An agent with no memory starts every conversation from zero — it can't remember your name, your last request, or what it did five minutes ago. Adding memory is what makes an agent feel coherent. Here is how to do it in two layers.
Short-Term Memory: The Conversation History
The simplest memory is keeping the running conversation in the context window. Each turn, you append the new exchange and send the whole history back:
messages.append({"role": "user", "content": user_input}) response = model.generate(messages=messages) messages.append({"role": "assistant", "content": response.text})
This works until the conversation gets long enough to overflow the context window — which is where the next step comes in.
Managing a Growing Context
You can't keep an unbounded history in the prompt. Two common strategies:
- ▸Summarize older turns into a compact recap and keep only recent turns verbatim.
- ▸Trim to a sliding window of the last N messages.
if token_count(messages) > LIMIT: summary = model.generate(f"Summarize this conversation: {old_messages}") messages = [summary_message(summary)] + recent_messages
Long-Term Memory: Store and Retrieve
For facts that should survive across sessions — user preferences, past decisions — you need persistent memory. Store facts as embeddings in a vector store, then retrieve the relevant ones at the start of each conversation:
# Save a durable fact memory_store.add("User prefers metric units", user_id="123") # Recall relevant facts for a new conversation relevant = memory_store.search(query=user_input, user_id="123", k=3)
Inject those retrieved facts into the system prompt so the agent "remembers" them without carrying the entire history.
Summary
Give your agent two layers of memory: short-term, the running conversation kept in context and summarized or trimmed before it overflows; and long-term, durable facts stored as embeddings and retrieved when relevant. Together they let an agent stay coherent within a conversation and remember what matters across them — without ever blowing your context budget. Memory is what turns a stateless model into something that feels like it knows you.