Memory: Short-Term to Long-Term

Here is the simplest way to think about an agent's mind: at any given moment, it can only "think about" the words currently in front of it. That's the context window — the block of text (system prompt, conversation, tool results) handed to the model on each call. There is no hidden brain it quietly consults on the side.

Stated precisely: the only thing an agent can reason over in a given step is what is physically present in its context window. If a fact isn't in the prompt, embedded in the system message, or returned by a tool call, the model cannot use it — full stop. The context window is the agent's working memory, the equivalent of RAM in a computer.

This is why "memory" in agents is a bit of a misnomer. Everything else we call memory — conversation history, retrieved documents, summaries, prior outcomes — is really machinery for deciding what to load into that window and when. All of it has to be selected, formatted, and injected into the context before each model call. Memory engineering is, at bottom, context-loading engineering.

That reframing drives the whole lesson. The question is never "does the agent remember?" It is "at this step, did the right information get into the window, and was there room for it?" The window is finite and shared with the system prompt, tool definitions, and the current task. Memory systems exist to fill it well.

The most basic memory an agent has is just the chat so far — the running list of messages in this session. The catch: that list grows forever, but the window doesn't. So you need a policy for what to keep once the conversation outgrows the budget. Three patterns dominate:

1. Sliding window — keep only the last N messages. Cheap and lossless for recent turns, but it amnesically forgets anything older.
2. Summarization — compress older turns into a compact running summary, then prepend it. Preserves the gist across long conversations at the cost of detail.
3. Summary-buffer (hybrid) — keep recent turns verbatim and a summary of everything older. This is the production default; it gives you precise recent context plus durable older context.

Concretely: imagine a 40-turn support chat. A sliding window of 8 remembers the last few exchanges but forgets the original complaint. A pure summary remembers the complaint but blurs the exact error message. The hybrid keeps both — recent turns word-for-word, plus a summary of everything before. A minimal version in Python:

def build_context(history, summary, keep=8, summarize_fn=None):
    recent = history[-keep:]
    older = history[:-keep]
    if older and summarize_fn:
        summary = summarize_fn(summary, older)  # recursive: fold old into running summary
    msgs = []
    if summary:
        msgs.append({"role": "system", "content": f"Summary so far:\n{summary}"})
    msgs.extend(recent)
    return msgs, summary

Recursive summarization — repeatedly folding older messages into the running summary — keeps cost bounded but introduces drift: each compression cycle can erase a low-frequency detail that mattered.

Short-term memory dies when the session ends. To remember a user across days or conversations, you need something durable that lives outside the window. The standard trick: turn each memory into a list of numbers (an embedding) that captures its meaning, store those numbers, and later fetch the ones whose meaning is closest to the current question. That's the whole idea behind a vector database.

Long-term memory persists across sessions, and the pipeline is consistent across the ecosystem:

text → embedding model → vector → store in a vector DB → at query time, embed the query and do approximate-nearest-neighbor (ANN) search → inject the top-k retrieved chunks into the context.

("Approximate-nearest-neighbor" just means: find the closest vectors fast, trading a little exactness for speed.) Common backends include Qdrant, Weaviate, Milvus, Chroma (popular for prototyping), pgvector (Postgres-native), and Redis. The memory layer Mem0 alone supports 20+ such backends.

# Write a memory
vec = embed("User prefers metric units and dark-mode UIs.")
store.upsert(id="mem_42", vector=vec, payload={
    "text": "User prefers metric units and dark-mode UIs.",
    "user_id": "u_7", "type": "semantic", "ts": now(),
})

# Read at query time
hits = store.search(embed(query), top_k=5,
                    filter={"user_id": "u_7"})  # metadata scoping is essential
context = "\n".join(h.payload["text"] for h in hits)

Note the metadata filter: in multi-tenant systems, scoping retrieval by user_id (or session, or agent) is not optional — it's how you stop one user's memories from leaking into another's context.

Not all memory is the same kind of thing. Your own mind keeps several flavors: what you're thinking about right now, specific things that happened, general facts you've learned, and skills you can just do. Agent memory borrows exactly this split. The field has converged on a taxonomy formalized in the CoALA paper (Sumers et al., 2023), now used across LangChain, Letta, Mem0, and others. Four memory types:

The distinctions are practical, not academic. Episodic memory is timestamped and specific — it answers "what happened?" Semantic memory is distilled and timeless — it answers "what is true?" You often derive semantic memory from many episodic events: notice the user mention metric units in three separate chats (episodic), and you can store "user prefers metric units" once (semantic).

Procedural memory is the one people get wrong: it stores reusable executable skills (think Voyager's growing skill library), not instructions or preferences. "The user likes terse replies" is semantic; a saved, callable routine for handling a refund is procedural.

Having a store isn't enough — how you read from it, write to it, and prune it is where agents live or die. Think of it as three jobs: pulling the right memory back in, deciding what's even worth saving, and throwing out what's gone stale. All three are real engineering problems.

Reading (retrieval). Pure embedding similarity is a weak default — it finds text that is similar, not necessarily correct or useful. (Ask "what's the refund policy?" and similarity might surface a chat that merely mentions refunds but answers nothing.) Production systems layer signals:

• Hybrid search — dense vectors + keyword (BM25) to catch exact terms embeddings miss.
• Reranking — a cross-encoder re-sorts the top-k candidates by true relevance.
• Metadata + temporal/entity filtering — scope by tenant, recency, or named entity.

Multi-signal retrieval (semantic + keyword + entity) is now the norm; Mem0 reports large gains on temporal and multi-hop reasoning from exactly this.

Writing (the write path). Naive append is the most common mistake. A good pipeline does: filter (is this worth keeping?), canonicalize (normalize entity names), deduplicate, resolve conflicts (version contradictory facts instead of silently overwriting), and priority-score. Writes are increasingly asynchronous (Mem0 defaults to async) so they don't block the agent loop.

Forgetting. Forgetting is a feature, not a bug. Without TTLs or versioning, outdated facts accumulate and the agent treats them as authoritative — it will happily quote last quarter's price forever. So expire stale entries, supersede old facts, and prune low-value memories on a schedule.

You rarely build all of this from scratch — a handful of architectures and frameworks already package these ideas. The most instructive is Letta (formerly the MemGPT research project), which borrows an idea from operating systems: treat memory as tiers, and let the agent move data between them like a computer pages between RAM and disk.

• Letta uses an OS-inspired hierarchy: core memory (always in-context, like RAM), recall memory (searchable conversation history), and archival memory (external vector store, like disk). The agent pages between tiers via tool calls — powerful, but a wrong paging decision is a silent failure.
• Mem0 is a framework-agnostic layer with user/session/agent scopes over a hybrid vector + graph + key-value store.
• LangGraph (v1.0 GA October 2025) gives you built-in checkpointing for within-thread state (Postgres, Redis, MongoDB backends) and a separate store for cross-session long-term memory. Since October 2025, LangChain's create_agent runs on LangGraph's execution engine under the hood, replacing the legacy AgentExecutor.
• A-MEM (NeurIPS 2025) advances memory evolution: a new memory triggers re-indexing and link formation with existing notes (Zettelkasten-style), so memory reorganizes itself rather than sitting static.

The three pitfalls to watch: stale memory (no TTL/versioning), retrieval misses (similar ≠ correct), and context bloat (add-all crowds out the task). Note MemGPT is now Letta — use the current name.

Don't forget evaluation. Agents can ace single-hop recall yet collapse on temporal and multi-hop, cross-session tasks. Benchmarks like LoCoMo, LongMemEval, MemoryArena, and BEAM test these distinct dimensions — measure the one you actually depend on.