Agent Memory, Explained: Context, Retrieval, and State

One word, four mechanisms

People naturally talk about memory as one capability: either the agent remembers or it does not. In implementation, it is more useful to separate four layers.

1. 1. Context windowThe tokens visible to the model for the current call: instructions, recent messages, tool results, and any retrieved material.
2. 2. Working stateStructured information about the current job, such as completed steps, pending questions, selected files, and intermediate results.
3. 3. RetrievalA search process that selects useful passages or records from a larger external collection and places them into the current context.
4. 4. Durable memoryInformation deliberately saved across sessions, such as an approved preference, a project decision, or a user-created note.

All four can make a system appear to remember, but the engineering questions are different. Context is constrained by capacity. Working state needs a schema. Retrieval needs relevance and provenance. Durable memory needs consent, updates, isolation, and deletion.

The context window is a workbench

A model generates its next output from the information currently in context. Even a large context window is finite, and filling it with every prior event can make the relevant evidence harder to find. A useful mental model is a workbench: bring in the materials needed for this step, keep the rest in organized storage, and clear items that no longer help.

Research systems such as MemGPT explored explicit movement between memory tiers to work around finite context, borrowing the idea of virtual memory from operating systems. The specific implementation is not universal, but the underlying constraint is: persistent information lives outside the model and must be selected back into context when needed.^[2]

• Keep the governing instructions stable and easy to distinguish from retrieved content.
• Preserve exact facts—IDs, amounts, dates, decisions—in structured state rather than repeatedly summarizing them.
• Summarize long history only when the original detail can be recovered if necessary.
• Budget context for the task instead of treating the token limit as a target to fill.

Working state answers: where are we now?

Conversation history is a poor substitute for task state. If an agent is preparing a report, it should not have to reread twenty messages to infer which sources were approved and which sections remain unfinished. Store those facts explicitly.

{
  "goal": "Compare three approaches to agent memory",
  "approvedSources": ["rag-paper", "memgpt-paper"],
  "openQuestions": ["How are stale facts replaced?"],
  "completedSections": ["definitions", "context-window"],
  "status": "researching"
}

Structured state makes pausing, resuming, inspection, and testing much easier. It also reduces a subtle failure mode: a fluent model can reconstruct a plausible task history that is not the actual history.

Retrieval is selective recall

Retrieval-augmented generation, usually shortened to RAG, combines a model with an external collection. At question time, a retrieval system finds relevant material and adds it to the model’s input. The original RAG work framed this as combining parametric memory inside the model with non-parametric memory outside it.^[1]

Retrieval is not the same as memory storage. It is the selection mechanism. A system can store the correct document and still fail because the query did not retrieve it, because a stale version ranked higher, or because too many weak matches crowded out the useful passage.

• Chunking decides what unit can be retrieved.
• Metadata filters decide which project, user, date range, or source type is eligible.
• Ranking decides which candidates enter the limited context.
• Citations let the answer point back to the underlying source.

Durable memory should be earned

Cross-session memory can make an assistant feel continuous, but “save everything” is a weak design. A passing remark can be mistaken for a lasting preference. An old address can override a new one. Information from one user or project can leak into another. Deletion becomes difficult if the same fact has been copied into transcripts, summaries, embeddings, and caches.

1. SelectDefine which facts deserve persistence. Prefer explicit user requests and confirmed project decisions over guesses.
2. ScopeAttach the memory to the correct user, workspace, project, and purpose.
3. TimestampStore when it was created, where it came from, and when it should be reviewed or expire.
4. ResolveDefine what happens when a new fact conflicts with an old one. Do not silently keep both as equally true.
5. ExposeLet users inspect, correct, and delete durable memory when the product context permits it.

The failure modes hide behind “it remembers”

• False recall: the agent invents a prior preference that was never stored.
• Missed recall: the right fact exists but retrieval does not surface it.
• Stale recall: an old fact wins after the user has supplied a replacement.
• Over-recall: irrelevant history enters context and distracts the current task.
• Cross-scope recall: data from the wrong user, project, or environment appears.
• Unverifiable recall: the system cannot show where a remembered claim came from.
• Undeletable recall: the visible record is removed but derived copies remain active.

A polished demo typically tests one happy path: tell the agent a favorite color, start a new chat, and ask for it. That proves persistence once. It does not test whether the system stores only appropriate information, updates it correctly, keeps users isolated, or forgets on request.

How to test memory as a system

1. RecallWhen the saved fact is relevant, does the system retrieve and use it?
2. PrecisionWhen the fact is irrelevant, does it stay out of the answer and context?
3. FreshnessAfter an update, does the new value replace the old one everywhere that matters?
4. IsolationCan one user, tenant, or project ever retrieve another scope’s records?
5. ProvenanceCan an operator and, where appropriate, the user see the source of a remembered fact?
6. DeletionAfter deletion, does the fact disappear from storage, retrieval indexes, summaries, and future responses?

The deeper lesson is that memory is not something a model simply has. It is a data system wrapped around the model. Its quality depends on selection, storage, retrieval, conflict handling, permissions, and lifecycle controls—and each part can be inspected separately.