Memory Systems
A model is stateless: between calls it remembers nothing. Yet a useful agent must recall what happened five minutes ago and what it learned last month. Memory is the platform's answer to the model's amnesia — and, given the fixed context budget of Chapter 7, memory is fundamentally a problem of deciding what to keep, where, and how to bring the right fragment back at the right moment.
10.1 The memory hierarchy
By analogy to a computer's registers–RAM–disk hierarchy, agent memory spans tiers of decreasing speed and increasing capacity. The context window is the working set — fast, tiny, expensive. Short-term memory is the current run's state and transcript (Ch. 5). Long-term memory persists across runs and sessions, and itself divides by what it holds: episodic (specific past events and interactions), semantic (durable facts about the user, domain, or world), and procedural (learned skills and instructions — often the system prompt and tool set themselves).
10.2 The two hard operations: writing and recalling
Memory poses two questions the platform must answer well. What is worth remembering? — naively storing every message is expensive and pollutes future retrieval with noise; mature systems extract salient facts and consolidate them, updating or superseding prior memories rather than appending blindly. What should we recall now? — at each step the system retrieves only the memories relevant to the current context (typically by semantic similarity, often blended with recency and importance), respecting the budget of Chapter 7. Done well, the agent appears to genuinely know the user; done poorly, it either forgets or drowns in stale recollections.
10.3 Approaches and systems
The influential idea from the MemGPT research — realized in Letta — is to treat the context window like virtual memory: an OS-like controller pages information between the limited window ("main memory") and external stores ("disk"), with the agent itself able to call tools to read and write its own memory. Other systems package the write/recall loop as a service: Mem0 offers a memory layer that extracts and stores facts with automatic consolidation; Zep builds a temporal knowledge graph over the conversation so memories carry validity-over-time. The right choice depends on whether you need a turnkey memory API, a stateful agent runtime, or graph-structured, time-aware recall.
| System / idea | Core model | Strength |
|---|---|---|
| MemGPT / Letta | Virtual-memory paging; self-editing memory; stateful agent server | Agents that manage their own long-horizon memory as a first-class runtime |
| Mem0 | Extraction + consolidation memory layer over a vector store | Drop-in memory API with automatic fact updating |
| Zep | Temporal knowledge graph of entities and facts over time | Time-aware recall; "what was true when" |
| Roll-your-own | Postgres + pgvector + summarization jobs | Full control; fewest dependencies; you own the write/recall policy |
The value of a memory system is in what it chooses to forget and what it surfaces at the right moment — not in how much it stores. Design the extraction and retrieval policies first; the storage engine is secondary.
Long-term memory about users is precisely the data DPDP, GDPR, and similar regimes govern. A user's right to erasure means you must be able to delete their memories on request — including derived and embedded copies — and prove it. Build deletion and provenance into the memory layer from day one (Ch. 17); retrofitting it is painful.
10.4 Ontology-grounded, auditable memory
Most memory systems are vector-first: store embeddings, retrieve by similarity. That is fast and simple, but it is also opaque — you cannot easily say why a memory was recalled, what it relates to, or where it came from. A more structured approach stores memory as an ontology-grounded knowledge graph: typed entities and relationships, layered over (not instead of) vector recall. The payoff is threefold and ties directly to themes elsewhere in the book. Recall becomes typed and explainable — you traverse relationships, not only nearest neighbours. Memory becomes auditable — each node and edge carries provenance back to the source that created it (the lineage of Ch. 27), so you can answer "what does the system believe about this user, and on what basis." And it becomes governable — because every derived memory traces to its origin, targeted erasure and residency (Ch. 16, 27) become tractable rather than best-effort.
Cognee is a representative implementation: it builds a typed memory graph over a graph backend (such as FalkorDB or Neo4j) alongside a vector store, with explicit ontologies. The honest tradeoff is that a graph-backed, ontology-grounded memory buys auditability and typed recall at the cost of modeling effort and a second store to operate — so it earns its place where explainability, provenance, or compliance matter, and is overkill where a vector store and a summary would do. It is the memory-layer counterpart to the GraphRAG of Ch. 11: the same graph-over-vector idea, applied to what the agent remembers rather than to the corpus it reads.
When memory drives consequential behaviour, prefer a typed, ontology-grounded graph that records provenance on every node — so recall is explainable, erasure is provable (Ch. 16, 27), and you can always answer what the system believes and why. Reach for it when auditability or compliance justify the extra store; stay vector-only when they do not.