Agent memory architecture

TL;DR

• Agent memory has four tiers: working (context window), episodic (vector DB), semantic (RAG knowledge base), and procedural (model weights). Each has different latency, cost, and persistence characteristics.
• Working memory is available without retrieval latency but clears on session end. Without episodic memory, each new session has no durable record of prior interaction.
• Episodic retrieval often takes milliseconds via ANN search, but latency depends on the index, filters, and deployment. Context capacity depends on the model and on the tokens reserved for instructions and output.
• Context-window pressure is an operational condition to manage proactively: summarize before forced truncation, and preserve the task, constraints, and current state.
• Forgetting, deduplication, and retention policies are usually necessary. Without them, episodic memory grows and retrieval quality can degrade.
• Memory synthesis (compressing raw turn logs into structured episodes) is often a high-leverage engineering investment, but it should have a safe fallback when the synthesizer is unavailable.

---

30-second mental model

Agent memory is a data-placement policy, not one database. Working memory holds the current run, episodic memory stores experiences about a user or task, semantic memory holds shared knowledge, and procedural memory is behavior encoded in model weights or application logic. Choose a tier by asking how quickly the data must be available, who may read it, how long it should live, and how it can be corrected or deleted.

5-minute explanation

On each turn, the agent reads a small working set, optionally retrieves relevant episodic or semantic records, and writes durable facts only when they meet an explicit storage policy. Retrieval needs tenant and authorization filters, freshness and confidence signals, and a fallback when no memory is relevant. Long sessions need controlled summarization before the context window becomes full; summaries should preserve task state and high-value facts, not merely shorten text.

Memory also needs lifecycle management: deduplication, conflict resolution, retention, deletion, and versioning. A system that remembers everything can be less useful than one that remembers a smaller, accurate, properly scoped set of facts.

---

The problem it solves

A user tells your agent: "I always want code examples in TypeScript, not JavaScript." The agent confirms, stores nothing, and the session ends. Next week the same user is back, asks a related question, and the agent responds with JavaScript examples. The user is annoyed and repeats the preference. This cycle happens forever.

This is not a prompt problem. It is a persistence problem. The agent has no mechanism to carry facts from one session to the next. The context window clears on session end, and with it everything the agent learned about this user.

The cost is retention. Users who experience repeated memory failures trust the agent less and re-type context that the system should have stored. At production scale, this is not just a UX annoyance: it is a measurable drop in task completion rate and session return rate.

---

What is it?

Agent memory architecture is the design of how an agent stores, retrieves, and manages information across its four storage tiers to maintain continuity within and across sessions. Each tier has a different access pattern, capacity, update cost, and appropriate use case.

Think of it like a doctor's practice. The doctor's mind (working memory) holds the current patient's situation. The patient chart (episodic memory) records past visits and key history. The medical textbooks on the shelf (semantic memory) provide general domain knowledge. The doctor's trained skills like how to read an X-ray (procedural memory) are always active without lookup. A good doctor uses all four without confusing them.

---

How it works

Working memory and the context window math

Working memory is everything currently in the context window: the system prompt, tool schemas, conversation turns, and all Observations from tool calls so far. It is fast, always available, and costs nothing to read. It is also the smallest of the four tiers by capacity, and it clears when the session ends.

At a 200K-token context limit (use the limit documented by your selected model), you have significant headroom for a single session. One conversation turn with a moderate tool call is roughly 300-500 tokens. One episode summary retrieved from the vector store is roughly 200 tokens. The system prompt and tool schemas for a real agent commonly consume 2,000-5,000 tokens. That leaves roughly 190K tokens for task context, which is ample for most tasks.

The problem is long-running agents. A session that spans 50 tool calls at 400 tokens each uses 20,000 tokens for observations alone. Add in multi-turn conversation history, retrieved episodes, and a dense system prompt, and many production agents hit 60% context utilization within 30-40 turns. After that, the agent is operating under pressure.

Production agents can fail silently when they exceed context limits. The model truncates old content from the beginning of the context, which means the original task description disappears and the agent starts answering a different question. The fix is proactive summarization in the yellow zone, not reactive truncation at the limit.

Episodic memory: retrieval pipeline

Episodic memory is the tier that gives agents cross-session continuity. After each session, the agent extracts key facts and outcomes, compresses them into a structured episode record, embeds the record, and writes it to a vector database keyed by user ID.

At query time, the agent embeds the current user message, runs an ANN search filtered by user ID, retrieves the top-K most relevant past episodes, and injects them into the current context window as "Prior context for this user."