The Memory Layer: What Should Survive Across Turns?

§ 00The uncomfortable result first

I gave my research agent three kinds of persistent memory: structured notes, Markdown files on disk, and a Chroma vector store. All three work. I tested them in isolation and they do exactly what they say.

Then I ran the full agent and looked at the memory directory.

agent_memory/
└── test_memory.md      # written by my test, not by the agent

The agent had done the research, written the report, and finished — without saving anything. The tools were there. The capability was there. The behaviour was not.

That gap is the actual subject of this article.

§ 01Why the default agent has no memory

An LLM is stateless. Every call sees only what you put in the context window. In a plain ReAct loop, the only memory is a growing list:

self.messages = [
    {"role": "system",    "content": SYSTEM_PROMPT},
    {"role": "user",      "content": query},
    # assistant tool_calls
    # tool observations
    # ... repeat until done
]

This is memory in the way a whiteboard is memory. It has three failure modes:

1. Ephemerality — the process ends, the knowledge is gone.
2. Noise accumulation — a 40k-character scraped page and a one-line insight occupy the same list, and the raw dump wins on volume. I truncate every observation at 8,000 characters, which controls the cost but not the signal-to-noise ratio.
3. No reuse — a second run on a related question starts from zero. So does every other agent I write.

§ 02What memory actually is

Before writing code I wanted a definition that could be tested against a design, not a vibe. The one I settled on:

Memory is externalized state that outlives a single model call, and that the agent can deliberately read from and write to.

Both halves matter. “Outlives the call” rules out the message list. “Deliberately read from and write to” rules out a RAG pipeline that silently injects chunks — that’s retrieval performed on the agent, not memory used by it.

That gives three layers:

Most tutorial agents stop at the first two. The third is the one that compounds.

§ 03The progression, in three stages

I deliberately did not start with a vector database. Each stage exists because the previous one was insufficient in a specific way.

Stage A — Forced compression (update_notes)

The first memory tool writes nothing to disk. It takes key_findings, important_sources, and open_questions, formats them into a Markdown block, and returns it as a tool observation.

That looks pointless. It is not, and understanding why changed how I write tools:

The value isn’t in the storage. It’s that calling the tool forces the model to produce the compression, and the compressed artifact lands back in context as a first-class observation.

It’s a prompt disguised as a tool. The schema — three named, typed fields, one of them “what do you still not know?” — does the real work. Raw pages go in, structured knowledge comes out, and the open-questions field gives the agent something to argue with itself about later.

Still ephemeral. But abstraction now happens during research instead of only at report time.

Stage B — File memory (save_memory / load_memory)

Then I pushed the notes onto the filesystem: one Markdown file per topic, named by the agent.

agent_memory/
└── solid_state_batteries.md

Four things change at once, and only the first is obvious:

• Knowledge survives the process.
• I can read it. An agent’s memory in a vector store is opaque to me; in Markdown I can see that it recorded something wrong.
• Another agent can read it — memory becomes a shared surface, not a private one.
• Naming becomes a decision. The agent must choose a key it can guess again later, which is a small, real act of knowledge organization.

Stage C — Vector memory (add_to_vector_memory / search_vector_memory)

File memory has an obvious ceiling: load_memory requires knowing the filename. Recall by exact key doesn’t scale, and it fails exactly when it matters — when the useful prior knowledge is under a name you wouldn’t think to ask for.

So, a persistent Chroma collection. Chroma handles embedding; the agent only ever handles text and metadata. Now recall works by meaning: a query about sodium-ion batteries can surface notes filed under solid-state.

§ 04The hybrid architecture

Nothing here replaces anything. Each layer covers a different failure of the others.

┌─────────────────────────────────────────────┐
│                 Agent Loop                  │
│  Plan → Act → Observe → Compress → Reflect  │
└─────────────────────────────────────────────┘
                      │
       ┌──────────────┼──────────────┐
       ▼              ▼              ▼
  Working         Structured      Vector
  Memory          Notes + Files   Memory
  (messages)      (Markdown)      (Chroma)

The intended loop:

1. create_research_plan
2. search_vector_memory / load_memory     ← recall
3. web_search + browse_page
4. update_notes                           ← compress
5. save_memory                            ← persist
6. add_to_vector_memory                   ← index
7. reflect
8. finish_research

Which closes the cycle:

Goal → Experience → Compression → Evaluation → Decision
                         ↑                        │
                         └────────────────────────┘

§ 05Tools don’t change behaviour. Constraints do.

Back to the empty directory. My system prompt already said “Treat memory as a core part of your research process, not an optional extra.” The agent ignored it, politely.

What actually works is making the undesired path unavailable. Two mechanisms are already in the loop, and they’re the template for fixing memory:

Force the entry point. On step 1 only, I don’t ask — I pin the tool:

if step == 1:
    tool_choice = {"type": "function",
                   "function": {"name": "create_research_plan"}}

The agent cannot begin without a plan. Not “should not” — cannot.

Gate the exit. finish_research is checked at runtime against the message history:

if name == "finish_research" and not self._has_reflected():
    result = ("You must call the `reflect` tool before finishing. "
              "Please evaluate your current research first.")

The call doesn’t fail — it returns an instruction. The agent reads its own blocked exit as an observation and re-plans. Prompt-level “please reflect” was ignored; a gated exit never is.

Memory needs the same treatment: finish_research should also refuse while findings exist that were never persisted. Steering belongs in the runtime, not only in the prompt.

§ 06The bug that cost me the most timegotcha

Not a memory bug, but the one worth passing on. With OpenAI-style tool calling, every tool_call in an assistant message must be answered by a matching tool message. Miss one and the next request 400s on malformed history.

The trap is the early return. My first version returned immediately when it saw finish_research — before appending that call’s response. If the model emitted finish_research alongside a parallel call, history was left invalid and the run was unrecoverable. The fix is one line of ordering: record the observation, then exit.

# Exit only after the tool response is recorded (keeps history valid
# if finish is one of several parallel calls).
if name == "finish_research" and self._has_reflected():
    return observation

Parallel tool calls turn “obviously correct” control flow into a landmine. Always close the loop before you leave it.

§ 07What I’d tell someone starting this

1. Memory is continuity, not storage. The question isn’t where bytes go, it’s what deserves to survive.
2. Compression beats persistence. update_notes improved reasoning quality before a single byte hit disk.
3. Hybrid wins. Files give inspectability, vectors give scale. Neither substitutes for the other.
4. A capability is not a behaviour. Shipping the tool is maybe 30% of the work. The rest is making its use structurally unavoidable.
5. Test tools in isolation first. Mine surfaced import and packaging problems well before the agent was in the picture.