TL;DR
Two October studies tested AI memory tools against search over the raw chat history. Search won 90 of 99 times.
Memory tools lose facts when they rewrite your history into short notes, and some take hours to build.
Keep the originals, measure your search, and add memory only where a test shows it wins.
Every week someone asks me which memory tool to add to their AI agent. Mem0, Letta, Zep, a graph database. My answer used to be a hunch. Now it has two studies behind it.
In October, two research teams tested AI memory tools against a simpler setup: keep the whole chat history and search it. Search won, and it wasn’t close.
I maintain obsidian-second-brain, an open-source memory setup for AI agents with over 5,000 stars on GitHub. In July I wrote about how I gave Claude Code memory without a vector database. This post covers what the new research found, and what I run instead of a memory layer.
What is AI agent memory?
AI agent memory is any system that lets an agent recall past chats and facts in a later session. Most memory tools rewrite your history into short facts, notes or a graph. Then they search that smaller store.
The other option is to keep the full history and search it directly.
Most of page one on Google sells the first option. Mem0, Letta, Zep and others call it a memory layer: a service between your agent and its past. The studies asked one question. Does the rewriting step pay for itself?
What did the two studies test?
Both studies put memory tools and search over the raw chat history through the same questions.
“Memory Tax” (arXiv, October 25, 2026), from a team at the University of Illinois Urbana-Champaign and partners. It tested 11 memory systems, including A-Mem, MemOS, MemoryOS, OpenViking, HippoRAG2 and Mem-T. Three benchmarks times three AI models gave 99 head-to-head results.
“Do Conversational Agents Need Explicit Memory?” (arXiv, October 17, 2026), by Yongjie Zhou, Shuai Wang, Bevan Koopman and Guido Zuccon. It tested six systems, including Mem0, A-Mem and LightMem, on two benchmarks.
The benchmarks are long chat histories with questions about them. Think “How long is my daily commute?”, with the answer buried somewhere in weeks of old chats.
Did AI memory tools beat search?
No. In the Memory Tax study, memory tools scored below search in 90 of the 99 results. In the second study, no memory tool beat the best search setup.
The numbers that stood out to me:
Best case for memory: on the LongMemEval test with GPT-5.6 Luna, search scored 90.0. The best memory tools scored 84.0.
Mem0: scored 57.0 where search scored 78.0, in the second study.
Build time: search took about 7 minutes to set up. A-Mem took 117 hours on the same 100 test questions.
Per chat history: Mem0 made 417 AI calls and spent 42 minutes building memory for each one. Search needed zero.
Mem0 reports higher scores on its own tests. Those use a different setup, so the numbers don’t compare directly.
Why did memory lose to search?
Memory tools rewrite your history before they store it. Each rewrite can drop a fact, bury it, or make it depend on the model. The Memory Tax paper calls this loss the memory tax, and it found three kinds:
Facts that never get stored. When the researchers sampled facts, OpenViking had no memory entry at all for 38% of them.
Facts that are stored but hard to find. In one example, “my commute is 45 minutes each way” ended up as one line in a 59-line profile of unrelated facts. Search over the raw chat found it on the first try.
Facts that depend on the model. The same memory tool can score well with one AI model and much worse with another.
None of this means the tools are badly built. Rewriting a history always loses some of it. The raw history loses nothing.
What does “search” mean in these studies?
The winner was not keyword search. Both studies used search that matches words and meaning together, then reorders the best hits:
Keyword search finds notes that share your exact words.
Meaning search (embeddings, a way to compare meaning) finds notes that say the same thing in other words.
A reranker reads the top hits and puts the best ones first.
The Memory Tax team also tested keyword-only search. It scored lower than the combined search on all four test sets they reported, for example 68.2 against 82.7. In the second study, swapping one meaning-search model for a better one added 14.8 points.
So the lesson is not “skip memory, use any search.” The quality of your search decides most of the result.
When does agent memory help?
Memory won only in narrow cases. In the second study, memory tools beat search on three question types:
Knowing when to say “I don’t know” (up to 15 points better).
Putting events in the right order (1 point).
Sorting out facts that contradict each other (2.4 points).
The Memory Tax paper shows examples where memory helps with dates and with forgetting old facts. On its preference test, a few tools gained 1 to 2 points, mostly with smaller AI models.
Its proposed fix is the part I’d copy. Keep a path back to the original history. When the team searched the raw history next to the memory store and ranked both together, several memory tools improved.
What I use instead of a memory layer
I don’t run a memory service. My agents work from a folder of plain markdown notes in an Obsidian vault, managed by obsidian-second-brain. That is a memory system too, so the studies apply to it. Here’s how it avoids the tax:
Originals never change. Every source I save goes into a
raw/folder untouched. Summary pages are built from it and can be rebuilt from it. That’s the paper’s path back to the original.Any agent can read it. It’s plain files, so Claude Code, Codex, Gemini, OpenCode, Antigravity, Hermes, Pi and Grok Bot all use the same notes. No service and no monthly bill.
Search is measured, not assumed. In June I tested my own search on 35 questions worded differently from their notes. It found the right note in the top 10 zero times. Word matching alone was the problem.
Search got fixed first. By July, with optional meaning search running on my own laptop, it found the right note 77% of the time. Private notes never leave the machine.
Old notes step back. A note marked superseded or declined ranks lower, so the agent doesn’t treat last spring’s decision as current.
The test I ran ships in the repo as /obsidian-retrieval-eval. If you want the full setup, here’s how to build an Obsidian second brain every AI agent can use.
How do you test your agent’s memory before buying a tool?
You can run a small version of these studies in an afternoon:
Write 30 questions about things your agent should remember. Word them differently from the original chats.
Note where each answer lives in your raw history.
Run each question through search over the raw history. Count the answers that land in the top 10.
Run the same questions through the memory tool. Count again.
Keep the memory tool only for question types where it wins, and keep the raw history either way.
If search wins, put your effort into better search, not a memory layer. That’s where both studies found the points.
Key takeaways
In the Memory Tax study, 11 AI memory tools scored below search over raw chat history in 90 of 99 tests.
Under its original setup, Mem0 scored 57.0 against 78.0 for search over the raw history on LongMemEval-S.
Memory tools lose accuracy when they rewrite history: facts go missing, get buried, or depend on the model.
The search that won combines keyword and meaning matching with a reranker. Keyword-only search scored lower.
Memory helped mainly with knowing when to say “I don’t know.” Other gains were small.
Keep the original history searchable, measure your search, and add memory only where a test shows it wins.
Further reading
Memory Tax: Dissecting the Net Value of Agent Memory (arXiv, October 25, 2026)
Do Conversational Agents Need Explicit Memory? A Controlled Comparison with Raw-History Retrieval (arXiv, October 17, 2026)
About the author
I’m Eugeniu Ghelbur. I build AI agent systems and the tooling around them, and I write The AI Operator. I maintain obsidian-second-brain, an open-source Claude Code skill that runs an Obsidian vault as a living second brain. It’s MIT-licensed and starred by over 5,000 developers on GitHub.