What Agent Memory Actually Fixes (and What It Doesn't)
I wanted to test a simple question, if we preprocess a large corpus and pre-extract facts into a memory layer, does that help downstream agents on document-heavy work ?
Motivation
The broader question is about document-heavy corpora, diligence folders, compliance archives, investigation productions, customer support histories, scientific corpora, internal wikis, contract repositories. These are environments where the answer is often not hidden behind one magic document. It is spread across many files, repeated in inconsistent language, and easy for an agent to miss even when it technically has file access.
So I wanted to test whether memory-style preprocessing changes the downstream task. Can we ingest a corpus once, extract or index useful facts, and then give the agent a better way to surface evidence when it is actually doing the work?
Why Legal Tasks?
I used legal tasks because legal work is an unusually good proxy for document-heavy knowledge work. The point was not that memory systems only matter for lawyers. The point was that legal tasks force the agent to do the thing this experiment cared about: navigate a corpus, identify the controlling facts, keep evidence grounded, and produce a reviewable work product.
For the task suite, I used Harvey’s 2026 Legal Agent Benchmark (LAB), an open-source benchmark for long-horizon legal-agent work. LAB tasks are structured around an instruction, a client matter with relevant materials, and a required work product. That structure is useful for this ablation because it looks much closer to real document work than a single-shot QA benchmark.
The specific tasks I sampled were intentionally varied:
Sparse clause hunts: change-of-control provisions and acquisition diligence, where the agent needs to locate important language across many contracts or data-room documents.
Red-flag and compliance reviews: data-room red flags and privacy program review, where the evidence is spread across policies, summaries, controls, and factual records.
Event reconstruction: litigation timeline building, where the task is to extract dated facts from multiple documents and organize them into a coherent chronology.
Document-by-document coding: relevance/privilege review, attorney production review, and privilege log review, where the model has to apply repeated classification cues across a production set.
Request matching: subpoena comparison, where the agent compares produced documents against request categories.
Compact legal-risk synthesis: FTC noncompete analysis, where the decisive facts fit into a smaller set of documents and reasoning quality can matter more than retrieval breadth.
That variety matters. “Document-heavy” is not one condition. A 55-document privilege-log task, a 19-contract clause hunt, and a compact FTC legal-risk memo stress different parts of an agent. Some reward raw lexical recall. Some reward evidence selection and organization. Some mostly test raw reasoning capability.
Choosing the Frameworks
Each framework was treated as an independent worktree, not as a single harness abstraction. The point was to let each system use its native shape as much as possible.
raw-rg: a simple lexical baseline using ripgrep-style search over normalized source text.
LightRAG: graph/vector-style retrieval over an indexed corpus.
Graphiti: episodic graph memory and search over ingested document episodes.
Mem0: memory search over native stored source chunk memories.
GBrain keyword: native GBrain keyword search.
GBrain Gemma: native GBrain query over converted markdown/indexed corpus artifacts.
ActiveGraph: an event-sourced structured memory profile, closer to matter-state modeling than a normal retrieval database.
This distinction matters. The comparison is not only “which retrieval algorithm is best?” It is also “what does this framework decide to preserve, chunk, extract, store, and return?” In practice, preprocessing is part of the product.
Ablation Setup
The core setup was deliberately narrow: same tasks, same generator model family, same judge family, same normalized corpora per task, and branch-local memory implementations.
The filtered comparison used GPT-5.5 as the generator with low reasoning effort and GPT-5.4-mini as the judge. I also ran a regular no-memory baseline with the same generator and judge combination. In that baseline, memory tools were disabled and the runs recorded zero memory search/read calls.
The metric below is criterion pass rate: how many rubric criteria the final answer satisfied. It is not Harvey’s binary all-pass task score. That is important, because many of these runs still fail at the all-pass level even when they satisfy a useful fraction of the rubric.
This is an exploratory ablation, not a clean production leaderboard. The honest question is not “which memory framework won?” It is “where did searchable preprocessed memory change what the agent surfaced?”
What Surfaced
The strongest signal was task-dependent. Memory/search helped most where the regular run either missed important parts of the corpus or failed to keep the right evidence selection in context.
Task Lift Observed
Framework Fit
Task-Winner Matrix
Interpreting the Results
The most interesting result is not that one framework dominates.
The interesting result is that the task shape matters a lot.
When the bottleneck was simply finding enough source material, raw lexical search could be very competitive. Acquisition diligence and subpoena comparison are good examples. There, raw-rg was either the best observed run or tied for best.
When the task required surfacing specific facts across a broader corpus, memory/search systems often helped. Privilege log, attorney production review, privacy program, litigation timeline, and relevance/privilege all showed meaningful lift over the regular no-memory baseline.
But memory did not always help. FTC noncompete is the warning label. The regular GPT-5.5 run did very well, and the best memory/search run was slightly lower. If the decisive documents already fit into the model’s workflow, memory can add overhead, distract, or just fail to add anything useful.
Task-specific read
Privilege log: this is the cleanest coverage-insurance case. The regular run read only 3 of 55 documents and scored 40.2%. GBrain keyword reached 59.8%, a +19.5 point lift, while raw-rg reached 50.0%. This is the kind of task where memory/search can rescue a run that simply did not inspect enough of the corpus.
Acquisition diligence: regular read 11 of 31 documents and scored 46.9%. raw-rg reached 64.1%, a +17.2 point lift. This points to the simplest explanation: direct lexical retrieval found more of the relevant source material. Graph semantics were not required for the best observed result.
Attorney production review: regular read all 18 documents but scored 58.3%. GBrain Gemma and LightRAG both reached 70.8%. Here the issue was not raw document exposure. The issue was likely salience: pulling the right classification cues back into the final review.
Relevance / privilege: regular read all 25 documents and scored 70.1%. GBrain keyword reached 79.1%, while raw-rg was much lower at 58.2%. That suggests retrieval formulation matters; simply searching the corpus was not enough.
Litigation timeline: regular read all 15 documents and scored 65.2%. GBrain keyword reached 75.8%, with raw-rg close behind at 74.2%. This looks like an ordering/extraction task where retrieval helped pull the relevant dated facts into a more complete timeline.
Change-of-control: regular read only 5 of 19 documents and scored 66.7%. GBrain keyword reached 73.7%, with raw-rg at 68.4%. This is a sparse clause-hunt task: the lift appears to come from reaching more clause-bearing contracts.
Data-room red flags: regular read all 13 documents but scored 52.0%. LightRAG reached 60.0%, while raw-rg fell to 44.0%. This is a good example of full document coverage not being enough; the task needed the right red flags to stay focused and gather the right evidence.
Privacy program: regular read 11 of 13 documents and scored 53.2%. ActiveGraph reached 66.1%, with raw-rg at 62.9%. The caveat is that ActiveGraph is not a normal retrieval DB, so I would interpret this as structured state helping on a compliance-mapping task, not as a clean retrieval leaderboard win.
Subpoena comparison: regular read 6 of 14 documents and scored 70.2%. raw-rg reached 79.0% and was the best observed run. This looks like a direct matching problem where lexical search is a very strong baseline.
FTC noncompete: regular read 15 of 22 documents and scored 80.7%. Graphiti was the best memory/search run at 79.0%, slightly below regular. This is the counterexample: if the regular model already finds the decisive documents and reasons well, memory does not automatically help.
What Kind of Tasks Did Memory Help?
It helped when the task needed coverage insurance, the agent had to find evidence scattered across many files, and missing a small document could cost many rubric criteria.
It also helped when the regular run technically read a lot of documents but still needed better focus. That is a subtle but important distinction. Document coverage is not the same as evidence use. A model can touch a file and still fail to bring the right fact into the final answer.
That is why I would not summarize the result as “memory helps on tasks with lots of docs.” The better version is: memory/search helps when the task punishes missed evidence, weak focus, or poor evidence organization. Lots of documents often create those conditions, but document count alone is not the mechanism.
Rules of Engagement
This experiment also made the evaluation pitfalls obvious.
Keep model and judge families fixed. Otherwise you are comparing model capability, not memory.
Do not mix no-memory runs with memory-framework results. A run with zero memory calls is a base model run, not a framework result.
Do not hide fallbacks. If a native memory path is unsupported or degraded, mark it that way.
Separate raw search from memory. raw-rg is a strong retrieval baseline. It is not the no-memory baseline.
Use criterion pass rate honestly. It is useful for diagnosis, but it is not the same as all-pass task success.
Token Budgets Matter. Can’t explain how many times I got burned by elapsing windows or upstream provider errors before settling upon Codex with Pro sub.
What I Would Rerun
If I were turning this into a more formal benchmark, I would rerun the promising cases by pinning every framework to a single commit, rejudge with at least one cross-family judge, and report latency/cost alongside criterion pass rate.
I would also separate framework effects from preprocessing effects. Right now, each system gets its own adapter and indexing strategy. That is realistic, but it means the measured object is “framework plus preprocessing recipe,” not a pure retrieval algorithm.
One important caveat: this was a low-repeat exploratory ablation, not a statistically powered benchmark. Treat the scores as directional evidence about task/framework fit, not as stable rankings.
The Takeaway
The cleanest takeaway is not that memory universally improves agent synthesis in document heavy domains.
The takeaway is that preprocessing a large corpus into a searchable memory layer can change what downstream agents surface, especially on document-heavy tasks where the cost of missing a fact is high.
That makes memory systems worth ablating early. Not because the word “memory” is magic, but because the retrieval surface, preprocessing recipe, and evidence organization can become part of the agent’s effective reasoning environment.
References
The structure of this post is inspired by the ablation sections in Hugging Face’s Smol Training Playbook.
All the experimental results documented here - Harvey memory ablations and benchmarks