Before you give an agent memory, price the raw-history baseline

Agent projects often settle their memory design in week one: extract facts, write summaries, build a graph, let the model update its own store. I’d hold off. Early on, keep the raw conversations, put a retrieval baseline on top, and measure what memory costs and how much it gets right. Only then decide whether you need a memory layer at all. Two papers from October make the case, and Latent Space’s AINews put it plainly: raw history remains a strong memory baseline.⁴

Raw history plus retrieval wins six of seven settings

The first paper’s baseline is deliberately plain. It splits timestamped conversations into 256-token windows, pulls the 128 nearest by embedding, and lets a reranker pick 15 for the model to answer from. Nothing is extracted, summarised or rewritten. Against 12 memory systems, including Mem0 and MemOS, across four benchmarks, it ranks first in six of seven benchmark–model settings, 0.13 to 5.53 points ahead of the strongest memory system. The one loss is DeepSeek-V4-Flash on MEME, by 1.52 points.¹

The second paper tests a different set of systems and lands in the same place. On LongMemEval-S, raw-history retrieval scores 78.0; the best of six explicit memory systems, LightMem, scores 64.0. In the authors’ words, no explicit memory system beats the strongest raw-history baseline.²

Writes are the expensive part: 0.33 hours against 26.95

FIG.1Time to ingest one conversation of about a million tokens (BEAM-1M). MemOS was the fastest memory system to finish, Memora the slowest.

Memory systems call a large model on every write to extract, merge and update. BEAM-1M is a single conversation of 1,710 messages, close to a million tokens. Raw-history retrieval ingests it in 0.33 hours; the memory systems take 3.25 to 26.95. On MemoryAgentBench, three systems with memory management burned 21.6 to 109.3 million tokens and 2.9 to 8.8 hours to ingest and search, against 12 minutes for the baseline.¹

FIG.2Six memory systems ordered by build time, with their accuracy on the right. The most expensive, A-Mem, is second from the bottom, and none reaches raw-history retrieval.

Spending more doesn’t buy accuracy either. Build times in figure 2 differ by 35x, and accuracy doesn’t follow.²

Every transformation drops some evidence

The first paper also handed each memory system’s entire exported store to the model, with no retrieval in the way. Accuracy still trailed reading the raw evidence by 7.58 to 29.38 points. The authors caution that not all of that gap is information lost at write time.¹ The second paper sees the same pattern under oracle retrieval.²

For audited work this matters most. A raw record traces back to what the user actually said; a summary or an extracted fact doesn’t. The price is storage and privacy: raw conversations must be deletable per user.¹

Complex memory helps in a narrower band than the pitch suggests

Explicit memory’s gains cluster in a few question types: abstaining when it should, resolving contradictions, ordering events, and settings with tight context budgets or smaller answer models. Even those gains don’t hold across both benchmarks.² A separate test called MemoryTax used one 419-turn conversation: Pi with Opus 5.5, searching raw history, scored 99% at under half the cost of full context, and a memory layer like Mem0 helped only weaker models. There is evidence the other way too: Toloka and xmemory report memory gains across 11 models.³

Measure four things early

Time and tokens to ingest one conversation.
Latency and cost per query.
Accuracy on your own questions, especially abstention, contradictions and time order.
How long raw records are kept, and what it takes to delete one user.

If the baseline is good enough, stop there. If one kind of question keeps failing, or your context budget really is tight, add structured memory for that case only, and keep it side by side with the baseline.