Why your AI agent has amnesia and why forgetting is the fix

Building a biologically-grounded memory architecture for AI agents that actually remember

By Yitzhak Kesselman, Alexei Robsky, Doga Kerestecioglu, and Clemens Vasters

Image created by the authors with Bing Image Creator.

Ask an AI agent what you discussed last Tuesday, and it will stare at you blankly. Ask what happened after that conversation, what actions were taken, or whether anything changed… and the answer is still nothing. Change the subject mid-conversation and return to it later… gone. Close the chat and reopen it… gone. Every session starts from zero.

In enterprise settings, this is more than an inconvenience. It breaks the illusion that agents can participate in real operational workflows, whether they be tracking incidents, following up on alerts, remembering decisions, or learning from past actions. Your most junior employee remembers what happened after yesterday’s meeting. Your most expensive AI agent does not.

We set out to fix this, not by expanding context windows or bolting on a vector database, but by studying how human brains actually manage memory and building those mechanisms into an AI agent architecture. We formalized the approach in a research paper and then set out to validate whether the theory holds up against real-world data at scale. The human brain doesn’t just store memories. It consolidates them during sleep, actively forgets outdated ones to keep retrieval sharp, and holds new memories in a “silent” state until they’ve been verified. These aren’t bugs in human cognition. They’re features we’ve been missing in AI. We came up with a counterintuitive conclusion: forgetting right is more important than remembering all.

The problem with how agents “remember” today

Today’s approaches to agent memory fall into a few categories, each with a critical weakness.

Stateless agents forget everything between sessions. Every conversation starts cold. Users re-explain context. The agent never learns.

Context window stuffing tries to solve this by expanding the prompt with history. Modern models accept millions of tokens, but this scales cost without scaling intelligence. When everything is treated as equally relevant context, the agent still can’t distinguish a critical customer complaint from a routine log entry three months ago.

This problem becomes more pronounced in enterprise settings, where agents are connected not just to chat history but to tools, logs, alerts, tickets, and action traces. As more information is appended into the context window, reasoning quality often degrades rather than improves. When the context eventually fills up, the default behavior in many frameworks, including Azure OpenAI, is truncation: “auto”, which silently drops the oldest content.

Whatever was filed most recently stays. Whatever matters most, but happened a while ago, is gone.

Vector databases (e.g. via RAG) improve things by retrieving relevant information via embedding similarity. But standard RAG treats all information equally. There’s no mechanism to consolidate related facts, forget outdated ones, or evolve memories when new information arrives, especially when facts are conflicting. Ask “What’s happening with the terminal?” and you might get a result from six months ago because it happens to have high cosine similarity even though the bug was fixed weeks ago.

We wanted something better. We looked at the system that’s been solving this problem for 500 million years: the human brain.

What neuroscience teaches us about memory

Human memory is not a filing cabinet. It’s a dynamic system that actively processes, transforms, and forgets information. Three insights from neuroscience shaped our architecture.

First: memories consolidate during sleep. The brain doesn’t store raw experiences permanently. During sleep, the hippocampus, the part of the brain that rapidly encodes recent experiences, “replays” the day’s events, and the important ones get transformed into compressed, abstracted knowledge in the neocortex, which is the part of the brain that supports longer-term, more generalized memory. The details fade; the meaning persists. We implemented this as a “sleep consolidation pipeline,” which is an offline batch process that deduplicates, filters, clusters, and merges recent events into a clean knowledge store.

Second: forgetting is a feature, not a failure. Neuroscience has demonstrated that active forgetting mechanisms (i.e., interference, decay, and graceful degradation, among others) are essential for effective retrieval (Richards & Frankland, 2017; Berry et al., 2012). A brain that remembered everything would be paralyzed by irrelevant information. We implemented three forgetting mechanisms:

1. Exponential decay (old resolved issues lose priority).
2. Interference (when a new fix supersedes an old bug report, the old one can be forgotten).
3. Graceful degradation (memories lose detail before disappearing entirely).

Passive decay:

where I₀ is the original importance, λ is the decay rate, and t is time since encoding.

Interference score:

where wⱼ represents directional weights (retroactive versus proactive) and sim is cosine similarity between memories.

Combined forgetting decision:

The three mechanisms are applied sequentially: Decay reduces scores over time, interference identifies redundant low-value memories, and degradation progressively strips detail from borderline memories before full removal. A memory only survives if it passes all three filters.

Third: memories mature before they’re trusted. Research from MIT’s Picower Institute showed that memory engrams form simultaneously in the hippocampus and cortex, but cortical engrams remain “silent” for about two weeks before becoming functionally mature (Kitamura et al., 2017). We implemented this as activation strength dynamics. New memories start silent and gradually become retrievable, which prevents premature influence of unverified information while still allowing them to influence related searches through “priming.”

Building it: From theory to forgetful agent

Imagine you’re a developer joining a large open-source project. Thousands of issues are filed every month: bug reports, feature requests, performance regressions, user complaints. Some get fixed quickly. Some linger. Some turn out to be duplicates of problems reported weeks ago. No single person can keep track of it all, yet every triager, every on-call engineer, every program manager needs to answer the same question: “What’s actually going on right now?” Using Microsoft Agent Framework, we built an agent designed to answer exactly that question, one that ingests the full stream of GitHub issue activity for microsoft/vscode, remembers what matters, forgets what doesn’t, and retrieves relevant context when asked. To test it, we fed it three months of real development activity: 13,127 issues with 120,000 timeline events.

The architecture has six components working together:

1. A consolidation pipeline that cleans and compresses incoming events.
2. Three forgetting mechanisms that prune the store.
3. A maturation system that gates when memories become retrievable.
4. A reconsolidation engine that updates memories when they’re retrieved alongside contradicting information.
5. A knowledge graph that captures relationships between issues and components.
6. A hybrid retriever that combines embedding similarity search with graph traversal.

How we evaluated it. Each issue carries natural importance signals: how many people reacted to it, how many comments it generated, whether it was milestoned, and whether it’s still open or was resolved. We combined these into a ground-truth importance score (on a 1–5 scale) and used three metrics to judge the pipeline:

• Retention precision: Of the memories the system kept, what fraction were actually important? (Higher is better.)
• Catastrophic forgetting rate: How often did the system forget something it absolutely should have kept, like a critical open bug? (Lower is better.)
• Store size: How many events remain in the memory store? (Smaller is better, as long as precision stays high.)

Evaluation was stateful: Once the pipeline forgot an event, it stayed forgotten. No do-overs. This mirrors how a real deployed agent would operate. We processed events in fixed-size batches of 200, simulating the paper’s periodic “sleep” consolidation cycles, and measured metrics after every batch across the full three-month window.

When we tested it on the VSCode issue data using stateful evaluation, where forgetting decisions are permanent, just like in a real system, the results were striking. At the optimal configuration, the pipeline achieved 97.2 percent retention precision with only a 3.6 percent catastrophic forgetting rate, a +21.8 percentage point improvement over keeping everything. In other words, the model forgot the unimportant things and retained the most important ones. The memory store self-regulated at roughly 400–500 events, automatically balancing new arrivals against forgetting to maintain a compact, high-quality knowledge base.

Getting the evaluation right

Early in the project, we were getting suspicious results. Our evaluation showed 84.4 percent precision and we were ready to declare victory. Then we asked a simple question: “Are we testing this the way it would actually run in production?”

The answer was No in three different ways: (1) Our evaluation reprocessed history from scratch at each time step, so events we “forgot” in January reappeared in February. We noticed we were testing batch processing, not persistent memory. (2) Our time windows were quarterly, but the architecture specified 6-hour cycles, making decay 365 times more aggressive than intended. (3) Our dataset was biased. The GitHub API sorts by most recently updated, so some months had three issues and others had 2,700.

Once we caught these errors, we rebuilt everything: the complete 13,127 issue dataset, stateful evaluation where forgetting decisions are permanent, and fixed-size batches matching the intended cadence. The results jumped from 84.4 percent to 97.2 percent precision. **The lesson: **When validating a system designed for continuous operation, your evaluation must also operate continuously.

Three approaches, one question

To understand whether this architecture actually helps, we built a live demo that asks the same question in three different ways.

Without memory, the LLM generates plausible-sounding answers from its training data. Asked “What bugs have been reported about the terminal?”, it produces a list of generic issues: “Terminal not launching,” “Text rendering issues,” “Input/output delay.” None of these are real. They sound convincing because the model has seen thousands of bug reports during training, but they don’t correspond to any actual issue in the VSCode tracker.

With truncation (the default approach), we pack the most recent events into context until the token budget fills. In our case, about 46 of 200 events fit, the other 154 are silently dropped. The LLM now has real data, but it’s whatever was filed most recently, not necessarily what’s relevant to the question. When we asked about terminals, three of the five items in the response weren’t about terminals at all.

With our pipeline, embedding similarity search finds the five most relevant memories regardless of when they were filed. For the terminal question, it retrieved: a terminal relaunch suggestion, a terminal task UI border bug, a terminal auto-completion issue, a terminal execution callback problem, and a no-feedback-on-Enter bug. All five are genuinely about terminals. All verified against the real issue dataset.

The pipeline doesn’t always win. For broad summarization queries like “summarize the most critical issues across all components,” truncation’s larger context window gives the LLM more material to synthesize. Our pipeline retrieved five results that happened to be bot auto-responses, which are semantically similar to “critical issues” but completely unhelpful. The right approach for that query type is more context, not more precise retrieval.

This reveals an interesting finding and future room for improvement: Intelligent retrieval excels at targeted, specific queries. Broad summarization benefits from volume. An optimal system would combine both.

What we didn’t expect: Insights from building three domain prototypes

To test whether these findings generalize beyond software development, we built three additional prototypes across very different domains:

1. A luxury fashion retail advisor that remembers returning customers across store visits.
2. An F1 race engineer tracking telemetry anomalies across a season.
3. A security operations analyst connecting threat campaigns across shifts.

All three used the same underlying architecture with different domain ontologies.

Several insights emerged from this parallel effort that reinforced lessons we had already learned the hard way in our own validation work confirming that these are fundamental truths about memory architectures, not quirks of a single implementation.

Attention turned out to be the missing keystone. Choosing which timeline event types should become memories (issue creations, comments, status changes) and which should be filtered out (subscriptions, mentions, cross-references) is critical. We imposed a graph schema that classified labels into areas, features, and platforms, discarding workflow tags like “verified” or “stale.” These are attention decisions. The prototype work formalized this into an attention controller that scores candidates on goal relevance, novelty, and significance. It made explicit what we had been doing implicitly.

Time needs to be domain-relative, not wall-clock. We discovered this when our quarterly evaluation cadence made the decay function 365 times more aggressive than intended. A memory with a 24-hour half-life should barely change over a single processing cycle, but our quarterly windows obliterated everything older than a week. The fix was straightforward once we identified the root cause: measure time in domain-meaningful units, not wall-clock hours. The prototype work confirmed this principle across domains. A fashion advisor should forget seasonal preferences at the rate of seasons. A race engineer’s tire observation decays per weekend, not per hour. Each domain has its own natural rhythm, and the decay function must match it.

Agents need to study before they interact. Before we could build the memory graph, we had to study the dataset’s embedding similarity distribution, estimate thresholds from data percentiles, and design entity resolution rules that mapped raw GitHub labels to structured component types. Without this upfront work, the consolidation pipeline would have created meaningless clusters. The prototype work also generalized this into a “curriculum engine” that reads domain literature and extracts structured knowledge triples. The system needs canonical vocabulary and domain structure before it can meaningfully process events. We are now actively working on automating this pre-processing and discovery stage.

What we learned

This project surfaced a small set of principles that consistently mattered more than any individual algorithmic choice. These lessons reflect what held across datasets, domains, and multiple rounds of evaluation — not just what worked once

Consolidation is the primary value driver

Deduplication, filtering, and semantic clustering of incoming events account for roughly 80 percent of the quality improvement. The sophisticated forgetting mechanisms add incremental value on top of a well-consolidated store. If you’re building agent memory and can only implement one thing, implement consolidation.

Decay rate is domain-relevant

We proposed a decay half-life of 24 hours. Empirically, 29 days performs far better. It is gentle enough to keep actively discussed issues alive, aggressive enough to deprioritize resolved ones over weeks. The right rate depends on your domain’s natural rhythm, which reinforces the insight about domain-relative time.

Evaluation methodology matters more than algorithm choice

The difference between our flawed evaluation (84.4 percent precision) and our corrected evaluation (97.2 percent precision) was larger than the difference between any two algorithm configurations. Getting the evaluation right with the correct stateful processing, complete datasets, and appropriate cadence is a prerequisite for drawing any valid conclusions about the algorithms themselves. As with most of our previous experience, getting the evaluation framework right is as crucial as building the product.

The memory store self-regulates

One of the most satisfying findings was that the system naturally converges to a stable memory size (400–500 events for our dataset). New arrivals are balanced by forgetting, and the store never grows unboundedly. This emergent behavior mirrors how human working memory maintains a relatively constant capacity despite continuous input.

Where this goes

Memory is the next frontier for AI agents. Not memory as in “store more tokens,” but memory as in:

1. Learn from experience
2. Prioritize what matters
3. Forget what doesn’t

We demonstrated a +21.8 percentage point improvement in retention precision with a memory store that self-regulates at 400–500 events. But the numbers are secondary to what they represent: an agent that can participate in real business operations over weeks and months. An agent that understands context accumulated across dozens of conversations, learns from outcomes it observes over time, and evolves as conditions change. That is the gap between today’s stateless chat interfaces and the autonomous agents that enterprises actually need.

Vector databases with cosine similarity are this era’s flat files. They work for simple cases but lack the sophistication that complex, long-running agent deployments require. The architecture we’ve described with consolidation, forgetting, maturation, and reconsolidation provides a principled foundation drawn from 500 million years of evolutionary R&D. The domain prototype work showed that attention mechanisms, domain-relative time, and pre-deployment knowledge acquisition are essential extensions that make the architecture portable across industries.

The tools exist. The neuroscience is well established. The engineering is tractable. What’s needed now is for the agent ecosystem to treat memory not as an afterthought, but as a first-class architectural concern.

References

• Richards, B.A. & Frankland, P.W. (2017). “The Persistence and Transience of Memory.” Neuron, 94(6), 1071–1084.
• Berry, J.A. et al. (2012). “Dopamine Is Required for Learning and Forgetting in Drosophila.” Neuron, 74(3), 530–542.
• Kitamura, T. et al. (2017). “Engrams and circuits crucial for systems consolidation of a memory.” Science, 356(6333), 73–78.