My agent finally gets déjà vu — and it’s saving us actual money

Hindsight Is 20/20 (Ours Also Has an API)

I got tired of re-solving the same incident every few months, so I built a system that remembers. Not “remembers” in the vague, marketing-slide sense — I mean it stores every incident we’ve resolved as a searchable, structured memory, and the next time something similar breaks, it tells us exactly what worked last time and why. This is the story of building that system, and of the one design decision — reaching for a dedicated memory layer instead of just a bigger vector index — that ended up mattering more than anything else in the stack.

What It Actually Does (No, Really)

The system, which I’ve been calling Incident Response, sits between your alerting and your on-call engineer. When an incident comes in — through a webhook that mimics how a real monitoring tool like PagerDuty would call it, or typed directly into a dashboard — it does four things in order: embeds the description, searches for similar past incidents, asks an LLM to draft a fix grounded in those matches, and lets a human approve or edit before anything gets written back permanently.

Under the hood it’s a small Flask app with two stores doing two different jobs. Postgres is the source of truth — every incident is a row with a title, description, root cause, resolution, service, severity, and a timestamp. Qdrant is the vector index — every incident also gets embedded and stored as a point, keyed by the exact same id as its Postgres row, so a similarity search on one side always maps cleanly back to the full record on the other. Nothing exotic. The interesting part isn’t the two stores — it’s what sits on top of them.

Cosine Similarity Is Not a Personality

Cosine similarity search gets you a long way. Embed a new incident, compare it against everything you’ve stored, return the closest matches — that’s maybe forty lines of code and it works well enough to demo. But it has a specific, structural blind spot: it only ever tells you about pairs. Incident A looks like incident B. It has no concept of “this is the fourth time this quarter that a connection pool has exhausted, across four different services, and nobody’s escalated it as a systemic problem.”

That’s a different kind of question, and it’s not one you can answer by ranking nearest neighbors. It requires something that looks across many retrieved memories, holds them together, and asks what they have in common — then keeps that synthesis around so it doesn’t have to be re-derived every time. That’s the actual job of a long-term memory system, and it’s a different problem than search. This is where Hindsight earned its place in the architecture instead of being an add-on.

Hindsight gives you three operations instead of one: retain, recall, and reflect. Retain is what you'd expect — store a piece of information. Recall is retrieval, but running four strategies in parallel (semantic, keyword, graph, and temporal) instead of pure vector similarity, which matters more than it sounds like it should — a plain-text error code match that vector search misses because the surrounding sentence is oddly phrased is exactly the kind of thing keyword recall catches. Reflect is the part that doesn't have a clean analog in a standard RAG pipeline: it takes several recalled memories and reasons over them together, producing a synthesized observation rather than a ranked list. That's the mechanism that actually answers "is this systemic," not just "is this similar."

Wiring It In Without Building a Single Point of Failure

The integration point is deliberately narrow. Every confirmed incident gets retained as a structured experience — service, severity, root cause, resolution, and whatever lesson the resolution implies:

hindsight_bank.retain(
    content=f"{incident['title']}: {incident['root_cause']} "
            f"Resolution: {incident['resolution']}",
    metadata={
        "service": incident["service"],
        "severity": incident["severity"],
        "incident_id": incident["id"],
    },
)

On the way in, before the LLM ever sees a prompt, the same incident description gets run through recall:

memories = hindsight_bank.recall(
    query=new_incident_description,
    limit=5,
)

And when there’s enough signal across those memories, reflect gets a chance to synthesize before the suggestion prompt is assembled:

if len(memories) >= 2:
    synthesis = hindsight_bank.reflect(
        query="What pattern connects these incidents?",
        memories=memories,
    )
    prompt_context["pattern"] = synthesis

The guard on len(memories) >= 2 is intentional and, honestly, took a couple of iterations to get right. Calling reflect on a single memory is wasted latency and money — you're asking a model to find a pattern in one data point. It also matters that this whole layer is optional at the code level, not just in spirit:

try:
    memories = hindsight_bank.recall(query=new_incident_description, limit=5)
except HindsightUnavailableError:
    memories = []

If Hindsight is down or misconfigured, the core retrieve-and-suggest loop against Postgres and Qdrant keeps running exactly as before. That fallback isn’t defensive paranoia — it’s the difference between a memory layer and a single point of failure sitting in the critical path of an incident response tool, which is a spectacularly bad thing to have go down during an actual incident.

Show Me the Receipts

Here’s a concrete run, not a hypothetical. I fed the system a description modeled on a real class of failure: a payment service timing out under load because its Postgres connection pool was capped too low. Resolved it, confirmed the fix, and it got retained.

A few incidents later, I gave it a checkout service throwing 504s during a traffic spike — different service, different-looking symptoms on the surface. Recall pulled the payments incident as the closest match by meaning, not by any shared keyword. The suggested fix cited it directly: same root cause pattern, connection pool exhaustion, apply the same fix. I confirmed it.

Then I gave it a third, unrelated-looking incident: a reporting service getting OOMKilled during nightly batch jobs. This is where reflect did something recall alone couldn’t — instead of just surfacing the reporting-specific match, it synthesized across the growing memory and flagged that this was starting to look less like three isolated incidents and more like a pattern of under-provisioned resource limits across multiple services, worth an infrastructure review rather than three separate one-off patches. Recall gets you “here’s what’s similar.” Reflect gets you “here’s what these similar things mean together,” and that distinction is the actual value of treating agent memory as its own subsystem instead of a side effect of your vector database.

Lessons I’d Yell at Past Me

Retrieval and memory are not the same thing, and conflating them costs you the interesting behavior. A vector index answers “what’s similar.” A memory layer answers “what have I learned.” You can build the first in an afternoon. The second is a different design problem, and bolting reflection onto a bare vector store as an afterthought produces worse results than building for it from the start.

Make the memory layer optional in code, not just in architecture diagrams. It’s easy to draw a box labeled “guarded and optional” on a whiteboard. It’s a different discipline to actually wrap every call in a try/except that degrades gracefully, and to test that degraded path as seriously as the happy path. An incident response tool that becomes another incident when its memory service hiccups has failed at its one job.

Don’t call the expensive reasoning step on insufficient data. The len(memories) >= 2 guard before reflect is a small line, but it's the difference between a system that's thoughtfully selective about when it synthesizes and one that burns latency and tokens finding "patterns" in noise.

Keep the two stores linked by a boring, explicit key. Using the same integer id for a Postgres row and its Qdrant point sounds too simple to mention, but it’s what makes the whole system debuggable. When retrieval returns something surprising, you can always trace it back to one row, one vector, no ambiguity.

Human confirmation before write-back is not a nice-to-have. Every fix that gets retained into long-term memory went through a person first. That’s what keeps the memory trustworthy as it grows — an agent that writes its own unverified guesses back into its own memory is building a feedback loop with no correction mechanism, and that compounds in the wrong direction just as reliably as it compounds in the right one.

The pattern underneath all of this isn’t specific to incident response. Any system that’s supposed to get better with use — not just bigger — needs somewhere to keep what it’s learned that isn’t simply “more rows in a table.” Hindsight’s documentation frames this as the difference between storage and memory, and building on top of it made that distinction concrete for me in a way that reading about it never quite did. Hindsight is supposed to be 20/20. It turns out you have to build the reflection step on purpose — it doesn’t show up for free just because you’re storing a lot of text in a database.