With Hindsight, My AI Remembered — and Never Let Me Down

https://github.com/durgam-sai-ashwidha/foundermind-ai

Building an AI Chief of Staff That Actually Remembers: What I Learned Wiring Persistent Memory Into an LLM App

The first time FounderMind correctly recalled a pricing decision I’d made three weeks earlier, in a session I’d completely forgotten existed, I stopped mid-sentence and just stared at the terminal for a second. That’s the moment I knew memory, not the model, was the actual product.

I’m a founder who also writes the code, and I built FounderMind — an AI Chief of Staff for founders — with my teammates Sirish and Priyanshu. The premise is simple to state and annoyingly hard to build: an assistant that remembers your business across sessions, not just within a single chat window.

Founders are already context-switching between term sheets, roadmaps, and marketing syncs all day. An AI tool that makes you re-explain your company from scratch every session isn’t saving you time — it’s adding a tax on top of the switching you’re already paying.

This is the story of how we built the memory layer, why the obvious approach — just paste more context — doesn’t scale, and what changed once we treated memory as an explicit system instead of an implicit side effect of a long prompt.

What FounderMind Does

FounderMind is a Flask app with three moving parts: a chat interface, an LLM (Llama 3.1 8B, served through Groq for low-latency streaming), and a memory layer built on Hindsight GitHub, an open-source agent memory system.

The chat interface is intentionally boring — vanilla HTML/CSS/JS, no framework — because the interesting engineering isn’t in the UI. It’s in what happens between a user’s message arriving and the LLM generating a response.

At a high level, every message goes through this sequence:

1. The user sends a message.
2. Before the LLM sees anything, FounderMind queries the memory layer for relevant prior context — past decisions, meetings, priorities — related to the current message.
3. That retrieved context gets folded into the system prompt.
4. The LLM generates a response, using both the current message and the retrieved memory.
5. The interaction itself gets written back into memory, so it’s retrievable in future sessions.

None of these steps is individually hard. Getting all five to work together reliably, quickly, and without the user noticing the machinery is where most of the engineering time went.

The Core Story: Memory as a First-Class System, Not a System Prompt Hack

The naive version of “AI with memory” is: keep a running log of everything the user has ever said, and stuff as much of it as fits into the context window. That works for a demo. It falls apart the moment your conversation history grows past a few dozen sessions — you’re paying for tokens on irrelevant history, diluting the model’s attention with noise, and eventually blowing past the context window entirely.

What you actually want is selective recall: given the current message, retrieve only the memories relevant to it, in a form compact enough not to crowd out the actual conversation. That’s a retrieval problem, not a “bigger context window” problem, and it’s why we built the memory layer around Hindsight instead of a homegrown embeddings table.

Here’s the retain/recall pattern we ended up with, using the Hindsight Python client:

from hindsight_client import Hindsight 

memory = Hindsight(base_url=HINDSIGHT_API_URL) 

def retain_interaction(bank_id: str, user_message: str, assistant_reply: str): 
    memory.retain( 
        bank_id=bank_id, 
        content=f"User asked: {user_message}\nAssistant replied: {assistant_reply}", 
    ) 

def recall_context(bank_id: str, query: str, token_budget: int = 800): 
    results = memory.recall( 
        bank_id=bank_id, 
        query=query, 
        budget=token_budget, 
    ) 
    return results

The bank_id is the piece that made multi-tenancy trivial. Each founder using FounderMind gets their own memory bank, so retrieval is automatically scoped to their business and never leaks across accounts — a detail that's easy to overlook until you're the one debugging why User A's confidential roadmap showed up in User B's context window.

The recall call runs semantic search under the hood, so a question like “what did we decide about pricing?” can retrieve a memory that never uses the word “pricing” at all — it just needs to be about the topic. That’s the difference between recall and plain keyword search, and it’s why FounderMind can answer a vague, human question instead of requiring you to phrase things exactly the way you did the first time.

Capturing Memory Without Blocking the Response

The other constraint was latency. We chose Llama 3.1 8B served through Groq specifically because Groq’s whole value proposition is fast token streaming. Adding a synchronous “write this to memory” call after every response would have reintroduced the exact lag I was trying to avoid.

So capture happens on a background thread, off the request/response path entirely:

import threading 

@app.route("/chat", methods=["POST"]) 
def chat(): 
    user_message = request.json["message"] 
    bank_id = get_bank_id_for_session() 

    retrieved = recall_context(bank_id, user_message) 
    system_prompt = build_system_prompt(retrieved) 

    reply = stream_llm_response(system_prompt, user_message) 

    threading.Thread( 
        target=retain_interaction, 
        args=(bank_id, user_message, reply), 
        daemon=True, 
    ).start() 

    return reply

The user sees the same instant streaming response they’d get without memory at all. The retain call happens after the fact, asynchronously — if it’s slow or briefly unavailable, it doesn’t touch the request the user is actually waiting on. That’s what keeps FounderMind feeling like a live participant in the conversation instead of a tool with a noticeable tax on every turn.

The Fallback: What Happens When Retrieval Fails

Early on, I hit a failure mode that made me rethink the whole design: what happens when the memory service is unreachable, or recall legitimately returns nothing useful?

The naive failure mode is the assistant just… doesn’t have memory that turn, silently. For a product whose entire pitch is “it remembers,” a silent degradation is worse than an honest one — you don’t find out until the AI confidently tells you it has no idea what you’re talking about, in a conversation where it obviously should.

So we added a local SQLite fallback that keeps its own lightweight archive of past interactions, searchable by keyword rather than semantics:

:

import sqlite3 

def local_fallback_search(bank_id: str, query: str, limit: int = 5): 
    conn = sqlite3.connect("session_archive.db") 
    cur = conn.cursor() 
    cur.execute( 
        """ 
        SELECT content FROM interactions 
        WHERE bank_id = ? AND content LIKE ? 
        ORDER BY created_at DESC LIMIT ? 
        """, 
        (bank_id, f"%{query}%", limit), 
    ) 
    rows = cur.fetchall() 
    conn.close() 
    return [r[0] for r in rows] 

def get_context(bank_id: str, query: str): 
    try: 
        results = recall_context(bank_id, query) 
        if results: 
            return results 
    except Exception: 
        pass 

    return local_fallback_search(bank_id, query)

It’s a downgrade, not a replacement — keyword matching over semantic recall is a real loss in quality. But it converts a hard failure into a soft one. The assistant still has something to work with, and the failure mode changes from “confidently clueless” to “slightly less precise.” For a tool that’s supposed to be trustworthy about what it does and doesn’t know, that distinction matters more than it sounds like it should.

What This Looks Like in Practice

A concrete example, roughly reconstructed from a real session: I asked FounderMind, in a brand-new chat session, “Did we ever settle on annual vs. monthly pricing?”

It had no prior turns in that specific session to draw on — exactly the case where a stock LLM chat would say “I don’t have any information about that.” Instead, the recall step pulled a memory from a session two weeks earlier where I’d walked through the tradeoffs and landed on annual-first with a monthly option for smaller teams. The response referenced that decision directly and asked whether anything had changed since — which is the actual behavior you want from a chief of staff, not a chatbot. It doesn’t just answer; it checks whether the standing decision still holds.

Lessons Learned

1. Memory is a retrieval problem, not a context-window problem. Bigger context windows don’t fix “the AI forgot,” because irrelevant history is noise even when it technically fits. You need something doing the job of deciding what’s relevant — see Vectorize agent memory for the broader argument, which matched what I ran into in practice.
2. Isolate memory writes from the response path. Anything that writes to a memory store should be async and non-blocking. If it’s on the critical path, you’ll eventually be forced to choose between latency and completeness, and you shouldn’t have to. The Hindsight docs cover this pattern if you want to implement it correctly the first time.
3. A degraded fallback beats a silent one. When retrieval fails, don’t let the system quietly behave as if it has no memory. Build a cheap, less-precise fallback and use it, so failure looks like “slightly worse” instead of “completely different tool.”
4. Scope memory per tenant from day one. Adding a bank_id (or equivalent) after the fact, once real user data is already mixed together, is a much worse problem than designing for isolation up front.
5. The LLM is the easy 10%. Getting a model to generate a fluent reply is table stakes at this point. The product-defining work is what context you retrieve, how you inject it, and what you do when that pipeline doesn’t behave — none of which the model itself has any say in.

FounderMind is still evolving, but the core lesson has stuck: if you’re building something people are supposed to use every day, the model is not the differentiator. The memory system is.