How I Architected a Dual-Layer Memory System for an AI Chief of Staff

https://github.com/ysirishchandra-lgtm/foundermind-ai

Founders don’t have time to wait for loading spinners. We needed an assistant that could remember complex business context across dozens of scattered chat sessions, but it had to stream responses instantly.

This is the story of how I architected a dual-layer memory system that decouples vector storage from the critical response path, turning a sluggish RAG pipeline into a lightning-fast, persistent agent.

What FounderMind Does FounderMind is built on a lightweight Flask backend and a vanilla HTML/JS frontend. We use Llama 3.1 8B served through Groq because we wanted ultra-low latency token streaming.

To give the AI persistent context, we integrated the Hindsight GitHub repository’s open-source memory system.

The architecture is simple in theory:

The user asks a question. FounderMind retrieves relevant past context. The LLM answers using that context. The new interaction is saved for the future. But making this pipeline robust and fast required treating memory as an explicit, asynchronous engineering problem.

The Core Story: Decoupling Memory from the Critical Path The biggest mistake you can make when building Vectorize agent memory into your app is putting the database writes on the critical response path.

If I add a synchronous “write this conversation to the vector database” call right after the LLM generates a response, I completely destroy the speed advantage of using Groq.

To solve this, I decoupled the memory capture. We use the Hindsight Python SDK, but we wrap the retention call in a daemonized background thread:

python

import threading from hindsight_client import Hindsight memory = Hindsight(base_url=HINDSIGHT_API_URL) def retain_interaction(bank_id: str, user_message: str, assistant_reply: str): memory.retain( bank_id=bank_id, content=f”User asked: {user_message}\nAssistant replied: {assistant_reply}”, ) @app.route(“/chat”, methods=[“POST”]) def chat(): user_message = request.json[“message”] bank_id = get_bank_id_for_session()

1. Retrieve past context (Fast)

retrieved = recall_context(bank_id, user_message) system_prompt = build_system_prompt(retrieved)

2. Stream response instantly

reply = stream_llm_response(system_prompt, user_message)

3. Save memory asynchronously in the background

threading.Thread( target=retain_interaction, args=(bank_id, user_message, reply), daemon=True, ).start()

return reply The user sees the streaming response instantly. The heavy lifting of semantic indexing happens silently in the background.

Architecting the Fallback Layer Retrieval is great, but what happens when the cloud vector database times out or network latency spikes? For an AI Chief of Staff, amnesia is a fatal flaw.

I engineered a local SQLite fallback that acts as a safety net. If the semantic recall fails or takes longer than 1.5 seconds, the system instantly executes a highly optimized keyword search across the local session archive:

python

import sqlite3 def local_fallback_search(bank_id: str, query: str, limit: int = 5): conn = sqlite3.connect(“session_archive.db”) cur = conn.cursor() cur.execute( “”” SELECT content FROM interactions WHERE bank_id = ? AND content LIKE ? ORDER BY created_at DESC LIMIT ? “””, (bank_id, f”%{query}%”, limit), ) rows = cur.fetchall() conn.close() return [r[0] for r in rows]

def recall_context(bank_id: str, query: str): try: results = memory.recall(bank_id=bank_id, query=query, budget=800) if results: return results except Exception: pass

return local_fallback_search(bank_id, query) It’s a downgrade in precision — keyword matching isn’t as smart as semantic search — but it guarantees the system never fails silently. A soft degradation is always better than a hard crash.

What This Looks Like in Practice Because of this dual-layer architecture, the user experience is magical.

In testing, I opened a completely fresh chat session and typed: “Did we ever settle on annual vs. monthly pricing?”

Because the async threads had perfectly indexed my sessions from two weeks ago, the recall function instantly grabbed the exact decision we made (annual-first with a monthly option). Groq streamed the answer back to the UI in milliseconds. It felt less like a chatbot and more like a real colleague who actually remembers our meetings.

Lessons Learned Async writes are mandatory for UX. Never put vector database writes on the critical response path. Users will tolerate a slightly delayed memory index, but they won’t tolerate a slow chat interface. (If you’re building this, check the Hindsight docs for proper implementation patterns). Cloud semantic search needs a local keyword fallback. You cannot trust the network 100% of the time. A cheap SQLite LIKE query will save your app’s reputation when an API inevitably hiccups. Multi-tenancy starts at the memory layer. Using bank_id to strictly isolate memories per user is a design decision you have to make on day one. LLMs are a commodity; memory pipelines are the product. Getting a model to generate text is easy. Getting it to retrieve the right context, instantly, without hallucinating, is what actually makes the software valuable. Building FounderMind proved to me that the future of AI isn’t just about larger context windows. It’s about engineering smart, lightning-fast retrieval pipelines.