How I Gave Our Training Bot a Six-Session Memory With Hindsight

How I Gave Our Training Bot a Six-Session Memory With Hindsight

The first version of WorkPod’s mentor was embarrassingly generic. An employee could fail their communication score three sessions in a row, ask the mentor “what should I work on?”, and get the same boilerplate coaching every time. The mentor had no idea this person had been struggling with the same skill for a week. It was a system prompt, a conversation window, and zero continuity between sessions.

I spent most of my time on WorkPod solving this one problem: making the system remember. Not remember within a session — that’s just chat history. Remember across sessions, across days and weeks, scoped tightly enough that an engineer’s SDE training memories don’t pollute their PM training memories. The tool that made this work is Hindsight, and this is the story of how I integrated it, what design decisions mattered, and what we are taking next.

The Problem in Concrete Terms

WorkPod is a workplace simulation platform. Employees pick a role — Software Engineer, Product Manager, HR Manager, or an intern track — run a 45-minute session with AI teammates, handle an emergency scenario, and get scored on communication, task management, and pressure handling. The scores and feedback are real, specific, and quantified. But without memory, every session starts from zero.

Here’s what “no memory” looked like in practice. A new SDE intern runs their first session and scores 45 on communication — they barely engaged with teammates during sprint tasks. Session two, they score 52. Slight improvement, but still weak. When they open the private mentor channel and ask for guidance, the mentor gives advice that could apply to literally anyone. It doesn’t know about the 45 or the 52. It doesn’t know this is their third session. It doesn’t know they’ve been steadily improving at task management while stagnating on communication.

A human mentor would know all of this. That’s the whole point of shadowing programs — you pay a senior engineer to be the memory layer, to track a new hire’s trajectory over weeks. The question was: could I automate that memory at a fraction of the cost?

Why Hindsight, and What It Actually Gives You

Hindsight is a long-term memory layer for agents. The API is two operations: retain (write a memory) and recall (query memories by semantic similarity). You organize memories into named banks, each with its own vector embedding space. You write memories as natural language strings, and you query them with natural language strings. Hindsight handles the embedding, indexing, and retrieval.

What attracted me to this model is that it maps cleanly onto how training memory should work. Each employee-role combination is its own memory bank. Memories are session summaries — scores, feedback, what was completed, what was struggled with. And recall is driven by whatever the employee is currently asking about.

The Scoping Decision That Changed Everything

The first design choice — and the one that had the biggest impact on quality — was how to scope memory banks. I started with one bank per user. Simple. But the results were bad. When an employee had done both SDE and HR sessions, asking the SDE mentor “how do I handle code review pushback?” would sometimes surface HR-specific memories about conflict resolution policies. The semantic space was too broad, and the recall results were noisy.

The fix was a one-line convention:

function buildBankId(mongoUserId, role) { return workpod${mongoUserId}${role}; }

Every user-role combination gets its own bank. An employee practicing as an SDE has a workpod_abc123_sde bank. The same employee practicing as a PM has a completely separate workpod_abc123_pm bank. The mentor for their SDE sessions only sees SDE history. This narrower scoping gave dramatically better recall results — when the embedding space is limited to one role’s worth of sessions, semantic search returns much more relevant memories.

The tradeoff is more banks to manage. With five roles and a growing user base, the bank count scales at users × roles. I handle this with a local in-memory cache and idempotent bank creation:

const knownBanks = new Set();

async function ensureBank(bankId, role) { if (knownBanks.has(bankId)) return;

try { await client.createBank(bankId, { name: WorkPod — ${role.toUpperCase()}, reflectMission: You are a career growth memory for a WorkPod user in the ${role} role. Remember their session scores, struggles, completed tasks, and feedback to help them improve over time., }); } catch (err) { if (err.statusCode === 409) { // Bank already exists — that’s fine } else { throw err; } } knownBanks.add(bankId); }

The reflectMission field tells Hindsight what this bank is for — it influences how memories are indexed and reflected on internally. I think of it as the bank’s purpose statement: “you’re tracking career growth for an SDE, focus on scores, struggles, and feedback.” After the first call, ensureBank is a no-op for the lifetime of the server process.

The Retain Phase: Writing Memories That Are Worth Recalling

When a session ends, scores get persisted to MongoDB first. That’s the source of truth. Then, as a non-blocking follow-up, I write a memory to Hindsight:

export async function retainSessionMemory({ mongoUserId, role, report, tasksCompleted, emergencyTriggered, durationSeconds, }) { const bankId = buildBankId(mongoUserId, role); await ensureBank(bankId, role);

const memoryText = [ WorkPod session completed as ${role.toUpperCase()}., Duration: ${Math.floor(durationSeconds / 60)} minutes., Tasks completed: ${(tasksCompleted || []).join(‘, ‘) || ‘none’}., Emergency handled: ${emergencyTriggered ? ‘Yes’ : ‘No’}., Overall score: ${report.overallScore}/100., Communication: ${report.communication}/100., Task management: ${report.taskManagement}/100., Pressure handling: ${report.pressureHandling}/100., Feedback: ${(report.feedback || []).join(‘ ‘)}, ].join(‘ ‘);

await client.retain(bankId, memoryText, { metadata: { source: ‘workpod-session’, role, overallScore: String(report.overallScore), timestamp: new Date().toISOString(), }, }); }

A few things are intentional here. First, the memory is written as a single natural-language string, not structured JSON. This is important because recall is semantic — Hindsight searches by meaning, not by field names. A string that says “Communication: 45/100. Feedback: Minimal engagement with team during collaborative tasks” will surface when someone later asks “how do I get better at working with my team?” A JSON blob with {“communication”: 45} won’t match as well semantically.

Second, the metadata is there for structured filtering if I ever need it, but the primary retrieval mechanism is vector search on the text content. This is a deliberate design choice: write memories for humans (and LLMs), not for databases.

Third — and this was a hard-won lesson — the retain call is wrapped in a try/catch in the session controller:

try { await retainSessionMemory({ mongoUserId, role, report, … }); } catch (hindsightErr) { console.error(‘[hindsight] Memory retention failed (non-blocking):’, hindsightErr.message); }

If Hindsight is down, slow, or throws an error, the employee still gets their report, their session is still saved in MongoDB, and the API response is unaffected. Memory is an enhancement, not a dependency. I learned this the hard way during early testing when a misconfigured Hindsight URL caused the entire session-end flow to fail silently. Now, MongoDB is the write-ahead log and Hindsight is the enrichment layer.

The Recall Phase: Injecting Memory Where It Matters

The payoff happens in the mentor channel. When an employee sends a message to their private mentor, I intercept it before the LLM call and check for relevant past memories:

// Inside the Socket.io user-message handler if (useMentor && currentUser?.userId) { try { const memories = await recallMemories( currentUser.userId, room.role, content, 3 ); if (memories.length > 0) { const memoryBlock = memories .map((m, i) => [Past Session ${i + 1}]: ${m}) .join(‘\n’); enrichedContent = [CONTEXT FROM PAST SESSIONS — use this to personalise your advice]:\n + ${memoryBlock}\n\n[CURRENT QUESTION]: ${content}; } } catch (recallErr) { console.warn(‘[hindsight] Recall failed for mentor (non-blocking):’, recallErr.message); } }

The recall query is just the employee’s message. If they type “what should I focus on improving?”, Hindsight returns the most semantically relevant past session summaries — probably the ones with the lowest scores or the most pointed feedback. If they ask “how do I handle production outages better?”, it surfaces sessions where pressure handling was scored, especially ones where they struggled.

I cap results at three memories. This keeps the prompt size manageable and avoids diluting the current conversation with too much historical context. Three past session summaries add roughly 300–400 tokens to the prompt — enough to give the mentor real trajectory data without blowing the context window.

The recall is also non-blocking. If it fails, the mentor falls back to its regular behavior — still useful, just not personalized. The employee never sees an error.

What Memory-Aware Coaching Actually Looks Like

Here’s the before and after.

BEFORE:

AFTER:

An employee has run two SDE sessions. Communication scores: 45, then 52. Pressure handling: 71, then 68. They open the mentor channel and type: “What should I focus on?”

Without Hindsight, the mentor responds with generic guidance: “Focus on clear communication, break down tasks into smaller pieces, and stay calm under pressure.” Applicable to everyone. Useful to no one specifically.

With Hindsight, the recallMemories call surfaces those two session summaries. The enriched prompt includes the actual trajectory. The mentor responds: “Your pressure handling has been consistently solid — above 65 in both sessions. But communication is your weak spot. You went from 45 to 52, which is progress, but you’re still not initiating enough in team discussions. Next session, try proactively asking teammates for input on at least two tasks before the emergency scenario fires.”

That’s the kind of feedback that normally requires a senior engineer observing a new hire for weeks. Here it happens on day three, automatically, at the cost of a few hundred tokens and a sub-second Hindsight recall call.

Where We Are Taking This Next

Per-dimension memory writes. Right now retainSessionMemory() writes one memory string per session that concatenates all three scores together — communication, taskManagement, and pressureHandling — into a single paragraph. The problem is that when someone asks the mentor “how do I get better at handling emergencies?”, Hindsight has to match that question against a blob that also contains their communication and task scores, which adds noise. The next version would call client.retain() three times — once per scored dimension — so semantic search has a much tighter target to hit.

Recency filtering using existing metadata. We’re already writing timestamp: new Date().toISOString() into every memory’s metadata field. We just aren’t using it on the recall side yet. The plan is to filter recallMemories() results by timestamp before returning them — prioritizing the last three sessions over older ones. For someone who’s done eight SDE sessions, the memories from sessions one and two are much less relevant than what happened last week.

Bringing memory into the team chat channel. Right now only the private mentor channel runs recallMemories() before making an LLM call. But the AI teammates — Anjali Kumar (Tech Lead) and Sam Park (Backend) — run through the same Socket.io handler and currently have zero session history. The goal is to inject a lightweight one-line memory summary into their system prompts too — something like “This employee scored 45 on communication in their last session” — so Anjali can proactively involve them in discussions during sprint tasks rather than waiting to be asked.

Surfacing recall results in the Portfolio page. The /portfolio page already pulls session history from MongoDB to render role cards and score breakdowns. The next step is to also pull the most recent Hindsight memory per role and display the mentor’s last personalized observation directly on the card — so employees can see what the system has learned about them without having to open the mentor channel and ask.