Designing an AI-Powered SOC Automation Platform with Splunk and Claude AI — Part 5

From context to memory and judgment — a local semantic retrieval layer that gives the AI precedent, a scope-aware knowledge base that separates legitimate infrastructure from real threats, and the closing of the “C2” false positive that ended Part 4.

New here? Part 1 covered the foundation: a brute-force rule, risk scoring, Claude AI triage, and a SOAR playbook that sends email alerts and logs incidents. Part 2 turned that into a MITRE ATT&CK detection engine — 12 detections converted from Sigma to SPL, index-time log normalization, and a test lab with Atomic Red Team. Part 3 gave the platform judgment — tamper-evident hash-chain logging, kill-chain correlation, and a score-based triage layer that decides what’s actually worth an AI’s attention. Part 4 gave it context — artifact-driven IOC enrichment, IOC-aware triage, and trend-based escalation. This part gives it two things it still lacked: memory, and the ability to tell its own infrastructure apart from an intruder’s.

Where Part 4 Left Off

Part 4 ended on a genuinely funny failure. I’d stood up an Oracle Cloud attacker box in Frankfurt, tunneled it to the lab over Tailscale, and finally had real external telemetry to reason about. Then Claude took one look at the Tailscale tunnel installing itself via a RunOnce registry key and confidently flagged it as attacker command-and-control.

And it was right. A remote-access tunnel persisting itself is textbook C2 behavior. The AI did flawless SOC reasoning. It just didn’t know that I — the analyst — installed the tunnel.

The platform reasoned about every event as if the world began five minutes ago. It had no memory of past cases it had already worked, and no place to record that “Tailscale is a sanctioned tool here.” Two missing faculties, one root cause: the system had context about indicators (Part 4’s enrichment) but no context about itself — its own history and its own environment.

Part 5 adds both:

• L3 — Semantic Retrieval. A local vector store that surfaces the nearest past cases for every new event, so the AI reasons with precedent instead of from a blank slate.
• The Knowledge Base. A scope-aware exception layer that lets an analyst record what’s known-legitimate — and finally closes the C2 loop.

Along the way, a VMware disk crisis, a “45x redundancy” discovery, and a statistical trap that almost had me ship a bad threshold.

L3 — Semantic Retrieval

From Part 1 the architecture diagram had an L3 slot labeled Semantic Retrieval (planned). It stayed planned for four parts on purpose. Semantic retrieval surfaces precedent from past cases — which requires having past cases. Building it over an empty vector DB would have been theater: an impressive-looking layer retrieving nothing. By Part 5 the incident log had matured enough to make it real.

The idea is simple. When a new event arrives, find the most similar past incidents and hand them to Claude as context — “here’s how cases like this played out before.” The implementation is where every interesting decision lived.

Decision 1: Local, not an embedding API.

The obvious path is OpenAI’s embedding endpoint. I went with a fully local model — ChromaDB’s default all-MiniLM-L6-v2 (ONNX, 384-dim) - for three reasons:

• Architectural consistency. Incident data is exactly the sensitive material a SOC shouldn’t ship to a third party: users, hosts, internal IPs, the AI’s own reasoning about your environment. The rest of the platform is on-prem-friendly; the retrieval layer had to be too. This runs air-gapped.
• The data doesn’t need the bigger model. For short technical strings — a technique ID, a tactic, a risk level — the practical difference between MiniLM’s 384 dimensions and a 1536-dim commercial embedding is negligible.
• Zero friction. pip install chromadb, and the model auto-downloads (~80 MB) on first run. No key, no quota, no per-embedding cost.

Decision 2: Index unique patterns, not raw rows.

The first time I looked at indexing the incident log, the numbers stopped me cold. 1,319 raw incident records — but only 29 unique (technique_id, user, host) patterns. A single technique (T1078) accounted for 975 of those rows. That's ~45x redundancy.

Indexing raw records would have filled the vector space with hundreds of copies of the same point. Every retrieval would just return more duplicates of whatever technique happened to be noisiest. So the index unit became the unique (technique_id, user, host) pattern, with one representative per pattern:

def select_representatives(records):
    # one representative per (technique_id, user, host)
    # prefer English analysis > longer analysis
    # drop generic "[AUTO-LOG]" analyses (they pollute embeddings)
    # drop <40-char analyses (junk)

That collapsed 1,319 raw records to 13 clean representatives. The vector space now holds signal instead of echoes.

Decision 3: The query and the document are deliberately asymmetric.

This one took a real debugging session to get right. My first version embedded the same rich text for both indexing and querying — technique + tactic + risk + the full AI analysis. It produced terrible distances.

The reason: a live event being triaged has no AI analysis yet — that’s the whole point, the analysis is what we’re deciding whether to spend. So querying with the rich builder meant comparing a mostly-empty 2,000-character document against full ones, and the emptiness dominated the distance.

The fix was to split the builders:

build_document(record)  # rich - for the index: technique + tactic + risk + full AI analysis
build_query(event)      # short - for the query: technique + tactic + risk, no analysis

Asymmetry by design: a short query against rich documents keeps the distances meaningful.

Decision 4: The language of the analysis was quietly breaking retrieval.

An embedding model clusters by meaning — but only in languages it understands well. My AI analyses had been written in Turkish, and all-MiniLM is English-weighted. In its vector space, semantically different Turkish analyses landed in nearly the same region: the model couldn't tell them apart, so retrieval returned near-random neighbors.

The better fix was cheaper and improved everything else too: standardize the AI's output language to English. No model swap, and the existing MiniLM is excellent in English.

So the single-event prompt, the kill-chain prompt, the console output, and the email template all moved to English. A backward-compatible parse regex ((?:OLAY|EVENT)\s*#\d+) keeps older Turkish records readable while new ones come out in English - and the language-preferring representative selection quietly retires the old Turkish cases as English data accumulates.

Decision 5: No fixed distance threshold

This is the decision I’m most glad I didn’t rush. The natural way to filter retrieval results is a distance cutoff: “only show cases closer than X.” So I measured the distribution, self-matches excluded:

SAME technique (signal):     min=0.561  median=0.707  max=0.727
DIFFERENT technique (noise): min=0.747  median=1.071

A clean gap — 0.727 vs 0.747. It looked like a textbook threshold sat right there at ~0.74.

It was a mirage. When I broke the numbers down per-technique, most of the 13 representatives were the only example of their technique in the index. A single-example technique can never produce a low distance — there’s no second instance to be close to. The entire “clean gap” was coming from the three techniques that happened to have multiple examples. On a small, mostly-single-example dataset, the statistic was describing an artifact of the data, not a real boundary.

So: no hard threshold. The pipeline retrieves the nearest two cases and hands them to Claude with an explicit label — “context hint, may be unrelated, use your own judgment, do not treat as ground truth.” Filtering by reasoning, not by a brittle number I’d have to defend. A threshold gets measured once each technique has 3–5+ examples and the statistics mean something.

Wiring it in.

Retrieval feeds L4 in both modes. For single events, each event pulls its nearest two past cases into a PAST SIMILAR CASES block. For kill-chains, the chain's first event (usually the initial access) is the anchor, and its neighbors go into a PAST SIMILAR CAMPAIGNS block. Match quality is shown as HIGH / MEDIUM / LOW rather than a raw distance - similarity=0.26 reads as "irrelevant" to a language model, which is the opposite of what a 0.74 distance means.

One debugging note worth passing on, because it cost me an hour: the retrieval block goes into the prompt sent to Claude, not the response printed from Claude. I kept grepping the console output for my injected context and not finding it — because I was looking at the output when the context was in the input. When your instrumentation shows nothing, check whether you’re inspecting the right side of the API call.

The Knowledge Base

Retrieval gives the platform memory of its own cases. The knowledge base gives it awareness of its own environment — and it’s what finally kills the C2 joke.

The premise: detection without context produces confident false positives. Tailscale’s 100.64.0.0/10 CGNAT range initiating SMB is, in the abstract, suspicious - it's a non-RFC1918 address doing lateral-movement-shaped things. The detection is correct. What's missing is a place to say "that range is mine."

So the knowledge base is an analyst-curated layer of exceptions. Two record types: legitimate_tool (an indicator that's known-good) and closed_case (a resolved incident recorded as reusable knowledge). But the design detail that matters most - the one I'd build this way in production - is scope.

Scope: how far does an exception reach?

Early on I had a choice. The simple version: an exception is an exception — if an IP or process is marked legitimate, it’s legitimate everywhere. It’s less code and it “just works” for the Tailscale case.

I built the harder version instead, because the simple one is a security hole. Consider: an analyst silences a T1078 (authentication) false positive by marking a service account’s login IP as legitimate. Reasonable. Months later, that service host is compromised and the attacker uses it as a pivot for C2 (T1071) or exfiltration (T1048). In the simple design, that IP is marked “legitimate” — globally — so it sails through, no threat-intel check, no flag. A narrow allowance given for one technique became a blind spot for all of them. That’s exactly the “trusted” pivot an attacker hunts for.

So every exception carries a scope:

• **infrastructure** - global, technique-independent. For things that are legitimate everywhere: a VPN range, a corporate proxy, an internal DNS resolver. Tailscale is infrastructure.
• **detection* - limited to specific techniques. For behavioral allowances: a signed updater that writes a particular Run key is fine for that persistence technique* - but if that same process later shows up doing process injection, the exception doesn't apply and it gets scrutinized like anything else. OneDrive's updater is detection, scoped to T1547.001.

The enforced rule: a detection-scoped exception never silently widens. Mark something legitimate for one technique and it stays legitimate only for that technique. The proof is in the behavior:

Tailscale (infrastructure)  under T1078  → legitimate ✓
Tailscale (infrastructure)  under T1071  → legitimate ✓   (global: any technique)
OneDrive  (detection)       under T1547.001 → legitimate ✓
OneDrive  (detection)       under T1055     → NOT legitimate ✗   (blind spot closed)

The knowledge base feeds two layers, with a division of labor. IOC enrichment checks it before any threat-intel call — a known-legitimate infrastructure match returns instantly (known_legitimate, tagged [KB]), saving both the API request and the false alarm. AI analysis injects matching, in-scope knowledge into the prompt as verified ground truth - explicitly distinct from the "may be unrelated" retrieval hints - so Claude reasons with confirmed context instead of guessing.

Exceptions go in one line, via a small CLI, with detection as the safe default so a scope has to be chosen to go global:

python3 kb_add.py --ip-prefix "100." --reason "Tailscale VPN range" --scope infrastructure
python3 kb_add.py --process "OneDriveSetup.exe" --reason "OneDrive updater" \
                  --techniques T1547.001 --scope detection

Closing the Loop

Here’s the payoff. The exact detection that made Claude cry C2 in Part 4, run again with the Tailscale range registered as infrastructure:

[ARTIFACT] IP 100.109.237.72 → KNOWN_LEGITIMATE (score:0) [KB]

AI ANALYSIS:
1. ATTACK OR FALSE POSITIVE?
   This is a FALSE POSITIVE. The source IP 100.109.237.72 is part of the Tailscale VPN CGNAT range (100.64.0.0/10), a verified legitimate internal test infrastructure...
2. RISK SEVERITY:
   Downgraded from HIGH to LOW based on the verified infrastructure context.
4. INTERNAL OR EXTERNAL THREAT?
   Internal source despite the non-private IP. 100.64.0.0/10 is the IANA-designated shared address space used by Tailscale's CGNAT overlay - functionally internal.

Same detection. Same telemetry. But now the system knows the difference between the analyst’s own tunnel and an intruder — and says so, with the reasoning to back it. The enrichment layer skips the pointless threat-intel lookup, and Claude, handed the range as ground truth, downgrades the event instead of escalating it.

The joke from Part 4 is now a closed case — literally, it’s the kind of thing the closed_case record type exists to remember.

The Bugs and Decisions That Taught Me the Most

1. A “clean” statistic can be an artifact of your data. The distance threshold looked obvious and defensible — a clean gap in the histogram. It was an illusion created by single-example techniques that structurally can’t score low distances. Aggregate numbers lie when the aggregate is tiny and lopsided. I almost shipped a threshold that would have quietly filtered out real matches. Lesson: before trusting a distribution, break it down by the dimension that actually varies — here, per-technique — and ask whether the pattern survives.

2. Instrument the right side of the call. I spent an hour convinced retrieval was broken because I couldn’t see the injected context in the console. It was working perfectly — the context was in the prompt going to Claude, not the response coming back. When your logging shows nothing, verify you’re inspecting input vs. output before you start “fixing” a system that isn’t broken.

3. The cheapest fix beat the fancy one. Bad retrieval from mixed-language analysis had an expensive fix (a 470 MB multilingual model) and a cheap one (write the analyses in English). The expensive fix triggered a disk crisis and I rejected it anyway; the cheap fix solved the retrieval problem and made the pipeline output more portable. Reach for the change that removes the problem, not the one that accommodates it.

4. Scope is a security boundary, not a convenience field. The tempting version of the knowledge base — “legitimate is legitimate” — is one compromised host away from becoming an attacker’s cloak. Making exceptions carry an explicit, enforced scope was more work for a feature that mostly needed to un-flag one VPN range. But an allowance that can’t silently widen is the difference between a tuning tool and a bypass primitive. In security, the boundary that stops the rare bad case is worth the friction in the common good one.

5. Deferring a layer can be the correct call. L3 sat “planned” for four parts. It would have been easy to build it early to complete the diagram — and it would have retrieved nothing meaningful from an empty store. Shipping the empty version would have been worse than shipping nothing. Some features are only honest once their prerequisites exist.

Where This Leaves Us

After Part 5, the platform has memory of its work and awareness of its environment:

✅ Semantic retrieval (L3) — a local, offline ChromaDB store surfaces the nearest past cases for every event and kill-chain, injected into Claude’s context as precedent

✅ Representative indexing — unique (technique_id, user, host) patterns instead of raw rows, collapsing ~45x redundancy into signal

✅ Scope-aware knowledge base — infrastructure (global) vs detection (technique-scoped) exceptions, where a narrow allowance can never silently widen into a blind spot

✅ Enrichment layer separation — IOC verdicts, AI reasoning, and asset data live in a dedicated block (schema v2.1); raw telemetry stays clean, multiple sources accumulate

✅ The “C2” false positive, closed — the analyst’s own Tailscale tunnel is now correctly resolved as a verified false positive, HIGH downgraded to LOW

✅ English standardization — analysis, prompts, and output unified for retrieval quality and portability

The code is fully open: **github.com/thesaep/ai-soc-automation**

The two ideas I’d most like to hear other people’s take on: how you scope allow-listing so it doesn’t rot into a bypass, and how you handle retrieval thresholds while your case history is still small. The comments are open.

This is Part 5 of a build-in-public series on building an AI-powered SOC automation platform from scratch. Parts 1–4 are linked at the top and on my profile.