Seasonality-Aware Log Anomaly Detection

Replacing hand-tuned static thresholds with a self-calibrating, per-pattern, time-of-day-aware alerting pipeline — using percentiles, a vector KB, and a daily cron.

Every observability platform eventually hits the same wall: the alert rules don’t scale with the system they’re supposed to watch.

Most log-alerting stacks start with a handful of rules built around static thresholds. It works — until the number of distinct error patterns outgrows the number of rules anyone is willing to maintain. Two failure modes dominate:

1. Coverage gaps. A new error pattern has no alerting until an engineer notices it and hand-writes a rule. Incidents slip through the seam between “pattern appeared” and “rule created.”
2. Threshold blindness. A fixed number like 1800 errors/hour treats Monday 9 AM peak traffic identically to Sunday 3 AM quiet hours — false positives during lulls, missed incidents during peaks.

This is the design of a system that replaces both: a dynamic-threshold anomaly detection pipeline that auto-calibrates per pattern, per hour-of-week, with zero manual rule maintenance

Design Principles

• No manual rules. Every pattern — including ones discovered five minutes ago — gets alerting coverage automatically.
• Seasonality-aware. Thresholds understand that traffic has a shape: weekday vs. weekend, peak vs. trough.
• Operationally cheap. Reuse the existing TSDB and key-value store. No new serving infrastructure, no model lifecycle to babysit.
• Explainable. Every fire-or-no-fire decision is reconstructable after the fact. A black box that pages you at 3 AM is worse than no system at all.

System Architecture

The pipeline splits into a hot path (every few minutes, makes decisions) and a cold path (daily, calibrates the thresholds the hot path reads). Both run as Kubernetes CronJobs.

The two paths share exactly two pieces of state: Mimir (hot path writes counts, cold path reads history) and the OCI NoSQL threshold store (cold path writes, hot path reads). Everything else is decoupled — which is what makes the system testable and lets either path fail without corrupting the other.

The Hot Path: From Raw Log to Paging Decision

1. Cluster, embed, match. Raw log lines are noisy — timestamps, request IDs, tenant identifiers. Tokenize and cluster structurally similar lines from Loki into patterns, collapsing variable tokens into <VAR> placeholders. Each pattern is embedded with Cohere (via AWS Bedrock) and matched against a vector knowledge base backed by S3 Vectors. A distance threshold separates "known pattern" from "novel"; novel patterns get inserted into the KB so the next cycle recognizes them. This embedding-based dedup is what makes "no manual rules" real — the KB discovers patterns, you never enumerate them by hand.

2. The noise gate. Not every recognized pattern deserves a page. A single boolean splits matched patterns three ways:

The subtlety: noise patterns are still pushed to Mimir. Muting a pattern shouldn’t make it invisible — “did this known-noisy pattern spike right before the outage?” is exactly the question an incident timeline needs. Mute the page, not the signal.

3. Bulk-fetch, then evaluate. A naive version issues N Mimir queries and N threshold lookups per cycle. Instead, do bulk queries — current counts and thresholds for all patterns at once — then evaluate via dictionary lookups. This gives every cycle a flat cost ceiling regardless of how many patterns are active, which is exactly what you want during an incident when everything fires at once.

The Cold Path: How Thresholds Are Born

This is where “seasonality-aware” lives. Once a day, rebuild every pattern’s thresholds from 28 days of Mimir history.

The 168-slot model

For each pattern, compute a threshold for every (day-of-week × hour-of-day) slot — 7 × 24 = 168 thresholds. Monday 9 AM and Sunday 3 AM are separate baselines.

We use the 95th percentile, not mean or max: the mean is dragged around by a single bad day; the max permanently raises the bar after one spike; p95 says “almost every normal Monday-9-AM sits below this” and is robust to outliers. Multiply by a per-pattern sensitivity multiplier — the one knob operators tune.

The fallback ladder

A pattern that’s existed for three days has zero samples in most slots. So threshold computation walks a fallback ladder, using progressively less-specific groupings until it finds enough data:

Every threshold is tagged with the rung that produced it. When an alert fires at 3 AM, the responder sees why that threshold was chosen — same_slot, n=22 versus flat, n=4 tell two different stories. The ladder is a ramp, not a cliff: a new pattern climbs from early_warning toward full same_slot precision around the 28-day mark, automatically.

The Hard Case: Rare-But-Critical Patterns

The most dangerous incidents aren’t the loud ones — they’re low-volume, high-severity: an auth subsystem failing for two requests, one tenant’s encryption silently breaking. A flat bootstrap misses these by construction.

For patterns with no calibrated history, the answer is a hybrid persistence-OR-burst rule:

fire  ⟺  (active in ≥6 of last 12 windows)  OR  (count ≥ floor)
              └──── persistent ────┘              └── burst ──┘

The insight: a count of 2 that recurs across six separate time windows is a slow-bleed incident; the same count of 2 in a single burst is probably noise. Persistence — not raw volume — distinguishes a real rare incident from a transient glitch.

The third case — count=4 across 8 windows — is the failure mode static thresholds are blind to. The persistence rule catches it.

Engineering for Production

A few decisions that separated “works in a notebook” from “runs unattended”:

Read-throughput as a first-class constraint. The hot-path threshold query in OCI NoSQL filters on (day_of_week, hour_of_day) — not the partition key — which is a full scan over thousands of rows every cycle, enough to trip provisioned read limits with 429 TooManyRequests. The fix: a secondary index on those columns, turning a full scan into a small indexed lookup (~100× fewer read units), backed by paginated reads with exponential backoff so a transient throttle degrades gracefully instead of blanking out every threshold.

Shadow mode. Before going live, the entire pipeline ran in production with one change: Zenduty calls were suppressed and logged as WOULD fire …. Comparing shadow output against the legacy rules surfaced bad multipliers and surprising patterns before they paged anyone. Shadow mode is the cheapest insurance on an alerting cutover — never deploy one without it.

Full recompute over incremental. The daily job recomputes all 168 slots from scratch rather than maintaining a rolling delta. It’s self-healing (one bad run is fixed by the next), trivially reasoned about (today’s threshold is a pure function of today’s window), and cheap. Incremental updates trade all three away for complexity you rarely need.

Honest Limitations

An architect who only sells the wins isn’t worth listening to. This is a volume-spike detector for known patterns — deliberately not a complete anomaly-detection solution. It does not yet catch:

• Drops — a pattern going silent (a fix, or a dead service?). Thresholds are upper bounds only.
• Slow drift — a baseline creeping up over weeks; the daily recompute silently absorbs it.
• Rate-of-change — a step change mid-hour that the rolling window averages away.
• Correlation — 20 patterns spiking together is one incident but fires as 20 alerts.

Each is addressable as an additive layer on the same architecture — none requires a rewrite. They’re the roadmap, not regrets.

Closing

The instinct on hearing “anomaly detection” is to reach for machine learning. For a large class of real problems — log-pattern volume spikes among them — that instinct is wrong. A per-pattern, per-time-slot percentile, with a graceful fallback ladder and a persistence rule for the rare cases, delivers explainable, operationally cheap, self-maintaining detection that an on-call engineer can actually trust.