I Broke My Own Serverless App on Purpose (Chaos Engineering on a Budget with AWS FIS)

Why I did this

Tyche has been running quietly in production for [6 months]. It hadn’t gone down. That should have felt reassuring, but it actually made me nervous, an app that has never failed is an app that has never been tested under failure. I didn’t know whether it was resilient or just lucky.

So I decided to find out the honest way: I used AWS Fault Injection Simulator (FIS) to break my own architecture on purpose, in a controlled experiment, and watch what happened.

This post walks through the experiment design, the faults I injected, what broke, what didn’t, and the three changes I made afterward that actually mattered.

The architecture under test

A quick sketch of what I was testing:

• API Gateway → Lambda (business logic) → DynamoDB (state) → EventBridge (async fan-out) → a second Lambda for downstream processing
• CloudWatch Alarms wired to SNS for on-call notification

Nothing exotic. This is the shape of a huge number of serverless AWS apps, which is part of why I think this experiment is useful beyond my specific project.

Designing the experiment, not just “turning things off”

Chaos engineering has a bad reputation among people who’ve only heard the pitch (“randomly kill things in prod!”) without the discipline behind it. The actual practice, and the reason FIS is built the way it is, comes down to four steps:

1. Define steady state: what does “healthy” look like, in metrics? For me: p99 latency under 800 ms, error rate under 0.5%, DLQ depth at zero.
2. Hypothesize: “If Lambda concurrency is throttled, I believe API Gateway will return 429s, but the DLQ will absorb failed EventBridge deliveries without data loss.”
3. Inject a real-world fault, not a random one. I picked faults that map to failures I could plausibly hit: throttling, added latency, and a dependency (DynamoDB) becoming briefly unavailable.
4. Learn and fix, whether or not the hypothesis held.

FIS experiment templates let you encode exactly this: a target (which Lambda, which percentage of traffic), an action (throttle, add latency, terminate), and a stop condition tied to a CloudWatch alarm so the experiment safely halts itself if things go sideways for real.

Experiment 1: Lambda throttling

Action: aws:lambda:invocation-error / concurrency throttling on the business-logic Lambda, targeting 50% of invocations for 5 minutes.

Hypothesis: API Gateway would surface 429s to the client, and the client-side retry logic (exponential backoff, already implemented) would mostly paper over it, but I expected the DLQ to catch a few messages.

What actually happened: Latency for successful requests barely moved, which was good, throttled invocations failed fast rather than degrading everything else. But my retry logic on the client used a fixed 3-retry cap with a short backoff window, and under sustained throttling that wasn’t enough headroom. A slice of legitimate requests errored out to the user instead of eventually succeeding.

Fix: Widened the backoff ceiling and added jitter. More importantly, I set a reserved concurrency floor on the Lambda so a burst elsewhere in the account couldn’t starve this function of capacity in the first place.

Experiment 2: Injected latency on the DynamoDB path

Action: Simulated added latency on calls hitting the table (via a Lambda-layer fault injection, since FIS doesn’t natively fault inject managed DynamoDB calls. More on that gap below).

Hypothesis: Lambda timeout (set to 10 s) would absorb it. Worst case, some requests time out, but nothing is left inconsistent.

What actually happened: This is where it got interesting. A subset of writes were partially applied. The DynamoDB write itself succeeded, but the Lambda timed out before it could publish the corresponding EventBridge event, because both were sequential inside a single invocation. The downstream processor never learned that state had changed. No error was ever logged, because nothing “failed.” It just silently never happened.

This is the kind of bug that chaos experiments are worth running specifically to find, because it doesn’t show up in a single unit test, a synthetic load test, or a code review. It only shows up when something slow happens at exactly the wrong moment between two operations that were implicitly assumed to be atomic.

Fix: Moved to a transactional outbox pattern: write state and the “event to publish” record to DynamoDB in a single transactional write, then let a DynamoDB Streams-triggered Lambda handle the actual EventBridge publish. Now a timeout anywhere downstream can’t create a silent gap.

Experiment 3: Regional dependency slowdown

Action: Latency injection targeting the second (downstream) Lambda, simulating a slow third-party API it calls.

Hypothesis: This should be contained. It’s async, off the critical path, and shouldn’t affect user-facing latency at all.

What actually happened: Correct, mostly. The user-facing path was unaffected. But I hadn’t set a DLQ redrive policy, so once messages started failing after Lambda’s max retry duration, they piled up in the DLQ with no automated remediation. In a real incident, that pile would have sat there until someone noticed manually.

Fix: Added a scheduled DLQ redrive Lambda plus a CloudWatch alarm on DLQ depth > 0, wired to SNS.

The honest limitations of FIS here

Worth being upfront about: FIS’s native fault library is strongest for EC2, ECS, EKS, and networking-level faults (packet loss, CPU stress, AZ disruption). For a fully serverless stack, a lot of the interesting faults, such as DynamoDB latency and third-party API slowness, require you to build your own fault injection into the code path (a feature flag or a Lambda extension that adds artificial delay) rather than relying purely on the managed FIS actions. That’s not a knock on FIS. The experiment framework (steady state, stop conditions, safety controls) is still genuinely useful even when the fault itself is hand-rolled.

What I’d tell someone starting this

• Don’t start with production. Run your first experiments against a staging environment with production-like traffic patterns, and only graduate to production once you trust your stop conditions.
• The bug worth finding isn’t “the service went down.” It’s “the service kept running and quietly did the wrong thing.” Latency and partial-failure faults find more of these than hard kill switches do.
• Write the hypothesis down before you run the experiment. It’s tempting to skip this, but the gap between what you predicted and what happened is where the actual learning is.