The Deployment That Broke Production (And the Feature Flags That Saved It)

It was a Thursday afternoon. Our team had just shipped a revamped loan application flow — a fairly large change that touched form validation, a new eligibility API call, and a redesigned step-progress indicator. QA had signed off. Staging looked clean. We deployed to production at 3:47 PM.

By 4:02 PM, our support channel was on fire.

The new eligibility API call was returning a 500 for roughly 30% of users — specifically users who had a pre-existing application in a legacy status code we hadn’t accounted for. Every one of those users saw a blank white screen and couldn’t complete an application. By the time we detected it, correlated it, rolled back, and redeployed, forty-seven minutes had passed. That’s forty-seven minutes of real loan applications not being submitted. The revenue impact was measurable. The post-mortem conversation was uncomfortable.

What made it worse: none of this was technically hard to prevent. We just hadn’t built the infrastructure to do it.

What Actually Went Wrong

The proximate cause was a missing status code in our eligibility mapper — one of those bugs that doesn’t surface in staging because the dataset doesn’t have the right edge-case data. But the root cause was architectural: we deployed to 100% of users immediately, with no way to stop it short of a full rollback.

Our deploy process was: build, push to S3, invalidate CloudFront. Done. That’s it. No canary. No kill switch. No feature flag governing the new code path. The only rollback mechanism was re-running the old build pipeline, which took 8–12 minutes on its own, and then another CloudFront propagation cycle.

Detection was also slow. We were relying on Sentry error volume alerts, which have a threshold and a delay before they fire. We didn’t have a business-metric alert — something like “application submission rate dropped by 20% in the last 5 minutes.” By the time the Sentry alert fired, users had already been hitting the bug for over ten minutes.

The Retrofit

We couldn’t rewrite our deploy pipeline overnight, but we could add kill switches within a week. We went with ConfigCat — lighter than LaunchDarkly, generous free tier, and the Angular SDK is straightforward.

Setup looked like this:

// feature-flag.service.ts
import { Injectable } from '@angular/core';
import * as configcat from 'configcat-js';
@Injectable({ providedIn: 'root' })
export class FeatureFlagService {
private client: configcat.IConfigCatClient;
constructor() {
this.client = configcat.getClient(
environment.configCatSdkKey,
configcat.PollingMode.AutoPoll,
{ pollIntervalSeconds: 60 }
);
}
async isEnabled(flagKey: string, user?: configcat.User): Promise<boolean> {
return this.client.getValueAsync(flagKey, false, user);
}
}

For the new application flow, we wrapped the entry point:

// application-flow.component.ts
async ngOnInit() {
const user = new configcat.User(this.authService.getUserId(), {
email: this.authService.getUserEmail(),
custom: { segment: this.authService.getUserSegment() }
});
this.useNewFlow = await this.featureFlagService.isEnabled(
'new_application_flow_v2',
user
);
}

The template delegates based on the flag:

<app-new-application-flow *ngIf="useNewFlow; else legacyFlow" />
<ng-template #legacyFlow>
<app-legacy-application-flow />
</ng-template>

This gives us a kill switch. If something breaks, we flip the flag to off in the ConfigCat dashboard and within 60 seconds (the poll interval) all users fall back to the legacy flow. No redeploy. No pipeline wait. Sixty seconds.

Canary Rollout via Percentage Targeting

The kill switch is reactive. The canary is proactive. ConfigCat’s percentage-based targeting let us configure rollouts without code changes:

• 0–5%: new flow enabled (internal users + early testers)

• 5–20%: expand after 24 hours of clean metrics

• 20–50%: expand after 48 more hours

• 50–100%: full rollout

The key discipline here is that “expand after clean metrics” is not informal. We wrote it into our deploy checklist as a literal gate: you must look at the dashboard, check the submission funnel conversion rate for the flag-enabled cohort vs. the control cohort, and explicitly approve the expansion in writing in the deploy Slack thread. No implicit “it seems fine.”

## The Alerting Gap

We added a business-metric alert that we should have had years ago. Using our analytics pipeline (we emit events to BigQuery via a proxy), we built a simple Datadog monitor on the metric application.form.submitted with a change_alert type: if submissions drop more than 25% compared to the same 15-minute window from the previous week, page the on-call engineer.

This would have fired within eight minutes of the original incident. We were getting paged after twenty-two. That’s a fourteen-minute gap that, in a loan funnel, is not acceptable.

The Rollback SLA

After the post-mortem, we wrote a one-page rollback SLA. The headline numbers:

• Detection SLA: 10 minutes from deploy to confirmation of user-facing regression

• Mitigation SLA: 5 minutes from detection to kill switch activated (flag off) or traffic rerouted

• Resolution SLA: 30 minutes from detection to full rollback if kill switch isn’t sufficient

The SLA also defined who owns each step. Detection is automated (alerting). Mitigation is the deploying engineer or on-call. Resolution is the tech lead. If any SLA is missed, it goes into the post-mortem log — not as a blame exercise, but as a signal that our tooling needs to improve.

The New Deploy Checklist

Every production deploy now requires:

1. The feature is gated behind a flag if it touches a critical user flow

2. The initial rollout percentage is set to 5% or less

3. A business-metric baseline has been captured for the canary cohort before expanding

4. The Datadog alert exists and has been tested (by temporarily lowering the threshold)

5. The deploying engineer has confirmed the rollback path (kill switch or re-deploy) in the Slack deploy thread

It sounds like overhead. It adds maybe fifteen minutes to a deploy. But the incident cost us close to two hours of engineering time across six people, plus the revenue impact. The math is not close.

What I’d Tell Myself Before That Deploy

The thing about this incident is that none of the infrastructure was expensive or hard to build. ConfigCat’s free tier covers what we needed. The alerting took half a day. The deploy checklist is a Notion template. The real cost was not having done it earlier — and the reason we hadn’t was that it felt like something to do “when we scale up” or “when we have more traffic.”

We had enough traffic. We just hadn’t taken the incident seriously enough as a possibility.

Gradual rollouts are not a big-company luxury. Kill switches are not over-engineering. The moment you have real users depending on your product, you have a production responsibility. The infrastructure to honor that responsibility is smaller than you think.

We were lucky the incident happened in the afternoon, not at 2 AM. We were lucky the affected user segment was 30%, not 100%. Luck is not a deploy strategy.