What a Production Incident Taught Me About Building Enterprise-Scale Rollout Systems.

What does a phased rollout system actually need to handle at enterprise scale?

How I built one inside an endpoint protection platform — and what a production incident taught me first

I want to start with an incident I witnessed.

An agent upgrade was released to all customer endpoints simultaneously. No staged rollout. No eligibility checks. Everyone got it at once.

Within hours: a tsunami of false positive alerts across customer environments. Security teams were flooded. Investigations were triggered on legitimate processes. In some cases, critical protection failures on customer endpoints.

Nobody intended this. The build was tested. The intentions were good. But production has a way of introducing variables that no test environment anticipates — and when you release to everyone at once, the blast radius is everyone at once.

That incident is why I built FRCS — the Feature Rollout & Control System. And it’s what this article is about.

The problem isn’t testing. It’s blast radius.

Here’s the thing most engineers get wrong about rollout systems: they think the goal is to catch bugs before release. It isn’t. That’s what testing is for.

The goal of a phased rollout system is to limit who gets hurt when something still goes wrong.

Because something will. Production surfaces variables that staging never will — edge case license configurations, account-specific data states, fleet sizes you never load tested against, endpoint OS combinations you didn’t anticipate. The question isn’t whether a release will cause issues. It’s how many accounts are affected when it does.

A staged rollout system is your answer to that question.

Two paths. I took the harder one. When I set out to build FRCS, two approaches were on the table.

Option A — An existing main server machine already fired upgrades to accounts periodically. I could hook into that, wire up account eligibility logic, and call it done. Fast. Familiar. Already in production.

Option B — Pack the rollout logic with the product itself. Invoke upgrades during a periodic scheduler. Manage account eligibility through runtime key patching — a 5 minute operation — rather than an actual server release, which took 1 to 3 days. Keep an adhoc API trigger for scenarios that couldn’t wait.

Option A was the quicker ship. Option B gave us something more valuable: runtime control without a release cycle.

I took Option B.

The reasoning was straightforward once I framed it right. If you discover a rollout is causing issues and you need to stop it, you don’t want to wait 3 days for a server release to flip a flag. You want to patch a key and halt in minutes. That’s the difference between a bad afternoon and a customer escalation that runs into the next week.

Think in accounts, not percentages

Most rollout systems let you deploy to 10%, then go to 20% of accounts and go on. Mine needed to deploy to 10% of the right accounts and know exactly which ones those were and then go on.

The problem is that percentages are anonymous. You don’t know which 5% got it. When something breaks, you can’t reason about why those specific users were affected. Were they on a particular OS? A specific license tier? A fleet above a certain size?

I flipped the mental model. Instead of percentages, FRCS thought in accounts — every rollout decision made at the individual customer account level.

This gave two things that percentages don’t:

Targeting — roll out to one specific account for validation, or exclude a problematic account from a rollout entirely without touching anyone else.

Auditability — you always know exactly which accounts are on which version. When something breaks at 2am, your logs tell you precisely who got what and when.

The atomic unit of the entire system was one question: is this particular account eligible to receive this upgrade right now? Everything else was built around answering that cleanly.

Sequencing: trial → small → large

Once you think in accounts, sequencing becomes a deliberate choice rather than a random sample.

FRCS rolled out in this order: trial accounts first, then smaller registered customers, then large enterprise customers last.

The logic is risk segmentation. Trial users are evaluating the product — they tolerate more friction than a security operations team running production workloads. Small customers have simpler fleet configurations, so issues surface clearly without too many compounding variables. Large enterprise customers have the most complex environments, the largest fleets, and the least tolerance for disruption.

By the time an update reaches your largest customers, it has already been validated across hundreds of accounts. The confidence level is real, not assumed.

This is also a trust decision. Your biggest customers trust you the most. They go last precisely because of that trust — not as an afterthought, but as a deliberate act of protecting them most.

Two triggers: scheduler and adhoc

FRCS had two ways to invoke a rollout.

Scheduler-based — a background job running on a 24-hour cycle, iterating over accounts and advancing the rollout in stages. Gradual, automatic, zero ops burden once configured. This is the default path for every feature rollout.

Adhoc API — a manual trigger that bypasses staged rollout and pushes to specific accounts immediately. For when a critical security fix needs to reach a customer right now, regardless of where they sit in the rollout sequence.

The adhoc path is necessary. But it’s dangerous if it becomes the default. The moment your team starts reaching for forced rollouts as a convenience, your staged system becomes theater — you have the infrastructure of control without actually using it.

Treat the adhoc trigger like a fire extinguisher. It should always be there. You should almost never need it.

The interface contract: rollout as infrastructure

One architectural decision I’d make from day one again: define a clean interface that any module implements to participate in the rollout system.

In FRCS, any module wanting staged rollout support implemented a common interface and registered itself. The rollout system knew nothing about what the module actually did — it just called the implementation at the right time, with the right account context.

New detection engine activation, agent upgrades, product integrations — all different modules, all going through the same gateway.

The principle: your rollout system should be infrastructure, not feature code. The moment you encode feature-specific logic into your rollout mechanism, you’ve coupled two things that will fight each other. Keep them separate from day one.

What a phased rollout system actually needs to handle

If you’re building one, here’s what it needs beyond the basic “deploy to X% and stop”:

Account-level targeting — not percentages, identities

Explicit exclusion — the ability to fence off specific accounts from a rollout entirely

License-aware sequencing — trial before small before enterprise

A scheduler for the default path — gradual, automatic, auditable

An adhoc escape hatch — with clear team discipline around when it’s appropriate

A kill switch — stop mid-rollout in minutes, without a code deploy

An extensible interface — so any module participates without touching rollout logic

Logs at every decision point — not just success/failure, but which account, which check, which outcome

The last one is the most underrated. At 2am when something is wrong, your logs are your only window into what the system did. Make them good before you need them.

Closing thought

The incident I opened with didn’t happen because someone was careless. It happened because the system allowed a release to reach everyone simultaneously with no circuit breaker in place.

A phased rollout system doesn’t make your releases perfect. It makes the consequences of imperfection survivable.

Think in accounts. Sequence deliberately. Build the kill switch before you need it. And treat the forced path as a last resort, not a shortcut.

Production will still surprise you. That’s not a problem you solve. It’s a risk you architect around.