We Migrated Our Node.js API to Rust at 10 Million Requests/Day. The Performance Gains Were Real. The Hidden Costs Were Bigger.

We benchmarked it for months. We planned for six. It took eleven — and forty-seven minutes that almost ended the project entirely.

3:17 AM, and the Color That Isn’t on the Runbook

3:17 AM, Tuesday in November. Our dashboards went amber.

Not red. Red has a runbook. Amber is the color that says nothing is technically broken, but something is wrong — and you have three hours to find out which thing.

p99 at 412ms. Fourteen new pods spun up by the autoscaler. Compute bill 22% over budget. It was the 18th of the month.

I muted Slack, made coffee, and stared at the flamegraph.

We had built our API on Node for the right reasons — fast to ship, easy to hire, three pivots without complaint. By the time it started cracking, it had earned the right to be defended.

But the cracks were widening. Quietly. The kind of failure that never earns a postmortem — just a sixth weekly retro in a row where someone shrugs and says “yeah, latency was weird again.”

This is the story of what happened next. The migration. The wins. And the forty-seven minutes that almost made us roll the whole thing back.

The Erosion Nobody Wrote a Ticket For

10 million requests a day sounds impressive in a postmortem. It’s pedestrian. The peaks were what hurt — 4,100 RPS at evening, and the long afternoon plateau that never let the GC settle.

Transactional API. JSON in. JSON out. Two database calls. Standard work. For a long time, Node handled it beautifully.

Then p99 drifted from 180ms to 240ms over a quarter. We added an index — it dropped to 210, crept back to 260. We added Redis — it dropped to 190, crept back to 270.

Every fix bought six weeks. Then the curve resumed. We were spending $42,000/month on a service that should have cost a quarter of that.

That was the moment the conversation changed.

What the Flamegraphs Refused to Hide

Two weekends with clinic.js, 0x, and perf. The answer was: everywhere and nowhere.

Mean latency looked fine. But every few hundred milliseconds, something stalled a request for 80, 120, 200ms.

Lined up against GC logs, the picture sharpened. Thousands of small allocations per request — ajv instantiating validator state on a couple of code paths we hadn't realized weren't cached, middleware context objects threading through six layers, JSON.parse trees the size of small forests for our larger payloads.

Free at 500 RPS. At 4,000 RPS, the young generation filled faster than minor GCs could drain. Objects got promoted, a major GC eventually landed, and the event loop froze for 60–180ms.

app.post('/v1/transactions', validate(schema), async (req, res) => {
  const enriched = await enrichContext(req.body);
  const result   = await persist(enriched);
  res.json(toResponseShape(result));
});

Four allocations per request. Tiny on their own. Murderous in aggregate.

The cost of garbage at scale isn’t the garbage. It’s the silence between collections.

The Whiteboard Meeting That Almost Didn’t Choose Rust

The migration debate did not start as a Rust debate.

Six engineers in a thirty-minute meeting that lasted two hours forty. One staff engineer allergic to rewrites after a Scala disaster at his last company. One platform lead who’d been waiting six months for an excuse to talk about Rust.

The responsible options went up first:

• Worker threads + clustering — ~30% headroom. Doesn’t fix GC.
• Aggressive caching — already doing it.
• Service decomposition — six months. Doesn’t address per-request cost.
• Bun — we wouldn’t put our paycheck on it that November.
• Go — boring, proven, fast enough. Almost won.
• A C++ native module via neon for the hot path — the staff engineer pushed hard for this.

Rust came in the back door.

The argument that finally moved the room wasn’t “Rust is faster.” Every language on the board was faster than Node.

The argument was: we don’t have a performance problem, we have a predictability problem. A runtime without a garbage collector was the only option that addressed predictability at the language level.

The staff engineer still voted no. “You’re going to regret the hiring,” he said.

He was half right.

Forty-Seven Minutes That Almost Ended It

Strangler-fig pattern. NGINX in front. Header-based routing. Five percent shadow first, then live, then gradual.

Elegant on the whiteboard. The first real production cutover went sideways within forty minutes.

The Rust service performed exactly as benchmarked — axum on tokio, sqlx against the same Postgres, tikv-jemallocator for the allocator. 47ms p99, down from 380ms. Memory holding at 180MB per pod against Node's 1.2GB. Everything we'd promised was happening.

Then the duplicate-charge alerts started firing.

Our internal SDK’s retry policy used a 50ms fixed delay with no jitter — fine when Node took 380ms to respond, catastrophic when Rust returned in 47. Retries were arriving before the original responses had finished serializing downstream. An auth service we depended on started 429-ing us because we’d quietly tripled our effective throughput against it without warning anyone, and our un-jittered fleet immediately retried in lockstep.

Then it got worse.

A finance engineer pinged me at minute thirty-one. “Are you seeing the reconciliation diff?”

I wasn’t. He sent a CSV. 0.3% of transactions on Rust had a subtly different response shape than Node — a timestamp with three extra digits of precision, a null where Node had emitted an empty string, an array key ordered differently. One field was an i64 transaction ID that serde_json was correctly emitting as a JSON number; Node, going through JSON.stringify, had been silently truncating it at Number.MAX_SAFE_INTEGER for two years. Three downstream consumers had built around the truncated values.

None of these were wrong. Several were technically more correct than Node had ever been. We had been quietly lying with JSON shapes for two years, and Node had been the cooperative liar.

I called the rollback at 11:53 AM. Forty-seven minutes in. p99 was beautiful. Everything was beautiful. The rollback was still the right call.

Performance is a property of a system, not a service. Make one component faster without warning the others, and you don’t reduce tail latency — you relocate it into the coordination layer.

Two weeks of fixing downstream contracts and shipping a serde compatibility layer that mimicked Node's permissive behavior on purpose — emitting strings instead of nulls, capping integers at the safe-integer boundary — quietly undoing a piece of the type safety Rust had just given us.

The staff engineer didn’t say I told you so. He didn’t have to.

The Six Weeks Where Half the Team Quietly Wondered

Between the rollback and the next attempt, something shifted that wasn’t on any roadmap.

Two engineers asked me, separately, whether we should rip out the Rust work and just buy more Node pods.

PRs slowed. Even the platform lead — our loudest Rust advocate — started saying ‘If we continue…’ instead of ‘When we ship…’

We never formally debated whether to abandon the migration. We just stopped acting like it was inevitable.

What broke the spell wasn’t a heroic decision. It was a re-rollout to 10% on a Wednesday that held clean for six hours. Then twelve. Then a week.

Migrations don’t die from technical failure. They die from the quiet weeks where nobody is willing to say what they’re actually feeling.

The Borrow Checker Became a Team Member

Within two months, code review velocity dropped by roughly half. Twenty-minute Node reviews were taking ninety. Reviewers were learning from the diff — lifetimes, Send bounds, whether a field needed Arc<Mutex<T>> or Arc<RwLock<T>> or an mpsc channel.

Bugs that would have shipped silently in Node were dying at compile time.

pub async fn enrich(
    ctx: Arc<Context>,
    req: TransactionRequest,
) -> Result<EnrichedTransaction, EnrichmentError> {
    let user   = ctx.users.fetch(&req.user_id).await?;
    let policy = ctx.policies.lookup(&user.tier).await?;
    Ok(EnrichedTransaction::new(req, user, policy))
}

The week was spent on the type design around it — four refactors of TransactionRequest before we stopped fighting clones.

The build pipeline took the rest of our patience. CI went from 4 minutes on the Node service to 19 on the Rust one. sccache helped, until a Cargo.lock bump invalidated the cache and we ate a cold rebuild on every PR for half a day. Docker layer caching helped less than we'd hoped — touching anything under src/ invalidated the build layer entirely until we split the workspace into crates and adopted cargo-chef for the dependency layer. Release builds with lto = "fat" peaked at 6GB of memory; we had to bump the CI runner tier twice.

Container images were a real win — 180MB Node images down to 38MB stripped Rust binaries.

But the product team had not signed up for a 50% velocity drop, and no amount of “we’re investing in correctness” softened the calendar.

What Got Better, Honestly

Eleven months in (we’d estimated six), here’s what the numbers said:

• p99 latency: 380ms → 47ms. p99.9: 940ms → 78ms.
• Memory: 1.2GB → 180MB per pod.
• Pod count: 24 → 6 for the same workload.
• Monthly compute: ~$42,000 → ~$11,400.
• Container image: 180MB → 38MB.
• GC-induced tail latency: not improved. Gone. That class of incident stopped existing.

Two of our most-paged alerts went silent for three consecutive months — the first time in two years.

The product team noticed almost nothing.

For us, internally, the texture of work was completely different. Not all of it was good.

The Costs No Dashboard Captured

• Hiring took three times longer. Node openings closed in 5–7 weeks. Our first Rust opening took 14, at the top of our band.
• Onboarding stretched from two weeks to six or eight. One mid-level hire needed eleven.
• Observability cost a full quarter to stabilize. tracing + opentelemetry-otlp worked on paper, but our context propagation lost parent span IDs across HTTP boundaries for the first three weeks — different conventions on the traceparent header between our Node middleware and the Rust crate. We ended up writing a custom propagator. Trace ingestion volume rose ~30% before we tuned sampling, and tokio-console became a daily tool because perf flamegraphs kept dead-ending in tokio runtime internals when we tried to debug async stalls.
• Hotfix ergonomics changed. A one-line patch used to ship in four minutes. Now it took twenty-two — compile, test, image build, rolling deploy. Once, during a real incident, that mattered.
• Knowledge concentrated. Three of six became the de facto experts. Bus factor got worse before it got better.

One engineer left quietly during the migration. In their exit conversation, they said something I haven’t forgotten:

“I used to enjoy ending the day. Now I end every day in the middle of fighting the compiler about something I’d already solved in my head three hours ago.

Migrations are organizational events that happen to be wearing technical clothes.

What We’d Do Again, and What We Wouldn’t

One sentence, if you’re considering this same migration:

Don’t do it for the performance. Do it for the predictability — and only if the predictability is worth the organizational tax.

We’d do it again for services where tail latency is product-defining. Payments. Real-time bidding. Anything where p99.9 is a customer experience, not a metric.

We wouldn’t for services bottlenecked on I/O, with small teams, or bursty-but-tolerant workloads. Node was still right for two-thirds of the services we considered migrating.

Six months was a fantasy. The technical work was on track. The hiring, the onboarding, the downstream contracts, the quiet weeks of doubt — that was the iceberg.

The Quiet Ending

The dashboards are boring now. That’s the truest thing I can tell you.

The amber Tuesdays stopped. On-call went from tense weekly handoffs to acknowledging routine alerts and going back to bed. The infrastructure bill has held at its new number for nineteen consecutive months.

When I think about 3:17 AM in November, I don’t think about the migration as a win or a loss. I think about it as a trade — between one kind of pain and another.

We took on slower hiring, longer onboarding, harder code review, and six brittle months of team morale, in exchange for a runtime that doesn’t lie to us about cost.

For our product, at our scale, with our team, it was the right trade. I am not certain it would be the right trade for yours.

The Rust service is still running. So is the part of our system we never migrated away from Node. They talk to each other every few milliseconds, in two different languages, with two different memory models, and they have not gone down together in over a year.

That’s the ending. Not a victory lap. Just two services doing their jobs — and a team that finally understands what every line of code is quietly costing.

If this resonated, I’d love to hear about your own migration scars — the ones that didn’t make it into the postmortem. The unwritten parts are usually where the real engineering lives.