From 10 AM to 1 AM: How We Survived a Migration Timeout Disaster

Last weekend was supposed to be a smooth, well-planned migration day.

We were onboarding a major new customer into our Order Management System (OMS). The team started at 10:00 AM, fully prepared:

• Migration scripts ready
• Business data validated
• Monitoring dashboards open
• Messaging channels buzzing

We knew it would be a long day — migrations always are — but we didn’t expect this adventure.

By late afternoon, everything looked great.

And then around 8:00 PM… right when we were preparing to start the bulk import… One tiny timeout setting changed the entire night.

The API Didn’t Fail — The Connection Died

Everything inside the backend was working correctly:

• The heavy migration job was still running
• The logs showed no exception
• CPU and memory were stable

But the HTTP clients kept reporting:

504 Gateway Time-out
The server didn't respond in time.

And it happened exactly 50 seconds later.

That number rang a bell.

We inspected the network layer (HAProxy), and there it was — the villain of the night:

timeout http-request 50000(50s)

HAProxy was silently killing every long-running HTTP request.

Our API wasn’t wrong. Our system wasn’t wrong. The network layer refused to keep connections alive for more than 50 seconds.

At 8:10 PM, the migration was stuck.

But the real journey was just beginning.

The Root Cause: HTTP Doesn’t Like Long Jobs

As we analyzed the failure, it became painfully clear:

HTTP is not designed to keep a connection open for 10 or 30 minutes.

Every layer in the chain has its own timeout:

Layer: Browsers — Typical Timeout: 1–2 minutes Layer: Mobile apps — Typical Timeout: 30–60 seconds Layer: Tomcat/Netty — Typical Timeout: ~30 seconds Layer: Nginx — Typical Timeout: 60–120 seconds Layer: HAProxy — Typical Timeout: 50 seconds Layer: Cloudflare — Typical Timeout: 100 seconds Layer: API Gateway — Typical Timeout: 29 seconds

Even if your backend works perfectly…

The network will kill it.

At 8:30 PM, we realized that continuing with a standard synchronous API call was impossible.

Then came the breakthrough.

The Ah-Ha Moment: We Need Fire-and-Forget

Somewhere around 9 PM, after hours of debugging and head-scratching, someone on the call finally said:

“Why are we keeping the HTTP request open at all? Just start the job and respond immediately!”

Silence. Then the agreement.

Of course.

We needed the same pattern used by:

• AWS
• Stripe
• GitHub Actions
• Google Cloud

The legendary:

Fire-and-Forget Pattern

The flow:

1. Client calls /start-migration
2. The server returns immediately with 202 ACCEPTED and a jobId
3. The server runs the long job in the background
4. Client polls /status/{jobId}
5. No more timeouts
6. No HAProxy drama
7. No waiting for 30 minutes in a single HTTP connection

This was the turning point.

We implemented it between 10 PM and 12:30 AM, working as fast as possible.

By 1:00 AM, the migration had completed successfully.

Reactive Fire-and-Forget (WebFlux Style)

This is the version we decided to use moving forward.

1. Start Job

    @PostMapping("/start-migration")
    public Mono<String> migration() {
        return assetPortfolioService.migration();

    }

2. Background Task

    public Mono<String> migration() {
        redisService.checkAndSetPortfolioSyncerKey()
                .filter(result -> result)
                ...  
                .subscribe();

        return Mono.just("TaskStarted...");
    }

3. Check Status

@GetMapping("/status/{id}")
public Mono<String> status(@PathVariable String id) {
    return Mono.just(jobStatus.getOrDefault(id, "UNKNOWN"));
}

Optional: Real-Time Notifications with WebSocket

Technically, we could add WebSocket support so clients receive a “DONE: jobId” message instantly when the job finishes.

We didn’t enable it in production — but it’s always an option.

End of the Night

By 1:00 AM, the final migration finished successfully.

We closed our laptops, stretched our backs, and finally relaxed.

The lesson from this adventure was clear:

Long-running jobs don’t belong inside HTTP requests lifecycles. Fire-and-forget is the right tool for the job.

And now, it has become our new standard for all heavy operations in the OMS.