How a Single async Keyword Crashed Our .NET Core Production Server

In .NET Core development, async/await is a blessing for us. We all use them with the belief that they improve performance, reduce thread blocking, and enhance scalability.

But this blog tells the story where a single async keyword 👉 crashed the production server 👉 increased API response time from 200ms to 30s+ 👉 caused CPU usage to reach 90%+ and the thread pool to become exhausted 👉 resulting in a hotfix at 2 AM.

All of this was caused by a single line of code.

[embed]Top NumPy Questions Interviewers Love to Ask Currently, Python is being used most in Data Science, Machine Learning, Artificial Intelligence, and Scientific…medium.com

The day of the production incident

The morning started completely normally. The deployment had been done the previous night, automated tests passed.

Suddenly —

• 504 Gateway Timeout from the API Gateway
• Application Insights shows:
• Thread Count ↑↑
• Requests Queued ↑↑
• Even after recycling IIS, the problem reappeared within 5 minutes.

The most terrifying thing? 👉 There are no exception logs.

[embed]The Magic Behind ASP.NET Routing: How It Works ASP.NET Routing is a mechanism that interprets URLs and maps them to specific controllers and action methods. It is…medium.com

What were the recent code changes?

A small performance refactor.

Previous code:

public List<Order> GetOrders(int userId)
{
    return _orderRepository.GetOrders(userId);
}

The new code (which we thought was “Better Practice”):

public async Task<List<Order>> GetOrders(int userId)
{
    return _orderRepository.GetOrders(userId);
}

As you can see, there is no ‘await’ here.

[embed]Understanding Task and Task<T> in C# Understanding Task and Task<T> (Created in Canva)medium.com

This is where the trouble began.

This code looks like:

“No problem, I’ve made it future-proof.”

But what actually happened?

async without await — what does it actually do?

When you write:

public async Task<T> Method()

But there is no await inside —

What does .NET do?

1. The compiler creates a state machine.
2. Even though the method is synchronous, an async wrapper is created.
3. Extra allocation and context capturing occur.

The ASP.NET Core request pipeline assumes: 👉 “This request is async, I can release the thread.”

But in reality —

• The internal work is completely synchronous.
• The thread is being blocked.
• Threads are not being released from the thread pool.

Domino Effect: Thread Pool Starvation

What happened under production load:

1. 100 concurrent requests
2. Each request has an async signature
3. Contains a blocking DB call internally
4. Thread pool exhausted
5. New requests got stuck in the queue
6. Timeout → Retry → More load

Perfect Storm

Why wasn’t it caught in local or QA testing?

Because —

• Local environment:
• Low concurrency

QA:

• Manual testing

Production:

• Real users
• Burst traffic
• Parallel requests

If there’s an error in asynchronous operations, it only gets detected under load.

The most dangerous part

This method —

public async Task<List<Order>> GetOrders(int userId)

• Calls were being made from 10+ controllers
• Also from background jobs
• Each call blocking a thread

One mistake → throughout the entire application.

How do we find the root cause?

Step 1: Thread Pool Metrics

• Available Worker Threads ≈ 0
• Queued Requests ↑↑

Step 2: Async Analyzer

• Searched: async task
• Checked: where await missing

Step 3: Hotfix

public Task<List<Order>> GetOrders(int userId)
{
    return Task.FromResult(_orderRepository.GetOrders(userId));
}

Or even better —

public async Task<List<Order>> GetOrders(int userId)
{
    return await _orderRepository.GetOrdersAsync(userId);
}

What happened after the deployment?

• CPU ↓ 90% → 25%
• API Response ↓ 30s → 180ms
• Thread Pool Stable
• No more timeouts

One line change. A night of panic.

What We Learned (Hard Lessons)

1️. async is not a magic bullet

Writing async doesn’t automatically make the code better.

2️. async means await

If you don’t await, don’t async

3️. Blocking calls are silent killers

• .Result
• .Wait()
• Sync DB call inside async

4️. Code Review Checklist Update

• Does it have an async method?
• Does it have await?
• True async dependency?

Golden Rules (Production Safe Async)

✔ Async all the way ✔ No fake async ✔ No blocking inside async ✔ Measure thread pool, not just CPU ✔ Load test async code

This story is not an example from a book. This is a real-world production failure.

A single little async keyword 👉 which we thought was a “best practice” 👉 proved that —

When async goes wrong, it’s not a bug — it’s a disaster.

If you work with .NET Core, remember this story.

Because production shows no mercy.