Image Edited With Canva

I Spent 8 Hours Debugging a Spring Boot Bug That Only Happened in Production

Timezone. UTF-8. Connection pool. Race condition.

Every test passed. QA approved the release. The code review took less than five minutes. Thirty minutes after deployment, production started returning inconsistent results — and nobody could reproduce the problem locally.

👉 Friend Link | Spring Boot — Crack Interview 14 -Days | Spring Boot — Production Ready Saas Codebase Bundle

One of the hardest bugs I’ve ever investigated wasn’t caused by Hibernate.

It wasn’t PostgreSQL. It wasn’t Kubernetes.

And it certainly wasn’t Spring Boot.

It was a perfectly valid piece of Java code that had survived months of development because it only failed under one condition:

Hundreds of users accessing the same singleton bean at exactly the same time.

The frustrating part was that every environment except production behaved perfectly.

My laptop worked. The QA environment worked.

The staging environment worked. Only production failed.

The First Alert

The incident started with a message from our support team.

“Some customers are receiving duplicate invoice numbers.”

Five minutes later another ticket arrived.

“A few invoices contain another customer’s data.”

That immediately raised concerns because invoice generation is deterministic. Given the same request, the service should always produce the same output.

We checked the database first. The records were correct.

Audit logs showed no unexpected updates.

The SQL statements looked normal. Yet the API responses were inconsistent.

Refreshing the page sometimes fixed the problem.

Sometimes it made it worse.

That randomness made the bug incredibly difficult to reason about.

Everything Looked Healthy

Like every production investigation, we started with infrastructure.

PostgreSQL? Healthy.

Redis? Healthy.

CPU utilization? Around 35%.

Memory? Plenty available.

Connection pool? No waiting threads.

Application logs? No exceptions.

No stack traces. No failed transactions.

Monitoring painted a picture of a perfectly healthy system.

And yet customers continued reporting incorrect responses.

The Endpoint Was Surprisingly Simple

The controller looked completely ordinary.

package com.example.invoice.controller;

import com.example.invoice.dto.InvoiceResponse;
import com.example.invoice.service.InvoiceService;
import lombok.RequiredArgsConstructor;
import org.springframework.web.bind.annotation.*;

@RestController
@RequiredArgsConstructor
@RequestMapping("/invoices")
public class InvoiceController {
    private final InvoiceService invoiceService;
    @PostMapping("/{id}")
    public InvoiceResponse generateInvoice(
            @PathVariable Long id) {
        return invoiceService.generate(id);
    }
}

Nothing unusual.

The service wasn’t much larger.

package com.example.invoice.service;

import com.example.invoice.dto.InvoiceResponse;
import lombok.Service;
import java.util.HashMap;
import java.util.Map;

@Service
public class InvoiceService {
    private final Map<Long, InvoiceResponse> cache = new HashMap<>();
    public InvoiceResponse generate(Long customerId) {
        InvoiceResponse response = cache.get(customerId);
        if (response != null) {
            return response;
        }
        response = createInvoice(customerId);
        cache.put(customerId, response);
        return response;
    }
    private InvoiceResponse createInvoice(Long customerId) {
        // Simulate expensive business logic
        return new InvoiceResponse(
                customerId,
                "INV-" + System.currentTimeMillis());
    }
}

During code review, nobody questioned this implementation.

Including me. It looked efficient.

Cache previously generated invoices. Avoid unnecessary database work.

Return the cached response.

Simple. Readable. Fast. Or so we thought.

Why Couldn’t We Reproduce It?

We copied the production database.

Ran the application locally.

Executed the endpoint. Everything worked.

We tried again. Still worked.

QA repeated the same tests. No issues.

At one point someone jokingly said,

“Maybe production is haunted.”

The truth was much less entertaining.

Our laptops processed one request at a time.

Production processed hundreds simultaneously.

That single difference changed everything.

The Missing Ingredient: Concurrency

The bug wasn’t caused by incorrect business logic.

It was caused by multiple threads executing the same code at the same time.

Spring Boot creates one instance of every @Service bean by default.

That means every incoming HTTP request shares the same object.

Conceptually, the application looked like this.

                  Spring Container
                         │
        ┌────────────────┴────────────────┐
        │                                 │
                InvoiceService
                (Singleton Bean)
        │        │        │        │
        ▼        ▼        ▼        ▼
 Request 1  Request 2  Request 3  Request 4

Every request entered the same InvoiceService.

Every request accessed the same HashMap.

At low traffic, nothing went wrong.

Under heavy load, multiple threads started reading and writing to the map simultaneously.

The Bug Was Hiding Here

Photo by Patrick Martin on Unsplash

This line looked harmless.

private final Map<Long, InvoiceResponse> cache = new HashMap<>();

And so did this.

cache.put(customerId, response);

The problem wasn’t the syntax.

The problem was that **HashMap is not thread-safe**.

Imagine two requests arriving at exactly the same time.

Request A
↓
cache.get(101)
↓
null
────────────────────────────
Request B
↓
cache.get(101)
↓
null

Neither request finds the value.

Both generate invoices.

Both attempt to update the same shared map.

Under concurrent traffic, operations begin interleaving in unpredictable ways.

Sometimes the result is duplicated work.

Sometimes stale data.

Sometimes corrupted state.

Sometimes everything appears to work.

That’s exactly why production bugs like this are so difficult to reproduce.

They depend on timing.

Why Unit Tests Didn’t Catch It

Every unit test executed sequentially.

invoiceService.generate(101L);

One thread. One request. One service instance.

No contention. Production looked very different.

Tomcat Thread Pool
↓
Thread-18
Thread-24
Thread-31
Thread-45
↓
InvoiceService
↓
Shared HashMap

The code wasn’t failing because it was incorrect.

It was failing because it had never been tested under realistic concurrency.

The Breakthrough

After almost eight hours, we finally reproduced the issue using a simple load test.

Within seconds, duplicate invoices started appearing. The database was innocent.

Spring Boot was behaving exactly as designed.

The real issue was our assumption that a singleton service could safely maintain mutable state using a regular HashMap.

That assumption cost us an entire day of debugging. And it taught me one lesson I’ll never forget:

The code that works perfectly with one request isn’t necessarily correct when one hundred requests arrive at the same time.

What’s Coming in Part 2

Now that we’ve identified the real culprit, we’ll step inside Spring Boot’s bean lifecycle and Tomcat’s request processing model to understand why singleton beans are shared across threads.

We’ll also answer some important questions:

• Why are @Service beans singleton by default?
• How does Tomcat assign threads to incoming requests?
• Why is HashMap unsafe under concurrent access?
• When should you use ConcurrentHashMap, synchronization, or stateless services?
• How we fixed the bug without sacrificing performance.

We’ll finish by comparing three production-ready solutions and discuss which one is appropriate for different workloads.