When Redisson Stole My Morning: Debugging a 4.5-Minute Startup Mystery

I was recently handed a challenging task: investigate why one of our services was taking a staggering 4.5 minutes to start up. In today’s fast-paced deployment environments, such a delay is more than just an inconvenience — it’s a serious bottleneck. My mission was clear: dig deep, uncover the root cause, and bring that startup time down to something far more reasonable. Here’s how I tackled the mystery.

The Scene: A Service with Too Many Friends

This particular service was a heavyweight, relying on a constellation of dependencies:

• ElasticSearch
• Redisson (Redis client)
• Kafka
• PostgreSQL
• AWS S3

With so many moving parts, my first instinct was to suspect one of the big players — maybe ElasticSearch, notorious for handling vast datasets, was taking its sweet time.

ElasticSearch: The Red Herring

I dove into the ElasticSearch integration, expecting to find it slogging through mountains of data at startup. But after reading through the internals of the ElasticSearch client, I realized it wasn’t loading any data at all during initialization — just setting up a connection. To be sure, I peppered the codebase with time logs around every ElasticSearch-related bean creation.

This was the Elastic Search connection initializer, taking almost negeligible time execute.

While investigating the startup delay, I closely examined the Elasticsearch **RestHighLevelClient** initialization. One line in particular caught my attention:

this.registry = new NamedXContentRegistry((List)Stream.of(
getDefaultNamedXContents().stream(), getProvidedNamedXContents().stream(), 
namedXContentEntries.stream()).flatMap(Function.identity()).collect(
Collectors.toList()));

This snippet initializes three key components:

• **getDefaultNamedXContents()**: Provides the default classes defined by the Elasticsearch client for serialization and deserialization.
• **getProvidedNamedXContents()**: Includes any custom classes we supply for handling specific object types.
• **namedXContentEntries**: This one stood out—it scans the entire project and all dependencies to discover custom classes created for serializing and deserializing particular data objects.

Given the scale of our service, I suspected that initializing **namedXContentEntries** could be a bottleneck, potentially consuming significant time as it scanned through all project classes and dependencies.

To test this theory, I wrote a small piece of code to measure how long this initialization actually took during startup. To my surprise, the process completed in just a few milliseconds. My suspicion was unfounded — this part of the Elasticsearch client was not responsible for the slow startup.

Spotting the Pattern: The Logging Blackout

After confirming that bean initialization wasn’t the source of our 4.5-minute startup lag, I decided to monitor the entire startup process more closely. What I found was both intriguing and frustrating: there were two distinct periods — first for 3 minutes, then for 1 minute — where the logs went completely silent. No output, no errors, just a total blackout.

This complete absence of logs strongly suggested that something specific (and blocking) was happening during those intervals. It wasn’t just slow — it was as if the application was frozen, waiting for a single operation to finish before moving on.

Thread Dump to the Rescue

To dig deeper, I took thread dumps right in the middle of each blackout. The evidence was clear: the main thread was in a blocked state, and the culprit looks to be redisson. This was a crucial clue—Redisson, our Redis client, was involved.

Reading further into the thread dump, I noticed that the main thread was stuck executing a function called hashCode. This was unexpected, as hashCode is typically a lightweight operation.

The Realization: Redisson’s Heavy hashCode

Curious, I dove into the Redisson client library’s source code. There, I discovered that this particular hashCode function wasn’t your typical hash calculation—it was traversing all the key-value pairs and generating a hash, a process that can be extremely time-consuming if the dataset is large or if network latency is involved.

After discovering that Redisson was blocking the main thread during startup, I needed to understand why this heavy operation was even running at that time. Digging deeper into the thread dump, I realized the culprit was Spring’s bean creation process.

How Spring Bean Creation Triggered the Problem

During bean creation, Spring records destruction information for each bean — essentially, instructions on how to clean up the bean when the application shuts down. To manage this, Spring uses a concurrent hashmap, where the key is generated by calling the **hashCode method on the bean name. Normally, Java’s default `hashCode** implementation is extremely fast. However, Redisson overrides this method in its map implementations. Instead of a simple calculation, Redisson’shashCode` traverses all entries in the map to compute the hash.

Why It Was So Slow

I investigated which caches were causing the slowdown and found two major offenders: one cache with 4.5 million entries and another with 2.1 million.

Every time Spring tried to store destruction info for a Redisson map bean, it triggered this overridden **hashCode** method, which then scanned through millions of entries.

This explained the long periods of inactivity in the logs—the application was essentially frozen while Redisson’s **hashCode** churned through massive caches.

Root Cause: Cache Configuration

The caches were configured with **LocalCachedMapOptions.defaults(), which sets the cache TTL (Time-To-Live)** to infinite. As a result, data was never evicted, and the caches kept growing unchecked.

The Solution

I immediately cleared the oversized caches.

I reconfigured the beans to use a sensible TTL, ensuring that cache entries would be evicted over time.

The Problem Was Titanic, The Solution Was… a Mop?

After hours of playing detective, analyzing thread dumps, and suspecting every service in the stack, I finally uncovered the villain behind our 4.5-minute startup: millions of dusty old cache entries clogging up Redisson. The drama! The suspense! The hours spent squinting at logs and stack traces!

And the solution? Not a new algorithm. Not a fancy optimization. Not even a heroic code refactor.

Nope. Just… clear the caches.

That’s right — after all that, the fix was basically the IT equivalent of “have you tried turning it off and on again?” I swept out those monstrous caches, and — BOOM! — startup time dropped from a glacial 4.5 minutes to a breezy 0.5 minutes.

Sometimes, the mountain you’re climbing is just a pile of junk you forgot to sweep. Lesson learned: when in doubt, bring a broom! 🧹