Technical Deep-Dive: Netty Threading & OS Realities

A Case Study on Thread Over-provisioning: Matching Threads to CPU Limits

Context: Why We Did This

This deep dive was triggered by specific customer-reported issues where our services experienced performance degradation. Upon investigation, we identified:

• OS Thread Spikes: Rapid increases in thread counts leading to instability and “thread exhaustion.”
• OOM (Out of Memory) Errors: The service was crashing due to the high memory overhead of maintaining too many thread stacks.
• The “High Default” Trap: We discovered the service was configured with excessive thread counts that far exceeded the available CPU limits.

We made critical changes to the service configuration to stop wasting OS resources and memory. This document explains the “why” behind those changes.

Summary

• Netty Framework Context: Our applications leverage Netty for high-performance networking. We observed that configuring more threads than required by the workload leads to OS thread leaks and resource exhaustion.
• The 1:1 Rule: Every Netty EventLoop creates one permanent, dedicated OS thread.
• The Over-provisioning Trap: Setting thread counts significantly higher than CPU limits causes “CPU Starvation.” Threads appear RUNNABLE but are stuck in a "traffic jam" waiting for a CPU slice.
• The Fix: Limit Boss threads to 1, size Workers based on CPU count, and scale horizontally (replicas) instead of vertically (threads).

Before vs. After: Performance Impact

The following graph illustrates the impact of these changes. Note the uncontrolled climb in thread counts (leaks) on the left, followed by the stable, flat line after we applied the optimised configuration.

1. The Architecture: Netty Boss vs. Worker Pools

Netty uses a “Parent-Child” model to separate connection handling from data processing.

The Boss Group (The Receptionist)

• Default Sizing: CPU * 2
• Responsibility: Listens for new connections (accept()).
• Best Practice: Set this to 1. One receptionist can handle thousands of connections/sec. Multiple boss threads can actually cause “lock contention” at the OS level.

2. Worker Pool Sizing: The Core Strategy

The sizing of your Worker Group must match the nature of your infrastructure interactions (gRPC, REST, Kafka, etc.).

Scenario A: Purely Non-Blocking (Reactive)

If your application never “pauses” the thread (e.g., using fully reactive clients):

• *Recommended Sizing: CPU 2**
• Impact: Adding more threads here decreases performance due to context switching.

Scenario B: Infrastructure Interaction (gRPC, REST, Kafka, DB & ElasticSearch etc.)

If your code involves waiting for external infrastructure that isn’t fully reactive:

• The Problem: The Worker thread “hangs” until the external infra responds. While it waits, it cannot process any other requests for other clients.
• The Calculation: *Threads = CPU Cores 8**
• The Limit: Even with blocking, do not exceed 128–150 threads. If latency persists, do not increase threads further; instead, scale horizontally with more replicas.

3. The “Stickiness” Principle

• Connection Marriage: Once a client connects and is assigned to a specific Worker thread, every single request from that client is handled by that same thread for the life of that connection.
• Why? This ensures Thread Safety. You don’t need complex locks because only one thread ever touches that specific client’s data.

4. The Impact of Excessive Threads vs. CPU Limits

The “Traffic Jam” (Context Switching)

When the thread count far exceeds the number of CPU cores, the OS spends more time “swapping” threads (saving/loading states) than actually executing code.

The Memory Tax

Each thread reserves ~1MB for its stack.

• Over-provisioned: ~750MB RAM consumed just for threads.
• Optimized: ~100MB RAM consumed.
• Savings: Reclaims hundreds of megabytes of RAM instantly.

5. Troubleshooting: How to Read a Thread Dump

• The “Runnable” Mirage: If hundreds of netty-worker threads are RUNNABLE but latency is high, it’s CPU Starvation.
• The “Wait” Sign: If threads are WAITING on external infra, your pool is "hanging." The fix is asynchronous I/O, not more threads.

6. Advanced Monitoring

Run this command to see if the OS is forcing your threads to stop (Involuntary Context Switching):

# Monitor context switches every second
top -d 1 -b | grep -i Cswch

• High cswch: Confirms you have too many threads for your CPU limit.

7. Scaling Best Practices Checklist

• Boss Count: Manually set to 1.
• Worker Count: Align with CPU cores (CPU 2 for I/O, max CPU 16 for infra-heavy apps).
• Avoid “Thread-per-Request”: Don’t try to solve external latency by adding more Netty threads.
• Horizontal Scaling: Use more replicas to distribute connections.

Final Thought

Netty is designed for efficiency through a few threads. By aligning our configuration with the physical reality of our CPU limits, we successfully addressed the customer-reported issues, reclaimed memory, and built a much more resilient service.