How Downsizing MSK Unleashed the CPU Credit Monster

Introduction

Cost optimization is often pursued with the best intentions, aiming to balance performance and expenses. In certain scenarios, adjustments such as downsizing AWS MSK broker instances may appear logical based on observed resource utilization. However, such changes can occasionally lead to unexpected outcomes that impact system stability.

Symptoms

Seeing Intermittent and frequent rebalance messages like below in your kafka logs ?

Offset commit failed on partition ... The request timed out.
Group coordinator ... is unavailable or invalid due to cause: REQUEST_TIMED_OUT
Preparing to rebalance group ... error when storing group assignment during SyncGroup

If yes, buckle up — you’re in for a mystery 🤯

Initial Checks

When all the usual checks flash green like below…

• DNS resolution: OK.
• Telnet to brokers: OK.
• Security groups: Correct.
• MSK cluster health: Looked fine at first glance.

Everything looked innocent. But when the green flags wave and the errors keep coming, it’s time to dig deeper.

Deep Dive

Let’s zoom in on this log line:

Preparing to rebalance group ... reason: error when storing group assignment during SyncGroup

This pointed to the GroupCoordinator failing to write to the __consumer_offsets topic.

Why? — Either disk I/O or CPU bottlenecks.

Next, check CloudWatch metrics:

• If CpuIdle dropping to zero or near zero frequently
• If CpuCreditBalance is almost zero

This opens the door to something big….and below could be your root cause.

Root Cause

• check if your instance type is burstable like (t3.small)
• t3.small brokers → limited CPU and burstable credits.
• When credits ran out, brokers throttled to baseline performance.
• Kafka couldn’t handle GroupCoordinator tasks, causing timeouts and rebalances.
• Low EBS IOPS (gp2 default) added to the pain.

Fix

• Upgrade brokers to m5.large (non-burstable, more CPU and RAM) or similar instance type as per the load.
• Increase EBS IOPS using gp3.

Result? A stable cluster and brokers that can finally breathe.

Lessons Learned

• Never use burstable instances (like t3) for Kafka in Critical environments.
• Monitor CPU credits and request handler idle time.
• Size brokers based on throughput and consumer group count.
• Enable auto-scaling for storage.

Key Metrics to Watch

• CPU Credit Balance (for burstable instances)
• BytesInPerSec / BytesOutPerSec
• CPU utilization and idle time

Conclusion

For Kafka, burstable instances aren’t just risky — they’re a ticking time bomb.