Practical Guide to Kafka Message Backlog Handling: Optimization Techniques for Emptying Million-Level Queues

Having worked as a Java developer for eight years, I have encountered at least eighty to a hundred middleware-related pitfalls. Among them, the most terrifying was a Kafka message backlog in the production environment — particularly receiving an alert at 3 a.m. and observing the number of unconsumed messages on the monitoring panel climb toward one million, while downstream business teams urgently demanded the data. That pressure remains unforgettable.

Last week, our team managed an emergency outage involving a backlog of millions of messages. From root cause identification to queue clearance, the resolution took only four hours, followed by code upgrades that fully prevented recurrence. Using this real incident as a case study, this article explains the message backlog handling logic, optimization techniques, and fallback solutions from a Java developer’s perspective.

I. Emergency Alert: First Determine Why the Backlog Is Happening (Business + Technical Analysis)

Upon receiving an alert, resist the urge to panic. Blindly restarting consumers often exacerbates the situation.

As a senior Java developer, the initial priority is always to locate the root cause.

A Kafka backlog fundamentally indicates: Production speed > Consumption speed

The source may lie with the producer, the consumer, or system configuration.

1. Three-Step Quick Troubleshooting Method (Essential for Java Developers)

Step 1: Check Monitoring Metrics Begin with Kafka monitoring tools such as Kafka Manager. Key metrics to examine:

• Sudden increase in production TPS? (Possible traffic spike)
• Decrease in consumption TPS? (Usually the primary issue)
• Normal consumer group heartbeat?
• Balanced partition consumption progress?

Step 2: Check Consumer Logs Log into the consumer node and review runtime status. Leverage Arthas and jstack to investigate:

• Large volumes of exception logs (e.g., DB timeouts, API failures)
• Thread states in WAITING or BLOCKED
• JVM health:
• GC frequency
• Heap usage
• Possible OOM restarts

Step 3: Check Business Dependencies Our consumer logic followed this flow:

Receive message → Parse → Call inventory service → Store in database

Investigation revealed that a master–slave database switch in the inventory service increased response times:

• Normal latency: 200 ms
• Failure latency: 5 seconds

This led to severe thread blocking in consumers.

Resulting metrics:

MetricValueProduction TPS2000+Consumption TPS1000+ → 50BacklogRapidly increasing

2. Common Causes of Kafka Message Backlog (Based on 8 Years of Experience)

Consumer-Side Problems (≈80%) Most frequent issues include:

• Redundant business logic
• Excessive external API calls
• Slow database queries
• Unstable dependent services
• Improper thread pool configuration
• Code bugs (e.g., null pointers, infinite loops)
• Frequent consumer rebalances

Producer-Side Problems (≈15%) Typical scenarios:

• Sudden traffic spikes (e.g., large promotions)
• Oversized messages (10MB+)
• Poor retry logic causing repeated failed productions

Configuration Problems (≈5%) Examples:

• Insufficient Kafka partitions
• fetch.min.bytes set excessively large
• max.poll.records too small

II. Rapid Bleeding Control: Emergency Plan for Million-Level Backlog

Once identified as downstream service timeout causing consumption blockage, the strategy prioritized: Ensure consumption capacity first, then optimize

Kafka retains messages for 7 days by default, making rapid backlog clearance essential.

Step 1: Temporary Isolation With the inventory service unstable, normal consumption would worsen the backlog. We executed:

1. Pause the original consumer service (via configuration center, no restart needed)
2. Deploy a temporary consumer service with simplified logic:

Pull message → Parse → Write to temporary MySQL table

External service calls were bypassed to accelerate draining.

Step 2: Optimize Consumer Parameters and Parallel Processing

Kafka Consumer Parameter Optimization Original:

properties.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 100);
properties.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, 10240);
properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 10000);

Optimized:

properties.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 1000);
properties.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, 1024);
properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 30000);
properties.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, 500);

Improvements: 10x more messages per poll, reduced fetch wait time, lower rebalance risk.

Java Thread Pool Optimization Original:

ExecutorService executor = new ThreadPoolExecutor(
        5, 10, 60L, TimeUnit.SECONDS,
        new LinkedBlockingQueue<>(1000));

Optimized:

ExecutorService executor = new ThreadPoolExecutor(
        20, 50, 60L, TimeUnit.SECONDS,
        new LinkedBlockingQueue<>(100),
        new ThreadPoolExecutor.DiscardOldestPolicy());

Rationale: Large queues mask thread starvation; smaller queues force scaling and boost throughput.

Increase Kafka Partition Parallelism Original: 8 partitions Temporary: Increased to 20 partitions using:

kafka-topics.sh --alter --topic xxx --partitions 20

Consumer threads scaled accordingly to 20.

Step 3: Process Temporary Data in Batches After two hours: Backlog reduced from 1.2 million to 300,000 messages. Inventory service recovered.

Subsequent actions:

1. Batch processing from temporary table (1000 records per batch)
2. Asynchronous handling via thread pools
3. Retry mechanism with Guava Retryer (3 attempts, 1-second interval; failures to dead letter table)

Final timeline:

StageTimeBacklog reduction2 hoursRemaining processing2 hoursTotal recovery4 hours

No data loss occurred.

III. Code Upgrade: Preventing Backlog at the Architectural Level

Emergency measures address immediate crises; architecture prevents recurrence.

1. Asynchronous Consumption Logic Original: Synchronous API calls Improved:

@Override
public void onMessage(ConsumerRecord<String, String> record) {
    try {
        OrderMessage message = JSON.parseObject(record.value(), OrderMessage.class);

        CompletableFuture.runAsync(() -> {
            try {
                inventoryService.updateStock(
                        message.getOrderId(),
                        message.getProductId(),
                        message.getNum());
                orderMapper.insert(message);
            } catch (Exception e) {
                deadLetterQueue.send(record.value(), e.getMessage());
                log.error("Message consumption failed", e);
            }
        }, businessExecutor);
    } catch (Exception e) {
        log.error("Message parsing failed", e);
    }
}

Outcome: Consumer threads no longer block on business logic.

2. Service Fault Tolerance (Circuit Breaking + Degradation) Integrated Sentinel:

@SentinelResource(
    value = "updateStock",
    fallback = "updateStockFallback",
    blockHandler = "updateStockBlockHandler"
)
public boolean updateStock(String orderId, String productId, int num) {
    return inventoryClient.updateStock(orderId, productId, num);
}

Fallback:

public boolean updateStockFallback(String orderId, String productId, int num, Throwable e) {
    stockCache.put(orderId, new StockDTO(productId, num));
    return true;
}

Circuit breaker triggers on >50% failure rate or >1-second latency.

3. Thread Pool Monitoring and Alerts Added Spring Boot Actuator + Prometheus + Grafana. Example metric:

meterRegistry.gauge("thread.pool.active.count",
        Tags.of("thread.pool.name", name),
        executor,
        ThreadPoolTaskExecutor::getActiveCount);

Alerts: Active threads >80%, queue capacity <100.

4. Message Reliability Guarantees

• Retry: Guava Retryer (3 attempts, intervals 1s/2s/3s)
• Dead Letter Queue: Failed messages to dedicated Kafka topic
• Idempotency: Unique index on orderId

IV. Practical Tips for Avoiding Kafka Backlog

Three core principles:

1. Keep Consumption Logic Lightweight Consumer threads handle only: Parse message + Submit task. Heavy operations must be asynchronous.
2. Protect All External Dependencies Never assume downstream stability. Implement circuit breaking, timeouts, and degradation.
3. Comprehensive Monitoring Track beyond backlog: thread pools, downstream latency, success rates, rebalance frequency.

Conclusion

A Kafka message backlog is not the core issue — the absence of a clear handling strategy and fallback plan is.

From eight years of Java development experience, one principle endures: Emergency response relies on experience, but long-term stability depends on architecture.

This incident followed a disciplined approach: Isolation → Optimization → Root Cause Fix

The subsequent upgrades — asynchronous processing, circuit breaking, and monitoring — eliminated recurrence risk.

If you have encountered Kafka backlog challenges in production, please share your experiences in the comments. Real-world accounts often provide the most valuable engineering insights.

🔖 Thanks for reading.

• If you enjoyed this article, please consider giving it a clap.👏
• I would appreciate hearing your thoughts in the comments below! 💭
• Follow me for ongoing learning and connection!🔔