Debugging a Zombie Flink Pipeline: When Your Job Looks Alive But Is Actually Dead

A real-world case study of network buffer exhaustion in Apache Flink

The Problem

Have you ever encountered a Flink job that appears perfectly healthy in the UI? The status shows RUNNING, all tasks are in RUNNING state, and there are no exceptions in the logs. Yet somehow, no data is being processed.

I call these “zombie” Flink pipelines.

This happened to me recently. My Flink pipeline was in a mysterious state:

✅ Job status: RUNNING ✅ All tasks: RUNNING ✅ No OOM errors ✅ No memory leaks ❌ Zero data processing

What the flink?

Initial Investigation: The Standard Checks

I started by checking the usual suspects.

Memory Issues?

# Check TaskManager heap memory
flink.taskmanager.Status.JVM.Memory.Heap.Used

# Check TaskManager non-heap memory  
flink.taskmanager.Status.JVM.Memory.NonHeap.Used

# Check managed memory (RocksDB, batch operations)
flink.taskmanager.Status.Memory.Managed.Used

Result: Nothing. Memory looked fine.

Checkpoint Problems?

# Checkpoint duration
flink.jobmanager.job.lastCheckpointDuration

# Failed checkpoints
flink.jobmanager.job.numberOfFailedCheckpoints

# Checkpoint alignment time
flink.jobmanager.job.lastCheckpointAlignmentBuffered

# Checkpoint size
flink.jobmanager.job.lastCheckpointSize

Result: Checkpoints were completing successfully.

Dead TaskManagers?

All TaskManagers reported healthy heartbeats. Everything looked normal on the surface.

The Breakthrough: Finding the Root Cause

Since traditional monitoring wasn’t showing anything, I turned to Flink’s OpenTelemetry metrics.

Step 1: Check Data Flow

# Are we processing ANY records?
sum(rate(flink_taskmanager_job_task_operator_numRecordsIn[5m]))

Result: 0.

The entire pipeline was processing zero records per second.

Step 2: Find the Stuck Operator

# Find operators receiving data but not outputting
(
  sum(rate(flink_taskmanager_job_task_operator_numRecordsIn[5m])) 
  by (operator_name) > 0
)
and
(
  sum(rate(flink_taskmanager_job_task_operator_numRecordsOut[5m])) 
  by (operator_name) == 0
)

This query revealed something: at some point, certain stateful transform operators were receiving data but producing nothing. When I checked again, even the input had dropped to zero.

The pipeline had gone completely silent.

Step 3: The Smoking Gun

I checked the buffer pool usage metrics:

# Check network buffer pool usage
flink_taskmanager_job_task_buffers_inPoolUsage
flink_taskmanager_job_task_buffers_outPoolUsage

The pattern was dramatic:

Time Buffer Pool Usage Success Rate Status T+0 0.3 300/sec Normal T+60m 0.8 150/sec Degrading T+120m 1.0 0/sec 🧟 Zombie

The moment buffer pool usage maxed out at 1.0, my success rate dropped to zero.

The pipeline had choked itself to death.

Understanding the Network Buffer Architecture

Here’s where things get interesting. Flink uses a shared network memory pool across all tasks in a TaskManager:

TaskManager Network Memory Pool (Total: 32,000 buffers)
├─ Task1 Input Buffers  (borrowed from pool)
├─ Task1 Output Buffers (borrowed from pool)
├─ Task2 Input Buffers  (borrowed from pool)
├─ Task2 Output Buffers (borrowed from pool)
└─ ... (all tasks share this pool)

When a task’s inPoolUsage reaches 1.0, it means:

• The task has borrowed its allocated buffers from the shared pool
• It’s holding onto them without releasing them
• Other tasks can’t get the buffers they need
• This creates a cascading effect throughout the pipeline

According to Apache Flink’s official documentation:

“The NetworkBufferPool is a fixed size pool of MemorySegment instances for the network stack. The NetworkBufferPool creates LocalBufferPools from which the individual tasks draw the buffers for the network data transfer.”

This is the key architectural detail that explains everything.

The Cascading Failure Sequence

Here’s what happened according to my metrics timeline:

T+0 minutes: Normal operation

Available Network Buffers: 32,000
Heavy transform operator inPoolUsage: 0.3
Data flowing normally

T+60 minutes: Trouble brewing

Available Network Buffers: 15,000
Heavy transform operator inPoolUsage: 0.8
Checkpoint started (state snapshot in progress)

T+120 minutes: Critical

Available Network Buffers: 100
Heavy transform operator inPoolUsage: 1.0
Multiple operators waiting for buffers

T+180 minutes: Complete failure

Available Network Buffers: 0
Pipeline zombie state
Success rate: 0

Why Did This Happen?

The heavy stateful transform operators in my pipeline were:

• Stateful — holding large amounts of state in memory
• Slow during checkpoints — taking significant time to snapshot state
• Holding buffers — unable to release input buffers while processing

When the checkpoint started, here’s the chain reaction:

Stateful operator receives data 
  → Input buffers fill up
  → Processing slows (snapshotting state)
  → Can't release buffers fast enough
  → Input buffer pool reaches 100%
  → Upstream operators can't send data
  → Their output buffers fill up too
  → Chain reaction continues
  → Network buffer pool exhausted
  → No available buffers anywhere
  → Source stops reading
  → Pipeline dead 🧟

Confirming the Configuration Issue

I checked the TaskManager configuration via the REST API:

curl http://<jobmanager>:8081/taskmanagers/<tm-id>

There it was. On a 64GB TaskManager, only 1GB was allocated for network buffers.

This imbalance was the core issue.

Understanding the Math

Current allocation:
- Network Memory: 1GB = ~32,000 buffers (32KB each)
- With ~1,500 tasks running
- Each task gets ~20 buffers on average

The Fix

I added these settings to flink-conf.yaml:

# Increase network memory from default 1GB cap
taskmanager.network.memory.fraction: 0.15
taskmanager.network.memory.max: 8gb

The Result

Metric Before After Change Network Memory 1 GB 8 GB 8x Total Buffers ~32,000 ~256,000 8x Buffer per Task ~20 ~170 8.5x

The zombie states disappeared. The pipeline ran smoothly through checkpoints without buffer exhaustion.

Key Takeaways

1. Zombie pipelines have subtle symptoms

Unlike crashes or obvious errors, zombie pipelines appear healthy. You need to actively monitor:

• Actual record processing rates (not just job status)
• Network buffer availability
• Per-operator throughput

2. Buffer exhaustion creates cascading failures

When one slow operator holds buffers:

Slow operator → holds input buffers
→ upstream blocks → holds output buffers
→ upstream's upstream blocks
→ cascade to source
→ entire pipeline stops

3. Default configurations don’t scale

The default taskmanager.network.memory.max: 1gb is fine for small jobs, but completely insufficient for:

• High parallelism jobs
• Stateful operators with large state
• Jobs with many operators
• TaskManagers with >32GB memory

4. State size correlates with buffer needs

If you’re using significant managed memory for state (10GB+), you probably need proportionally more network memory.

Rule of thumb: Managed Memory : Network Memory ratio should be roughly 5:1 to 7:1 (not 12:1 as my default was).

Diagnostic Queries for Your Toolbox

Here are the PromQL queries I now keep in my monitoring dashboard:

Pipeline Zombie Detection

# Alert when job is running but processing nothing
(
  flink_jobmanager_job_uptime > 300
)
and
(
  sum(rate(flink_taskmanager_job_task_operator_numRecordsIn[5m])) == 0
)

Buffer Exhaustion Warning

# Alert when buffers drop below 5%
(
  flink_taskmanager_Status_Network_AvailableMemorySegments / 
  flink_taskmanager_Status_Network_TotalMemorySegments
) < 0.05

Stuck Operator Detection

# Find operators with input but no output
(
  sum(rate(flink_taskmanager_job_task_operator_numRecordsIn[5m])) 
  by (operator_name) > 0
)
and
(
  sum(rate(flink_taskmanager_job_task_operator_numRecordsOut[5m])) 
  by (operator_name) == 0
)

Buffer Pool Saturation

# Alert when task buffer pools are maxed
avg(flink_taskmanager_job_task_buffers_inPoolUsage) 
by (operator_name) > 0.95

Conclusion

Debugging zombie Flink pipelines requires looking beyond traditional metrics. The combination of OpenTelemetry metrics, understanding Flink’s network buffer architecture, and recognizing the cascading nature of buffer exhaustion was key to solving this issue.

If your Flink job ever looks alive but processes no data, start with these questions:

1. Are records actually flowing? (Check numRecordsIn/Out metrics)
2. Are network buffers available? (Check AvailableMemorySegments)
3. Are any operators stuck? (Compare input vs output rates)
4. Is your network memory properly sized? (Should scale with TaskManager size)

And remember: a RUNNING status doesn’t mean your job is actually running.

References

1. Apache Flink — NetworkBufferPool API Documentation
2. Apache Flink — Network Buffer Tuning Guide
3. Apache Flink Blog — Network Stack Vol. 2: Monitoring, Metrics, and Backpressure

Have you encountered zombie Flink pipelines? What metrics did you use to debug them? Share your experiences in the comments below.