The Hidden Complexity of OOMKills: How to Properly Debug Memory Failures in Production

After a long time away from writing on Medium, I’m finally coming back.

The past months have been extremely intense. I’ve been deeply focused on personal projects related to observability, Kubernetes, eBPF, distributed systems, AI-assisted diagnostics, and low-level infrastructure tooling, so writing ended up taking a back seat for a while.

But during that time I accumulated a huge amount of real production experience and research that I want to start sharing again.

So for this return, I wanted to start with one of the most misunderstood production failures in modern infrastructure:

How to Properly Debug an OOMKill in Production

Most engineers think an OOM event simply means:

“The application used too much memory.”

But reality is far more complicated.

An OOMKill is not merely an application problem.

It is the result of interactions between:

• the Linux kernel,
• cgroups,
• container runtimes,
• allocators,
• garbage collectors,
• runtime memory models,
• Kubernetes,
• and the programming language itself.

Two services consuming the exact same amount of memory can behave completely differently depending on whether they are written in Go, Java, Rust, Python, or Node.js.

In this article we will go deep into:

• how Linux actually kills processes,
• how Kubernetes reports OOM events,
• how to recover the real kernel messages,
• how memory behaves differently across languages,
• and how to investigate real-world memory failures in production systems.

Understanding What Actually Happened

One of the biggest mistakes during incident response is assuming the application crashed by itself.

Usually it did not.

In many cases the Linux kernel forcibly terminated it.

That distinction matters enormously.

There are two completely different situations:

ScenarioWhat happenedApplication crashThe application detected an internal error and exitedOOMKillThe Linux kernel externally terminated the process

When the kernel kills a process, the application often has no chance to:

• flush logs,
• finish requests,
• emit telemetry,
• or even write a stack trace.

This is why many engineers start an investigation with:

There are no logs.

Because the process never had time to write them.

The Linux OOM Killer

Linux contains an internal subsystem called the OOM Killer.

Its purpose is brutal but necessary:

When memory is exhausted, something must die so the machine survives.

The kernel computes an internal score for processes called:

oom_score

Processes with higher scores are more likely to be killed.

You can inspect it directly:

cat /proc/<pid>/oom_score

And its adjustment value:

cat /proc/<pid>/oom_score_adj

This becomes especially important in Kubernetes because the kubelet automatically manipulates these values depending on QoS class.

QoS ClassOOM PriorityBestEffortHighest chance of deathBurstableMediumGuaranteedLowest

This is why pods without proper requests and limits tend to die first under memory pressure.

What Kubernetes Actually Shows You

Most people first see this:

kubectl describe pod my-app

And then:

Last State: Terminated
Reason: OOMKilled
Exit Code: 137

But this is only the surface.

The truly valuable information is usually inside the kernel logs.

Depending on the environment:

dmesg -T

Or:

journalctl -k

Or inside node logs in managed Kubernetes environments.

A real kernel OOM message often looks like this:

Out of memory: Killed process 18273 (java)
total-vm:4097152kB,
anon-rss:1983240kB,
file-rss:0kB,
shmem-rss:0kB

This single line contains an enormous amount of information.

Understanding the OOM Message

Most engineers see these fields and ignore them.

But they are critical.

Example:

anon-rss:1983240kB

This represents anonymous resident memory:

• heap allocations,
• runtime allocations,
• malloc usage,
• non-file-backed memory.

This is usually the most important field during debugging.

Meanwhile:

file-rss

Represents file-backed memory:

• mapped binaries,
• mmap’ed files,
• shared libraries,
• page cache mappings.

And:

total-vm

Represents virtual memory.

This is where many people get confused.

Virtual Memory Is NOT Real Memory Usage

One of the biggest misconceptions in Linux debugging is assuming virtual memory equals physical RAM usage.

It does not.

A process may reserve enormous address spaces without actually consuming physical memory.

For example:

• Java reserves huge heaps,
• Go reserves arenas,
• Rust allocators reserve regions,
• Node.js reserves V8 spaces.

You might see:

total-vm: 20GB

While actual RSS is only:

2GB

This distinction is critical.

RSS vs Heap vs Cache

Another major source of confusion is that memory usage is not a single thing.

People often say:

“The process is using 8GB.”

But which 8GB?

There are multiple categories:

TypeMeaningRSSResident physical memoryHeapAllocated runtime memoryVirtual MemoryReserved address spacePage CacheFile cache maintained by kernelShared MemoryShared pages between processesAnonymous MemoryHeap-like allocations

Many debugging mistakes happen because engineers look only at container memory graphs without understanding what is actually being measured.

Why OOM Debugging Changes Completely Depending on the Language

This is where things become extremely interesting.

Different runtimes manage memory in radically different ways.

The exact same workload can produce entirely different OOM behavior depending on the language.

Java OOMs

Java is one of the most misunderstood environments during memory investigations.

People assume:

-Xmx = total memory usage

This is false.

Java memory usage includes:

• heap,
• metaspace,
• thread stacks,
• direct buffers,
• JIT compiler memory,
• native allocations,
• GC structures.

A container limited to:

2GB

Can still OOMKill even with:

-Xmx1500m

Because the JVM itself consumes additional native memory outside the heap.

You can inspect native memory usage with:

jcmd VM.native_memory summary

And heap usage with:

jmap -heap <pid>

Go OOMs

Go behaves very differently.

The Go runtime aggressively reserves memory arenas from the OS.

This often causes confusion because RSS may continue growing even after garbage collection occurs.

A common misunderstanding is:

“GC ran, why didn’t memory drop?”

Because Go frequently keeps memory reserved for future allocations instead of immediately returning it to the operating system.

This behavior changed significantly across Go versions.

Modern Go versions improved scavenging behavior dramatically.

You can inspect runtime memory using:

GODEBUG=gctrace=1

Or export runtime metrics via Prometheus:

runtime.ReadMemStats()

Very often Go OOMs are caused by:

• massive slice growth,
• unbounded caches,
• goroutine explosions,
• protobuf allocations,
• JSON parsing,
• or large buffer retention.

Node.js OOMs

Node.js introduces another type of confusion.

V8 uses its own memory model with:

• young generation,
• old generation,
• large object space,
• code space,
• and external memory.

A process may crash with:

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Before Kubernetes itself triggers an OOMKill.

This distinction matters:

• application-level OOM,
• versus kernel-enforced OOMKill.

Node.js also suffers heavily from:

• buffer retention,
• unresolved promises,
• huge JSON objects,
• accidental global references,
• and memory leaks caused by closures.

Heap snapshots become extremely important here.

Python OOMs

Python is particularly deceptive during memory debugging.

Python applications often appear lightweight initially but suffer from:

• object overhead,
• fragmentation,
• reference cycles,
• C-extension leaks,
• Pandas allocations,
• NumPy buffers,
• and multiprocessing duplication.

Another major issue:

• CPython does not always return freed memory back to the OS.

So RSS may remain permanently elevated even after objects are released.

This creates many false assumptions during incident response.

Rust OOMs

Rust changes the problem entirely.

Rust avoids garbage collection, but that does not mean Rust applications cannot OOM.

They absolutely can.

The difference is usually:

• allocator behavior,
• uncontrolled buffering,
• memory fragmentation,
• huge Vec growth,
• mmap usage,
• or cache retention.

One particularly interesting case in Rust is allocator selection.

Different allocators behave radically differently under pressure:

• glibc malloc,
• jemalloc,
• mimalloc,
• tcmalloc.

In high-throughput systems this can completely change RSS behavior.

Cgroups and Why Containers Complicate Everything

Containers do not see memory the same way the host does.

The kernel enforces limits through cgroups.

You can inspect limits with:

cat /sys/fs/cgroup/memory.max

Or in older systems:

cat /sys/fs/cgroup/memory/memory.limit_in_bytes

And current usage:

cat /sys/fs/cgroup/memory.current

This becomes critical because:

• the kernel may kill the process,
• even while the node still has free memory.

Why?

Because the container exceeded its cgroup limit.

This is one of the most misunderstood parts of Kubernetes memory debugging.

Recovering the Real OOM Event in Kubernetes

Many engineers stop at:

kubectl describe pod

But the real investigation usually requires:

• node logs,
• kubelet logs,
• container runtime logs,
• and kernel messages.

Useful commands:

journalctl -u kubelet

crictl inspect <container>

dmesg -T | grep -i kill

And in managed Kubernetes:

• GKE node logs,
• EKS kernel logs,
• AKS VMSS diagnostics.

The Most Common Real Causes of OOMs

In production, most OOMs are not caused by “one giant allocation.”

Usually they are caused by slow uncontrolled growth patterns.

Examples:

CauseExampleUnbounded cacheMap grows foreverBuffer retentionRequest bodies never releasedQueue explosionBackpressure failureGoroutine leakInfinite background workersMemory fragmentationRSS never decreasesNative allocationsRuntime unaware memoryHuge JSON payloadsDeserialization spikesCompression buffersLarge temporary allocationsMetrics cardinality explosionMassive in-memory label maps

Many “memory leaks” are not leaks at all.

They are:

• retention problems,
• fragmentation,
• or allocator behavior.

One of the Most Dangerous Misconceptions

Many engineers think:

“If memory usage is stable, there is no leak.”

This is completely false.

A process can:

• stabilize at a dangerously high RSS,
• fragment memory,
• retain buffers,
• or continuously pressure the kernel reclaim system.

Then a small traffic spike arrives:

And suddenly:

• reclaim fails,
• memory compaction stalls,
• cgroup limit exceeded,
• kernel kills process.

The actual root cause may have started hours earlier.

Why OOMKills Are Becoming More Important

Modern systems are increasingly memory-sensitive.

Especially:

• Kubernetes,
• AI workloads,
• observability pipelines,
• vector databases,
• streaming systems,
• and eBPF-heavy agents.

Today memory debugging is no longer optional infrastructure knowledge.

It is becoming a core production engineering skill.

Because modern incidents are rarely:

• simple crashes,
• or obvious stack traces.

Increasingly, they are:

• resource exhaustion,
• allocator behavior,
• runtime pressure,
• and kernel-level decisions.

Final Thoughts

OOM debugging is one of the best examples of why modern infrastructure engineering requires understanding systems far below the application layer.

To debug OOMs properly you need to understand:

• Linux internals,
• memory accounting,
• container isolation,
• runtime behavior,
• garbage collection,
• allocators,
• and observability itself.

The application log alone is often useless.

The real story usually exists:

• inside the kernel,
• inside cgroups,
• inside allocators,
• and inside runtime memory models.

And once you start reading OOM events correctly, you realize something important:

An OOMKill is not merely a crash.

It is the kernel telling you a story about how your system behaves under pressure.