When Kafka Streams Silently Lost Its State: The /tmp Trap

A Kafka Streams join that ran fine for three months suddenly stopped producing output after a restart. No exceptions, no error logs, offsets still committing. The culprit turned out to be /tmp.

The Symptom

I have a Kafka Streams application that joins a metric_source KTable with a metric_value stream to parse and enrich incoming measurement data. It ran stably for about three months.

Then, after a routine restart, the parsed output silently stopped. The strange part:

• No exceptions were thrown.
• Offset commits kept advancing normally.
• The application looked perfectly healthy from the outside.
• But every join against metric_source came back empty — the key was simply not in the state store.

A pipeline that produces nothing while reporting that everything is fine is the worst kind of bug. Nothing points you at the cause.

Background: Where Kafka Streams Keeps Its State

To understand what went wrong, you need two facts about how Kafka Streams persists state.

1. Local state lives in RocksDB. Aggregations, joins, and KTables are materialized into a local RocksDB instance on disk, under the directory configured by state.dir.
2. The changelog topic is the source of truth. Every write to a state store is also written to a compacted Kafka topic (the changelog). If the local RocksDB data is lost, Kafka Streams can rebuild it by replaying the changelog.

There is also a small but important file in each RocksDB directory: the **.checkpoint* file. It records the changelog offset that the local state has been restored up to. On restart, Kafka Streams reads this file and only replays the changelog from that offset onward* — an incremental restore, not a full replay.

That checkpoint optimization is exactly what made this failure invisible.

The Clue: Only the .sst Files Were Gone

When I finally inspected the state directory, the picture was bizarre. The RocksDB metadata files — CURRENT, MANIFEST, LOG, the write-ahead log, and the .checkpoint — were all still there. But the **.sst files**, which hold the actual key-value data, were gone.

bash

# all the metadata was present, but this returned 0
find /tmp/kafka-streams -name "*.sst" | wc -l

Selective deletion like that is not something a human does with rm. It is a fingerprint of a process that deletes files based on age.

Root Cause: A Dangerous Default Plus a Checkpoint That Lied

Here is the part that matters most: I had never set state.dir at all. I did not point it at /tmp — I simply left it unconfigured. And the default value of state.dir is /tmp/kafka-streams, with Importance: high in the official configuration reference.

properties

# what I actually had: nothing.
# state.dir was never set, so Kafka Streams silently used:
#   state.dir = /tmp/kafka-streams   (the default)

That distinction is the whole lesson. The failure was not “I chose a bad path.” It was “I chose no path, so the framework chose /tmp for me." A default that is fine for a local dev run becomes a time bomb the moment the same code reaches a long-running production host — and nothing in the code makes that risk visible.

On this server, /tmp is managed by systemd-tmpfiles, with a policy along these lines:

# /usr/lib/tmpfiles.d/tmp.conf
D /tmp 1777 root root 30d

That 30d means: any file under /tmp not accessed or modified for 30 days gets purged, and systemd-tmpfiles-clean.service runs daily to enforce it. The cleanup works per file, by age — not per directory.

Now look at the two kinds of files in a RocksDB directory:

• **.sst files** are written once and then only read. For a slow-moving table like metric_source, their access and modification timestamps quickly age past 30 days.
• Metadata files (CURRENT, MANIFEST, LOG, the WAL, and especially .checkpoint) are rewritten constantly. RocksDB and Kafka Streams touch them on every commit — by default every 30 seconds (commit.interval.ms).

So the cleanup found exactly one set of stale files — the .sst data — and deleted them, while leaving every fresh metadata file untouched. That perfectly explains "only the .sst files vanished."

Here is the trap that turned data loss into silent data loss:

• The .sst files were deleted, but the .checkpoint file survived.
• On restart, Kafka Streams read the checkpoint, saw an offset that said “this store is already restored up to here,” and therefore skipped restoring from the changelog.
• RocksDB came up with no data, the checkpoint insisted everything was fine, and every join quietly missed.

The changelog — the actual source of truth — was sitting safely in Kafka the whole time. Kafka Streams just never read it, because a stale checkpoint told it not to.

The Fix

There were two things to fix: recover now, and make sure it never happens again.

Recovering the state

You cannot recreate .sst files by hand — that is not how RocksDB works, and you do not need to. The changelog can rebuild everything. The key is to force a full restore by removing the inconsistent local state, checkpoint included:

changelog topic (safe in the Kafka brokers)
        ↓  replayed on restart
RocksDB state rebuilt from scratch (.sst files regenerated)

Because the changelog topic used cleanup.policy=compact, every key was still retained. Deleting the affected state-store directory entirely — so the misleading checkpoint goes away too — and restarting let Kafka Streams replay the changelog from the beginning and regenerate the .sst files. The joins started matching again.

The permanent fix: always set state.dir explicitly

The real bug was not a wrong path — it was a missing one. The fix is to treat state.dir as mandatory the moment your topology uses any state store, and to point it somewhere that is never auto-cleaned or wiped on reboot:

properties

# before: state.dir was never set
#   → Kafka Streams silently defaults to /tmp/kafka-streams

# after: explicitly set to a stable, non-volatile path
state.dir=/data/kafka-streams

Anything under a durable path (/data, /var/lib/kafka-streams, etc.) is safe. This single line removes the entire class of failure — and, just as importantly, it makes the decision visible in your config instead of leaving it to a default nobody reviewed.

Takeaways

• If your topology uses a state store, setting state.dir is not optional. The default is /tmp/kafka-streams, and /tmp will eventually delete your state — by reboot or by age-based cleanup. The danger is that the path is implicit: you never typed /tmp, so nothing in your code warns you. Set it explicitly, every time.
• A surviving checkpoint can hide missing data. The incremental-restore optimization assumes the local data is consistent with the checkpoint. If something deletes .sst files out from under a live process, that assumption breaks and the failure is silent.
• Selective file deletion points at age-based cleanup. When only one category of file disappears, suspect a policy that deletes by access/modification time, not a person.
• The changelog is your safety net — if it is compacted and intact. Confirm cleanup.policy=compact and that no time-based deletion is in play, then a full restore is just "delete the local state directory and restart."

The bug looked like a Kafka Streams problem for a long time. In the end, it was an operating-system housekeeping policy quietly eating files that should never have been in its path.