Recovering Lost Malware and Fixing a Watchdog That Watched the Wrong Thing

Twelve days of honeypot data never reached Elasticsearch. Most of it was routine scan noise I was happy to write off. But twenty of those lost events were captured malware downloads, the exact thing this honeypot exists to collect. Here is how I got them back, and how I rebuilt my monitoring so this never goes unnoticed again.

This continues from Part 3, where I diagnosed a silent shipping failure by reading Elasticsearch index boundaries. The data was on disk. The question was how to recover what mattered.

Triage: What Is Actually Worth Recovering

The gap held roughly 354,000 connection events and 20 malware download events. I made a deliberate call here. The connection events are commodity scan traffic, replaceable in spirit and grep-able from disk if I ever need stats. The downloads are irreplaceable signal. So I recovered the twenty and documented the rest as a known gap.

This is the same triage logic from my download-queue work: lead with consequence, not volume. Twenty download events outrank 354,000 connection events because a download means an attacker’s intent became an action.

The Recovery

The downloads were already sitting in dionaea-downloads.json on the Pi, with full metadata. Filebeat had read past them, so it would never re-ship them on its own. I wrote a small script to extract just the gap-window lines and bulk-load them straight into Elasticsearch through the _bulk API, stamping each document's @timestamp from its original event time so they would land on their real June dates.

I loaded them into a dedicated index, isolated from my live data, so the whole operation was trivially reversible.

The Gotcha: Data Streams Reject Plain Writes

My first attempt failed instantly with a clear error:

only write ops with an op_type of create are allowed in data streams

Here is what bit me. My Filebeat template forces anything matching honeypot-* to become a data stream, and data streams reject the standard bulk index action. My backfill index name matched that pattern, so it got converted and then refused my writes. The fix was simple once I understood it: name the backfill index outside the honeypot-* pattern. One rename, and all twenty documents loaded cleanly.

The lesson there is worth keeping. Index templates apply by name pattern, silently, and a data stream is not a normal index. If a bulk load fails on op_type, your target got templated into something you did not intend.

The Real Fix: Watch the Right Thing

Recovering the data was the easy part. The harder question was why nothing told me for twelve days.

My old watchdog checked whether the local log file had been modified recently. During the outage, the file was being modified constantly, because the honeypot never stopped capturing. The check saw a fresh file and reported healthy the entire time. It was watching a proxy near the start of the pipeline, not the end.

So I built a new watchdog that asks the only question that matters: has a new document actually landed in Elasticsearch recently? It queries the cluster for the newest honeypot timestamp and alerts if nothing has arrived in two hours.

One deliberate design choice: it runs on the Elasticsearch host and queries locally, not on the Pi across the tunnel. The tunnel going down is one of the failure modes I most need to catch, and a watchdog that depends on the tunnel would go blind in exactly that scenario. During the June outage, this check would have fired within two hours instead of staying silent for twelve days.

The Lesson

Monitoring is only as good as the question it asks. A healthy sensor, a fresh log file, a running agent- none of those prove your data arrived. The only honest signal is a document landing where you actually query it. Check the end of the pipeline, not the start, and run the check from somewhere that survives the outage you are trying to detect.

Next up: those twenty recovered downloads include one URL-staged outlier that does not match the usual SMB-worm pattern. Time to find out what it is.

Part of my hands-on Honeypot Investigations series. Real infrastructure, real failures, real lessons.