The Ghost Event: How a Missing Read Preference Silently Broke Our Event Pipeline

It started with a single error log on a Yesterday morning. A validation failure — two required fields showing up as undefined. The kind of error that should never happen, because the code that produces those fields had been working for months.

Replication lag

The Setup

We run an event-sourced microservices built on Node.js and MongoDB. If you’re not familiar with event sourcing, the core idea is simple: instead of storing the current state of your data, you store every change as an immutable event. To get the current state, you replay the events from the beginning.

Our system has a twist. Some of our events contain personally identifiable information — think national ID numbers, dates of birth, names. Privacy regulations require us to handle this data carefully, so we split it.

When an event is saved, our EventStoreRepository separates it into two parts:

• EventCollection — the main event, with PII fields stripped out
• EventPIICollection — just the PII fields, stored in a separate collection

Both records share the same _id, written in a single database transaction. Atomic. Clean. Simple.

When the events need to be read — whether for rebuilding aggregate state or for publishing to our message bus — the PII data is fetched and merged back in. The application never knows the difference.

At least, that’s how it’s supposed to work.

The Error

The error appeared in our event subscriber — a service that listens for events on Pub/Sub and triggers follow-up actions.

ValidationError: Schema validation failed
  Missing required property: national_id
  Missing required property: date_of_birth
  Expected type string but found type undefined

An event was being consumed, and the handler was trying to build a follow-up action from the event data. Two fields that should have been strings were undefined.

My first instinct: maybe the data was never saved. I checked the database.

EventCollection — the event was there. The non-PII fields looked correct. The PII fields were absent, as expected — they’d been separated out.

EventPIICollection — the PII record was there too. Same _id. The fields were present and correct.

So the data existed in both places. The transaction had worked. But somewhere between the database and the subscriber, those fields had vanished.

The Pipeline

To understand what went wrong, I had to trace the entire path an event takes from creation to consumption. Let me walk you through it.

Step 1: Save. A handler builds the event with all fields — PII included. The repository separates PII from non-PII, generates a shared _id, and writes both records in a single transaction.

Step 2: Change Stream. Our publisher watches the EventCollection using MongoDB’s change stream feature. When a new document is inserted, the change stream fires and hands the publisher the full document — minus the PII fields.

Step 3: PII Enrichment. Before publishing, the publisher checks if the event type is registered as containing PII. If it is, the publisher fetches the matching record from EventPIICollection by _id, merges the PII fields back in, and publishes the complete event to Pub/Sub.

Step 4: Consume. The subscriber receives the event from Pub/Sub, reads the data (including what should be the merged PII fields), and uses them to build a follow-up actions.

The logic was sound. Every step had been tested. So where did the ghost come from?

The Clue

I pulled the publisher logs for the failing event. For every PII event that gets successfully enriched, the publisher logs a debug message: “*PII merge complete*”.

For the failing event? No such log. Just the publish confirmation.

I pulled logs for a different event of the exact same type, published the next day. That one had the merge log, followed by the publish log. Both events were the same type, processed by the same code path.

One worked. One didn’t.

The publisher had attempted to fetch the PII data for the failing event, received nothing back, and silently moved on. It published the event without PII fields, and once published to Pub/Sub, the message is immutable. Every retry of that message would deliver the same incomplete data. The subscriber would fail, retry, fail again, forever.

The Hunt

I stared at the enrichment code. It was straightforward:

async getPIIInfo(recordId) {
  const result = await this.piiDataCollection.findOne(
    { _id: recordId },
    { projection: { data: 1 } }
  );

  return result;
}

A simple findOne by _id. The record existed — I had verified it. Why would this return null?

• I checked if the _id types matched. They did — both ObjectIds.
• I checked if the event type was correctly registered. It was.
• I checked if PII redaction had run before the publisher processed the event. It hadn’t — redaction only runs for older records.

I was stuck. The code was correct. The data was correct. The behavior was wrong.

Then I looked at the connection configuration.

The One Line

Deep in the config file, the database connection had this:

{
  "database": {
    "options": {
      "readPreference": "secondaryPreferred"
    }
  }
}

secondaryPreferred. This tells the MongoDB driver: “Read from a secondary replica if one is available. Only fall back to the primary if no secondary is reachable.”

This is a perfectly reasonable setting for most read operations. It distributes load across the replica set. For eventually-consistent reads — dashboards, reports, projections — it works great.

But the publisher’s PII fetch wasn’t a casual read. It was a read that had to see data that was just written by a transaction on the primary.

Here’s the sequence:

1. The transaction commits on the primary — both EventCollection and EventPIICollection records are written atomically.

2. The change stream fires — change streams read from the oplog, so they see the committed data immediately.

3. The publisher receives the change stream event and calls findOne on EventPIICollection.

4. That findOne inherits secondaryPreferred from the connection. It routes to a secondary replica.

5. The secondary hasn’t replicated the EventPIICollection write yet.

6. findOne returns null.

7. The event is published without PII.

The window is tiny — milliseconds of replication lag. That’s why it almost always works. The secondary usually catches up before the publisher gets around to querying. But “usually” isn’t “always.”

The Irony

The funny part? Our own EventRepository — the one used by handlers and aggregate replay — explicitly overrides the read preference on every query:

const events = await this.eventCollection
  .find(query, null, {readPreference: 'primary'})
  .sort({event_seq: 1})
  .lean();

const piiRecords = await this.eventPiiCollection
  .find(piiQuery, {_id: 1, data: 1}, {readPreference: 'primary'})
  .lean();

Someone had thought about this exact problem for the application layer. Every read that needed consistency was pinned to the primary.

But the publisher wasn’t using our repository. It was using the native MongoDB driver directly — a Collection object obtained from the same connection pool. And on that findOne, there was no override. It just… inherited the default.

One line. One missing option. In a query that looked entirely correct.

The Fix

The fix itself was almost anticlimactic:

const piiInfo = await this.eventPiiCollection.findOne(
  {_id: eventId},
  {projection: {data: 1}, readPreference: 'primary'}
);

One option added to a query. That’s it.

The Lesson

I’ve been thinking about why this bug was so hard to find, and I think it comes down to three things.

First, the failure was silent. The publisher didn’t error. It didn’t log a warning. It just skipped the merge and moved on. The event looked like any other event on Pub/Sub. You wouldn’t know it was broken until a downstream consumer tried to use the missing fields.

Second, the failure was rare. Replication lag is typically sub-millisecond. This bug might fire once in ten thousand events. You can’t catch it in tests. You can’t catch it in staging with a single-node replica set. You can only catch it in production, under real load, with real network conditions.

Third, the failure was permanent. A Pub/Sub message, once published, cannot be modified. Every retry delivers the same broken payload. The subscriber will fail on this message forever, or until someone manually intervenes. A transient infrastructure hiccup becomes a permanent data corruption.

If there’s a single takeaway, it’s this:

When you split data across collections and reunite it later, the read that reunites them must have the same consistency guarantees as the write that split them.

A transaction on the primary means nothing if the subsequent read goes to a secondary.

This incident occurred in a Event Sourcing system handling PII data separation. The fix was a one-line change to a database query in a shared library. The bug had been latent since PII separation was introduced, waiting for the right millisecond of replication lag to strike.

Thanks for reading this far. If you’ve ever spent a day chasing a bug that turned out to be one line — I’d love to hear your story.