Should I Be in This Meeting?

I was invited to a meeting that clearly wasn’t meant for my team.

Different roadmap. Different OKRs.

My pair looked at the invite and said, “Probably a mistake,” and skipped it.

I almost did the same.

But then I heard one sentence:

“We’re planning to introduce an automated real-time deletion flow across all modules.”

And I stayed.

Two hours later, I had redrawn the entire proposal.

This isn’t a story about heroics. It’s a story about something much more mundane.

Every system is a fan-out. And every implementation detail leaks into another implementation.

The moment I realized why I was still in the meeting.

Because this wasn’t about indexing.

This wasn’t about the indexer module.

This wasn’t about crawling.

This was about the entire search pipeline.

Distributed systems with especially high throughput are not isolated boxes.

They are layered fan-outs.

For some context: I was working on a search engine project and the architecture of this kind of system is typically built around the following stages:

Crawl → Parse → Enrich → Index → Replicate → Rank → Serve

Since I had been part of the project from the very beginning, I was involved in almost every stage of this pipeline. However, toward the end, I was primarily responsible for the ranking component and Elastic search clusters health.

The Meeting Proposal: “Just Delete It”

The Crawler/Indexer teams wanted something reasonable.

When a web page becomes invalid — removed, outdated, policy-violating — delete it everywhere. Immediately. Automatically. In real time.

What “everywhere” means in a search engine:

• Multiple modules
• Multiple indices
• CDN for images
• Multiple clusters
• Replications (CCR)
• Analytics dashboards querying them live

We can phrase it clearly and powerfully like this:

“Just delete it” was about to become something much heavier. The system wasn’t just going to perform deletions — it was about to execute deletion requests in the middle of a write storm(new/updated webpages) and a search storm(queries).

Elasticsearch Is Not a CRUD Database

The first misconception was subtle but critical.

The proposal treated Elasticsearch like this:

DELETE FROM pages WHERE id = X;

But Elasticsearch doesn’t “delete” documents the way a relational database does. It creates more work behind the scenes.

It:

1. Marks documents as deleted.
2. Writes tombstones.
3. Triggers segment merges.
4. Consumes I/O.
5. Rewrites data structures in the background.

Deletion is not free.

At low volume? Fine.

At high-frequency, real-time across multiple indices?

You’re not cleaning data.

You’re manufacturing I/O pressure.

And in search systems, I/O pressure leads directly into:

• Query performance
• Refresh latency
• P99 latency
• Segment merge backlog
• Ranking freshness

Which means:

A deletion implementation can degrade search quality and performance.

Then Someone Said “CCR”

Cross-Cluster Replication.

And that’s when the real problem surfaced.

CCR does not replicate data. It replicates operations.

Which means:

A deletion spike on the leader cluster → becomes a deletion spike on the follower clusters → becomes double the write amplification → becomes doubled merge pressure

If the leader cluster struggles, the follower struggles. If the follower lags, replication queues grow. If replication lags, ranking sees stale documents and it also suffers from increased latency and degraded performance.

Suddenly, deletion strategy is no longer an indexing detail.

It’s a ranking performance and stability issue.

The Fan-Out Fallacy

This is the default human instinct:

“Let’s just send the deletion request to all indices.”

But in distributed systems?

Spray and pray.

Broadcast and hope.

But every broadcast is a fan-out.

Every cascading failure multiplies cost.

Every layer multiplies cost downstream.

Every decision upstream reshapes relevance downstream.

If replication lags:

• Search consistency degrades.
• Metrics lie

If merges spike:

• Query latency increases.
• Tail latencies hurt CTR.
• Ranking performance inefficiencies.

That’s the connection.

The solutions I implemented on our end right after this meeting immediately: Buffer, Target, Observe

Instead of real-time per-document deletion across all modules, I proposed introducing a middle layer service in between indexing and ES clusters. When we are planning the components, the priority for real-time ingestion forced a direct coupling to indexing component.

The change was:

1. Buffer Instead of Spike

In most distributed systems, adding a messaging or buffering layer is standard practice. We initially bypassed this to ensure that newly discovered or updated pages were reflected in our search results with near-zero latency. Since the Indexer service was already directly feeding the Elasticsearch clusters, it seemed efficient. However, this left us with a dangerous blind spot: we had overlooked the reality that an upstream service, managed by a different team, could trigger massive spikes.

Elasticsearch was a strategic choice for real-time visibility. However, this created a tight coupling that ignored the risk of uncontrolled ingestion spikes.

Using Kafka as a buffer.

• Aggregate deletion events.
• Process in bulk.
• Execute during controlled windows.
• Tune batch size.
• Protect refresh intervals.

Bulk deletes reduce merge fragmentation.

This solution transforms chaos into predictable load.

2. Target Instead of Broadcast

Maintain lightweight metadata:

• Which page lives in which index?
• Which language cluster owns it?
• Which module actually stored it?

Turn this:

Delete everywhere.

Into this:

Delete exactly where needed.

The most expensive request is the one you didn’t need to send.

A 9-index broadcast per deletion may seem like a practical implementation detail. However, at scale, such a pattern can unintentionally resemble a distributed denial-of-service against the system itself.

3. Track Deletions as a First-Class Signal

Elasticsearch already gives you dashboards.

Indexing rate, search latency, segment count, JVM pressure, replication lag… all of it is visible in Kibana. On paper, we had observability.

But visibility is not the same as protection.

Dashboards are reactive. They tell you something is wrong after it starts going wrong.

What we needed wasn’t another graph. We needed guardrails.

So instead of relying solely on cluster-level metrics, we introduced a preventive layer:

• We tracked deletion events separately.
• We measured deletion rate per minute/hour.
• We correlated spikes with indexing throughput and merge pressure.
• We defined safe operational thresholds.

All Systems Are Fan-Out

Here’s what that meeting reminded me:

There are no local decisions in distributed systems. Because in distributed systems, prevention is cheaper than recovery.

Every “simple” feature is a multiplier.

Every multiplier touches:

• Latency
• Stability
• Relevance
• Infrastructure cost
• Experiment quality

Everything feeds back.

Engineering isn’t about knowing more Elasticsearch internals.

It’s about recognizing when:

A local optimization is about to become a global instability.

By decoupling our deletion logic, we successfully mitigated the system during heavy indexing windows and eliminated cluster instability during cleanup tasks.

Thanks for reading…