ElasticSearch: Navigating Lucene Segment Merges

Introduction: Beyond the Search Box — Understanding Lucene’s Core

At the heart of powerful search and analytics engines like Elasticsearch and OpenSearch lies Apache Lucene, a robust full-text search library. A fundamental concept in Lucene’s design, crucial for its efficiency and performance, is the “segment.” Segments are essentially immutable, self-contained mini-indices that collectively form a larger Lucene index. When new documents are added, or existing ones are updated or deleted, Lucene does not modify the existing segments. Instead, it employs an append-only strategy: new documents are written to new segments, and updated or deleted documents are merely marked as deleted within their original segments.

This immutable, append-only architecture, while simplifying write operations and ensuring data integrity, inherently leads to the accumulation of numerous small segments and “logically” deleted documents. Without a mechanism to manage this growth, search performance would degrade significantly as the system would have to query an ever-increasing number of files and filter out deleted entries. This is where segment merging, often an unseen background process comes into play. It acts as a vital choreographer, continuously combining smaller segments into larger, more efficient ones, reclaiming space, and optimising the index structure.

Segment merges in ElasticSearch

The Mechanics of Merging: How Lucene Keeps Indexes Lean

A Lucene index, which corresponds to a shard in Elasticsearch or OpenSearch, is built from these smaller, independent segments. When documents are ingested, they are initially buffered in memory and then periodically flushed to disk as new segments. The primary reasons why segment merging is essential include:

• Improved Search Performance: Searching across numerous small segments is inherently inefficient. Each segment must be opened, and searched independently, and its results then combined, consuming considerable CPU and memory resources.
• Disk Space Reclamation: As documents are updated or deleted, they are not immediately removed from their segments. Instead, a “delete marker” is applied. These marked documents continue to occupy disk space and consume memory in per-document data structures until the segments containing them are merged.
• Resource Consumption Management: Each segment, regardless of its size, consumes system resources such as memory and CPU cycles for its metadata and file handles. By consolidating many small segments into fewer, larger ones, merging reduces the overall resource overhead associated with managing the index.

Lucene’s IndexWriter automatically orchestrates this merging process based on a configured merge policy. The TieredMergePolicy is the default in recent Lucene versions and, consequently, in Elasticsearch and OpenSearch. This policy intelligently selects segments for merging, typically favouring those of approximately equal size and those with a higher percentage of deleted documents. The merge operation involves reading the content of the selected segments, re-indexing all their documents (excluding the marked-for-deletion ones) into a new, larger segment, and then deleting the original, smaller segments. Once the new segment is committed, it replaces the old ones, becoming part of the active index. This continuous background process, occurring both during active indexing and idle times, is fundamental to maintaining the long-term health and efficiency of a Lucene-based index.

The design choice of immutable segments, while simplifying the write path by making it append-only, shifts the complexity and resource burden to the merge process. The cost of cleaning up “dead” data and optimising the index is deferred and aggregated into these resource-intensive merge operations. This represents a classic engineering trade-off: optimising for writing simplicity by offloading the complexity to a background process.

The Impediments: When Merges Become a Bottleneck

Credit: Vespa Blog

Despite their necessity, Lucene segment merges are not without their challenges. They can become significant performance bottlenecks, particularly in high-throughput or write-heavy environments, due to their inherent resource demands.

Merge operations consume substantial system resources:

• CPU, I/O, and Memory Overhead: The process of reading multiple segments, re-indexing their contents, and writing a brand new, larger segment is computationally intensive. This involves significant CPU cycles for re-indexing, extensive disk I/O for reading and writing segment files, and temporary disk space for the new segments before the old ones are deleted.
• Degraded Search Performance: An excessive number of small segments directly increases search latency because the system must iterate through more files to find relevant documents. Moreover, active merge operations compete with ongoing search queries for shared resources like CPU and disk I/O, potentially leading to increased query latency and overall system instability.
• The “Deleted Document” Dilemma: The append-only nature of Lucene means that documents marked for deletion continue to occupy disk space and consume RAM in in-memory data structures (like norms or field data) until a merge physically removes them. In dynamic indices with frequent updates or deletions, the percentage of “dead” documents can fluctuate significantly, sometimes hovering between 35% and 60% in tests. A large segment, by default, might only be eligible for merging once it accumulates 50% deletions. If the rate of deletions outpaces the merge rate, this can lead to persistent resource waste and reduced search efficiency.

Elasticsearch’s Strategy: Fine-Tuning and Evolving Merge Policies

Elasticsearch, being built directly on Apache Lucene, primarily manages segment merges by exposing and optimising Lucene’s TieredMergePolicy (and LogByteSizeMergePolicy for time-based data, which favours adjacent segment merging to maintain time-ordered data. Elasticsearch's approach revolves around providing granular control over merge behaviour through a suite of dynamic settings and integrating continuous improvements from the Lucene project.

Practical Optimisations and Best Practices

Beyond direct merge policy tuning, several practical strategies can mitigate merge-related impediments in Elasticsearch:

• Reducing Shard Count: An excessive number of shards, each representing a Lucene index, consumes significant system resources (memory, CPU, file handles) and can lead to slow query performance. Reducing the shard count, ideally to match the number of nodes, can significantly lower resource overhead and improve query speed.
• Increasing Refresh Interval: Elasticsearch, by default, refreshes an index every second if there’s search activity, creating new segments.1 Increasing this refresh interval (or temporarily disabling it to -1 during bulk indexing) reduces the frequency of new segment creation, thereby decreasing merge pressure and improving write throughput. This comes at the cost of slightly increased replication lag and eventual consistency.
• Bulk Indexing and Disabling Replicas: For large data ingestions, using Elasticsearch’s Bulk API and parallelising indexing jobs is highly recommended. Furthermore, temporarily disabling replicas ( number_of_replicas to 0) during a bulk indexing job can greatly increase indexing speed. This is because each replica would otherwise perform the same indexing work, effectively multiplying the effort by the number of replicas. This, however, is not very practical in live traffic use cases. You would also want to try blue/green deployment in such a scenario.
• Judicious Use of _forcemerge: While the _forcemerge API can consolidate all segments in an index into a single segment (max_num_segments=1), which is ideal for read-heavy, static indices, it is a highly resource-intensive operation. It can temporarily double the storage required for a shard and block other merge requests until it completes. Therefore, it should be used sparingly and only during periods of low cluster traffic, typically after all write requests to an index are complete. This highlights that while it offers immediate search optimisation, it comes with a significant, temporary performance and resource spike, making it a specialised tool rather than a general-purpose optimisation.

Future-Proofing: Lucene Innovations Adopted by Elasticsearch

Elasticsearch’s evolution is closely tied to advancements in Apache Lucene. Recent Lucene versions, particularly Lucene 10, have introduced significant improvements that directly address common performance bottlenecks, including those related to segment management and I/O:

• Enhanced Search Parallelism: Traditionally, search parallelism in Lucene was coupled with the index’s segment geometry, meaning a force-merged index with a single segment couldn’t leverage multiple CPU cores for search. Lucene 10 overcomes this by allowing queries to split an index into logical partitions, independent of physical segments. This enables parallel searching even on indices with few segments, significantly increasing search parallelism on multi-core CPUs.
• Improved I/O Parallelism: For indices larger than the operating system’s page cache, I/O latency can become a bottleneck. Lucene 10 introduces an IndexInput#prefetch API, which allows the OS to parallelise the retrieval of data pages from disk within a single thread. This directly addresses the I/O bottleneck observed in merge operations, particularly benefiting fast local NVMe disks and, more importantly, network-attached storage like Amazon EBS or Google Cloud Persistent Disk, which often have higher latencies but good parallelism. This demonstrates a reactive development cycle where core Lucene improvements directly target known performance impediments in higher-level systems like Elasticsearch.
• HNSW Graph Merging Optimisations: For vector search use cases, work is ongoing to speed up the merging of Hierarchical Navigable Small World (HNSW) graphs, which are used for approximate nearest neighbour (ANN) search. Efforts are focused on enabling multi-threaded merging to reduce the overhead of building multiple HNSW graphs.
• Merge Executor and Scheduler Enhancements: Elasticsearch’s release notes indicate continuous improvements to its thread pool merge executor and scheduler, making them aware of available disk space to prevent issues during merge operations.

These ongoing integrations from Lucene demonstrate Elasticsearch’s commitment to refining and enhancing the core segment merging mechanism, ensuring it remains efficient and performant as hardware and workload patterns evolve.

OpenSearch’s Alternative: Segment Replication

While Elasticsearch focuses on optimizing the existing Lucene merge paradigm, OpenSearch has introduced a fundamentally different strategy for managing index copies (replicas) to address the inherent bottlenecks of traditional replication.

ElasticSearch vs OpenSearch Comparison

The Impediment in Traditional Document Replication

In the traditional “document replication” model, which Elasticsearch (and OpenSearch historically) employs, every replica node in a cluster performs the same indexing operation as the primary shard. This means that when a document is indexed, updated, or deleted, the primary shard processes the operation and then sends the document data to each replica. Each replica then independently parses the data, builds its own Lucene segments, and performs its merge operations. This duplicated effort across all replica nodes leads to:

• High Compute Costs: Every replica consumes significant CPU and memory resources to perform redundant indexing work.
• Indexing Throughput Bottlenecks: As the number of replicas increases, the total CPU and I/O demand for indexing scales linearly, potentially overwhelming the cluster and limiting overall ingestion rates.

Introducing Segment Replication

OpenSearch 2.7 introduced Segment Replication as a generally available alternative to document replication, built upon Lucene’s Near-Real-Time (NRT) Segment Index Replication API. This new strategy fundamentally changes how replicas are updated:

• Primary-Centric Indexing: With segment replication, only the primary shard performs the actual indexing operation, creating the Lucene segment files.
• Direct Segment Copying: Instead of re-indexing documents, the primary node then directly copies these pre-built segment files remotely to each node in the replica group. Replicas simply fetch and apply these segment files, making them searchable. This is possible because Lucene’s index is append-only, meaning existing segments are untouched, and new segments are created for changes.

Benefits and Implications

Segment replication offers several compelling advantages, particularly for specific use cases:

• Significantly Increased Indexing Throughput: By offloading the indexing CPU work from replicas to the primary, segment replication can achieve substantially higher ingestion rates. Initial benchmarks reported up to 40% higher throughput compared to document replication with the same cluster setup.
• Lower Compute Costs: Reducing duplicated CPU effort across replica nodes directly translates to lower compute resource utilisation for indexing operations.
• Decoupling Reads and Writes: Segment replication is positioned as the first step in OpenSearch’s broader vision to decouple read and write operations, allowing for more flexible scaling of compute and storage resources.
• Optimised for High Write Loads: It is particularly well-suited for scenarios with high ingestion rates and relatively low search volumes, or clusters with low replica counts (1–2 replicas), such as those used for log analytics. It also allows for easier addition of new nodes under high load without needing to re-index all data immediately.

However, this architectural shift also introduces new trade-offs:

• Increased Network Utilisation: The primary shard now sends larger blocks of segment data to its replicas, leading to increased network congestion on the primary node, especially as replica counts grow.
• Primary Shard Bottleneck: If replica counts are high (more than 3), the primary shard can become a network bottleneck due to the increased replication traffic.
• Replication Lag Sensitivity: Applications that require very low replication lag might find segment replication less suitable, as the data transfer is less frequent but in larger blocks compared to per-document replication.
• Remote-backed Storage Integration: To mitigate the network bottleneck on the primary, OpenSearch integrates segment replication with remote-backed storage (e.g., Amazon S3). In this model, the primary uploads segments to the object store, and replicas fetch updates from this shared remote storage, reducing direct primary-to-replica network traffic.
• Shard Indexing Backpressure: OpenSearch includes mechanisms like shard-level indexing backpressure and segment replication backpressure to dynamically reject indexing requests or slow down ingestion if replicas fall too far behind the primary, ensuring cluster stability under strain.

This adoption of segment replication represents a fundamental architectural divergence for OpenSearch, directly addressing the traditional impediment of duplicated indexing work on replicas. It allows OpenSearch to achieve higher indexing throughput and lower compute costs for specific workloads. This move, particularly when combined with remote-backed storage, hints at a long-term vision within OpenSearch to decouple compute and storage, enabling more flexible scaling and potentially different segment management strategies across various data tiers (hot, warm, cold).

Conclusion

Lucene segment merges are the unsung heroes of search engine efficiency, perpetually working in the background to maintain index performance and reclaim resources. Both Elasticsearch and OpenSearch, built upon the same Lucene foundation, grapple with the inherent challenges posed by Lucene’s immutable segment architecture, particularly the resource intensity of merges and the accumulation of deleted documents.

Elasticsearch’s strategy to address these challenges primarily involves the continuous refinement and granular tuning of Lucene’s TieredMergePolicy. Through a rich set of configurable settings, Elasticsearch empowers users to optimise merge behaviour for their specific workloads, balancing disk space efficiency, CPU, and I/O consumption. Furthermore, Elasticsearch consistently integrates the latest advancements from Apache Lucene, such as enhanced search and I/O parallelism, and specialised HNSW graph merging, to future-proof its segment management capabilities. This approach focuses on iterative improvements and maximising efficiency within the established Lucene framework.

In contrast, OpenSearch has taken a more architecturally divergent path with the introduction of Segment Replication. While traditional document replication duplicates indexing work across all replicas, segment replication offloads this heavy lifting to the primary shard, with replicas simply copying the resulting segment files. This fundamental shift significantly boosts indexing throughput and reduces compute costs, particularly beneficial for write-heavy workloads with lower replica counts. This move, especially when combined with remote-backed storage, signals OpenSearch’s long-term vision to decouple computing from storage, offering a distinct path for scalability and resource optimisation.

Ultimately, understanding the nuances of Lucene segment merges and the differing strategies employed by Elasticsearch and OpenSearch is crucial for architects and developers. Whether fine-tuning merge policies in Elasticsearch or leveraging segment replication in OpenSearch, informed decisions about cluster configuration and design are paramount to achieving optimal performance, resource utilisation, and overall system health in distributed search environments.

References

1. This gives detailed insights into how Vespa performs against ES: https://blog.vespa.ai/elasticsearch-vs-vespa-performance-comparison/
2. https://www.glassthought.com/notes/goy21eko6xk63f4vmz5fhmx/
3. https://luis-sena.medium.com/the-complete-guide-to-increase-your-elasticsearch-write-throughput-e3da4c1f9e92
4. https://www.elastic.co/search-labs/blog/apache-lucene-10-release-highlights
5. https://docs.opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/segment-replication/index/
6. https://docs.opensearch.org/docs/latest/api-reference/index-apis/force-merge/
7. https://opensearch.org/blog/segment-replication/

If you like my posts, don’t forget to subscribe to my profile for technical analysis on ElasticSearch and Apache lucene