How We Unified Single and Multi Tenant Elasticsearch Clusters with 3–5× Performance Gains at Just 9% Latency Overhead

Before defining the problem, it is necessary to explain what our team does at Insider One.

We manage an API that serves thousands of partners and handles up to millions of requests per minute. This API is called the Smart Recommender API and, in simple terms, provides personalized product recommendations for our e-commerce vendor partners through a widget embedded on their websites, as well as across other channels such as email, price-drop and back-in-stock notifications, web/app push, and in-app recommendations. These recommendations must not only be smart, accurate, and relevant, but also delivered with very low latency so users can see them instantly and consistently across all touchpoints.

The Problem

We store our partners’ product catalogs in an ElasticSearch database. Each index represents the catalog of a single partner. At this point, the core problem should already be clear: all partners are hosted on the same cluster, meaning we are running a multi-tenant cluster.

Not every partner has the same usage pattern. While some partners only need around 100 requests per minute, others may require up to 120,000 requests per minute. Supporting such vastly different workloads within the same cluster can be challenging. When one partner heavily consumes cluster resources, it can negatively impact other partners, creating unfair resource usage.

Additionally, based on our internal performance and scalability assessments, we identified that certain high-usage workloads require isolation to prevent them from being affected by other partners — or from affecting them.

For these large-scale partners, we need to manage single-tenant clusters. Our current system, in its simplest form, looks like this:

Now, imagine that we create a new cluster for a specific partner. In that case, every service would need to communicate with this partner’s cluster separately. It would look something like this:

It would be a complete mess. Each service would also need to depend on an additional service just to determine which cluster a specific partner is using.

The system keeps getting more and more complex…

Okay, okay. So how should we manage a system that sometimes needs to run in a multi-tenant mode, and other times in a single-tenant mode?

Exactly. We need a central hub that abstracts away the underlying data flow and keeps track of which partner is assigned to which cluster, as well as the metadata of all available clusters.

With this approach, complex and operationally challenging tasks — such as onboarding a new cluster or routing a partner to a specific cluster — can be handled easily through a single, centralized system.

So, what is the additional latency introduced by this hub? After all, we’re no longer talking directly to the database — there’s now an extra hop in between. Shouldn’t that slow things down?

The highest latency we have observed is approximately 80 ms at p99, while p95 is around 10 ms. Compared to direct database calls that take roughly 110 ms, the additional 10 ms introduced by the DataHub corresponds to about a 9% latency overhead which is considered an acceptable trade-off given the added benefits

In other words, this DataHub not only simplifies operational management, but also does not introduce any significant additional latency to the system.

With routing solved, we turned our attention to the next bottleneck: query performance at the Elasticsearch layer.

Additional Lucene (Elasticsearch) Index Optimizations

While designing this system, we implemented several improvements at the search layer (Elasticsearch, built on top of Lucene). One of the most impactful optimizations we focused on was Lucene segment compaction via controlled force merges.

In simple terms, force merge reduces multiple segments (N segments) within a shard of an Elasticsearch index into a smaller number of segments — ideally down to a single segment. This significantly improves query performance by reducing the number of segments that need to be searched. However, aggressive segment merging is most effective for read-heavy, mostly immutable indexes. For indexes that receive continuous writes and updates, the cost of frequent merging can outweigh the performance benefits.

Our existing catalog database is highly write-intensive: new products are added continuously, existing products are updated, and various attributes change throughout the day. As a result, applying force merge to this shared catalog index would not yield meaningful performance gains and would introduce unnecessary indexing overhead.

For the new partner-specific indexes created specifically for the Smart Recommender API, we designed the system to minimize indexing activity in order to fully benefit from Lucene’s segment compaction model. We achieved this by updating only critical fields that genuinely need to be reflected in real time. For example, a product’s price or availability may change during the day, and these fields are updated immediately. On the other hand, changes to non-critical fields such as product names or descriptions are not reflected instantly.

All non-critical fields are updated through a daily batch job. This job runs every night for each single-tenant partner. It fetches the products used by the Smart Recommender API from the main catalog database, writes them into a newly created index on the partner’s dedicated Elasticsearch cluster, performs a controlled force merge to compact segments, and then routes traffic to this new index.

As a result, non-critical product attributes are refreshed daily, and any segments created by critical updates during the day are also compacted during the nightly merge process. This allows us to build a synchronization mechanism that effectively refreshes and optimizes itself on a daily basis.

Since these new clusters store only the subset of products served by the Smart Recommender API, other clients continue to use the main catalog database. Because these clients do not have strict low-latency or high-throughput requirements, there was no need to redesign their architecture for this use case.

After applying all of these optimizations, we were able to build an architecture capable of delivering the required performance for partners of any size. When we compared the legacy catalog database with the optimized setup under identical conditions, we observed 3×–5× improvements in response times, achieved purely through the optimizations described above.

The performance benefits introduced by isolating partners into dedicated Elasticsearch clusters are theoretically unbounded, as cluster resources can be scaled independently based on each partner’s traffic patterns, data size, and latency requirements.

Index Lifecycle

① Initial Sync: We create a new index in the single-tenant cluster and start the initial bulk copy from the source catalog database (read via /_search, write via /_bulk). During this step, traffic continues to flow to the previous day’s active index.

② Catch-up Sync #1: After the bulk copy completes, we run a catch-up sync to apply all updates that happened between t0 (when indexing started) and t1 (when it finished), ensuring we don’t miss any changes.

③ Force Merge: Once the index is consistent up to t1, we run /_forcemerge to compact Lucene segments and optimize the index for low-latency reads.

④ Catch-up Sync #2: When force merge completes at t2, we run a second catch-up sync to apply updates that occurred between t1 and t2 (i.e., during the merge window).

⑤ Switch Traffic (Atomic): Finally, we perform an atomic switch by updating the partner configuration in our DataHub (the table where we manage partner routing). After this update, all read and write traffic for that partner is routed to the new index.

Conclusion

By combining single-tenant and multi-tenant models, we built an architecture that adapts to partner needs instead of forcing them into a fixed shape. A centralized routing layer helped reduce operational overhead, while focused optimizations delivered clear improvements in latency and throughput. As a result, we were able to meet our SLAs consistently, even as traffic volume and workload complexity continued to increase.

The main takeaway for us was not to resist multi-tenancy, but to hide its complexity behind the right abstractions. Treating each partner as “just another cluster” through a single routing layer gave us flexibility without adding friction. Looking back, we would prioritize this abstraction earlier — it ended up being the key enabler for both scalability and day-to-day reliability.

We’d be interested to hear how others have tackled similar challenges. Have you gone fully multi-tenant, kept strict isolation, or landed somewhere in between? What design choices made the biggest difference for you?

See other articles from our brilliant engineers at Insider One.

[embed]Scaling Recommendation API: How We Cut Elasticsearch CPU Load by 50% with External Query Caching… The Challenge: Elasticsearch Under Pressuremedium.com

[embed]An Unexpected Bottleneck of A Recommendation System and 33% Cost Reduction As a Senior Software Engineer on the Algorithms team, I primarily focus on the infrastructure, release processes…medium.com

[embed]Managing Large Datasets in Go with Streaming Queries and Batch Inserts Handling large datasets in Go can be tricky — especially when data involves joins across multiple tables and needs to…medium.com