Real-time ML Ranking for Autocomplete: Deploying Learning-to-Rank inside OpenSearch (Part 1)

Co-authored with Srinivas Nagamalla. Special mentions to Yawan Gupta and the Search-engineering-team for their contributions.

Autocomplete is one of the most latency-sensitive surfaces in any consumer app. At Swiggy, autocomplete is triggered on every keystroke, so ranking has to fit within a tiny latency budget while serving far more traffic than a typical search endpoint.

Image 1: AutoComplete in Swiggy’s Search

That makes autocomplete an interesting machine learning problem. Better ranking can improve what users discover and how quickly they act, but the system still has to behave like infrastructure: predictable, cheap to serve, and fast enough to respond instantly.

In part 1 of this post, we walk through how we moved from a hand-tuned heuristic ranking formula to a learning-to-rank (LTR) model running inside OpenSearch, and in part 2, we will see how we later extended this with more capabilities like real-time personalization.

Before ML ranking: Early Approach and Challenges

Before introducing ML ranking, autocomplete relied on a simple heuristic scoring formula using OpenSearch’s function_score query. The approach combined text-match rules with precomputed weights for each suggestion, making it fast, easy to run, and effective for basic cases.

However, as more signals were added, the system quickly became brittle:

• Balancing fuzzy matches with other signals: Fuzzy matching was easy to implement, but tuning its influence against popularity or other relevance signals was tricky. Poor balance often caused less relevant suggestions to outrank popular ones.
• Lack of conversion signal integration: The heuristic system could not incorporate click-through rates or order completions, relying only on static weights and raw click counts.
• Cross-type ranking challenges: Ranking multiple suggestion types — dishes, restaurants, cuisines, and brands — consistently was difficult, leading to skewed or unpredictable results.
• Manual tuning overhead: Each new signal required careful adjustment and regression testing, making the system fragile and costly to maintain.

In short, what started as a simple, single-signal ranking formula became increasingly complex and hard to manage as more signals were added — highlighting the key limitation of heuristic autocomplete.

Performance Constraints for Real-Time Autocomplete

Autocomplete is not a typical search-ranking problem. Unlike a full search results page, autocomplete fires on every keystroke. Typing “biryani”, for example, can trigger seven ranking requests (b, bi, bir, …). That changes both the latency envelope and the cost of serving. In practice, it imposes two hard requirements:

1. Sub-10ms ranking latency. Any model inference that adds perceptible delay to keystroke response makes the UI feel sluggish. Network round-trips to an external model serving layer (like a dedicated ML inference service) are too expensive at this call frequency.

2. Extremely high throughput. At Swiggy’s scale, autocomplete handles orders of magnitude more requests than the main search endpoint. The ranking solution must scale horizontally with the OpenSearch cluster itself, not bottleneck on a separate service.

We evaluated our internal ML inference stack for the autocomplete ranking model, but given the latency and cost implications of making an external inference call on every keystroke, we explored alternative approaches better suited to this use case.

Why Learning-to-Rank Inside OpenSearch?

To support ranking based on multiple weighted parameters — rather than relying on a small set of heuristic rules — we needed an approach that could incorporate richer, real-time signals like textual match, all without impacting latency. OpenSearch’s learning-to-rank (LTR) plugin fit these requirements well.

Key advantages:

• In-engine inference. Model inference runs at query time within OpenSearch. No external service call is required. The plugin persists feature and model metadata in its feature store and can apply trained models during search-time rescoring.

Image 2: Typical ML ranking service vs OpenSearch LTR latency

• Two-phase scoring with the rescore API. A fast first-stage retrieval query fetches candidate suggestions, and the LTR model rescores only the top-k candidates. This two-phase approach keeps latency bounded regardless of model complexity.
• RankLib model training compatibility. Supports externally trained models in formats such as RankLib and XGBoost, and also supports simple linear models.
• Flexible feature definitions. Features can be defined using OpenSearch LTR feature definitions, commonly with Mustache templates and, where needed, expressions or scripts. Both query-dependent features (text match scores) and query-independent features (document-level signals) are supported and evaluated at query time.

That made it a practical choice for our use case: ML-driven ranking, but with search-engine-like latency characteristics.

Autocomplete Architecture

High-level overview

At a high level, our autocomplete system operates through two retrieval flows:

1. Partial match flow — When user intent is still ambiguous (short prefixes like “bi” or “ch”), results are primarily retrieved by prefix match against indexed suggestion terms.
2. Semantic template flow — When intent is clearer, for example after a more complete query like “biryani”, semantic templates retrieve related suggestions, such as restaurants known for that item.

Image 3: Left: Partial match flow. Right: Semantic intent flow

Both flows query the same OpenSearch index. Each document, whether it represents a dish, restaurant, cuisine, or category, carries precomputed signals such as click counts, conversion rates, order volumes, and ratings.

The ranking pipeline follows a standard retrieve-then-rescore pattern:

Image 4: retrieve-then-rescore pipeline for autocomplete ranking

How the LTR Pipeline Works

The OpenSearch query structure

The retrieval template uses function_scorefor first-stage scoring, blending exact-match boosts and heuristic weights, and then applies LTR-based reranking via the rescoreAPI.

Image 5: Example OpenSearch autocomplete retrieval template with ML model re-ranking using the LTR plugin.

Model Training Pipeline

From an ML systems perspective, deploying the model required a full pipeline spanning retrieval, feature engineering, offline training, and online serving:

Image 6: Training and deployment setup for the Autocomplete ML model using Opensearch LTR.

Step 1 — Initialize the LTR Store and Index Features:

Initialize the default LTR feature store in OpenSearch, then index the autocomplete corpus with every field needed for ranking. Besides suggestion text, this includes precomputed signals such as clicks, conversions, orders, ratings, and popularity metrics. At inference time, features must be computable from indexed fields, request parameters, or other supported LTR feature definitions.

Initialize the default store using:

PUT _ltr

Step 2 — Pre-ranking Query Finalization:

The ML model does not score every document. A fast pre-ranking query first narrows the candidate set using prefix, fuzzy, or exact matches. It is worth stabilizing this retrieval logic early, because feature logging, training data generation, and evaluation all depend on it. In other words, retrieval defines the universe that the model ever gets to reorder. In our system, this is very similar to “Stage 1: Retrieval” in Image 5.

Step 3 — Feature Set Creation:

Image 7: Feature categories that can be used by the LTR model to improve relevance.

Define the features, then upload the feature set to OpenSearch’s LTR plugin. In practice, many features are expressed as Mustache templates, though the plugin also supports other feature-definition mechanisms. Each feature computes a numeric score for a query-suggestion pair.

These features are the model inputs. Some are query dependent, such as text-match strength. Others are query independent, such as ratings or popularity.

For example:

{
  "featureset": {
    "name": "example_featureset",
    "features": [
      // Query-dependent feature
      {
        "name": "prefix_match_score",
        "params": ["query"],
        "template": {
          "match_phrase_prefix": {
            "inputTermsRaw": {
              "query": "{{query}}",
              "max_expansions": 1
            }
          }
        }
      },
      // Query-independent feature (e.g., CTR, rating)
      {
        "name": "rating_signal",
        "params": [],
        "template": {
          "script_score": {
            "query": {"match_all": {}},
            "script": {
              "source": "doc['ctr'].value + 0.0"
            }
          }
        }
      }
    ]
  }
}

Store the completed feature set using:

POST _ltr/_featureset/<feature_set_name>
{
  "featureset": {
    "name": "example_featureset",
    ... 
}

Step 4 — Training Data Generation:

This is the bridge between OpenSearch and offline model training. For each historical query, fetch the top-k results from OpenSearch using the pre-ranking query, log feature values for each result via the LTR feature-logging flow ( sltrplus ltr_log), and then join those logged features with click, add-to-cart, or order data to create training examples.

This step is where the ranking problem becomes a supervised ML problem: historical user behavior becomes labels, and logged ranking signals become features.

When training with RankLib, the resulting judgment file is typically represented in RankLib’s SVM-rank-style format:

3  qid:1  1:7.23  2:10.08  3:0.0  4:1.0  ...  # biryani DISH
2  qid:1  1:5.83  2:0.0    3:0.0  4:1.0  ...  # manis_dum_biryani RESTAURANT
0  qid:1  1:6.87  2:10.08  3:0.0  4:1.0  ...  # birthday_cake DISH

Each line has a label, a query group ID ( qid ), feature values, and a comment identifying the suggestion.

For more details on training data format, check here.

Step 5 — Model Training and Validation:

Model training happens outside OpenSearch, for example on Databricks, using the RankLib library from the Lemur project. OpenSearch LTR can consume RankLib models, XGBoost models, and simple linear models, but in our case we trained with RankLib.

We experimented with multiple models from the Ranklib library (few examples include LambdaMART, Random Forests, RankNet), and evaluated it different metrics like MRR, NDCG across multiple funnels such as click, add-to-cart, and order.

Example training command (for LambdaMART model):

java -jar RankLib-2.8.jar -ranker 6 -train train.txt -save model.txt -bag 20 -frate 0.9 //additional flags as required. 

See the RankLib usage guide for full details. In practice, this step is iterative: experiment with feature sets, label schemes, and hyperparameters, then retrain and reevaluate until the metrics stabilize.

Step 6 — Deploy and Rescore:

Upload the trained model to OpenSearch via the model-creation API:

POST _ltr/_featureset/<feature_set_name>/_createmodel
{
  "model": {
    "name": "my_ranklib_model",
    "model": {
      "type": "model/ranklib",
      "definition": "<ranklib-model-contents>"
    }
  }
}

Once the model is live, it is invoked at query time through the rescoreAPI. The pre-ranking query fetches candidates, and the model rescoring happens on the top-k within the same OpenSearch request, with no external call required.

This separation of concerns matters. Retrieval remains optimized for speed and recall, while the ML model focuses on ranking quality over a much smaller candidate set.

Post deployment, the model can be called from within the OpenSearch Template as shown in Image 5.

Wrapping Up Part 1

With this setup, we replaced a hand-tuned heuristic with a learned ranking model that runs entirely inside OpenSearch — no additional services, no extra network hops, and no compromise on latency. The result was improved ranking quality through a richer set of learned signals, better handling of ambiguity and typos, and a production system that stayed within strict latency budgets.

But ranking quality is only part of the story. In Part 2, we cover how we later extended this with more capabilities including real-time personalization.

References

• Learning to Rank documentation.
• Lemur project, Ranklib, SVM-rank and Ranklib JAR files.
• Elasticsearch python client, Elasticsearch templates and Rescore API.