From Elasticsearch to Vespa: Rebuilding the Kleinanzeigen Homepage Feed — Part 2

In the first part of this series, Andre Charton laid out the foundation: why we moved away from Elasticsearch, how we structured the data model, and how WAND-based click-profile retrieval went into production. The core principle was to keep profile learning, retrieval, and ranking inside the serving platform itself. That principle carries through everything in this part.

In this second part, we go further: how we incorporated explicit user preferences, how Vespa’s ranking primitives changed the way we think about retrieval, and how we introduced semantic embedding search for long-tail discovery. Along the way, we share the operational lessons learned from running these systems in production at scale.

Phase 2: Adding Explicit User Preferences

Not everything is derivable from click history. A user who explicitly states they’re looking for women’s fashion shouldn’t need to click a dozen items before we learn that. We had a user preferences service, but connecting it to Vespa required plumbing.

Preference updates flow in via Kafka, get consumed by a dedicated processor, and land directly on the user’s profile document in Vespa.

At query time, this spawns a second WAND query running alongside the click profile WAND. Both are normalized to the same scale and merged with a round-robin shuffle. The equal split was deliberate: we wanted preference signals to be genuinely influential, not drowned out by click history.

Fashion turned out to be the most demanding case. Preferences there carry structural meaning: gender is orthogonal to item type, and many tokens interact. We apply context-sensitive boost logic where certain item types get amplified depending on whether a gender context is present or absent. An A/B test validated the approach before full rollout.

Over time, fashion got its own consolidated query path. Instead of merging two separate WAND queries post-retrieval, preference tokens are injected directly into the click-profile WAND as a seed weight — a calibrated bias that influences retrieval from the start. A second labeled WAND is added as a RANK operator child, making its score available in the final rank expression alongside keyword relevance.

The path to the final design involved several iterations. We tried a preference boost multiplier and a dominant-signal boost before settling on seed weight injection, which produced the most stable ranking behavior. The preference floor was tuned dynamically as we learned how different preference strengths interacted with the click profile.

The RANK operator separates retrieval from scoring cleanly: the first child determines which documents are candidates, while additional children contribute scores without affecting the candidate set. This makes it straightforward to layer multiple scoring signals on top of a single retrieval query, something that requires significant workarounds in Elasticsearch.

Phase 3: Semantic Embedding Search

The click profile approach works well for structured attributes. But “Home & Garden” is full of long-tail queries, and “vintage oak sideboard with glass doors” doesn’t map cleanly to a set of attribute tokens.

We introduced a German-language dense retrieval model. Ads are indexed with binary-quantized embeddings stored under an HNSW index using Hamming distance. Binary quantization keeps memory usage manageable and retrieval fast, with acceptable recall loss for our use case.

User search keywords are embedded at write time, each keyword in the user’s recent search history gets its own embedding vector, stored on the profile document. This was a deliberate schema decision: rather than embedding a single aggregated query vector, we preserve individual keyword vectors so that retrieval at query time reflects the full breadth of recent intent, not just its centroid.

Keyword vectors are produced inside the same profile update path as click events. The processor reads the incoming search event, runs inference, and writes the resulting vectors back to the user document. This keeps the embedding step inside the serving cluster and avoids an external inference call on the query path.

At query time, we extract those vectors and run a separate nearest-neighbor query per keyword, then merge the results with the same equal-weight mechanism used elsewhere.

Getting performance right here was non-trivial. We initially ran the keyword queries sequentially, as we had in earlier processors, but that choked the latency for ANN queries. Parallelizing them within the Searcher brought it back under our SLO. Production runs the embedder on GPU; staging runs a lighter quantized version on CPU to keep costs down.

What We Learned

WAND is powerful but requires careful normalization. The raw scores from a WAND query are dot products, they’re meaningful only relative to the query norm and the document norm. We normalize everything to a fixed L2 magnitude before combining sources, otherwise the 50/50 merge degenerates into whichever source has higher raw values winning.

Vespa’s document processors are underrated. The ability to intercept a document update, read other documents asynchronously, and write back, all inside the serving cluster. This eliminated an entire class of consistency problems we’d had with external profile stores. The AsyncValueUpdate state machine is verbose to write but reliable to run.

The RANK operator changes how you think about retrieval vs. scoring. In Elasticsearch, the query both retrieves and scores. In Vespa, you can separate these cleanly: WAND retrieves, additional RANK children score. This unlocked the labeled WAND pattern for fashion preferences.

Infrastructure took longer than expected. Enclave setup (AWS Private Link, token auth, separate AWS accounts for staging and production), update windows to avoid rolling restarts during business hours — these are solved problems once done, but they took real time.

Offline evaluation is the next frontier. We shipped fast and measured live. The next phase is a proper offline evaluation pipeline with a frozen corpus snapshot, temporal ground-truth qrels, and reconstructed user profiles at historical evaluation cutoffs. We want to measure before we ship.

Where We Are Now

The homepage feed is live on Vespa for multiple major categories. Motors use click-profile WAND. Fashion uses consolidated preference-injected WAND with labeled scoring. Home & Garden uses a parallel ANN with a WAND-in-rank fallback.

The Elasticsearch path is gone. The external profile store is gone. Profile learning, retrieval, and ranking happen in one place.

The journey was not smooth, but Vespa gave us primitives we didn’t have before, and the team learned to use them. If you’re considering a similar migration, the honest advice is: the concepts (WAND, tensors, document processors, rank operators) are powerful and well-documented, but adopting them properly requires time and experimentation. Plan for it.