Real-Time Qdrant Search

Kafka Just Moves Bytes: Qdrant as the Whole Search Stack

The reference architecture for “search over a live stream” is five boxes. A message bus. An embedding service. A vector store. A reranker. And something with a cron in it to expire old data. Five boxes, five failure modes, five things to deploy.

I wanted to know how many of those Qdrant could absorb.

So I pointed a Hacker News firehose at it. Every story, comment, Ask HN, job, and poll that lands on HN arrives in the system within seconds, gets embedded twice, and becomes searchable immediately. Then it expires 24 hours later without anything resembling a cron job.

The final count:

Kafka moves bytes. FastAPI shuttles HTTP. Everything that looks like intelligence happens inside one Qdrant collection.

Ranking, fusion, filtering, deduplication, and retention. Not “Qdrant plus a reranking service.” Not “Qdrant plus Redis for the TTL window.” One collection, one client, four jobs.

This post is about those four jobs

The stack, minus the boxes I didn’t need

Everything crimson is Qdrant. That’s the whole argument for this architecture the grey boxes move bytes, and the red column does the thinking.

Kafka is a KRaft single-node container with exactly zero business logic in it it exists so the producer and the consumer can fail independently. Embeddings run in-process via fastembed, so there’s no embedding API key and no network hop for vectors. FastAPI owns three endpoints and a WebSocket.

Which leaves Qdrant holding the parts that actually matter.

Job 1: two vectors, one point, one round trip

The collection is hybrid from birth:

await client.create_collection(
    collection_name=settings.collection_name,
    vectors_config={
        "dense": qm.VectorParams(size=384, distance=qm.Distance.COSINE),
    },
    sparse_vectors_config={
        "sparse": qm.SparseVectorParams(modifier=qm.Modifier.IDF),
    },
)

BAAI/bge-small-en-v1.5 gives 384 dense dimensions for meaning. Qdrant/bm25 gives the sparse side for exact tokens — a username, a library name, a company nobody has heard of yet. Both live as named vectors on the same point, which is the design decision that removes an entire box from the diagram: there is no second index to keep in sync, no dual-write consistency problem, no “the lexical index is 40 seconds behind the vector index” incident at 3am.

modifier=IDF hands corpus statistics to the server. That matters more here than in a static corpus, because this corpus is a window — points are being inserted and deleted continuously, so document frequency is genuinely moving. The client cannot maintain an IDF table for a collection whose contents churn every few seconds. Qdrant recomputes rarity against the collection as it is at query time, which is the only correct answer.

Then the read path is one call:

res = await client.query_points(
    collection_name=settings.collection_name,
    prefetch=[
        qm.Prefetch(query=vector, using="dense",
                    filter=qm.Filter(must=must), limit=prefetch_limit),
        qm.Prefetch(query=sparse_vector, using="sparse",
                    filter=qm.Filter(must=must), limit=prefetch_limit),
    ],
    query=qm.FusionQuery(fusion=qm.Fusion.RRF),
    limit=prefetch_limit,
    with_payload=True,
)

Two retrieval arms, fused with Reciprocal Rank Fusion, inside the engine. One HTTP round trip from FastAPI. No reranking microservice, no manual score blending, no normalizing two incompatible score scales in Python and hoping.

Note where filter sits: inside each Prefetch, not on the outer query. This is easy to get wrong and the failure is silent. Filter on the outside and each arm happily retrieves 100 candidates, RRF fuses 200 candidates, and then you discard the ones that don’t match so your effective recall inside the filtered set collapses. Filter at the branch level and both arms spend their whole candidate budget on documents that can actually be returned.

The RRF plot twist

I’ll flag something, because I recently argued the opposite.

In a personal bookmark vault I built on the same engine, I ripped RRF out. It ranked badly, and I replaced it with a hand-rolled sparse-priority gate.

Here, native RRF works. Same fusion algorithm, same database, opposite conclusion.

The variable is corpus geometry. The bookmark vault is a few thousand links, all AI and engineering so homogeneous that dense cosine scores compressed into a narrow band and stopped discriminating between documents. Fusing a meaningless ranking with a meaningful one just dilutes the meaningful one

The HN firehose is the opposite: tens of thousands of points, wide open topically, everything from Rust compiler internals to somebody’s opinion about standing desks. Dense similarity is a real signal on that distribution, and RRF has two informative rankings to fuse. It behaves the way the paper says it does.

The lesson I’d extract: fusion strategy is a property of your corpus, not a default you inherit. Measure it on your own data before you pick

That said, I do keep one thin correction on top of RRF:

def _requires_lexical_match(query: str) -> bool:
    terms = _query_terms(query)
    return len(terms) == 1 and len(terms[0]) >= 4

If someone types a single substantial word — kafka, rust, zig — they are naming a thing, and a result that never mentions that thing is wrong no matter how good its cosine is. So single-term queries require a lexical hit; everything else rides pure RRF. Results also carry their match evidence back to the UI (exact_query_match, matched_terms, match_type: lexical | semantic), so the ranking is inspectable rather than a black-box float.

Roughly forty lines of Python guarding a decision Qdrant made in one call. That ratio is the point.

Job 2: writing to a collection while reading from it

Every Kafka batch 32 messages or 500 milliseconds, whichever comes first gets embedded and upserted straight into the live collection:

points.append(qm.PointStruct(
    id=str(uuid.uuid5(POINT_ID_NAMESPACE, f"{m.get('source')}:{hn_id}")),
    vector={"dense": v, "sparse": sv},
    payload={"text": ..., "ts": ..., "source": ..., "author": ...,
             "hn_url": ..., "parent_id": ...},
))
await client.upsert(collection_name=settings.collection_name, points=points)

No staging collection. No reindex-and-swap. No “search is degraded during ingest” banner. Qdrant serves consistent reads against a collection being written to several times a second, and that is the whole reason this project is only a few hundred lines.

The uuid5 is quiet but load-bearing. Point IDs are derived deterministically from source:hn_id, so upsert is the dedup layer. Kafka gives at-least-once delivery; replay the same offsets and you get one point, not two. The idempotency story that would normally need a seen_ids set in Redis is instead a property of how the ID is computed

Meanwhile “live search” turns out not to need a streaming query engine at all. The WebSocket handler waits on an asyncio event that the batch writer sets after each flush, re-runs the same hybrid query, diffs the returned ID set, and pushes only if it changed:

hits = await search(query, k, source)
ids = {h["id"] for h in hits}
if ids != last_ids:
    await ws.send_json({"query": query, "hits": hits})
    last_ids = ids

Real-time search here is just a fast query, fired again when there’s news. Qdrant is fast enough that this is indistinguishable from something far more sophisticated.

Job 3: Qdrant as the retention policy

This is my favourite part, and the one I see people bolt extra infrastructure onto most often.

The system keeps a sliding window: the last 24 hours, capped at 10,000 points. That’s two rules, and both are Qdrant calls.

Time-based eviction delete by filter, server-side, one request, no scan in application code:

cutoff = int(time.time()) - settings.window_seconds
await client.delete(
    collection_name=settings.collection_name,
    points_selector=qm.FilterSelector(
        filter=qm.Filter(must=[qm.FieldCondition(key="ts", range=qm.Range(lt=cutoff))])
    ),
)

Size-based trimming :scroll with order_by to get the oldest N, then delete them by ID:

scroll, _ = await client.scroll(
    collection_name=settings.collection_name,
    limit=over,
    with_payload=False,
    with_vectors=False,
    order_by=qm.OrderBy(key="ts", direction=qm.Direction.ASC),
)

The same ts index is read by the query path and written by the cleaner. One is about correctness, the other about disk.

An ordered scroll over an indexed payload field turns the collection into a size-capped ring buffer. Together, a 5-second background loop and those two calls replace what would otherwise be a TTL daemon, a cleanup cron, or a Redis sorted set shadowing the vector store.

And then there’s the part that makes it actually correct rather than merely tidy. The cleaner runs every 5 seconds, so between sweeps there is always a sliver of expired data sitting in the collection. So search filters the window too:

must = [qm.FieldCondition(key="ts", range=qm.Range(gte=cutoff))]

The cleaner is for storage. The filter is for correctness. The sweep reclaims space on a best-effort schedule; the query-time filter guarantees a stale point is never returned, sweep or no sweep. Same payload field, two different jobs. Getting this split right is the difference between a retention window and a retention suggestion.

Same for observability: /health is a count(exact=False) against the collection. Approximate count is cheap, and it’s the only “how much data is live right now” metric the app needs. No metrics store either.

Three things that bit me

**order_by on scroll needs a payload index.** Ordered scroll isn’t free the field has to be indexed. Both ts (integer) and source (keyword) get indexes at startup, wrapped in a try/except because index creation is idempotent-ish and raises if it already exists. Miss the ts index and size-trimming quietly fails while time-eviction keeps working, which is a confusing shape of bug.

Filters belong in the prefetch branches. Covered above, but worth repeating because nothing errors. You just get worse results than you should and no signal telling you why.

The schema guard is destructive on purpose. On startup, _has_hybrid_schema() checks that the existing collection really has a dense vector of the right dimension plus a sparse config. If not, it drops and recreates:

if not _has_hybrid_schema(info):
    log.warning("collection%s is not hybrid dense+sparse schema recreating", ...)
    await client.delete_collection(settings.collection_name)

Fine here — the data is a 24-hour window off a public firehose, and it refills in minutes. On a durable store this would be a loaded gun pointed at production. If you lift this pattern, make it a startup assertion that refuses to boot rather than one that deletes.

What Qdrant gave me for free

• Named vectors: dense + sparse on one point. No dual-index sync.
• **Fusion.RRF in query_points** :server-side hybrid ranking in one round trip. This is the reranking service I didn’t deploy.
• **modifier=IDF** :correct corpus statistics against a collection that churns continuously
• Live upserts :consistent reads while writing several times a second. No reindex window.
• **FilterSelector deletes** TTL as a database operation. This is the cron I didn’t write.
• **scroll + order_by** :a size-capped window without an external sorted set.
• **uuid5 point IDs + upsert** : at-least-once ingestion becomes exactly-once storage.
• Payload filters that compose :source and the time window ride the same hybrid query.
• **count(exact=False)** :the health endpoint, free.
• Cloud or self-hosted, one env var :QDRANT_URL is the only difference. Identical code, identical behaviour.

Next

• Multitenancy. source is already a keyword-indexed tenant discriminator. Turning that into proper isolation is the tiered multitenancy pattern, no new collections needed.
• Quantization on the window. 384-d is small, but the window is designed to grow. Scalar quantization is one config block, and I’ve measured what it costs
• More firehoses. The producer contract is four fields (text, source, ts, id). Nothing downstream knows it’s Hacker News. Point it at anything.

The repo is MIT: Realtime-search-kafka-Qdrant. A free Qdrant Cloud cluster, one Kafka container, three terminals, and you’re searching HN as it happens.

References

• Realtime-search-kafka-Qdrant : the project
• Qdrant hybrid queries: prefetch, Fusion.RRF, and where filters go
• Sparse vectors and BM25 :the IDF modifier
• Points and payload operations : filter-based delete, ordered scroll
• Inside a Search Engine: Qdrant Anatomy
• A Foundational Guide to Neural Search and Relevance Feedback