Why Static Vector Database Benchmarks Miss Production Reality

I have stopped trusting vector database benchmarks that only measure a fully loaded, fully optimized, read-only collection.

That setup is convenient for charts, but it does not look like production. Production has writes during reads. Production has filters. Production has cold data, warm caches, schema changes, index rebuilds, client concurrency, and operators asking how long the system is unusable while it “optimizes.”

The interesting thing here is that a benchmark can be technically correct and still operationally misleading. A database can report low query latency after ingest has finished, while hiding the fact that index optimization took hours. For an AI application with fresh documents arriving all day, that hidden time matters.

The benchmark shape is usually wrong

Most benchmarks I see follow a simple structure:

1. Load a static dataset.

2. Build or optimize the index.

3. Run nearest-neighbor queries.

4. Report latency, QPS, and sometimes recall.

That is useful as a microbenchmark. It is not enough for capacity planning.

In my experiments, the missing dimensions usually fall into three buckets.

First, the datasets are stale. SIFT and GloVe were useful for comparing algorithms, but many current systems store 768, 1024, 1536, or 3072 dimensional vectors produced by modern embedding models. Memory bandwidth, cache behavior, and index layout change materially when dimensions increase.

Second, the metrics overfocus on averages. Average latency hides tail behavior. A system that returns 95% of queries in 40ms and 5% in 2s does not feel like a 138ms system to the user who hits the slow path.

Third, the workload is too clean. Real applications rarely issue pure vector queries. They combine vector search with metadata filters, continuous ingestion, deletes, and ranking logic.

The metrics I actually want

For a vector store benchmark to be useful, I want at least this table:

The last one is the metric most benchmarks bury.

If Elasticsearch reports strong QPS after a long optimization phase, and Pinecone reports lower peak QPS but becomes queryable much earlier, the right answer depends on the application. A nightly analytics job may accept delayed optimization. A support chatbot indexing new tickets every minute may not.

Static tests are only the baseline

A static collection test still has value. It tells you what the system can do once ingestion and optimization are complete.

But I treat it as baseline data, not a decision. A static test should report:

• dataset size and vector dimension

• index type and parameters

• build time

• memory footprint

• p95 and p99 latency

• recall at top-k

• concurrency level

• client count and connection behavior

Here is the kind of small load harness I use for early testing. It is intentionally boring. I want repeatability before sophistication.

import time
from concurrent.futures import ThreadPoolExecutor

def run_query(client, collection, vector, top_k):
    start = time.perf_counter()
    result = client.search(
        collection_name=collection,
        data=[vector],
        limit=top_k,
        output_fields=["id"],
    )
    elapsed_ms = (time.perf_counter() - start) * 1000
    return elapsed_ms, result

def concurrent_latency(client, collection, queries, top_k=10, workers=32):
    with ThreadPoolExecutor(max_workers=workers) as pool:
        futures = [
            pool.submit(run_query, client, collection, q, top_k)
            for q in queries
        ]
    latencies = sorted(f.result()[0] for f in futures)
    p99 = latencies[int(len(latencies) * 0.99) - 1]
    p95 = latencies[int(len(latencies) * 0.95) - 1]
    return {"p95_ms": p95, "p99_ms": p99, "count": len(latencies)}

This does not replace a benchmark suite. It is a sanity check. If a system cannot behave predictably under this kind of basic concurrency, I do not trust the prettier numbers.

Filtering is where many results fall apart

Filtered vector search is the production path people under-test.

“Find similar documents” becomes “find similar documents from this customer, in this region, created after this date, excluding archived records.” The vector index and scalar filter now have to cooperate.

Filter selectivity changes everything. A filter that keeps 50% of rows is not the same as a filter that keeps 0.1% of rows. Depending on the execution strategy, the database may:

• search first, then filter

• filter first, then search

• use scalar indexes to prune candidates

• iterate the vector index until enough filtered results appear

The best plan depends on selectivity and top-k. A benchmark that tests only unfiltered search misses this entire query-planning problem.

For this reason, I like benchmarks that sweep selectivity levels: 50%, 10%, 1%, 0.1%. The source article I studied emphasizes this point with VDBBench, which explicitly tests filtering across selectivity levels and pairs QPS with recall. That is the right direction.

Streaming tests expose operational truth

The hardest case is search while inserting.

Production systems do not pause ingestion so the benchmark can look clean. New documents arrive while users query. Index segments grow. Compaction or optimization runs in the background. CPU, memory, and disk bandwidth are shared.

A meaningful streaming benchmark should define:

1. A fixed insert rate, such as 500 rows/sec.

2. A concurrent query workload.

3. Measurement checkpoints after each data increment.

4. Recall, QPS, p99 latency, and elapsed wall-clock time.

5. Optional post-ingest optimization time.

The elapsed time is not a detail. If one system reaches higher QPS only after a long optimization stage, that may be a good tradeoff for offline search and a bad tradeoff for live ingestion.

This is why I like the idea behind VDBBench: static, filtering, and streaming scenarios in the same framework. I do not care whether the tool makes one vendor look good. I care whether the methodology makes the tradeoffs visible.

What I would change in vendor benchmark pages

If I could rewrite every vector database benchmark page, I would require four disclosures.

First, disclose optimization time separately from query time. Do not hide it in setup.

Second, report recall beside every throughput number. A fast approximate nearest neighbor search with poor recall is just a lossy filter.

Third, show tail latency under concurrency. Median latency is not enough.

Fourth, include a mixed workload: reads, writes, and filters together.

That mixed workload does not have to match every production system. It just needs to prevent a benchmark from rewarding systems that only behave well in a static read-only state.

The decision process I use

When I evaluate a vector database, I now separate questions into three groups.

Serving:

• What is p99 latency at expected concurrency?

• What recall do I get at that latency?

• What happens when filters become selective?

Ingestion:

• How long until new data is searchable?

• Does indexing block queries?

• What is the write path bottleneck?

Operations:

• How much memory is required for the index?

• What background jobs compete with serving?

• How long does recovery take after node failure?

The final choice usually comes down to which tradeoff hurts least. Milvus, Qdrant, Pinecone, Elasticsearch, OpenSearch, and pgvector all have different operating envelopes. A benchmark should help identify that envelope, not pretend there is one universal winner.

Next, I want to run a streaming benchmark with filtered queries and deletes enabled at the same time. Deletes are often where clean benchmark assumptions meet storage-engine reality.