Beyond the Bit: Why We Traded Hamming Distance for Modern Vector Retrieval

How we evolved our Vision API from hardware-level optimization to semantic-first architecture.

When we first built our Vision API, the constraints were clear: minimize storage, maximize speed, and keep the compute footprint light. In that era, binary embeddings and Hamming distance were the gold standard for efficiency. They allowed us to perform similarity searches at the bit-level, squeezing performance out of every byte.

But as the AI landscape shifted toward LLMs and multimodal models, our “efficient” architecture became a “semantic” bottleneck. We realized that while bits are cheap, losing the nuance of a high-dimensional vector is expensive.

Here is how — and why — we evolved our Vision API from a binary-first system to a modern retrieval stack.

1. The Legacy: When Hamming Was King

Our initial architecture treated image embeddings as binary feature vectors (typically 1024-bit strings). We leaned heavily into OpenSearch’s space_type: hammingbit, leveraging script_score queries to calculate similarity.

The logic was elegant in its simplicity:

• Storage: Tiny. 1024 bits is just 128 bytes.
• Speed: XOR operations and bit-counting (Popcount) are incredibly fast at the CPU level.
• Cost: We could pack millions of vectors into a relatively small memory footprint.

For basic visual similarity, it worked. But as our use cases grew more complex — requiring fine-grained category prediction and cross-modal search — the cracks began to show.

2. The Breaking Point: Accuracy vs. Efficiency

we eventually hit the wall where “fast” isn’t “good enough.” We encountered three primary points of failure:

Loss of Semantic Nuance

Binary embeddings are a “hard” compression. By binarizing a float, you lose the magnitude and directional gradients that define modern embeddings. You aren’t just compressing data; you’re flattening the relationship between concepts.

The RAG Gap

Our Vision API needed to talk to LLM-driven systems (RAG). LLMs expect dense float embeddings to perform reasoning. Our binary vectors were “speaking a different language,” making it impossible to align our vision data with textual semantic spaces like CLIP.

The Recall Plateau

We observed that while our system was fast, it had a “quality ceiling.” Similar images with slight lighting variations were often missed because the binarization process pushed them into different bit-buckets. We were optimizing for hardware at the expense of the user experience.

3. The Modern Stack: Semantic-First Retrieval

We moved away from bitwise math to a tiered retrieval architecture:

Float Embeddings → ANN (HNSW) → Quantization → Rerank

Step 1: Preserving the Signal (Float Embeddings)

We moved to dense float embeddings (768–1536 dimensions). By keeping the vectors in continuous space and using cosine similarity, we preserved the delicate semantic relationships that binary vectors discard.

Step 2: Scaling with HNSW

To handle the increased computational load of floats, we implemented Hierarchical Navigable Small Worlds (HNSW) in OpenSearch. This gave us sub-linear search speeds without the quality loss of binarization.

JSON

PUT /vision-index
{
  "settings": { "index.knn": true },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 1536,
        "method": {
          "name": "hnsw",
          "space_type": "cosinesimil",
          "engine": "nmslib"
        }
      }
    }
  }
}

Step 3: Intelligence via Quantization

We didn’t want to completely abandon storage efficiency. Instead of “hard” binarization, we adopted 8-bit Scalar Quantization.

• The Difference: Unlike binary (1-bit), 8-bit quantization reduces memory by 4x while preserving the vector’s geometry. It’s the “sweet spot” between the efficiency of our old system and the precision of our new one.

Step 4: The Precision Multiplier (Reranking)

The final piece of the puzzle was adding a reranking layer. Our ANN search retrieves the top 100 candidates, which are then passed to a cross-encoder. This ensures that the final Top-10 results aren’t just “mathematically close,” but “semantically relevant.”

4. The Results: Engineering Impact

The transition was more than just a code change; it was a shift in how we think about data density.

• Accuracy: We saw a significant jump in category prediction, especially in edge cases where binary vectors were too “blunt.”
• Future-Proofing: Our Vision API now integrates natively with multimodal LLMs.
• Stability: Ranking behavior is now more predictable and less sensitive to minor image noise.

On the other side, cost of RAM in our new system compared to Hamming Distance:

| Dimensions | Format | Bytes per Vector | RAM for 1M Vectors |

| 1024 | Binary | 128 B | 128 MB |

| 768 | Float32 | 3,072 B | ~3 GB |

| 1536 | Float32 | 6,144 B | ~6 GB |

When Do We Still Use Hamming?

We haven’t deleted the Hamming code from our library. It still has a place in:

• Near-duplicate detection: Finding exact copies of images.
• Perceptual hashing: Checking for copyright infringement.

Final Thoughts

Our original exploration of Hamming distance wasn’t a mistake — it was a necessary step in our evolution. It taught us where the limits of “hardware-optimized” search lie.

By moving to a Float-ANN-Rerank model, we’ve built a system that is as smart as it is fast.

Thanks for reading…