Stop Storing Duplicates: How Deduplication Engines Cut Cloud Costs by up to 95%

Photo by Adrian Siaril on Unsplash

In storage systems, a dedupe engine:

Detects identical data chunks and stores only one physical copy, while maintaining logical references for all duplicates.

If 1 TB of data contains 400 GB repeated blocks → you only store unique blocks.

Advantages :

• Storage Reduction — 50–95% savings (depending on workload)
• Network Savings — Client-side dedupe avoids re-uploading existing data
• Faster Backups — Only new unique blocks are transmitted
• Lower Cloud Cost — Less object storage used (S3/Blob/etc.)

A dedupe system accepts:

• File data stream
• VM disk snapshot blocks
• Database dump stream
• Raw byte stream

Example:

Backup agent reads file → sends stream to dedupe engine

Architecture

                ┌────────────────────┐
                │   Backup Agent     │
                └─────────┬──────────┘
                          │ (stream)
                          ▼
                ┌────────────────────┐
                │  Ingestion API     │  (Go HTTP/gRPC)
                └─────────┬──────────┘
                          ▼
                ┌────────────────────┐
                │ Chunking Pipeline  │
                └─────────┬──────────┘
                          ▼
                ┌────────────────────┐
                │ Hashing Workers    │
                └─────────┬──────────┘
                          ▼
                ┌────────────────────┐
                │ Dedupe Service     │
                │ (Index Lookup)     │
                └──────┬─────────────┘
                       │
        ┌──────────────┴──────────────┐
        ▼                             ▼
┌───────────────┐             ┌────────────────┐
│ Metadata Store│             │ Object Storage │
│ (RocksDB etc) │             │ (S3/Blob)      │
└───────────────┘             └────────────────┘

Working

Step 1: File Read

Backup agent reads file in stream.

Step 2: Chunking

The engine splits incoming data into chunks. File is split:

Chunk A
Chunk B
Chunk C
Chunk D

Two strategies:

1. Fixed-size chunking :

Simple but weak when data shifts.

Example: 4KB / 8KB / 1MB blocks

2. Variable-size chunking (Content Defined Chunking) :

Uses rolling hash (e.g., Rabin fingerprint). Much better dedupe ratio. Most modern backup systems use variable chunking.

Step 3: Hashing

For every chunk:

hash = SHA-256(chunk)

This becomes the chunk ID. If two chunks have identical content → identical hash.

Hash(A) → h1
Hash(B) → h2
Hash(C) → h3
Hash(D) → h2   ← duplicate of B

Fingerprint Index (Core of Dedupe Engine)

This is the most critical component. It answers — Have we already stored this chunk? Fingerprint Index Options :

A. In-Memory + Persistent Log

• RAM for fast lookup
• WAL for durability

B. LSM-tree (RocksDB-like)

• Good for massive scale
• Write-heavy workloads

C. Distributed KV Store

• Needed at scale
• Partitioned by hash prefix

Example:

hash[0:2] → shard routing

Step 4: Index Lookup

Check index:

• h1 → not found → store
• h2 → not found → store
• h3 → not found → store
• h2 → found → skip storing

Step 5: Store Metadata

Instead of storing data again, Metadata stores logical file layout:

File1:
[h1, h2, h3, h2]

Only unique blocks are stored physically once.

Client-Side vs Server-Side Dedupe

1. Server-Side Dedupe :

Client uploads full data. Server dedupes. Simple but wastes network bandwidth.

2. Client-Side Dedupe (More Advanced)

Client:

• Chunks
• Hashes
• Queries server: “Do you have this hash?”

Server:

• Returns missing hashes

Client:

• Uploads only missing chunks

This dramatically reduces bandwidth.

Global vs Local Dedupe

Local Dedupe — Only within one backup job.

Global Dedupe — Across:

• All users
• All VMs
• All organizations

This is extremely powerful in SaaS backup.

Garbage Collection

When backups expire:

• Reference count of chunks decreases
• If ref_count == 0 → delete chunk

This is a critical background job.

Performance Optimizations

• Bloom Filters — Fast reject before expensive lookup.
• SSD tier for hot index — RAM + NVMe tiering.
• Batch index lookups — Reduce IOPS.
• Prefetch on restore — Sequential reconstruction.

Photo by Bernard Hermant on Unsplash