From Strings to BitSets: A Lesson in Data Representation

You may also like: — Platform Engineering Didn’t Fail — We Misunderstood the Problem — *A Year of Platform Engineering, GCP Migration, and Hard Lessons Learned*

Optimizing data formats often starts with good intentions — reduce payload size, speed up comparisons, and improve storage efficiency.

But not all optimizations are equal.

In this article, I’ll share a real-world scenario where we reduced JSON storage by ~75%, not by compressing strings, but by changing how the data was represented.

The Problem We Were Solving

We had technical data stored in JSON with a structure similar to this:

[
  { "condition": ["c1", "c2", "c3"], "value": 3 },
  { "condition": ["c3", "c4", "c5"], "value": 3 }
]

Key characteristics:

• Conditions are combinations
• Combinations vary
• Values can repeat
• Comparisons happen frequently
• Data is persisted and queried from a database

This wasn’t just storage — it was comparison-heavy data.

Initial Optimization Attempt: String-Based Replacement

Our first instinct was straightforward.

We tried to:

• Replace strings with shorter identifiers
• Map conditions to compact characters
• Reduce visual JSON size

Example idea:

["c1","c2","c3"] → "ABC"

What Improved

• JSON looked smaller
• Payloads appeared cleaner

What Didn’t

• Java comparisons still required string processing
• Equality checks remained expensive
• Database size barely improved in practice

This was an important realization:

Cosmetic compression does not change computational complexity.

We optimized how it looked, not how it behaved.

Rethinking the Problem: Representation Over Compression

The real question became:

What are we actually doing with this data?

The answer:

• Set membership checks
• Combination comparisons
• Equality and overlap detection

That’s when a BitSet-based approach became obvious.

The BitSet-Based Approach

Instead of storing conditions as lists or strings, we assigned each condition a fixed position.

Example:

c1 → bit 0
c2 → bit 1
c3 → bit 2
c4 → bit 3
c5 → bit 4

Now the same data becomes:

["c1","c2","c3"] → 00111
["c3","c4","c5"] → 11100

Represented internally as a BitSet.

Why BitSets Changed Everything

1️⃣ Storage Efficiency

• Compact binary representation
• Significant reduction in persisted size
• ~75% reduction in database footprint

2️⃣ Comparison Speed

• Equality checks are constant-time
• Overlaps become bitwise AND operations
• No string parsing, no regex, no iteration

3️⃣ Semantic Clarity

• Data now represents capabilities, not text
• Behavior aligns with intent

This was not just optimization — it was correct modeling.

Why the First Approach Failed

The string-based approach failed because:

• Strings are still strings
• Comparisons still scale with length
• Databases don’t magically optimize semantics

We optimized representation after serialization, instead of before it.

The Broader Lesson

This wasn’t a JSON problem.

It was a data-structure problem.

When systems grow, these questions matter more than tooling:

• What operations dominate?
• What comparisons are frequent?
• What representation best matches intent?

Compression hides inefficiency. Representation removes it.

Closing Thought

Optimizing systems isn’t about making data smaller.

It’s about making data behave better.

About the author Sriram is a platform and backend engineer with 20+ years of experience building large-scale systems, cloud platforms, and developer enablement tooling. He writes about real-world engineering lessons, architecture trade-offs, and the “why” behind systems.

If you found this helpful, feel free to follow for more writing on platform engineering and system design.