Modern Columnar Storage: Exploring Parquet and Light Weight Compression Techniques

Storage and Compute Seperation

The most recent development in data platforms is the separation of computation and storage. Storage and computation can be used, scaled, and priced separately if they are kept apart. As a result, companies can only pay for what they actually utilise.

Many organisations rely on public clouds like AWS, GCP they use ObjectStrore like S3, GCS as core storage to store their data which provides ability to scale storage independently and also provide three main advantages:

1. Replicated data across multi-region (DR)
2. Versioning
3. Durability

Columnar Storage

Columnar storage formats have been the foundation for modern data analytics systems. Parquet and ORC have dominated this ecosystem for more than a decade. Both these file formats have proven to be successful in DataWarehouses with modernisation in the lake-house architecture they lack in certain capabilities.

For saving large datasets, parquet stores data in the form of RowGroups and each rowgroup is composed of columns organised as pages. Each page holds statistics about each column value like min, max, sum, types of compression etc.

Inefficiencies with existing columnar storages:

General purpose compression — Both parquet and orc use general purpose compressions like zstd, snappy etc to compress. This requires additional cost during decompression resulting in more IO operations.

Column Encoding: While Parquet has pre-defined encodings(RLE, Bit Packing, Delta Encoding) for each column types this can result choosing in-efficient encoding scheme for that specific column type.

Random Access: Updates to file at random offsets requires us to load the entire page of 128MB, this requires us to load page from decompress them and then update and compress it again. Or even for small subset reads this turns to have multiple IO operations.

Next!

The growing interest in artificial intelligence has created workloads that require both sequential and random access. This allows us to use modern hardware to perform more than traditional CPU bound hardwares.

With the availability NVMe-backed storage solutions, now that we have data in a columnar format, we have a new world of optimization to explore, starting with vectorization and SIMD (Single Instruction Multiple Data). Current CPUs & GPUs supporting SIMD instruction sets that can process multiple data points with a single instruction, both working faster and giving much better memory and cache locality.

To address these inefficiencies, there are numerous new file formats originating so that these new formats can leverage modern hardware to provide maximum processing.

Most of these new file formats rely on Light weight compression techniques like RLE, BitPacking, Frame of Reference, Dict Encoding etc and use specific algorithm to identify the right encoding scheme to encode data. It does by choosing sample data and passing them through multiple encoding schemes and in few cases they use combination of encoding schemes.

Some of the new file formats evolving are F3(Future proof File Format) BTRBlocks, Nimble, Lance, TsFile etc

Here are the references:

F3 — https://db.cs.cmu.edu/papers/2025/zeng-sigmod2025.pdf

BTRBlocks — https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/papers/btrblocks.pdf