🧱How Columnar Storage Actually Works

Parquet, ORC, Arrow — Internal Anatomy Explained (No Hand-Waving)

Figure 1 — Columnar analytics across layers: Parquet and ORC optimize disk reads, query engines orchestrate execution, and Apache Arrow powers fast in-memory columnar computation.

How to read this diagram: Parquet and ORC store data on disk. The query engine (Spark / Trino / Presto) plans the query and reads only what’s needed. During execution, data is loaded into memory and processed using Apache Arrow’s columnar layout. BI, ML, and APIs consume the results of this execution — not Arrow directly.

TL;DR

Row storage is built for transactions. Columnar storage is built for analytics. Parquet and ORC optimize disk + cloud analytics. Arrow optimizes in-memory analytics + zero-copy execution. If you don’t understand how columns are stored, compressed, and read — you will never truly optimize Spark, Trino, or warehouses.

Introduction — Why You’re Still Slow Even “Using Parquet”

Everyone says:

“We use Parquet, so performance should be good.”

That’s lazy thinking.

Most slow analytics pipelines aren’t slow because of compute — they’re slow because:

• you’re reading too much data
• you’re decompressing useless columns
• you’re spilling memory unnecessarily
• you don’t understand file internals

Columnar storage is not magic. It’s a set of trade-offs.

Today, we’re going inside the file.

🧠 1. Row Storage vs Column Storage (The Fundamental Shift)

Row-Based Storage (OLTP mindset)

Row 1 → [id, name, country, revenue]
Row 2 → [id, name, country, revenue]
Row 3 → [id, name, country, revenue]

Good for:

• inserts
• point lookups
• transactions

Terrible for analytics Why?

• Every query reads full rows
• Even if you need just one column

Columnar Storage (OLAP mindset)

Column id      → [1,2,3,4,5]
Column country → [IN,US,IN,IN,US]
Column revenue → [100,200,50,400,80]

Why this wins:

• Read only what you query
• Better compression
• CPU cache friendly
• Perfect for scans & aggregations

📦 2. Parquet — Columnar Storage on Disk (Analytics First)

Parquet is not just “columns in files”. It’s a hierarchical binary format.

🧩 Parquet Internal Layout

Parquet File
 ├── Row Groups
 │    ├── Column Chunks
 │    │     ├── Pages
 │    │     │     ├── Data Page
 │    │     │     └── Dictionary Page
 └── Footer (Schema + Statistics)

Key ideas that actually matter:

• Row Groups = unit of parallelism
• Column Chunks = per-column storage
• Pages = compression + encoding unit
• Footer = query optimizer’s best friend

⚠️ Dictionary Encoding Isn’t Free

Dictionary encoding is extremely effective for low-cardinality columns (country, status, enums).

But for high-cardinality strings (UUIDs, URLs, request IDs): • dictionaries grow large • decoding cost increases • memory pressure spikes • late materialization can backfire

Columnar formats amplify good schemas — and punish high-entropy ones.

📏 Parquet Row Group Sizing (This Matters More Than You Think)

Rule of thumb:

• 128MB–256MB row groups for analytics
• Smaller = more metadata overhead
• Larger = worse parallelism

Why this matters:

• Row group = unit of skipping + parallelism
• Bad sizing = wasted IO + CPU

If your Parquet files are “slow”, check row group size before blaming Spark.

🔍 Why Parquet is fast

1. Column pruning

• Query only reads required columns

2. Predicate pushdown

• Skip row groups using min/max stats

3. Compression

• Same-type data compresses extremely well

4. Late materialization

• Decode only columns that survive filters

🧠 When Parquet shines

• Spark / Trino / Presto
• Cloud data lakes
• Batch analytics
• BI workloads

⚠️ When Parquet hurts

• Small files
• Frequent updates
• Low-latency writes

Parquet is append-friendly, not mutation-friendly.

🧊 3. ORC — Columnar Storage with Smarter Indexing

ORC is similar to Parquet — but more aggressive.

🧩 ORC Internal Layout

ORC File
 ├── Stripes
 │    ├── Index Data
 │    ├── Row Data
 │    └── Stripe Footer
 └── File Footer

📏 ORC Stripe Sizing (Aggressive by Design)

• Stripes are bigger than Parquet row groups
• Designed for long scans
• Excellent for Hive-style workloads

Trade-off: Better compression, slower writes.

Why ORC exists

ORC was designed for:

• Hive
• heavy aggregations
• extreme compression
• smarter skipping

🔥 ORC Advantages

• Built-in indexes per stripe
• Better compression ratios
• Faster scans for wide tables
• More aggressive predicate pruning

⚠️ ORC Trade-offs

• Heavier metadata
• Less flexible than Parquet
• Slower write paths in some engines

Parquet vs ORC

Table 1 - Parquet and ORC solve the same problem but make different trade-offs — Parquet favors ecosystem flexibility and write performance, while ORC pushes harder on compression and predicate pruning.

In practice: Parquet wins because of ecosystem dominance.

🧪 Compression Reality Check (Don’t Fool Yourself)

Compression helps only when data is similar.

Table 2- Compression effectiveness depends more on data shape than format — columnar storage amplifies good schemas and exposes bad ones.

Compression amplifies good schemas and punishes bad ones.

If your data is garbage, Parquet won’t save you.

⚡ 4. Apache Arrow — Columnar Data in Memory

Arrow is not a file format.

Let that sink in.

Arrow is a language-agnostic, in-memory columnar layout.

🧩 Arrow Memory Layout

Arrow Array
 ├── Validity Bitmap
 ├── Offsets Buffer
 └── Values Buffer

This allows:

• zero-copy reads
• vectorized execution
• cross-language data sharing

🔥 Why Arrow matters

• Pandas ↔ Spark ↔ Python ↔ JVM without serialization
• GPU acceleration
• ML pipelines
• Interactive analytics

Arrow eliminates:

• JSON
• Pickle
• JVM ↔ Python conversion hell

🧠 Where Arrow is used

• Spark (Arrow optimization)
• Pandas
• DuckDB
• Ray
• ML pipelines

Arrow is compute-speed, not storage optimization.

Important clarification: Arrow is one of the dominant in-memory columnar formats, but not all engines execute directly on Arrow buffers. Some engines (like Trino) use their own internal columnar representations — the invariant is columnar, vectorized memory execution, not Arrow itself.

🧠 What Actually Happens When You Run a Query (Step-by-Step)

SELECT country, SUM(revenue)
FROM sales
WHERE date >= '2024-01-01'
GROUP BY country;

What the engine actually does:

1️⃣ Metadata scan

• Reads Parquet/ORC footers
• Identifies row groups/stripes to skip

2️⃣ Column pruning

• Only loads country, revenue, date
• Ignores every other column

3️⃣ Predicate pushdown

• Uses min/max stats to skip files
• Avoids scanning cold data

4️⃣ Disk → Memory transfer

• Compressed column chunks loaded
• Decompressed column-by-column

5️⃣ Arrow in-memory execution

• Columns converted to Arrow buffers
• Vectorized aggregation happens

6️⃣ Result materialization

• Aggregated result returned to engine
• Minimal memory footprint

👉 Performance comes from skipping work, not doing work faster.

🧠 5. Disk vs Memory — Stop Comparing the Wrong Things

Table 3 - Parquet and ORC optimize how data is stored and read from disk, while Arrow optimizes how columnar data is processed in memory during query execution.

If you compare Arrow to Parquet, you already missed the point.

🏗️ 6. Real-World Architecture (How This Actually Fits Together)

Columnar storage is not a single system — it’s a pipeline of decisions from disk to CPU.

Here’s how a real analytical query actually flows in production:

Raw Data
   ↓
Parquet / ORC (Lakehouse Storage)
   ↓
Query Engine (Spark / Trino)
   ↓
Arrow In-Memory Execution
   ↓
BI / ML / APIs

• Parquet / ORC decide what data can be skipped
• Query engines decide how execution is planned
• Arrow decides how fast computation happens in memory

Disk formats optimize IO. Arrow optimizes CPU & memory. You need both for real performance.

🧩 What Happens Inside a Single Query (No Magic, Just Mechanics)

This is the part most engineers never see — and where performance is actually won or lost.

Query
 ↓
Metadata (Footer)
 ↓
Column Pruning
 ↓
Row Group / Stripe Skipping
 ↓
Arrow In-Memory Execution
 ↓
Result

Step-by-step, what this means:

1️⃣ Query hits metadata first The engine reads Parquet/ORC footers — not the data. This tells it:

• schema
• column stats
• min/max values

2️⃣ Column pruning kicks in Only the columns referenced in the query are read. Everything else is ignored — zero IO cost.

3️⃣ Row group / stripe skipping Entire chunks of data are skipped using statistics. This is why predicate pushdown matters more than CPU.

4️⃣ Data is loaded into memory Only surviving column chunks are decompressed.

5️⃣ Arrow takes over Data is laid out in contiguous columnar buffers. Vectorized execution runs without serialization overhead.

6️⃣ Result is produced Minimal memory, minimal CPU, minimal latency.

👉 Most performance gains come from skipping work, not speeding it up.

🧠 The Mental Model to Lock In

Parquet / ORC = data access optimization

Arrow = execution optimization

Query engines = orchestration

If you tune only Spark configs but ignore file layout, you’re tuning the wrong layer.

🧨 7. Common Mistakes Engineers Make (Learn These Now)

❌ Too many small Parquet files ❌ Wrong row group size ❌ No partition pruning ❌ Assuming compression fixes bad schema ❌ Treating Arrow like a storage format

Columnar storage amplifies good design — and punishes bad design.

🚨 8. Why Columnar Storage Is Bad for Streaming Writes

Columnar formats are read-optimized, not write-optimized.

Streaming problems:

• Small files explosion
• Frequent metadata updates
• Poor compaction behavior

Correct pattern:

Stream → Buffer → Micro-batch → Parquet/ORC

In practice, this looks like: • Kafka → Flink micro-batches → Parquet files • Spark Structured Streaming → checkpointed file sinks • Iceberg / Delta → periodic compaction jobs

Columnar formats want fewer, larger, well-structured writes — not event-by-event chaos. If you write Parquet per event, you’re doing it wrong.

🎯 9. When to Use What (Rules of Thumb)

• Lakehouse analytics → Parquet
• Hive-heavy workloads → ORC
• Python / ML / cross-language → Arrow
• Streaming ingestion → Write Parquet later, not first

🧠 Mental Model to Remember

• Parquet / ORC decide what gets read
• Arrow decides how fast computation happens
• Engines decide execution strategy

If you confuse these layers, performance tuning becomes guesswork

Conclusion — Columnar Storage Is a Weapon

Columnar storage is not a checkbox. It’s a design philosophy.

If you understand:

• how columns are laid out
• how metadata skips data
• how memory is accessed

If you don’t understand file layout, metadata skipping, and in-memory execution, you’re not tuning systems — you’re guessing and hoping.

Columnar storage rewards engineers who understand the full path from disk to CPU — and exposes everyone else.

That’s the difference between a data engineer and a pipeline babysitter.

🔥 Next in Series

⏱️ Time-Series Databases for Data Engineers

InfluxDB, TimescaleDB, ClickHouse — when OLAP meets time

Next, we’ll cover:

• why time-series data breaks traditional warehouses
• ingestion vs query trade-offs
• retention, rollups, and downsampling
• real-world use cases (metrics, IoT, logs, finance)

This is where real-time analytics gets serious.

🔔 Follow for More

Follow **TheDataForge** for high-signal deep dives on Spark, Kafka, Streaming, Lakehouse, and real-world data engineering — no fluff, only engineering truth.

⚡️Let’s connect — I post daily Big Data & Spark insights at DataForgeX on X.

💬 Comment Below:What confused you most about Parquet or Arrow before reading this — compression, memory, or execution?

🏷️ Tags

DataEngineering #ColumnarStorage #Parquet #ORC #ApacheArrow #Lakehouse #BigData #Analytics #Spark #DataArchitecture