Parquet From Scratch: The Problem First, Then the Solution

Imagine you run a website with lots of users. You have been saving user data the obvious way, a giant CSV file. It has:

• 200 Columns per user: Name, age, city, signup date, last login, total purchases, favorite category, device, country, … 200 fields.
• 500 million rows.
• On disk, this file is about 1 terabyte of text.

One morning, an analyst asks a dead simple question

“What the average age of our users?”

That’s one column out of 200. The actual age numbers, if you could grab just them, might be 5 GB of data. So this should be quick, right? it isn’t. It’s painfully slow and expensive.

Why? because of how the data is stored. Let’s list exactly what goes wrong, because each pain point is a reason Parquet exists.

Problem 1: To read a little, you are forced to read everything:

Remember, a CSV stores data row by row:

Alice,30,NYC,2021–04–01,…(196 more fields)… Bob,25,LA,2020–11–13,…(196 more fields)… Carol,41,SF,2019–01–07,…(196 more fields)…

The age values are 30,25,41 are scattered, one buried inside each row, sandwiched between 199 fields you do not care about. There is no way to grab just the ages without dragging every other field along with them.

So the answer “average age,” the computer must:

1. Read the entire 1 TB of the disk.
2. Parse every comma in every row.
3. Pick out the age from each row.
4. Throw away 99% what it just read and only keep only the ages.

You read a terabyte to answer a 5 GB question. This is the core wound, and it has a name once you see it: The data is scattered, so selective reading is impossible.

Problem 2: The file is bloated — it wastes space

CSV is a plain text, and text is wasteful way to store data:

1. The number 1000000 takes 7 bytes as text (1,0,0,0,0,0,0), but only 4 bytes as an actual integer.
2. Dates, booleans, everything stored as a long human readable strings.

More bytes on disk means more to store and more to read. (Problem — 1 gets worse).

Problem 3: It barely compresses:

Just zip the CSV! — helps a little, but not much. Compression works by finding repetition, and a row is a jumble of unrelated things: a name, then an age, then a city, then a timestamp.

Mixed types sitting next to each other give a compressor very little repeating pattern to exploit. The redundancy is there in your data, but the row layout hides it, because similar values are never next to each other.

Problem 4: The file does not know what it contains (no schema, no types):

A csv is just characters. Nothing in the file says “age is a whole number” or signup_date is a date”. So every program that opens it has to guess and parse:

• Is 2024 the number two-thousand-twenty-four, or the text “2024”?
• Is 01 the number 1, or a zip code or product code where the leading zero matters?
• Is an empty field a zero, an empty string, or missing data?

Every reader re-guesses, re-parses, and can get it wrong. The structure does not travel with the file, you need outside knowledge (Like a separate document related to structure) to interpret it correctly.

Problem 5: You cannot skip anything:

Now ask a filtered question: “How many users are older than 90?”

Almost nobody is over 90, the answer touches a tiny silver of your data. But the CSV has no summaries, no “table of contents”. So the computer still reads all 500 million rows and checks each one, just to discover the 499.99 million of them do not qualify. Every question, no matter how selective, costs a full scan.

Problem 6: In the cloud, all of this is literallyt money:

Modern data lives in cloud storage and cloud query tools often charge you per byte scanned. Problem 1 to 5 means you scan the whole file for every question. So a one column question does not just waste minutes, it wastes real dollars, over and over, every time anyone runs a query.

Every problem traces to one root cause:

1. Read everything to get one column, because data stored by row (columns are scattered).
2. File is bloated, because plain text, and no smart encoding.
3. Compresses poorly, because mixed types sit next to each other.
4. No types/schema, because nothing describes the file.
5. cannot skip the data, because no summaries or statistics.
6. Slow and costy, because all of the above, at scale.

The enemy is the row-by-row layout itself. What if we stored the data grouped by column instead of by row?

Flip the layout, and the whole list of problems dissolves at once:

• All the ages sit together -> read just that block -> Problem 1 is gone.
• A block of all similar values -> encode compactly (numbers as numbers “US x 1,000,000”)-> problems 2 and 3 are gone.
• While we are rebuilding the format, bake the schema into the file -> Problem 4 is gone.
• Store a min/max summary per block so we can skip blocks that cannot match -> problem 5 is gone.
• Fewer bytes read + fewer bytes scanned -> Problem 6 is gone.

That is exactly what parquet is: It is not random collection of clever tricks, it is the direct, point by point answer to this list of problems. This is the real history, too: engineers at twitter and cloudera around 2012 were drowning in precisely this pain, and parquet was their fix.

The published payoff is dramatic in one databricks benchmark, converting 1 TB to CSV to parquet shrank storage from 1 TB to 130 GB and cut a query from 236 seconds to under 7 seconds while scanning 99% less data.