What I Learned Deconstructing “Liquid Clustering”

It started with two words: “liquid clustering.” They came up in a conversation about supporting automation through agentic AI on our own stack — building with the Mosaic AI Agent Framework. I nodded along, but honestly didn’t know what liquid clustering was, and that nagged at me. So this weekend I followed the thread into Databricks’ best practices for performance efficiency, and ended up learning the whole picture.

The way I actually learn something is to explain it back to myself like I’m talking to a friend . So that’s what I did and here’s what I worked out.

How you scale

There are two ways to give a system more power. Vertical scaling means making one machine bigger — more CPU, more memory. Horizontal means adding more machines and splitting the work across them. Vertical is simple, but eventually you run out of “bigger machine” to buy. Horizontal keeps going, as long as the work can be done in parallel.

What you’re really chasing is linear scalability: double the resources, roughly double the output. The catch — that only holds when the tasks are independent. The moment they have to talk to each other, coordination overhead starts eating the gains.

Let the platform run the machines

Serverless compute means you don’t set up or manage infrastructure at all. It starts fast, scales on its own, and is the default recommendation for SQL, jobs, and notebooks — though for notebooks it comes with an asterisk: depending on how your org is set up, governance rules or custom-library needs can get in the way.

Model serving works the same way for deployed models — scaling up and down with demand.

The bit that surprised me: a bigger cluster usually isn’t more expensive than a small one. You rent it for the duration of the job, so if it finishes in a quarter of the time, the bill comes out about the same. One honest caveat, though — that assumes the work actually parallelizes. If a job has serial bottlenecks (a big final write, or a heavy shuffle), doubling the cluster won’t halve the time: a cluster 4x the size might deliver only a 2x speedup, and then it does cost you more. So faster and cheaper isn’t a free lunch — but when the work splits cleanly, they really aren’t a trade-off.

Do less work, and do it in parallel

Most of the speed comes from two engines. Spark splits a query into pieces and runs them across every node in the cluster at once, instead of one machine grinding through it alone. Photon is a faster, built-in version of that engine — you essentially just switch it on, no code changes, and queries run quicker at lower cost.

The other half is writing queries that don’t fight the engine. Prefer built-in functions over custom Python ones: when you write your own Python function (a UDF), the data has to be handed out of the engine into Python and back, and that packing-and-unpacking — serialization — quietly adds up.

Lay the data out so it’s easy to find

The recurring theme here is read less data. The less the engine has to open and scan, the faster and cheaper everything is. A few ideas that made it click:

• Liquid clustering — the thing that started all this — is about how rows are physically grouped in storage. If you tell the table to cluster by, say, customer ID, rows for the same customer get stored near each other, so a query filtering on that doesn’t have to look everywhere. It’s the modern replacement for partitioning: partitioning locks you into a rigid folder structure that’s painful to change later, while clustering keys can be adjusted as your queries evolve. (Partitioning a table under 1TB is usually a mistake for exactly that reason.)
• Data skipping is the engine being lazy in a good way. Databricks keeps simple min/max stats for each file — so if you ask for orders over $500 and a file’s max value is $200, it skips that file entirely without opening it.
• OPTIMIZE fixes the “small files” problem. Tables often end up as thousands of tiny files, and opening each one has overhead; OPTIMIZE merges them into fewer, bigger files so reads are smoother.
• Predictive optimization just does that housekeeping for you. On managed tables, Databricks notices when maintenance like OPTIMIZE would help and runs it automatically — so the table stays fast without anyone scheduling it by hand.
• On caching: disk cache copies remote data onto the cluster’s local SSD, so repeat reads are much faster — worth turning on. Manual Spark caching (pinning data in memory yourself) is the opposite: easy to misuse and burn memory, so the advice is to mostly avoid it. And leave adaptive query execution on — it watches the real data as the query runs and re-optimizes the plan on the fly, instead of committing to a guess made before it saw anything.

Trust numbers, not vibes

Performance testing only counts on production-like data — same volume, same file layout, same skew (a few values that dominate the rest). Test on a tidy little sample and the results just lie to you. And when something’s slow, you don’t guess: the system tables and the query profile show you exactly where the time, rows, and memory actually went.

If you work on performance — what would you add, or what did I miss?

Source: Best practises for performance efficiency

Previously: The harness your model needs