I Read About DeepSeek’s File System and Couldn’t Stop Thinking About Databricks

A data engineer’s take on what 3FS taught me about the tools I use every day.

Something clicked for me this past weekend.

I was reading about DeepSeek’s newly open-sourced file system called 3FS (Fire-Flyer File System) and about ten minutes in, I had this strange feeling of familiarity. Not because I’d seen 3FS before. But because the problem it was solving was one data engineers have been living with foryears, just from a different angle.

I’m a data engineer. I work with Databricks. And what DeepSeek built for AI researchers is, at its heart, the same magic trick Databricks performs for people like me.

Let me explain.

The Problem Both Are Solving

Imagine you have data ,a lot of it. Not “a lot” like a big Excel file. A lot like hundreds of terabytes, maybe petabytes, spread across dozens or hundreds of machines in a data center somewhere.

Now imagine you need to work with that data as if it’s just… sitting on your laptop. No thinking about which server it’s on. No worrying about what happens if one of those servers crashes. No manually stitching together pieces from different machines.

That’s the problem. And it’s a genuinely hard one.

3FS solves it for AI teams training massive models. When a researcher needs to feed data into a training run, they just point to a file path. The fact that the data is actually fragmented across hundreds of machines completely invisible to them.

Databricks solves it for data teams running pipelines and analytics. When I query a Delta table, I’m not thinking about which storage bucket a particular chunk of data lives in, or how many copies exist for redundancy. I just write my query and it works.

Same problem. Different audiences. Different design choices underneath.

Why This Realization Matters

Here’s the thing about working with platforms like Databricks: they’re so good at hiding complexity that you stop thinking about what’s actually happening.

That’s by design and it’s genuinely impressive engineering. But it also means we can start treating our tools like black boxes. We know the inputs and outputs. We forget there’s an entire world of decisions, tradeoffs, and engineering effort happening in between.

Reading about how DeepSeek built 3FS from scratch what problems they ran into, what tradeoffs they made cracked that black box open a little for me.

For example: one of the core challenges in any distributed system is fault tolerance. What happens when a machine goes down mid-write? How do you make sure your data isn’t corrupted or lost?

DeepSeek’s answer in 3FS is a replication approach where data is copied across multiple nodes in a chain, and writes aren’t considered complete until every node in that chain confirms it. It’s a deliberate bet on consistency over speed.

Databricks makes a similar bet through Delta Lake’s transaction log every write is recorded, every change is tracked, so even if something goes wrong mid-operation, your data stays clean and queryable.

Different implementations. Same underlying philosophy: don’t lose people’s data, ever.

Where They Diverge

The most interesting part of comparing the two isn’t where they overlap — it’s where they deliberately go in different directions.

3FS is built for AI training workloads. That means massive, fast, sequential reads. A model training run might need to stream through terabytes of data continuously, as fast as the hardware allows. 3FS is engineered around that optimized for the kind of throughput that keeps GPUs fed and not sitting idle.

Databricks is built for analytical workloads. That means a different kind of access pattern lots of queries hitting different slices of data, filters, aggregations, joins. Delta Lake’s columnar storage format is a direct response to that. You’re not reading everything sequentially; you’re reading smart.

This is the part that I think gets underappreciated in data conversations: storage systems aren’t generic. The choices made at the infrastructure level reflect deep assumptions about how data will be used. When those assumptions match your workload, things fly. When they don’t, you hit walls.

What I Took Away as a Data Engineer

Reading deeply about a system I’ll never directly use taught me more about the systems I use every day than a lot of documentation ever has.

A few things that stuck with me:

The abstraction is the product. The reason Databricks is valuable isn’t just that it stores data. It’s that it makes distributed storage feel local and simple. That abstraction is the hard part. 3FS is doing the same thing in a different domain.

Tradeoffs are everywhere. Every design decision in a distributed system trades something off against something else — speed vs. consistency, simplicity vs. flexibility, write performance vs. read performance. Understanding those tradeoffs helps you use your own tools more intentionally.

Curiosity compounds. The more you understand about how things work under the hood, the better your instincts get when things go wrong or when you’re trying to squeeze more performance out of a pipeline.

Worth Reading

If you’re in data and curious about what’s under the hood of systems like these, Henry Zhu’s blog series on 3FS is a great starting point. It’s technical but accessible, and it goes deep in a way that’s genuinely rare.

👉 An Intro to DeepSeek’s Distributed File System

You don’t need to be building distributed systems to get something out of it. Sometimes the best way to understand the tools you use is to look at how someone else solved the same problem from scratch, for a completely different purpose.