Three Lessons Learned Building Data Platforms From Scratch

The case for boring tools, smaller ambitions, and building for the team you have

Image generated using AI

Why Greenfield Projects Are Uniquely Hard

Building a data platform from scratch is a dangerous endeavor. Not because it’s technically hard (though not easy either), but because it gives you the freedom to make every mistake yourself.

Over the years I’ve seen the same patterns repeat:

• Teams that built for elegance before they even understood their consumers.
• Architectures designed around one brilliant engineer rather than the team as a whole.
• Platforms assembled from best-in-class tools that turned out to be nearly impossible to operate together.

Here’s what I’d do differently.

Start With the Data Consumers, Not the Architecture

When any engineer starts greenfield project, their instinct is to immediately start designing. I’m guilty as charged, too. The problem is that you can build something extremely well that nobody needs.

The most humbling version of this is that your end users may just want to export data to Excel. It already fits their workflow, they know how to use it, and it solves their problem. Your platform, however elegant, does not. That’s not their fault.

Before you write a single line of infrastructure code, understand exactly how your consumers work. What decisions does the data need to support? Who is making them? Where does data fit into their current workflow? The answers might be uncomfortable, but they’re real.

Build for that reality, not the one you assumed.

Pick Boring Tools for the Foundation

The data tooling landscape is exciting, and evaluating new tools feels like progress. Teams build comparison matrices and land on the best tool for each individual job. Each decision, made in isolation, is defensible. Collectively, they produce an architecture that’s nearly impossible to maintain, let alone extend.

That’s the integration tax.

Every tool you add multiplies the complexity of everything around it, such as authentication, monitoring, but also onboarding. In a startup that’s painful, but you’ll manage. In an enterprise, however, this becomes crippling: security reviews, procurement overhead, and institutional knowledge that walks out the door with every engineer that leaves.

This best of breed is contrasted to best of suite.

A single platform doing many things at 80% quality almost always beats ten “best” tools each doing one thing at 100%.

Snowflake is a good example. Its Spark environment isn’t the best. Its data quality features aren’t as powerful as Soda. Its orchestration isn’t Dagster. And yet, consolidating around Snowflake in an enterprise context might be exactly the right call. Not because it wins every battle, but because the cost of stitching together the tools that do is enormous and compounds over time.

Here’s a simple rule: if removing a tool would require rewriting a significant portion of your platform, it deserves some serious looking into. If you could swap it out in a sprint, the no harm done.

Boring foundational infrastructure is a feature, not a limitation.

Design for the Team You Have, Not the Team You Imagine

The architect who designs a system understands every trade-off that went into it. The rest of the team inherits the outcome, often without that context. The knowledge gap that this creates is often much wider than the designer realizes.

The most common symptom of this is tooling choices that outmatch the team’s familiarity with them. A senior engineer proposes something they know well, makes a compelling case, implements it, and inadvertently becomes the only person who can maintain it.

Terraform is a good example. Legitimate, widely adopted, genuinely useful. But if nobody on the team has prior IaC experience, you haven’t just added a tool. You’ve created a single point of failure, and decreased the team’s buy-in.

If I left tomorrow, could this team operate and extend what I built?

The Checklist: What to Validate Before You Commit

On your consumers

• Have you spoken to end users about their actual daily workflow? This should go beyond a requirements meeting where everyone agrees the platform sounds great.
• Are you building for the consumers you have, or the ones you imagine you’ll have? Saves you from over-engineering.

On your tooling

• Does each tool solve a problem your current stack genuinely cannot solve? “It solves it less elegantly” is not enough.
• How many tools does an engineer need to understand to debug a failure end to end?
• Could you consolidate two or more tools into something you already own? Fewer moving parts to deal with.

On your team

• If you left tomorrow, could your team operate and extend what you built?
• Was the team involved in tooling decisions, or presented with the outcome? Involvement builds ownership, which leads to adoption and support.

Good Platforms Are Designed, Great Platforms Are Grown

The most humbling question I can ask at the start of any project isn’t “what’s the best architecture for this?” Rather, it’s “does anyone actually want what I’m about to build?”

The best platform builders are the ones who check their plans and ambitions against reality early and often. And they’re willing to build something smaller and simpler than they are capable of, because that’s what the situation calls for. Extend it when there’s a real need.

The engineer who understands this will build something that lasts far longer than the one who got the architecture perfect on paper.