Lakebase: Why we picked Lakebase: fast benchmarks weren’t the whole story

What we built, what surprised us, and what we’d think harder about next time

[Wipro WealthAI Solution — Field Experience Note]

Where We Started?

Honestly, the starting point wasn’t a clean architecture decision — it was frustration. Our trade break workflow was spread across too many surfaces. Confirmations in one place, case notes somewhere else, status updates in a tool that wasn’t built for this. Delta Lake was doing fine on the historical and reporting side, but the operational layer — the bit that case managers actually write to all day — kept causing problems.

The issue was the data itself. A single break can change status a dozen times in a day. Two analysts might be updating the same case at the same time. Resolutions tick forward with every counterparty response. That’s not a batch workload. Treating it like one meant compaction overhead, write contention, and a case management UI that was noticeably sluggish on busy days. We needed something built for mutable, transactional data — and Lakebase was the answer that kept coming up

What Actually We Found?

We were sceptical going in. The pitch for Lakebase sounded a lot like “Delta Lake but faster for writes” and we’d heard variations of that before. What changed our minds was actually running it against our specific workload.

⦿ The upsert performance was genuinely different. Frequent small updates to trade_breaks and resolutions — the kind that pile up compaction debt in Delta — just worked without the overhead we’d been managing around.

⦿ Concurrent writes held up. Our operations team covers multiple time zones so parallel case access isn’t a stress test scenario, it’s Tuesday morning. No lock contention issues we’d been papering over.

⦿ Cross-table atomicity mattered more than we expected. When a break resolves, three tables need to update together. Getting a consistent snapshot in the Lakehouse promotion job used to be fiddly. Now it isn’t.

The table mapping we landed on after a few iterations:

The Path We Are Taking

The design we ended up with is simpler to describe than it was to agree on: data flows one way. Lakebase takes all the operational writes. When a case closes, a promotion job picks it up, validates it, and lands it in the Lakehouse. From there it feeds the historical tables, the curated views, and anything downstream. Nothing goes back the other way. The Unity Catalog permission model enforces that — the API can’t accidentally write to the Lakehouse and the BI tools can’t query Lakebase directly.

We keep a 90-day rolling window in Lakebase for recent-history lookups before archiving to the Lakehouse. That was a pragmatic call — ops teams ask “what happened to this break last week” more often than the architecture anticipated.

What’s next is straightforward: stress-test the promotion pipeline at three times current volume, and get sync lag instrumented as a proper operational alert rather than something we check manually.

One Honest Caveat

Lakebase is not right for every situation and we’d be doing you a disservice pretending otherwise. If your tables don’t change much, or your write volumes are modest, a well-tuned Delta layer will get you there without the added complexity. What we didn’t fully anticipate was how much operational overhead comes with running two storage surfaces — schema drift between layers, sync lag under load, two sets of alerts to manage. We hit all three in the first month. Eyes open on that going in.

That said — if the workload fits, the payoff is real. We’re running one platform, one governance model, one lineage graph across both the operational and analytical surfaces. No external database sitting alongside Databricks, no separate access control layer to maintain. The consolidation doesn’t show up in the architecture diagram but it shows up in how much time we’re not spending keeping two systems in sync manually.

Thanks for reading. This article represents the personal views of the author and is intended for informational purposes only. It does not constitute an official position or endorsement by any organization. In case you want to share your case studies and thoughts or want to connect, please ping me via LinkedIn