Databricks series and production stories : 3

Databricks is evolving fast, make sure your platform understanding does too!

Its important to be up-to-date

If we don’t keep up with how our tools evolve, we end up trusting an old picture of how things work. New updates can quietly change the meaning of configs we set months or years ago.

Everything looks fine — until the day you need that behaviour in production and realise it doesn’t work the way you thought anymore.

This is a story about how that happened to us with Delta Lake time travel in a live trading environment in February 2026.

The Production incident: a pricing issue and a 55‑day question

Fast‑forward to February 2026.

A pricing issue is raised on one of our products. The business team wants to know:

“Can you replay the trades table from about 55 days ago so we can compare PnL before and after we fix this model?”

This is the kind of question time travel is made for. We’d done similar things before. Nobody was nervous.

We jumped into a notebook, pointed at the trades table, and tried to query it “as of” that date.

And nothing was there.

• No valid table version at that point in time.
• No way to build a consistent snapshot as‑of day‑55 from Delta alone.
• Time travel stopped hard at around 40 days.

At first we assumed we’d mis‑typed the timestamp or hit the wrong cluster. Then, as you do in production, we started checking the less comfortable things:

• DESCRIBE HISTORY trades to see how far back history actually went.
• SHOW TBLPROPERTIES trades to inspect retention configuration.

Let me tell you the complete story :

The context: a “boring” but critical trades table

We have a Delta table called trades that sits at the heart of a few things:

• intraday and end‑of‑day PnL
• risk and exposure reporting
• ad‑hoc investigations from risk and compliance
• a couple of downstream analytics jobs

Nothing fancy — just one of those core tables you expect to “always be there” when someone asks a hard question.

From the beginning, we were careful about retention. Storage isn’t free, but neither is missing data during an investigation.

So we settled on a simple rule: we keep 60 days of history in Delta.

We enforced that with a nightly job:

%sql

VACUUM trades RETAIN 60 DAYS;

The mental model across the team was very clear:

• The platform keeps two months of data.
• If something looks off, we can time travel back and replay.
• Risk and product can safely assume “last ~60 days” are available.

And for a long time, that matched reality.

“We’ll clean this up later”: moving to table properties

As Delta Lake evolved, guidance started to shift towards using table properties to manage retention instead of baking it into every VACUUM call. That made sense: one source of truth on the table, simpler jobs, easier automation.

So we did what a lot of teams do in real life:

1. We left the existing VACUUM trades RETAIN 60 DAYS; job in place “for now”.
2. We added table properties to standardise settings and save some storage:

%sql

ALTER TABLE trades SET TBLPROPERTIES (   'delta.deletedFileRetentionDuration' = 'interval 40 days',   'delta.logRetentionDuration'        = 'interval 40 days' );

At this point, our thinking was roughly:

• “We know VACUUM ... RETAIN takes precedence, so we’re still safe at 60 days.”
• “The 40‑day table properties are mainly for consistency and future‑proofing.”
• “We’ll come back and clean this up properly when we have time.”

Nothing broke. Jobs were green. Dashboards looked normal. Recent time‑travel queries worked fine. There was zero pressure to revisit this decision.

What we didn’t fully internalise was that with the newer Delta behaviour, those table properties had quietly become the real limit for how long data and logs were kept.

The incident: a pricing issue and a 55‑day question

Fast‑forward to February 2026.

A pricing issue is raised on one of our products. The risk team wants to know:

“Can you replay the trades table from about 55 days ago so we can compare PnL before and after we fix this model?”

This is the kind of question time travel is made for. We’d done similar things before. Nobody was nervous.

We jumped into a notebook, pointed at the trades table, and tried to query it “as of” that date.

And nothing was there.

• No valid table version at that point in time.
• No way to build a consistent snapshot as‑of day‑55 from Delta alone.
• Time travel stopped hard at around 40 days.

At first we assumed we’d mis‑typed the timestamp or hit the wrong cluster. Then, as you do in production, we started checking the less comfortable things:

• DESCRIBE HISTORY trades to see how far back history actually went.
• SHOW TBLPROPERTIES trades to inspect retention configuration.

That’s when the story finally clicked:

• The table was configured for 40 days via delta.deletedFileRetentionDuration = 'interval 40 days' delta.logRetentionDuration = 'interval 40 days'
• Old data and log files beyond 40 days had already been cleaned up.
• Our nightly VACUUM trades RETAIN 60 DAYS; was no longer the thing really protecting us.

The platform was doing exactly what we told it to do now, not what we used to tell it months ago.

We still “knew” we had 60 days. Delta was only keeping 40.

The immediate fallout:

• We couldn’t replay the exact PnL state from the requested date.
• We had to lean on archived extracts and external audit feeds to piece things together.
• Business quite reasonably asked: “We thought you kept 60 days — when did that change?”

Nothing about this was a dramatic outage, but it was exactly the kind of moment you don’t want in front of risk and audit.

What changed under the hood

This incident sat right in the middle of Delta Lake’s shift towards more predictable, property‑driven retention and time travel.

The key changes you need to know about:

1. delta.deletedFileRetentionDuration defines the hard time‑travel window
2. You can only query history within that interval.
3. If it’s 40 days, queries 55 days back are blocked or simply have no data.
4. delta.logRetentionDuration must be ≥ delta.deletedFileRetentionDuration

• This keeps transaction logs and data files in sync.
• If logs are kept for less time than data, you can’t reliably reconstruct older versions.

5. VACUUM retention argument is ignored (except RETAIN 0 HOURS)

VACUUM table RETAIN 60 DAYS no longer “wins” over table properties.

The engine uses the table’s delta.deletedFileRetentionDuration instead.

VACUUM becomes “apply whatever the table says”, not “define retention here”.

Before this, actual behaviour depended on:

• What your table properties said.
• How and when VACUUM last ran.
• Which RETAIN value you’d given it.

That mix is exactly how you end up with people confidently saying “we keep 60 days” while the system has been quietly cleaned down to 40.

Lessons we took from this

A few things we wrote down after this incident:

• People remember behaviour, not table properties. Risk remembered “we can go back around two months” because that’s what had worked historically, not what the table said last week.
• Old habits linger in code. The VACUUM trades RETAIN 60 DAYS; job stayed around long after the real retention policy moved into table properties.
• New platform rules can silently flip the source of truth. Once Delta shifted to property‑driven retention, our assumptions about RETAIN taking precedence became wrong overnight.
• Incidents are when assumptions show up. Everything looked good — jobs, dashboards, SLAs — until we needed exactly the slice of data that no longer existed.

None of these are unique to Delta. The same thing happens with security defaults, retry policies, scheduler semantics, you name it. But time travel and retention are particularly sneaky because they only matter in stressful moments.

A simple checklist you can run today

If you’re using Delta Lake in production, , here’s a quick checklist to avoid our mistake:

1. List your critical tables
2. Check what the business thinks you keep Ask the relevant teams: “When you ask us to go back in time, how far back do you expect we can go?”
3. Compare that with your actual config

sql

SHOW TBLPROPERTIES trades;

Look at:

• delta.deletedFileRetentionDuration
• delta.logRetentionDuration

4. Align the numbers

%sql

ALTER TABLE trades SET TBLPROPERTIES (   'delta.deletedFileRetentionDuration' = 'interval 60 days',   'delta.logRetentionDuration'        = 'interval 60 days' );

5. Simplify your maintenance jobs

• Drop custom RETAIN values in VACUUM unless you truly intend to use RETAIN 0 HOURS.
• Standardise on VACUUM table_name; and let the table properties drive behaviour.

Closing thought

The main takeaway for us was not “Delta behaved unexpectedly” — it didn’t. The platform did exactly what our latest configuration said.

The real issue was that our understanding hadn’t kept up with how the platform had evolved.

If your teams are planning around “we have 60–90 days of history”, but your Delta tables are actually configured for 30 or 40, that’s a hidden production risk waiting for the next incident, investigation, or audit.

👉 Take a few minutes this week to check what delta.deletedFileRetentionDuration and delta.logRetentionDuration really are on your most important tables—before a future update or cleanup run turns an old assumption into your next production story.

Series 1 : Databricks/ Spark series and production stories — 1 | by Amit Dass | Medium

Series 2 :Databricks/ Spark series and production stories — 2 | by Amit Dass | Medium

These new, stricter time‑travel and VACUUM rules are tied to specific Databricks runtimes, not retro‑applied to absolutely everything at once.

• For Databricks Runtime 18.0 and above, time‑travel queries are now blocked if they go beyond delta.deletedFileRetentionDuration, and VACUUM ignores custom RETAIN values except 0 HOURS.
• For Unity Catalog / serverless / Databricks SQL, the same behaviour is being applied to serverless compute, Databricks SQL, and Runtime 12.2+ workloads starting around Jan 2026.

So: it’s not every old runtime, but anything on DBR 18.0+ (and UC/serverless/SQL on 12.2+) will follow the new rules, and that’s what you should reference in your story.

If you enjoy these kinds of real-world Databricks/Delta Lake stories — where configs, platform behaviour, and production reality all collide — let me know. I’m happy to share more war stories, deep dives, and small checks you can run to avoid surprises in your own environment.