From Replay Recovery to Durable Customer State: Why We Adopted Lakebase

Three days after Christmas 2025, a schema change in our Okta Customer Identity and Access Management (CIAM) event stream caused eleven Spark Structured Streaming pipelines to fail simultaneously.

The change itself was minor. Okta introduced several nested consent attributes into an existing Kafka payload, but parts of our downstream identity stitching logic still expected a flatter structure. Several reconciliation pipelines failed during deserialization and downstream MERGE processing.

Recovery required replaying nearly 1.2 billion Kafka events.

During the incident:

• customer profile freshness degraded from our normal 18–25 minute window to nearly four hours
• engineers spent most of the day rebuilding checkpoints and replaying Kafka offsets

The incident highlighted a deeper architectural issue that had been accumulating over time. Six months later, after introducing Lakebase as a transactional ingestion layer between Kafka and our lakehouse, steady-state profile freshness dropped below two minutes, reconciliation pipelines reduced from fourteen to three, and replay recovery became deterministic instead of operationally risky.

This post explains what broke in our original architecture, why late-arriving events created persistent profile drift, and how introducing a transactional boundary changed the operational behavior of our streaming platform.

What CustomerDNA actually does

CustomerDNA is our internal customer identity platform.

It combines signals from:

• Okta CIAM authentication and consent events
• e-commerce order and return systems
• website clickstream activity
• mobile engagement events
• Salesforce CRM and loyalty systems

A single customer can appear:

• anonymously on mobile through cookie identifiers
• authenticated on web through Okta identity IDs
• in CRM through customer IDs
• in loyalty systems through loyalty IDs

Those events arrive from different systems, at different times, and without guaranteed ordering.

The platform’s core job is turning fragmented event streams into a durable customer identity graph.

Original architecture

Original architecture showing Kafka streams continuously rebuilding customer state downstream through Spark reconciliation jobs.

Here’s how the system was structured before the change:

Okta CIAM + Orders + Web/Mobile Events
                ↓
            Kafka Topics
                ↓
    Spark Structured Streaming
         (~52 streaming jobs)
                ↓
      Delta Lake Bronze/Silver
                ↓
 Identity Resolution & MERGE Jobs
       (14 reconciliation pipelines)
                ↓
      CustomerDNA Gold Profiles

At peak scale, the platform processed:

• ~180 million Kafka events daily
• ~8 TB of streaming data per day
• 40+ Kafka topics
• ~52 Spark Structured Streaming jobs

Most of the streaming platform worked well operationally. Kafka ingestion remained stable during campaign spikes, and Spark clusters recovered normally after infrastructure failures.

The difficult part was maintaining consistent customer state across asynchronous streams.

Where the architecture struggled

A typical customer journey rarely arrived in sequence.

For example:

1. Anonymous mobile events arrived first
2. Web login events arrived later from Okta
3. Order events arrived from commerce systems
4. Loyalty updates appeared several minutes afterward

Spark Structured Streaming jobs continuously stitched identities across:

• email
• cookie ID
• loyalty ID
• CRM customer ID
• Okta identity ID

Correctness depended heavily on:

• watermark configuration
• streaming state windows
• replay ordering
• checkpoint integrity

That dependency became fragile during edge conditions.

The incident exposed this clearly.

When the Okta schema changed:

• 11 downstream streaming jobs failed
• several streaming checkpoints became invalid
• replay required rebuilding large sections of the identity graph
• downstream MERGE operations regenerated duplicate customer profiles during replay

By that point:

• reconciliation pipelines consumed ~30% of total streaming compute
• most operational incidents originated from identity correction workflows rather than ingestion itself
• engineers spent more time maintaining replay consistency than improving identity models

The platform had no durable customer state at ingestion.

State only existed after downstream Spark jobs reconstructed it.

That design decision created most of the operational instability we were seeing.

What changed with Lakebase

We did not replace Kafka.

We did not replace Spark.

We inserted a transactional boundary between ingestion and downstream profile reconstruction.

Instead of reconstructing customer state downstream, customer entities became durable during ingestion itself.

Late-arriving events updated existing customer entities transactionally instead of reopening downstream profile re-computation workflows.

Identity correction logic that previously existed across fourteen reconciliation pipelines was consolidated into transactional entity updates.

For example, our ingestion MERGE logic now looks closer to this:

MERGE INTO customer_identity t
USING incoming_identity_events s
ON t.customer_id = s.customer_id

MERGE INTO customer_identity t
USING incoming_identity_events s
ON t.customer_id = s.customer_id

WHEN MATCHED THEN
  UPDATE SET *

WHEN NOT MATCHED THEN
  INSERT *

Replay behavior also became significantly simpler because replayed events updated existing entities idempotently instead of regenerating profile state from streaming windows.

Importantly, most of the streaming platform still remained in place. Behavioral enrichment, feature engineering, and downstream analytics pipelines continued running in Spark Structured Streaming. The biggest reduction came specifically from reconciliation and profile-correction workflows.

Before & after Lakebase operational metrics

Migration tradeoffs

This was not a lightweight migration.

The project required:

• ~16 weeks
• 4 engineers
• ~8 weeks of dual-running both architectures
• ~1.6× temporary infrastructure cost during validation

One of the hardest tasks was extracting identity logic buried across ~52 streaming jobs and formalizing it into transactional entity models.

Not all workloads moved into transactional ingestion.

High-volume clickstream logs remained append-only in Delta Lake because transactional MERGE overhead was unnecessary for behavioral telemetry. Transactional ingestion was used specifically for identity-bearing entity streams.

Operational impact

After introducing Lakebase:

• steady-state profile freshness improved from 18–25 minutes to under 2 minutes
• reconciliation pipelines reduced from 14 to 3
• duplicate profile incidents dropped by ~57%
• replay recovery time reduced from several hours to under 30 minutes
• streaming compute dedicated to identity correction dropped from ~30% to ~18%

The total number of Spark streaming jobs reduced only modestly because enrichment and analytical processing still remained downstream. The major operational improvement came from eliminating repeated identity reconstruction and replay-driven profile correction workflows.

Engineering effort shifted away from replay recovery and duplicate cleanup toward improving identity quality and cross-device resolution.

The lesson

Our biggest production problems were not related to streaming scale.

They came from trying to reconstruct durable customer state downstream from unordered event streams.

Once customer identity became transactionally durable earlier in the ingestion flow, late-arriving events, replay recovery, and profile reconciliation stopped dominating platform operations.

The most impactful architectural change was not adding more reconciliation.

It was reducing the need for reconciliation entirely.