Databricks Lakebase: Instant OLTP for Data Apps and AI Agents

Databricks’ Unified Approach to Operational and Analytical Convergence

Ankur Kumar, Max Mogenis & Sandeep Upreti | Authors

“Operational-analytical convergence is not about building complex pipelines; it is about closing the execution gap. By merging live applications with the Lakehouse, we transform data from a static record into an immediate business driver “

The Divide That Persists

Every enterprise data architecture carries the same structural compromise: one system for transactions, another for analytics. OLTP databases handle the real-time writes — policy issuances, payment authorizations, claims submissions, customer interactions. Analytical platforms handle the reads — dashboards, models, strategic reporting. Connecting them requires ETL pipelines, CDC infrastructure, staging logic, and reconciliation processes that are expensive to build, fragile to maintain, and perpetually behind. This is not a theoretical concern. In a recent platform modernization for a global specialty insurer, our team inherited exactly this pattern — two parallel platforms, one for warehousing and one for data lake processing, with governance fragmented across both. Migrating to a unified Lakehouse on Databricks resolved the analytical side: workload processing dropped from 4.5 hours to approximately 45 minutes, lineage became transparent through Unity Catalog, and maintenance collapsed from two platforms to one. But the transactional systems that fed the Lakehouse still lived outside it. They required separate governance, separate infrastructure, and a pipeline that needed constant monitoring. That is the gap Databricks Lakebase addresses.

Figure 1: Traditional fragmented OLTP-to-OLAP architecture versus the Lakebase unified model. By bringing transactions and analytics onto a single platform, Lakebase eliminates the ETL pipeline, governance gaps, and operational overhead that characterize conventional architectures.

Introduction:

What is Databricks Lakebase?

Lakebase is a fully managed, serverless PostgreSQL 17 database engine integrated directly into the Databricks Data Intelligence Platform. It is built on technology from Databricks’ acquisitions of Neon (serverless Postgres with separated compute and storage) and Mooncake (analytics optimized storage).

A few important characteristics:

• Sub-10ms query latency and 10,000+ queries per second under production conditions
• Up to 8TB of storage per instance with 1,000 concurrent connections
• Full compute-storage separation — compute scales independently and can scale to zero when idle
• Two deployment modes: Autoscaling (default since March 2026) and Provisioned
• PostgreSQL ecosystem compatibility — existing drivers, tools (pgAdmin, DBeaver, psql), and libraries work out of the box

What matters most is not the database engine itself — it is where it lives. Lakebase runs inside the same platform where data engineering, analytics, and AI already happen. There is no middleware to maintain, no replication pipeline to monitor, and no separate catalog to reconcile.

For teams already on Databricks, Lakebase replaces the need for a standalone managed Postgres instance, custom CDC pipelines, separate OLTP governance models, and dedicated infrastructure teams managing operational databases independently.

Lakebase Architecture — Core Features

Sync Tables: Eliminating the Pipeline(Zero-ETL Approach)

This is Lakebase’s most architecturally significant capability. Sync tables enable bidirectional data movement between Lakebase and the Lakehouse without custom pipelines.

Lakehouse → Lakebase (Reverse ETL): Unity Catalog tables backed by Delta Lake can be synchronized into Lakebase Postgres instances, making analytical data available to applications at low latency. Sync pipelines run on managed Lakeflow Declarative Pipelines, support three modes:

• Snapshot — periodic full refreshes
• Triggered — incremental updates on demand or schedule
• Continuous — real-time streaming

Lakebase → Lakehouse (Forward CDC): Lakebase Change Data Feed automatically captures row-level changes and stores them as Unity Catalog Delta tables — no custom CDC infra required.

In a recent platform modernization, our team employed a metadata-driven approach to automate data movement across platforms. The principle was identical: reduce manual pipeline construction, automate through configuration rather than code, preserve governance end to end. Sync tables are the productized, platform-native evolution of that pattern.

Figure 2: Bidirectional data flow between Lakebase (OLTP) and the Lakehouse (OLAP). Sync tables enable multiple modes of reverse ETL for serving analytical data in applications, while Change Data Feed captures transactional mutations for downstream analytics — all governed through Unity Catalog.

Autoscaling and Scale-to-Zero

Compute adjusts dynamically based on demand and scales to zero when idle. For organizations running dozens of development and staging databases alongside production, the compounding cost savings are immediate and meaningful.

Database Branching

Teams can create instant branches from production for development or testing with instant copy-on-write clone of data — version the database like code, iterate in isolation, and discard cleanly. This closes a significant gap for organizations that have CI/CD for application code but still manage database changes manually and also allows for isolated feature testing without impacting production.

Instant Restore

Automated snapshots with point-in-time recovery with retention period configurable up to 30 days. Critically, restores can be validated in an isolated branch before being promoted to production — turning recovery from a high-stakes operation into a controlled, verifiable workflow.

Postgres Extensions

pgvector enables vector similarity search directly within Lakebase — useful for RAG pipelines and semantic search without a standalone vector database. PostGIS adds geospatial processing for location-aware applications.

The Lakebase Multiplier: Governance, AI, TCO Unified

Governance: Unity Catalog for OLTP

Lakebase databases register in Unity Catalog, extending the same governance model that covers the Lakehouse to transactional workloads. This means unified access control, consistent lineage, and a single audit surface across OLTP and OLAP.

Figure 3: Unified governance through Unity Catalog spanning Lakebase (OLTP) and Lakehouse (OLAP), ensuring consistent access control, lineage, and auditability across all workloads.

For organizations already invested in Unity Catalog — defining policies, building lineage, establishing audit workflows — Lakebase is not a new governance surface. It is a natural extension of what they already operate.

A Healthcare Perspective: Operationalizing AI-Driven Outreach

In a recent healthcare use case, Lakebase played a central role in operationalizing AI-driven population health outreach for Value-Based Care (VBC) programs. The challenge was not generating insights — it was turning fragmented patient data, SDOH signals, and engagement history into timely, actionable interventions.

While the Databricks Lakehouse unified clinical, survey, and engagement data into a longitudinal patient intelligence layer, Lakebase served as the operational system of record for executing those insights.

AI models generated “Next Best Actions” such as medication reminders, preventive screening outreach, and care coordination triggers. These recommendations were persisted in Lakebase, where care managers could review, approve, or modify actions before triggering outreach across SMS, email, or voice channels.

This human-in-the-loop design ensured that AI remained clinically safe, explainable, and governable, while still enabling scalable, real-time intervention workflows.

All patient data access, outreach decisions, and workflow actions were governed through Unity Catalog, maintaining a single audit trail across both analytical and operational workloads.

The key takeaway was clear: Lakebase did not replace the Lakehouse — it completed it, by bridging analytical intelligence with low-latency operational execution. Lakebase enabled transactional workflows for approvals, outreach status tracking, and engagement orchestration — functionality that analytical pipelines alone could not support.

Figure 4: Lakebase operationalizing AI-driven healthcare outreach by bridging analytical intelligence with real-time engagement workflows. In this architecture, the Lakehouse serves as the analytical intelligence layer while Lakebase powers real-time operational engagement workflows.

AI Integration

Lakebase serves as the online feature store for ML models, a persistence layer for AI agent state (supporting frameworks like LangGraph and OpenAI Agents SDK), and an embedding store via pgvector for RAG workflows. The inference loop - feature retrieval, model scoring, result storage - stays within the platform without external orchestration.

Cost Efficiency

The cost argument is not about cheaper Postgres. It is about eliminating the systems and pipelines that exist solely to bridge the OLTP–OLAP divide: standalone databases, CDC infrastructure, separate feature stores, and duplicate governance models. Scale-to-zero compounds the savings across non-production environments.

Lakebase: Use Cases & Industry Applications

Insurance

Real-time claims triage, UW decision support, fraud detection at point of claim

Financial Services

Real-time fraud scoring, streaming risk analytics, regulatory reporting

Healthcare

Patient record serving, clinical trial data management, care recommendations, AI-driven population health outreach

Retail

Personalized recommendations, dynamic pricing, inventory optimization

Manufacturing

Machine telemetry, predictive maintenance, supply chain optimization

On the Horizon: Early Access and Upcoming Engineering Features

Sync Back from Lakebase to Lakehouse- Allows operational data to be used by internal databricks tools like Databricks AI/BI Dashboard & Genie.

CMEK Key for Postgres data: Allows for customized and more controlled data security using customer managed encryption keys for your Postgres data.

Evolved Database Branching: Currently database branching is just limited to single workspace limiting DEV->QA->PROD promotion.

Conclusion: The Layer That Was Missing

The Lakehouse unified analytics. Unity Catalog unified governance. Lakebase unifies operations.

For enterprises on Databricks, Lakebase brings the one workload category that historically remained outside the platform — real-time transactions — into the same governed environment. The result is fewer — systems, pipelines, governance surfaces, and faster time from transaction to intelligence.

Not every OLTP workload will migrate overnight. Legacy systems and vendor-specific databases will continue to exist. But for new applications, AI-native workflows, and workloads where transactional and analytical data need to be tightly coupled — Lakebase offers a materially better architecture than bridging two separate systems with custom infrastructure.

The question is no longer “how do we connect our transactional database to our analytical platform?” It is “why are they separate in the first place?”