Databricks News: watermark-based incremental ingestion, MCP in AI gateway, Genie, Vector Search

[embed]

My favorite update is watermark-based incremental ingestion. I will record a video during the weekend and update the article content.

If you have not yet subscribed to Medium, here is a friend’s link for you.

Genie App

The Genie app is already available as an early release in the Play Store, but your account executive has to activate it for you.

Problems with Power BI

It is a bit wild: I planned to write this week that metric views were working with Power BI, but they were actually blocked by Power BI/Fabric after a short time. I hope that companies will work together to create a standard for a semantic layer and open-source it. If not, Databricks will create another open-source product.

Community connectors

Databricks is built on open-source. Now, let’s change how we ingest data so anyone can build connectors. Community connectors are here! For me, it is one of the most important news stories of the year as soon as we can have 1000s connectors and I count on contributions from all SAAS platforms!

Watermark-based incremental ingestion with soft deletes

Now Lakeflow Connect supports incremental ingestion for non-CDF tables. What is needed is a classic approach: a primary key, a last-modified column (cursor column), and soft deletes. You can read more details here https://medium.com/@databrickster/watermark-based-incremental-ingestion-lakeflow-connect-query-based-capture-91836fbaa453

Apps have a new home

A dedicated section for Databricks Apps has been added

MCP in AI gateway

We can now also govern MCP servers, including external ones

Vector Search

Now we can start the process to evaluate the quality of Vector Search. Based on it, we can adjust the settings of our vector search and, if required, use, for example, rerunker.

Also, the Vector Search endpoint can now be managed via DABS.

SQL-governed tags

Now we can manage governed tags by using SQL commands

CREATE GOVERNED TAG;
ALTER GOVERNED TAG;
DROP GOVERNED TAG;
DESCRIBE GOVERNED TAG;
SHOW GOVERNED TAGS;

ai_prep_search()

Transform the structured output of ai_parse_document() into a format optimised for RAG vector search

WITH parsed_documents AS (
  SELECT ai_parse_document(content) AS parsed
  FROM READ_FILES('/Volumes/mydata/documents/', format => 'binaryFile')
),
prepped_documents AS (
  SELECT ai_prep_search(parsed) AS result
  FROM parsed_documents
)

Lakebase to Lakehouse sync

Is now really easy with the Lakehouse Sync option. Sync is based on CDC.

Void columns

Delta support now includes VOID columns, which are empty columns in our Delta (can be kept for future use or for schema match). VOID is a new datatype; the only accepted value is NULL.

CREATE TABLE my_table(id INT, v VOID, name STRING) USING DELTA;

-- Before DBR 18.2: returns (id, name)
-- After DBR 18.2:  returns (id, v, name)
SELECT * FROM my_table VERSION AS OF 0;

If you like this blog post, consider buying me a coffee :-) https://ko-fi.com/hubertdudek