Databricks News: Lakeflow Designer, UV package manager, DABs templates, Genie scheduled tasks

[embed]

Platform development is not slowing down, and we have some exciting news. Databricks Summit is approaching, and in one month, we will have even more exciting news.

If you have not yet subscribed to Medium, here is a friend’s link for you.

Dais talk

If you are going to DAIS and want to know my favorite DABs tips & tricks, I welcome you to my session https://www.databricks.com/dataaisummit/session/dabs-do-pro-all-best-tips-and-tricks

Lakeflow Designer

Lakeflow designer is available. It looks really nice. Sadly, the generated code is not something you want to use in production. In my opinion, Canvas (Visual data prep) should consist of many files, like a zip, and some of them should be standard Python/SQL. Something that could be easily handed off to data engineers to productionize. Currently, the code is in a long notebook. There should be an option to configure how Visual Canvas is saved, such as notebook, wheel, SQL, and other options. That would allow us to develop outside Designer. Also, the reverse approach would be useful so that the designer could read existing projects (by using AI) and use the Designer with them. I have a feeling that with the current approach, it could repeat the mistakes of Alteryx and ADF. I hope that the next versions will be more powerful. There is an option to partially bypass that and use the Lakeflow designer as a temporary solution, then ask Genie to generate SQL or PySpark syntax based on the Lakeflow designer.

%uv pip

In a serverless environment 5 (soon, probably in other environments as well), we can also install packages using the UV package manager. Tests show that it is a few times faster!

DABs templates

We can configure a folder in Workspace for custom DABs templates. If we click Create Bundle in the git repo, we can use our ready DABs template.

Genie Schedule Tasks

Report every week for the given topic, now it is really easy, just ask for it!

and it starts to look busy in Genie One Menu:

Native Notebook Profiling

In SQL editor and Notebook, we can choose columns and see profiling.

Connect to on-premises resources using an SSH reverse tunnel

It is quite an interesting option, as sometimes I am asked how to connect databricks to on-prem without opening a port on the firewall https://docs.databricks.com/aws/en/security/network/serverless-network-security/ssh-reverse-tunnel?language=Lakeflow+Connect+CDC

Tag UC Functions

Now we can also tag functions

If you like this blog post, consider buying me a coffee :-) https://ko-fi.com/hubertdudek