Implementing Enterprise Naming Convention, Agentic Way

Recently, Databricks added Workspace skills, which are used by Genie Code

SKILLS.md is a simple Markdown file that the LLM prompt uses if the request is related to that skill.

If you are not yet a member of Medium, you can access the extended version on the SunnyData blog for free.

Markdown includes metadata fields at the beginning and explains the prompt to the agent when the skill is used. I decided to use a skill file to help me to follow the naming convention used in the enterprise:

---
name: enterprise-naming-convention
description: Apply enterprise naming standards to Databricks SQL DDL, especially CREATE TABLE statements. Use when a user asks to standardize, fix, or review schema, table, or column names.
---

# Enterprise naming convention
Use this skill whenever the user asks to create or rewrite Databricks SQL so object names follow the enterprise naming convention.
## Rules
1. Use lowercase snake_case for schemas, tables, and columns.
2. Schema names must follow `<domain>_<layer>` where layer is one of `raw`, `refined`, or `serving`.
3. If a schema only contains a domain name, convert it to `<domain>_serving`.
4. Table names must start with `tbl_`.
5. Column naming rules:
- identifiers end with `_id`
- date columns end with `_dt`
- timestamp columns end with `_ts`
- boolean columns start with `is_`
- monetary amount columns end with `_amt`
etc...

whole file here: https://github.com/hubert-dudek/medium/tree/main/topics/202604/agent-non-conversational/skills/enterprise-naming-convention

Genie Code helping

Now we can ask questions to Genie Code, and it will help to adjust SQL to our naming convention

What if we can audit the whole company

That skill is working great, but the problem is that not everyone will use it, and also, we have a lot of legacy objects in our cataloge. So it makes me think of writing a really simple agent which will

• Take all tables from the information schema
• read SKILL
• pass it to LLM
• Generate JSON with results for all tables, display it, and save to volumes

Of course, it is easier said than done, as we need to remember that our agent needs to have at least USE CATALOG, USE SCHEMA and BROWSE permissions and also has access to VOLUME and SQL WAREHOUSE.

Use the template

On the page https://github.com/databricks/app-templates/, we can find a lot of templates. We can use agent conversational ones (if we want to integrate a lot of skills), but for one skill, I used just https://github.com/databricks/app-templates/tree/main/agent-non-conversational

The easiest way is just to clone the entire repo and start experimenting with templates.

And now we have to copy the template to our repo and edit a few files

databricks.yml

• databricks.yml is the control file for the app.

We use it to define:

• the app resource (including start command)
• environment variables
• the SQL warehouse resource
• the UC volume resource

In our bundle file, it passes values such as:

• API_PROXY
• SCAN_WAREHOUSE_ID
• REPORT_VOLUME_PATH
• TARGET_CATALOGS
• SCAN_REPORT_DIR
• SKILL_FILE_PATH

It also binds:

• a SQL warehouse with CAN_USE
• a volume with WRITE_VOLUME

resources:
  apps:
    agent-naming-convention:
      name: ${var.app_name}
      description: "agent application to scan UC for naming convention"
      source_code_path: .
      config:
        command: ["uv", "run", "start-server"]
        env:
          - name: MLFLOW_TRACKING_URI
            value: "databricks"
          - name: MLFLOW_REGISTRY_URI
            value: "databricks-uc"
          - name: MLFLOW_EXPERIMENT_ID
            value: ${var.experiment_id}
          - name: AGENT_MODEL
            value: ${var.agent_model}
          - name: SCAN_WAREHOUSE_ID
            value: ${var.scan_warehouse_id}
          - name: REPORT_VOLUME_PATH
            value: ${var.report_volume_full_name}
          - name: TARGET_CATALOGS
            value: ${var.target_catalogs}
          - name: SCAN_REPORT_DIR
            value: ${var.scan_report_dir}

      # Resources which this app has access to
      resources:
        - name: "experiment"
          experiment:
            experiment_id: ${var.experiment_id}
            permission: "CAN_MANAGE"

        - name: "scan_warehouse"
          sql_warehouse:
            id: ${var.scan_warehouse_id}
            permission: "CAN_USE"

        - name: "report_volume"
          uc_securable:
            securable_full_name: ${var.report_volume_full_name}
            securable_type: "VOLUME"
            permission: "WRITE_VOLUME"

agent.py

We keep agent.py small on purpose. We changeagent.py into a scan entry point.

from mlflow.genai.agent_server import invoke
from pydantic import BaseModel

from agent_server.scan_catalogs import run_scan
class AgentInput(BaseModel):
    catalogs: list[str] | None = None
@invoke()
async def invoke_handler(data: dict) -> dict:
    payload = AgentInput(**(data or {}))
    return run_scan(catalogs=payload.catalogs)

scan_catalogs.py

This file contains the real work which our agent has to do — it is the tools used by the agent, which can be listed below:

def load_skill_file(path: str) -> str:
    ...
def read_information_schema(workspace_client, warehouse_id, catalogs):
    ...
def evaluate_naming(rows, skill_text):
    ...
def save_report(rows, volume_path, report_dir):
    ...
def run_scan(catalogs=None):
    ...

full file here: https://github.com/hubert-dudek/medium/blob/main/topics/202604/agent-non-conversational/agent_server/scan_catalogs.py

read_information_schema will execute simple SQL to get our table and column names

SELECT
  table_catalog,
  table_schema,
  table_name,
  column_name,
  data_type
FROM workspace.information_schema.columns
ORDER BY table_schema, table_name, ordinal_position

Writing these functions is the biggest challenge, and you need to customise them to adjust to your needs once they are ready. Only one thing remains: deploying.

Deployment

databricks bundle validate
databricks bundle deploy -t dev
databricks bundle run <app_resource_key> -t dev

Here are some important aspects that make the app different from other resources. Deploy will only create the computer and all settings you need to do “bundle run: to… make deployment. It is easy to get lost in it.

Additionally, the app is automatically creating a service principal. We need to add that SP to the group or give direct access to read the schema of our tables

GRANT USE CATALOG ON CATALOG workspace TO `<app-service-principal>`;
GRANT USE SCHEMA ON CATALOG workspace TO `<app-service-principal>`;
GRANT BROWSE ON CATALOG workspace TO `<app-service-principal>`;

or grant via UI:

Trigger scan

Once the app is deployed, we get the URL, and we can trigger the POST invocation endpoint

It will return us JSON with audit:

and also save it to the volume:

Not perfect, but working

Probably I could polish it for days. Add UI, more checks for tables, keep state maybe even in Lakebase, but what was important for me was to prove how easy it is to achieve agentic app in databricks thanks to the complete stack, especially thanks to Apps and DABS, one of my favourite pieces of the platform. One thing worth remembering is that apps haven’t yet scaled to zero, so after running an audit, the best is to pause the app compute.

Hubert Dudek (author)

If you like this blog post, consider buying me a coffee :-) https://ko-fi.com/hubertdudek