Passing custom variables to Databricks DLT pipelines using Asset Bundles

How to manage catalog names, schema names, and environment-specific config without hardcoding a single value — and actually sleep at night after promoting to prod.

The hardcoding problem

Every Databricks project starts the same way. You write a DLT notebook, it runs perfectly in dev, you feel great about yourself. Then you realise every table reference has dev_catalog.my_schema.my_table baked directly into the SQL. Promoting to production becomes a game of find-and-replace across a dozen files, hoping nothing gets missed, knowing something will.

Databricks Asset Bundles (DABs) solve this cleanly. You define variables once, inject them into pipelines at deploy time, and never touch an environment-specific string again. No runtime parameter hacks, no environment-specific notebooks, no “wait which file has the prod catalog name” moments at 11pm.

What this covers: defining variables in databricks.yml, consuming them in pipeline YAML files, referencing them inside SQL DLT notebooks, and handling cross-pipeline references between layers.

How the variable system works

Before diving into YAML, it helps to see the full picture. Variables flow through three distinct files at two distinct times — and mixing up which system resolves what is the source of most confusion:

Step 1: the databricks.yml file

The bundle root file is where all variables are declared. Each variable gets a description and a default value. Per-target overrides live in the targets block — only the values that actually differ between environments need to be listed there.

bundle:
  name: my_data_product

workspace:
  host: https://my-workspace.azuredatabricks.net

variables:
  catalog:
    description: Unity Catalog name for this environment
    default: dev_catalog
  bronze_schema:
    description: Schema name for the bronze layer
    default: sch_bronze_sales
  silver_schema:
    description: Schema for the silver layer
    default: sch_silver_sales
  gold_schema:
    description: Schema for the gold layer
    default: sch_gold_sales
  bucket_name:
    description: Cloud storage bucket for raw ingestion
    default: my-company-data-dev

resources:
  pipelines:
    pipeline_bronze: { source: resources/pipeline_bronze_sales.yml }
    pipeline_silver: { source: resources/pipeline_silver_sales.yml }
    pipeline_gold:   { source: resources/pipeline_gold_sales.yml }

targets:
  dev:
    mode: development
    default: true
    variables:
      catalog: dev_catalog
      bucket_name: my-company-data-dev

  prod:
    mode: production
    variables:
      catalog: prod_catalog
      bucket_name: my-company-data-prod

Notice: schema names are identical in both environments. Unity Catalog’s catalog.schema.table namespace means dev_catalog.sch_bronze_sales and prod_catalog.sch_bronze_sales are completely separate. You only override the things that actually differ.

Step 2: pipeline YAML files

Each pipeline gets its own YAML file. This is where bundle variables get consumed and passed into the DLT pipeline’s configuration block — which is the mechanism that makes them available inside SQL notebooks.

# resources/pipeline_bronze_sales.yml
resources:
  pipelines:
    pipeline_bronze_sales:
      name: "pipeline_bronze_sales_${bundle.target}"

      target: ${var.bronze_schema}    # Where DLT writes tables
      catalog: ${var.catalog}         # Which Unity Catalog

      serverless: true
      photon: true
      channel: "PREVIEW"

      configuration:
        catalog: ${var.catalog}
        bronze_schema: ${var.bronze_schema}
        bucket_name: ${var.bucket_name}

      libraries:
        - notebook:
            path: ${workspace.file_path}/src/bronze/nb_bronze_autoloader.sql

Three bundle-specific substitutions: ${var.variable_name} references a declared variable. ${bundle.target} injects the target name (dev/prod) — great for naming resources so they're visually distinct in the workspace UI. ${workspace.file_path} resolves the bundle's root path in the workspace, avoiding hardcoded absolute paths.

Step 3: referencing variables inside SQL notebooks

This is where the magic lands. DLT injects whatever is in the pipeline’s configuration block into the notebook at runtime. Inside SQL, you reference those values using ${key} syntax — no imports, no special setup needed.

-- Bronze streaming table using injected variables
CREATE OR REFRESH STREAMING TABLE __tbl_raw_orders
COMMENT "Raw order records from ERP export"
TBLPROPERTIES ("delta.appendOnly" = "true")
AS SELECT
  *,
  CURRENT_TIMESTAMP() AS __record_created_time,
  input_file_name()   AS __source_file
FROM cloud_files(
  "s3://${bucket_name}/orders/incoming/",   -- injected at runtime
  "json",
  map(
    "cloudFiles.inferSchema",       "true",
    "cloudFiles.schemaLocation",    "dbfs:/checkpoints/${catalog}/${bronze_schema}/orders/schema",
    "cloudFiles.schemaEvolutionMode", "addNewColumns",
    "cloudFiles.includeExistingFiles", "true"
  )
);

The syntax distinction that trips everyone up: ${var.catalog} in YAML is resolved by the bundle at deploy time. ${catalog} in SQL is resolved by DLT at pipeline runtime. Both end up with the same value — but if you write ${var.catalog} in your SQL notebook, DLT won't know what to do with it.

Cross-pipeline references: reading from upstream layers

Here’s a wrinkle that catches people out. When the silver pipeline needs to read from bronze tables, you cannot use LIVE.table_name — that syntax only works within the same pipeline. Instead, you construct the full Unity Catalog path using injected variables.

-- Silver notebook: reading from the bronze layer
CREATE OR REFRESH STREAMING TABLE __tbl_orders_typed AS
SELECT
  CAST(order_id AS STRING)            AS order_id,
  CAST(order_date AS DATE)            AS order_date,
  CAST(total_amount AS DECIMAL(12,2)) AS total_amount,
  UPPER(TRIM(status))                 AS status
FROM STREAM(${catalog}.${bronze_schema}.__tbl_raw_orders)
WHERE order_id IS NOT NULL;

And the silver pipeline YAML must pass bronze_schema in its config block, even though the pipeline itself writes to silver_schema:

# resources/pipeline_silver_sales.yml
configuration:
  catalog: ${var.catalog}
  bronze_schema: ${var.bronze_schema}   # needed to READ from upstream
  silver_schema: ${var.silver_schema}   # needed to WRITE here

Multiple notebooks in one pipeline

The gold layer often spans multiple notebooks — dimensions, facts, reporting views. You can list them all in a single pipeline; they execute in order and share the same configuration values. Tables created in notebook one are accessible via LIVE.table_name in notebook two.

# resources/pipeline_gold_sales.yml
libraries:
  - notebook:
      path: ${workspace.file_path}/src/gold/nb_gold_create_dimensions.sql
  - notebook:
      path: ${workspace.file_path}/src/gold/nb_gold_load_fact_sales.sql
  - notebook:
      path: ${workspace.file_path}/src/gold/nb_gold_create_reporting_views.sql

Overriding variables at deploy time

Beyond environment targets, you can override any variable directly from the CLI. Handy for testing against a non-standard bucket, pointing at a QA catalog, or debugging a specific notebook in isolation:

databricks bundle deploy --target dev           # default variables
databricks bundle deploy --target prod          # prod overrides
databricks bundle deploy --target dev --var="catalog=test_catalog"  # one-off override
databricks bundle validate --target prod        # dry run, no deploy

A practical gotcha: schema creation

Bundles do not create Unity Catalog schemas automatically. If the target schema doesn’t exist when the pipeline runs, it will fail immediately with a not-very-helpful error message. You have two options:

Option 1: pre-deployment SQL in CI/CD

databricks sql execute --warehouse-id <id> \
  "CREATE SCHEMA IF NOT EXISTS ${CATALOG}.${SCHEMA}"

Option 2: declare schemas in the bundle (recommended)

resources:
  schemas:
    bronze_schema:
      name: ${var.bronze_schema}
      catalog_name: ${var.catalog}
      comment: "Bronze layer for sales pipeline"

Option 2 is cleaner. The schema is created (if missing) when you run databricks bundle deploy — everything stays in version control, nothing needs to be run manually.

The full variable flow at a glance

Key takeaways

Define all variables in databricks.yml. Pass them into DLT notebooks through the pipeline YAML's configuration block. Reference them inside SQL with ${key} syntax. Keep per-environment differences in the targets block — only override what actually differs.

Cross-layer reads use full Unity Catalog paths, not LIVE.. That means the silver pipeline config needs to know about the bronze schema even though it writes to silver. And always declare your schemas as bundle resources, or add a schema-creation step to CI/CD — pipelines don't create missing schemas for you.

Once this pattern is in place, promoting from dev to prod is genuinely one command: databricks bundle deploy --target prod. Which means you can actually do it on a Friday without dreading Monday morning.

Next in this series: Deploying multi-pipeline Databricks bundles — keeping all your pipelines and workflows in sync without accidentally deploying the bronze layer three times.