Pass data from Terraform Infrastructure to DABS variables

There are more and more resources available in DABS, and I have to say, defining them is much nicer and easier to manage than in Terraform. We will continue using Terraform to deploy Azure or AWS resources, but we need to pass data from Terraform to DABS.

If you are not yet a member of Medium, you can access the extended version on the SunnyData blog for free.

Let’s say we register Azure Key Vault and then pass the ID and DNS name to DABS to register the secret_scope.

You can use DABS as per project resources, but you can also use it to define “infrastructure” things, which we did in Terraform before, such as secret scopes, catalogues, and external locations.

I created an infra repo with two folders

|
|-infra
|-bundle

I start with deploying the infrastructure by using

terraform apply

at the end of the main.tf I defined output in DABS-friendly style

output "dabs_variables" {
  value = {
    key_vault_dns_name    = local.key_vault_dns_name
    key_vault_resource_id = azurerm_key_vault.kv.id
  }
}

After deployment, we can read the state. In fact, we can read it even without a deployment; that command just returns the state's output.

terraform output -json dabs_variables

which is returning nice JSON:

{"key_vault_dns_name":"<https://kvbrickhsodw9.vault.azure.net/>",
"key_vault_resource_id":"/subscriptions/acbc7e82-ebf1-4daf-96ab-26c1a4cb38be/resourceGroups/brickster_westus2/providers/Microsoft.KeyVault/vaults/kvbrickhsodw9"}

Variables overrides in DABS

There are a few ways to override variables in DABS:

• Within any var options specified as part of the bundle command.
• Within any environment variables set that begin with BUNDLE_VAR_.
• Within the variable-overrides.json file, if it exists.

In that scenario, we will use variable-overrides.json

which by default reside in ./.databricks/bundle/<target> directory.

Pipeline behavior

Before taking data from Terraform, you need to be sure that the folder exists for your target

mkdir -p ../bundle/.databricks/bundle/prod

And after that, we can save their output:

terraform output -json dabs_variables > ../bundle/.databricks/bundle/dev/variable-overrides.json

My recommendation: do not commit the generated .databricks/bundle/dev/variable-overrides.json. Generate it in the runner every time from the state. It keeps the repo cleaner and makes the file clearly environment-specific.

The whole pipeline can look as follows:

- name: Create DABS variable overrides from Terraform
  run: |
    mkdir -p bundle/.databricks/bundle/dev
    cd infra
    terraform output -json dabs_variables > ../bundle/.databricks/bundle/dev/variable-overrides.json

- name: Validate and deploy bundle
  working-directory: bundle
  run: |
    databricks bundle validate -t dev
    databricks bundle deploy -t dev

This pipeline will replace variables in our databricks.yml with output from Terraform

bundle:
  name: azure-keyvault-secret-scope-bundle

# Variable definition - this one will come from output variable-overrides.json file
variables:
  key_vault_dns_name:
    description: Azure Key Vault DNS name (e.g. <https://my-kv.vault.azure.net/>)
  key_vault_resource_id:
    description: Azure Key Vault resource ID

resources:
  secret_scopes:
    secret_scope_azure:
      name: test-secrets-azure-backend
      backend_type: AZURE_KEYVAULT
      keyvault_metadata:
        resource_id: ${var.key_vault_resource_id} #this one will come from output variable-overrides.json file
        dns_name: ${var.key_vault_dns_name} #this one will come from output variable-overrides.json file

targets:
  prod:
    mode: production
    default: true

Remember that you can always validate the replacement by using databricks bundle validate --output json

What to say about Terraform “inside” bundles

One easy-to-miss background detail:

• Bundles were originally built on top of the Databricks Terraform provider, but newer Databricks CLI versions support a “direct deployment engine” that does not depend on Terraform and is expected to become the default over time.

Even if you still use Bundle with Terraform, it keeps its own DABS state. Terraform mentioned here is related to the state used to deploy general Azure infrastructure.

Hubert Dudek (author)

If you like this blog post, consider buying me a coffee :-) https://ko-fi.com/hubertdudek