Databricks Docker: from runtime CI/CD to compliance

So far, we’ve had ML and non-ML for classic dedicated compute, but Databricks dropped 12! different Docker images. Most importantly, it is not just 12; we can also build our own routines for it. It opens the door to many new use cases. I already heard that it solved compliance problems for many companies, as now you can, for example, include certificates in your runtime. I also heard about really awesome solutions like combining DuckDB with Databricks.

If you are not yet a member of Medium, you can access the extended version on the SunnyData blog for free.

Let’s look at the list of 12 images that we can find on https://hub.docker.com/u/databricksruntime

[embed]

[embed]

First, install Docker; for Windows users, the Linux Subsystem is also useful.

wsl --install

Now, create a Dockerfile based on the standard runtime.

We will copy the test text file into the image and install DuckDB:

FROM databricksruntime/standard:16.4-LTS
USER root
RUN apt-get update && apt-get install -y --no-install-recommends \    jq \    ripgrep \    unixodbc \    curl \    ca-certificates \    unzip \    && rm -rf /var/lib/apt/lists/*
# Install DuckDB CLI manuallyARG DUCKDB_VERSION=1.4.4
RUN curl -L \    "<https://github.com/duckdb/duckdb/releases/download/v${DUCKDB_VERSION}/duckdb_cli-linux-amd64.zip>" \    -o /tmp/duckdb.zip \    && unzip /tmp/duckdb.zip -d /usr/local/bin \    && chmod +x /usr/local/bin/duckdb \    && rm /tmp/duckdb.zip
COPY hello.txt /opt/demo/hello.txt
RUN chmod 644 /opt/demo/hello.txt
RUN duckdb --version

Let’s build it locally:

docker buildx build --platform linux/amd64 -t hdudek/dbx-hello-container:0.1 .

Now test it locally by executing DuckDB:

docker run --rm hdudek/dbx-hello-container:0.1 \ duckdb -c "select 42 as answer, current_date as run_date;"

Now we need to push to the cloud. If you use Docker, you will need a higher-tier plan due to limitations for production use. Alternatively, you can use another Docker hosting service:

Let’s move to Databricks first. What we need to do is enable Docker container service through the CLI.

databricks workspace-conf set-status --json '{"enableDcs": "true"}’

Now in Databricks in cluster Advanced settings, let set our Docker Image URL and also remember about setting Access mode to dedicated/

Once the cluster starts, let’s test it by displaying /opt/demo/hello.txt in the notebook (the file which we included in our image):

and also run the DuckDB command:

Databricks Docker Use cases

What are real production use cases? I specified nine

1. Golden runtime for regulated environments

A custom container can act as an approved runtime image: labeled, built in CI, scanned, and reused across environments.

FROM databricksruntime/standard:16.4-LTS

LABEL owner="data-platform"
LABEL purpose="approved-databricks-runtime-demo"
LABEL version="2026.05.27"

2. Run a native executable next to Spark

DuckDB is a good demo because it is a native executable, not just a Python package.

ARG DUCKDB_VERSION=1.4.4
RUN curl -L \
  "<https://github.com/duckdb/duckdb/releases/download/v${DUCKDB_VERSION}/duckdb_cli-linux-amd64.zip>" \
  -o /tmp/duckdb.zip \
  && unzip /tmp/duckdb.zip -d /usr/local/bin \
  && chmod +x /usr/local/bin/duckdb \
  && rm /tmp/duckdb.zip

3. Use DuckDB to inspect files written by Spark

Spark writes the data:

df.write.mode("overwrite").parquet("file:/tmp/duckdb_demo/orders")

DuckDB inspects it:

%sh
duckdb -c "
select country, count(*) as rows, sum(amount) as total_amount
from read_parquet('/tmp/duckdb_demo/orders/*.parquet')
group by country
order by country;
"

4. Offline or restricted network environments

In restricted environments, clusters should not download random packages at runtime. The approved image already contains the tools.

RUN apt-get update && apt-get install -y --no-install-recommends \
    jq \
    ripgrep \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

5. CI/CD for the runtime, not only the code

The runtime image can be built, tagged, scanned, and promoted like application code.

docker buildx build \
  --platform linux/amd64 \
  -t hdudek/dbx-custom-container-fakers:2026.05.27 \
  --load .

6. Legacy system integration

Many enterprise pipelines still depend on command-line tools, vendor utilities, or small internal binaries.

COPY bin/dqcheck /usr/local/bin/dqcheck
RUN chmod +x /usr/local/bin/dqcheck

Notebook:

%sh
dqcheck scan --input /tmp/duckdb_demo/orders --output /tmp/dq_report.json
cat /tmp/dq_report.json | jq .

Spark can read the report:

report = spark.read.json("file:/tmp/dq_report.json")
display(report)

7. Add enterprise certificates once

COPY certs/company-root-ca.crt /usr/local/share/ca-certificates/company-root-ca.crt
RUN update-ca-certificates

8. Internal packages and private wheels

A custom container can carry an approved internal package version.

COPY dist/company_quality_rules-0.1.0-py3-none-any.whl /opt/wheels/
RUN /databricks/python3/bin/pip install --no-cache-dir \
    /opt/wheels/company_quality_rules-0.1.0-py3-none-any.whl

Notebook:

from company_quality_rules import validate_table_name

validate_table_name("customer_orders")

9. Enterprise database drivers/client tools

The container should carry the client runtime, not credentials.

RUN apt-get update && apt-get install -y --no-install-recommends \
    postgresql-client \
    unixodbc \
    && rm -rf /var/lib/apt/lists/*

TL;DR

Databricks custom containers are not mainly about replacing pip install; they are about controlling the runtime.

You can start from different Databricks base images and build an approved environment with native tools, internal wheels, certificates, database clients, and validation utilities.

The result is a runtime that can be built in CI/CD, scanned, versioned, reused across jobs, and aligned with compliance and enterprise requirements.

Now you can treat your runtime as code.

Hubert Dudek (author)

If you like this blog post, consider buying me a coffee :-) https://ko-fi.com/hubertdudek