Running LLM Inference Safely with Docker Sandboxes and Model Runner

Docker shipped two features in 2026 that fundamentally change how we run AI workloads locally: Docker Model Runner and Docker Sandboxes. Model Runner lets you pull and run LLMs the same way you pull container images. Sandboxes give you process-level isolation for untrusted AI agent code. Together, they solve the two hardest problems in local AI development: dependency hell and security.

I’ve spent the last year building GPU infrastructure tools for Kubernetes — a GPU autoscaler for KEDA, an OpenTelemetry GPU receiver, and GPU NUMA topology scheduling for Volcano. All of that work happens in containers. Docker is the runtime. This post walks through how Docker’s new AI features fit into a production GPU workflow.

Docker Model Runner: LLMs as First-Class Docker Citizens

Docker Model Runner uses the same OCI registry infrastructure as container images. You interact with models the same way you interact with images — pull, list, run, remove.

Pull a model

docker model pull ai/llama3.2:1B-Q8_0

This downloads the quantized model weights from Docker Hub, just like docker pull for an image. Models are content-addressable and layer-deduplicated — if two models share base weights, you only download the diff.

List local models

docker model ls

NAME                      SIZE      CREATED
ai/llama3.2:1B-Q8_0      1.3 GB    2 minutes ago
ai/mistral:7B-Q4_K_M      4.1 GB    1 hour ago

Run inference

docker model run ai/llama3.2:1B-Q8_0 "Explain NUMA topology in GPU scheduling"

Under the hood, Docker Model Runner uses llama.cpp with automatic hardware detection — Apple Silicon (Metal/MLX), NVIDIA CUDA, or CPU fallback. The model runs as a Docker-managed process with the same lifecycle semantics as a container: start, stop, logs, resource limits.

OpenAI-compatible API

Model Runner also exposes a local API endpoint that’s compatible with the OpenAI SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:12434/engines/llama3.2/v1/",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="ai/llama3.2:1B-Q8_0",
    messages=[{"role": "user", "content": "Explain GPU memory fragmentation"}]
)
print(response.choices[0].message.content)

This means your application code doesn’t change between local development (Model Runner) and production (OpenAI, Anthropic, or self-hosted vLLM). Same SDK, same API shape, different backend.

Why This Matters for GPU Engineers

If you run GPU inference in production on Kubernetes, you’ve dealt with all of this:

1. Dependency conflicts — CUDA 12.x vs 11.x, cuDNN version mismatches, PyTorch builds tied to specific CUDA versions
2. Image bloat — a typical vLLM image is 15+ GB with the full CUDA toolkit, Python, and model weights
3. Local dev-prod gap — your MacBook doesn’t have an A100, so you can’t test inference locally without a cloud GPU

Docker Model Runner sidesteps all three for the development inner loop. You don’t build a 15GB container image to test a prompt template change. Pull the model, run it locally, iterate on your application code, then deploy the production-grade containerized version to your GPU cluster.

The development flow looks like this:

Local (Docker Desktop)                Production (Kubernetes)
┌──────────────────────────┐         ┌─────────────────────────────┐
│ docker model run          │         │ vLLM container (A100/H100)  │
│ (Apple Silicon / local GPU│  ──→   │ + keda-gpu-scaler           │
│  via Model Runner)        │         │ + otel-gpu-receiver         │
│                           │         │ + Volcano GPU NUMA sched    │
│ Same OpenAI API shape     │         │ + Same OpenAI API shape     │
└──────────────────────────┘         └─────────────────────────────┘

Same application code. Same API. Different runtime. Docker on both ends.

Docker Sandboxes: Isolation for AI Agents

The second piece is Docker Sandboxes. If you’re building agentic workflows — LLMs that execute code, call APIs, modify files, or orchestrate other tools — you need isolation. Running exec() on LLM-generated code in your host environment is a security incident waiting to happen.

Docker Sandboxes provide:

• Filesystem isolation — the agent sees its own filesystem, can’t touch your host
• Network isolation — configurable egress rules (whitelist which APIs the agent can call)
• Resource limits — CPU, memory, and GPU constraints per sandbox
• Ephemeral execution — sandbox is destroyed after the task completes, no state leaks

Example: Isolating an AI coding agent

# docker-compose.sandbox.yml
services:
  coding-agent:
    image: my-coding-agent:latest
    sandbox:
      enabled: true
      network:
        egress:
          - "api.openai.com:443"
          - "pypi.org:443"
          - "github.com:443"
      resources:
        memory: 4g
        cpus: 2
    volumes:
      - ./workspace:/workspace  # Agent can only access this directory
    environment:
      - MODEL_ENDPOINT=http://host.docker.internal:12434

The agent can:

• Read and write files in /workspace
• Call OpenAI’s API, install Python packages, and access GitHub
• Use up to 4GB RAM and 2 CPU cores

The agent cannot:

• Access your SSH keys, browser cookies, or credential files
• Make network requests to arbitrary hosts
• Consume unbounded resources
• Persist state between runs (unless you mount a volume)

This is the same isolation model Kubernetes provides with SecurityContexts, PodSecurityStandards, and NetworkPolicies — but for local development. Your agent code runs in the same security boundary locally that it’ll run in on your cluster.

Why this matters for production

In production, I run GPU inference containers with strict security policies:

# Kubernetes PodSecurityContext
securityContext:
  runAsNonRoot: true
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]

Docker Sandboxes let me apply the same security posture during local development. The gap between “works on my machine” and “works in production” shrinks because the isolation model is consistent.

Connecting It All: The GPU AI Stack on Docker

Here’s the full picture of how Docker fits into GPU/AI infrastructure, from local development to production:

Local Development (Docker Desktop)

ToolRoleDocker Model RunnerPull and run LLMs locallyDocker SandboxesIsolate AI agent code executionDocker ScoutScan GPU container images for CVEsDocker ExtensionsGPU monitoring dashboard (real-time NVML metrics)Docker ComposeMulti-service GPU development (inference + app + monitoring)

Production (Kubernetes)

ToolRoleDocker containers (via containerd)Runtime for inference servers (vLLM, Triton)keda-gpu-scalerAutoscale based on real GPU utilizationotel-gpu-receiverGPU metrics → OpenTelemetry pipelineVolcano GPU NUMA schedulingPlace training jobs on NUMA-aligned GPUsNetworkPolicy + PodSecurityProduction isolation (same model as Sandboxes)

The container is the common unit from your laptop to the GPU cluster. Docker owns the development experience; Kubernetes owns the production orchestration. Both use the same images, same security boundaries, same observability stack.

What I’d Like to See Next

Docker Model Runner is great for single-model inference. A few things that would make it production-grade for local GPU development:

1. GPU memory reporting — docker model stats showing VRAM usage per model, like docker stats for containers
2. Multi-model serving — run multiple models concurrently with resource limits per model (mimicking what Triton Inference Server does in production)
3. OTel integration — emit inference latency, throughput, and token/second metrics to an OpenTelemetry collector. My otel-gpu-receiver already collects hardware-level GPU metrics — having application-level model metrics would complete the picture.
4. Sandbox GPU passthrough — enable GPU access inside sandboxes for agents that need to run inference locally (currently sandboxes are CPU-only)

I’ve shared this feedback in the Docker community forums. If you’re building GPU/AI infrastructure on Docker, I’d love to hear what gaps you’re hitting.

Getting Started

# 1. Update Docker Desktop to 4.40+
# 2. Enable Model Runner in Settings → Features in Development

# Pull a model
docker model pull ai/llama3.2:1B-Q8_0

# Run inference
docker model run ai/llama3.2:1B-Q8_0 "What is NUMA topology?"

# Use the OpenAI-compatible API from your code
curl http://localhost:12434/engines/llama3.2/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/llama3.2:1B-Q8_0",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Docker is becoming the full-stack runtime for AI development. Model Runner handles inference. Sandboxes handle security. Scout handles supply chain. And when you’re ready for production, the same containers deploy to your GPU Kubernetes cluster.

The gap between local AI development and production GPU infrastructure just got a lot smaller.

Pavan Madduri is a Senior Cloud Platform Engineer at W.W. Grainger, Inc., CNCF Golden Kubestronaut, and Oracle ACE Associate. He maintains keda-gpu-scaler and otel-gpu-receiver, contributed GPU NUMA topology scheduling to Volcano, and is a Dragonfly Community Member. His work focuses on GPU/AI infrastructure for Kubernetes.