Stop Writing YAML: Build Your CI/CD Pipelines in Real Code with Dagger

Last Tuesday I spent three hours debugging a CI pipeline. Not my code — the code was fine. I was debugging the 200-line GitHub Actions YAML file that tests the code. Someone wrote it 18 months ago, nobody fully understood it anymore, and it refused to pass despite nothing changing in the application.

Here’s the bug. See if you can spot it:

steps:
  - name: Run tests
    run: go test ./...
  - name: Build binary
     run: go build -o myapp .

One extra space before run on the last line. That's it. YAML doesn't tell you where the problem is — it just doesn't work. No type checking, no compiler error, no red squiggly line in your editor. You find out by pushing to CI, waiting five minutes, reading a cryptic error, guessing, and pushing again.

That afternoon, I rewrote the pipeline in Go using Dagger. It ran on my laptop in 8 seconds. I pushed it. It ran in GitHub Actions. Same result. Same time. No YAML. No guessing.

This article explains what Dagger is, why it exists, and how to use it — with real examples from a real project.

The Problem: CI/CD Is Stuck in 2015

Every major CI/CD platform — GitHub Actions, GitLab CI, Jenkins, CircleCI — uses the same model. You write a configuration file that describes your pipeline. The platform reads it, runs it on a cloud server, and reports back.

This model has three fundamental problems.

YAML Isn’t a Programming Language

YAML has no functions, no real variables, no loops, no error handling, no imports, and no tests. Every complex pipeline eventually becomes a wall of copy-pasted steps with bash scripts embedded in multiline strings. Your IDE can check for syntax errors, but it can’t tell you that your pipeline logic is wrong.

When a pipeline grows past 50 lines, you start fighting the format instead of solving the problem. When it grows past 200 lines, nobody wants to touch it.

You Can’t Run It Locally

When your pipeline fails, the debugging loop looks like this:

1. Change the YAML (30 seconds)
2. Commit and push (15 seconds)
3. Wait for a CI runner to pick it up (30–120 seconds)
4. Wait for the pipeline to run (3–10 minutes)
5. Read the logs (60 seconds)
6. Realize the fix was wrong (5 seconds of quiet despair)
7. Go back to step 1

Average cycle: 5–10 minutes per attempt. Average attempts to fix a CI issue: 3–8. Total time wasted: 15–80 minutes on something that should take 30 seconds with a local run.

GitHub Actions workflows don’t run on your laptop. GitLab CI files don’t run on your laptop. You’re developing blind — the feedback loop is push-and-pray.

Vendor Lock-In Is Real

Your pipeline logic — install dependencies, run tests, build a binary, push a Docker image — is universal. But expressing it differs wildly across platforms. GitHub Actions uses uses: and run:. GitLab CI uses image: and script:. CircleCI has its own structure. Jenkins uses Groovy. The differences aren't just cosmetic — they extend to how you define service containers, secrets, caching, matrix builds, and conditional logic.

Switching providers means rewriting everything from scratch. More practically, it means you never switch, even when you should.

Dagger: Write Your Pipeline in Real Code

Dagger was created by Solomon Hykes, Sam Alba, and Andrea Luzzardi — the same team behind Docker. Docker solved “works on my machine” for applications. Dagger solves “works on my machine” for pipelines.

The concept: instead of writing YAML, you write your pipeline in a real programming language — Go, Python, TypeScript, Java, PHP, Rust, or Elixir — using the Dagger SDK. The SDK talks to the Dagger Engine, which runs everything inside standard OCI containers. Because it’s containers all the way down, the pipeline runs identically on your laptop, in GitHub Actions, in GitLab CI, or anywhere Docker runs.

┌──────────────────────────────────────────┐
│         YOUR PIPELINE CODE               │
│       (Go / Python / TypeScript)         │
│                                          │
│  Real functions, variables, types        │
│  IDE autocomplete and error checking     │
│  Import shared modules                   │
│  Write tests for your pipeline           │
└────────────────────┬─────────────────────┘
                     │ calls Dagger SDK
                     ▼
┌──────────────────────────────────────────┐
│           DAGGER ENGINE                  │
│                                          │
│  Translates code → container operations  │
│  Content-addressed caching by default    │
│  Built-in tracing and observability      │
└────────────────────┬─────────────────────┘
                     │ runs containers
           ┌─────────┼─────────┐
           ▼         ▼         ▼
       Your       GitHub     GitLab
       Laptop     Actions      CI
     Same code   Same code  Same code

Use Case 1: Go Microservice Pipeline

The most common scenario — build, test, and publish a Go service.

package main
import (
    "context"
    "fmt"
    "os"
    "dagger.io/dagger"
)
func main() {
    ctx := context.Background()
    // Connect to the Dagger engine
    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
    if err != nil {
        panic(err)
    }
    defer client.Close()
    // Load source code, excluding things the build doesn't need
    src := client.Host().Directory(".", dagger.HostDirectoryOpts{
        Exclude: []string{".git", "ci/", "*.md"},
    })
    // Persistent caches for Go modules and build artifacts
    goModCache := client.CacheVolume("go-mod-cache")
    goBuildCache := client.CacheVolume("go-build-cache")
    // Base container with source code and caches mounted
    golang := client.Container().
        From("golang:1.24-alpine").
        WithMountedCache("/go/pkg/mod", goModCache).
        WithMountedCache("/root/.cache/go-build", goBuildCache).
        WithDirectory("/app", src).
        WithWorkdir("/app")
    // Run tests
    fmt.Println("Running tests...")
    _, err = golang.
        WithExec([]string{"go", "test", "-v", "-race", "./..."}).
        Stdout(ctx)
    if err != nil {
        fmt.Println("Tests failed!")
        os.Exit(1)
    }
    // Build a statically-linked binary
    fmt.Println("Building binary...")
    builder := golang.
        WithEnvVariable("CGO_ENABLED", "0").
        WithEnvVariable("GOOS", "linux").
        WithExec([]string{
            "go", "build",
            "-ldflags", "-s -w",
            "-o", "/app/bin/server",
            "./cmd/server",
        })
    // Package into a minimal production image
    fmt.Println("Building production image...")
    prodImage := client.Container().
        From("gcr.io/distroless/static-debian12").
        WithFile("/server", builder.File("/app/bin/server")).
        WithEntrypoint([]string{"/server"}).
        WithExposedPort(8080)
    // Publish
    ref, err := prodImage.Publish(ctx, "ghcr.io/myorg/myservice:latest")
    if err != nil {
        fmt.Println("Publish failed!")
        os.Exit(1)
    }
    fmt.Printf("Published: %s\n", ref)
}

Run it locally with instant feedback:

dagger call build-and-publish

Then wire it into GitHub Actions — the entire CI file:

# .github/workflows/ci.yml
name: CI
on: push
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dagger/dagger-for-github@v5
        with:
          verb: call
          args: build-and-publish

Five lines of YAML. All the logic lives in Go code you already tested on your laptop.

Use Case 2: Integration Tests with Real Databases

This is where Dagger pulls ahead dramatically. Your tests need a PostgreSQL database and a Redis cache. In traditional CI, you’d configure service containers in platform-specific YAML (and each platform does it differently). In Dagger, it’s just code:

func runIntegrationTests(ctx context.Context, client *dagger.Client, src *dagger.Directory) error {
    // Spin up real PostgreSQL and Redis containers
    postgres := client.Container().
        From("postgres:16-alpine").
        WithEnvVariable("POSTGRES_PASSWORD", "testpass").
        WithEnvVariable("POSTGRES_DB", "myapp_test").
        WithExposedPort(5432).
        AsService()

   redis := client.Container().
        From("redis:7-alpine").
        WithExposedPort(6379).
        AsService()
    // Run tests against real dependencies - not mocks
    _, err := client.Container().
        From("golang:1.24-alpine").
        WithMountedCache("/go/pkg/mod", client.CacheVolume("go-mod")).
        WithDirectory("/app", src).
        WithWorkdir("/app").
        WithServiceBinding("postgres", postgres).
        WithServiceBinding("redis", redis).
        WithEnvVariable("DATABASE_URL",
            "postgres://postgres:testpass@postgres:5432/myapp_test?sslmode=disable").
        WithEnvVariable("REDIS_URL", "redis://redis:6379").
        WithExec([]string{"go", "test", "-v", "-tags=integration", "./..."}).
        Stdout(ctx)
    return err
}

Dagger spins up real Postgres and Redis containers, your tests connect via service bindings, and everything gets cleaned up automatically when the pipeline finishes. You run this on your laptop and get the exact same containers, the exact same network topology, and the exact same test results as CI. No more “it passes locally but fails in the pipeline” — because “locally” and “the pipeline” are the same thing.

Use Case 3: Multi-Platform Docker Builds

Building images for both AMD64 (Intel/AMD servers) and ARM64 (AWS Graviton, Apple Silicon):

func buildMultiPlatform(ctx context.Context, client *dagger.Client, src *dagger.Directory) error {
    platforms := []dagger.Platform{"linux/amd64", "linux/arm64"}
    variants := make([]*dagger.Container, len(platforms))
    for i, platform := range platforms {
        // Build the Go binary targeting this specific platform
        builder := client.Container(dagger.ContainerOpts{Platform: platform}).
            From("golang:1.24-alpine").
            WithDirectory("/app", src).
            WithWorkdir("/app").
            WithEnvVariable("CGO_ENABLED", "0").
            WithExec([]string{
                "go", "build", "-ldflags", "-s -w",
                "-o", "/app/bin/server", "./cmd/server",
            })
        // Package into a distroless image for this platform
        variants[i] = client.Container(dagger.ContainerOpts{Platform: platform}).
            From("gcr.io/distroless/static-debian12").
            WithFile("/server", builder.File("/app/bin/server")).
            WithEntrypoint([]string{"/server"}).
            WithExposedPort(8080)
    }
    // Publish as a multi-platform manifest
    _, err := client.Container().
        Publish(ctx, "ghcr.io/myorg/myservice:latest",
            dagger.ContainerPublishOpts{PlatformVariants: variants})
    return err
}

Both the builder and the production container receive the Platform option so the Go toolchain cross-compiles correctly, and the final image has the right architecture metadata. Try expressing that cleanly in YAML.

Use Case 4: Monorepo with Conditional Builds

You have a monorepo with three services. You only want to build the ones whose code actually changed:

func buildChangedServices(ctx context.Context, client *dagger.Client) error {
    services := []struct {
        Name string
        Path string
    }{
        {"api-gateway", "services/api-gateway"},
        {"user-service", "services/user-service"},
        {"notification-service", "services/notification-service"},
    }

    // Get changed files from the last commit
    // Use merge-base for PRs to handle merge commits correctly
    changedFiles, err := client.Container().
        From("alpine/git").
        WithDirectory("/repo", client.Host().Directory(".")).
        WithWorkdir("/repo").
        WithExec([]string{"git", "diff", "--name-only", "HEAD~1", "--diff-filter=ACMRT"}).
        Stdout(ctx)
    if err != nil {
        // Fallback: build everything if git diff fails (first commit, shallow clone, etc.)
        changedFiles = "services/"
    }
    for _, svc := range services {
        if !containsPrefix(changedFiles, svc.Path) {
            fmt.Printf("Skipping %s (no changes)\n", svc.Name)
            continue
        }
        fmt.Printf("Building %s...\n", svc.Name)
        src := client.Host().Directory(svc.Path)
        _, err := client.Container().
            From("golang:1.24-alpine").
            WithDirectory("/app", src).
            WithWorkdir("/app").
            WithExec([]string{"go", "test", "./..."}).
            WithExec([]string{"go", "build", "-o", "/app/bin/" + svc.Name, "."}).
            Stdout(ctx)
        if err != nil {
            return fmt.Errorf("%s failed: %w", svc.Name, err)
        }
    }
    return nil
}
func containsPrefix(files, prefix string) bool {
    for _, line := range strings.Split(files, "\n") {
        if strings.HasPrefix(strings.TrimSpace(line), prefix) {
            return true
        }
    }
    return false
}

This is just Go. Loops, conditionals, string manipulation, error handling — things that are painful or impossible in YAML. The fallback on git diff failure handles edge cases (first commit, shallow clones, merge commits with multiple parents) that would silently break a YAML-only approach.

Use Case 5: Reusable Pipeline Modules

Your organization has 30 Go services. They all need the same pipeline: lint, test, build, scan, publish. Instead of copying a YAML file into every repo, you create a shared Dagger module:

// Shared module: github.com/myorg/ci-modules/golang
package golang
import (
    "context"
    "dagger.io/dagger"
)
type GoPipeline struct {
    Source    *dagger.Directory
    GoVersion string
    Registry  string
}
func (p *GoPipeline) Lint(ctx context.Context, client *dagger.Client) error {
    _, err := p.baseContainer(client).
        WithExec([]string{
            "go", "install",
            "github.com/golangci/golangci-lint/cmd/golangci-lint@latest",
        }).
        WithExec([]string{"golangci-lint", "run", "--timeout", "3m"}).
        Stdout(ctx)
    return err
}
func (p *GoPipeline) Test(ctx context.Context, client *dagger.Client) error {
    _, err := p.baseContainer(client).
        WithExec([]string{
            "go", "test", "-v", "-race", "-coverprofile=coverage.out", "./...",
        }).
        Stdout(ctx)
    return err
}
func (p *GoPipeline) Build(ctx context.Context, client *dagger.Client) (*dagger.Container, error) {
    binary := p.baseContainer(client).
        WithEnvVariable("CGO_ENABLED", "0").
        WithExec([]string{"go", "build", "-ldflags", "-s -w", "-o", "/app/bin/server", "."})
    return client.Container().
        From("gcr.io/distroless/static-debian12").
        WithFile("/server", binary.File("/app/bin/server")).
        WithEntrypoint([]string{"/server"}), nil
}
func (p *GoPipeline) baseContainer(client *dagger.Client) *dagger.Container {
    return client.Container().
        From("golang:" + p.GoVersion + "-alpine").
        WithMountedCache("/go/pkg/mod", client.CacheVolume("go-mod")).
        WithMountedCache("/root/.cache/go-build", client.CacheVolume("go-build")).
        WithDirectory("/app", p.Source).
        WithWorkdir("/app")
}

Every service consumes it in about 10 lines:

func main() {
    ctx := context.Background()
    client, _ := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
    defer client.Close()

    pipeline := &golang.GoPipeline{
        Source:    client.Host().Directory("."),
        GoVersion: "1.24",
        Registry:  "ghcr.io/myorg",
    }
    pipeline.Lint(ctx, client)
    pipeline.Test(ctx, client)
    pipeline.Build(ctx, client)
}

Update the shared module once, and every service gets the improvement on its next run. This is how platform teams scale CI/CD across an organization — with real code, versioned and tested like any other library, not copy-pasted YAML.

Why Second Runs Are Fast: Content-Addressed Caching

Dagger caches every operation by default using content-addressed hashing. It doesn’t check whether “the same command ran before” — it checks whether the inputs that would affect the output have changed. File contents, environment variables, base images, and commands all factor into the cache key.

Here’s what that looks like in practice:

The CacheVolume for Go modules persists across runs — first run downloads everything, every subsequent run reuses it. This works on your laptop and in CI if you configure the Dagger cache to persist between workflow runs.

Built-In Observability

Every pipeline run produces a structured trace showing what happened, how long each step took, and where failures occurred:

┌─ Pipeline
├─┬─ Pull golang:1.24-alpine                    [0.0s] ✅ cached
├─┬─ Mount source code                          [0.1s] ✅
├─┬─ go test ./...                              [3.4s] ✅
│     ├── internal/handler      PASS   0.234s
│     ├── internal/store        PASS   0.112s
│     ├── internal/middleware   PASS   0.089s
│     └── cmd/server            PASS   0.451s
├─┬─ go build -o server                         [2.8s] ✅
├─┬─ Build distroless image                     [1.1s] ✅
└─┬─ Publish to ghcr.io                         [4.2s] ✅
      └── ghcr.io/myorg/myservice:latest@sha256:abc123
Total: 11.6s

When something fails, you see exactly which step broke, with full logs, without scrolling through hundreds of lines of output. Dagger Cloud provides a web UI for persistent, shareable traces if you need that across a team.

Migrating Without Rewriting Everything

You don’t have to go all-in on day one. The migration path is gradual.

Phase 1 — Move your most painful pipeline step to Dagger. Usually that’s “build and test.” Keep the rest of your YAML as-is, and replace one step with a dagger call invocation. You immediately get local reproducibility for that step.

Phase 2 — Move all pipeline logic into Dagger. Your CI YAML shrinks to a thin wrapper — checkout code, call Dagger, done. All logic lives in code you can test, review, and refactor like any other part of your codebase.

Phase 3 — Run it everywhere. The same Dagger pipeline now works on every developer’s laptop, in GitHub Actions, in GitLab CI, on self-hosted runners, or in Dagger Cloud. Switching CI providers becomes a five-minute YAML swap instead of a multi-week rewrite.

When to Use Dagger (and When Not To)

Dagger is the right tool when your pipelines are complex enough that YAML becomes a liability. Here’s a practical split:

Dagger makes sense when you have pipelines with 20+ steps, the “works in CI but not locally” problem is costing your team real time, you need to run the same pipeline across multiple CI providers, you’re managing a monorepo with conditional build logic, your platform team supports 30+ repositories, integration tests require real service dependencies, you need multi-platform builds, or your pipeline logic should have its own unit tests.

Sticking with YAML is fine when your pipeline is simple (test and deploy in 10 lines), it rarely changes, you’re committed to a single CI provider, you’re a solo developer or small team, and the debugging loop doesn’t bother you.

The Ecosystem

Daggerverse (daggerverse.dev) is a registry of community-built modules. Need to deploy to Kubernetes, scan with Trivy, or push to AWS ECR? Import a module instead of writing it from scratch — think npm for CI/CD components.

Dagger Cloud provides persistent caching across CI runs, visual pipeline traces, and team collaboration. Free tier available.

Language SDKs cover Go, Python, TypeScript, PHP, Java, Elixir, and Rust. Write your pipeline in whatever your team already knows.

Getting Started

brew install dagger/tap/dagger
dagger init --sdk=go
# Write your pipeline. Run it locally. Push with confidence.

OR

curl -fsSL https://dl.dagger.io/dagger/install.sh | BIN_DIR=/usr/local/bin sudo -E sh

Dagger is open source under Apache 2.0. The source is at github.com/dagger/dagger and the documentation at docs.dagger.io.

If you’ve ever spent an hour debugging a YAML indent, pushed six commits to get a CI step right, or wished you could just run the pipeline on your laptop — that’s the experience Dagger eliminates. It doesn’t replace your CI provider. It replaces the YAML you write for your CI provider, and turns your pipeline into real, testable, portable code.