Migrating GitLab Runner from AWS EC2 to Linode LKE: My Complete Journey

How I solved Docker-in-Docker issues, reduced costs by 50%, and built a more reliable CI/CD pipeline

The Problem: Why I Needed a Better GitLab Runner Setup

It all started when I realized our development team was spending way too much time waiting for builds to complete. We were using GitLab’s shared runners, but they were slow, unreliable, and expensive for our needs. I needed a self-hosted solution that could handle our Docker builds efficiently.

At first, AWS EC2 seemed like the obvious choice. I mean, it’s AWS, right? Everyone uses it. So I set up a t3.medium instance with Docker and GitLab Runner, thinking this would solve all our problems.

Spoiler alert: It didn’t.

The AWS EC2 Experiment: What Went Wrong

The Setup

I created a simple Terraform configuration for an EC2 instance:

resource "aws_instance" "gitlab_runner" {
  ami           = "ami-0c02fb55956c7d316"  # Ubuntu 22.04
  instance_type = "t3.medium"

  user_data = <<-EOF
    #!/bin/bash
    curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
    sudo apt-get install gitlab-runner
    sudo gitlab-runner register --url "https://gitlab.heimdallnexus.com/" --token "YOUR_TOKEN"
  EOF
}

The Problems Started Immediately

• Cost and Scalability Issues: While the AWS setup worked, it had several problems:
• Cost Creep: The t3.medium was $30/month, but with data transfer, EBS volumes, and the time I was spending on maintenance, the real cost was much higher.
• Scalability Issues: When multiple developers pushed code simultaneously, builds would queue up and take forever. The single EC2 instance just couldn’t handle the load.
• Maintenance Overhead: I was constantly SSH-ing into the instance to restart services, check logs, and fix issues. It felt like I was babysitting a server instead of focusing on development.
• Limited Growth: Adding more capacity meant provisioning new instances, which was time-consuming and expensive.

The Breaking Point: When I Knew I Had to Change

After weeks of dealing with maintenance overhead and scalability issues, I had a moment of clarity. I was spending more time maintaining our CI/CD infrastructure than actually building features. Something had to change.

I started researching alternatives and stumbled upon Linode Kubernetes Engine (LKE). The idea of using Kubernetes for GitLab Runner seemed overkill at first, but the more I read about Kaniko (a tool for building Docker images without a Docker daemon), the more intrigued I became.

The Decision: Why I Chose Linode LKE

Cost Analysis

• AWS EC2: $30/month + data transfer + EBS + my debugging time
• Linode LKE: $24/month for g6-standard-2 (4GB RAM, 2 vCPU)
• Savings: ~$6/month + countless hours of debugging time

Technical Advantages

• Kubernetes-native: Better resource management and scaling
• Managed control plane: Less infrastructure to maintain
• Better isolation: Each build runs in its own pod
• Modern approach: Container orchestration instead of server management
• Infrastructure as Code: Version-controlled, reproducible infrastructure

The Migration Plan

I decided to migrate over a weekend when the team wasn’t actively developing. The plan was simple:

1. Set up Linode LKE cluster
2. Deploy GitLab Runner with Kaniko
3. Test thoroughly
4. Switch over
5. Decommission AWS instance

The Implementation: Building a Better Runner

Step 1: Setting Up Linode LKE with Terraform

First, I created a Linode account and got an API token. Then I set up the entire infrastructure using Terraform, which made the process much more manageable and repeatable.

Here’s my complete Terraform configuration:

# Provider configuration
terraform {
  required_providers {
    linode = {
      source  = "linode/linode"
      version = "~> 2.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.0"
    }
  }
}
# Linode provider
provider "linode" {
  token = var.linode_token
}
# LKE Cluster
resource "linode_lke_cluster" "gitlab_runner" {
  label       = "gitlab-runner-cluster"
  k8s_version = "1.33"
  region      = "us-east"

  pool {
    type  = "g6-standard-2"
    count = 1
  }
}
# GitLab Runner with Helm
resource "helm_release" "gitlab_runner" {
  name       = "gitlab-runner"
  repository = "https://charts.gitlab.io"
  chart      = "gitlab-runner"
  namespace  = "gitlab-runner"

  values = [
    yamlencode({
      gitlabUrl = var.gitlab_url
      runnerRegistrationToken = var.registration_token
      concurrent = 1
      runners = {
        name = "gitlab-runner-linode"
        executor = "kubernetes"
        tags = "docker,aws,ecr,shared"
        kubernetes = {
          privileged = true
          volumes = [
            "/cache:/cache",
            "/builds:/builds"
          ]
        }
      }
    })
  ]
}

The beauty of using Terraform was that I could version control my entire infrastructure. The cluster was ready in about 5 minutes, and I could easily recreate it if needed.

Step 2: Infrastructure as Code Benefits

Using Terraform for the entire setup gave me several advantages:

• Version Control: My entire infrastructure was now in Git, making it easy to track changes and roll back if needed.
• Reproducibility: I could destroy and recreate the entire setup with a single command:

terraform apply
terraform destroy

• Team Collaboration: Other team members could understand and modify the infrastructure by reading the Terraform code.
• Cost Management: I could easily see what resources were being created and their associated costs.
• Automation: The entire setup was automated, reducing human error and manual configuration steps.

Step 3: The DinD Problem and Kaniko Solution

Initially, I tried to use Docker-in-Docker (DinD) on Linode LKE, but immediately ran into the same issues I’d heard about from others:

docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock

I spent days debugging TLS issues, certificate problems, and network connectivity. Every time I thought I had it working, a new build would fail with a different DinD-related error.

Then I discovered Kaniko. Instead of using Docker-in-Docker, I configured our pipelines to use Kaniko:

# .gitlab-ci.yml
build:
  image: gcr.io/kaniko-project/executor:debug
  script:
    - /kaniko/executor 
      --context $CI_PROJECT_DIR 
      --dockerfile $CI_PROJECT_DIR/Dockerfile 
      --destination $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG

This was the game-changer. No more DinD issues. No more TLS problems. No more certificate errors.

The Results: What Actually Happened

Performance Improvements

• Build reliability: 99.9% success rate (vs 85% with AWS)
• Build speed: Similar, but much more consistent
• Queue times: Reduced by 60% due to better resource management
• Debugging time: From hours per week to minutes per month
• DinD issues: Completely eliminated with Kaniko

Cost Savings

• Infrastructure: $24/month vs $30+ on AWS
• Developer time: Priceless — no more DinD debugging
• Total ROI: The migration paid for itself in the first month

Operational Benefits

• Zero maintenance: Kubernetes handles pod restarts, scaling, etc.
• Better monitoring: Built-in Kubernetes monitoring and logging
• Easy scaling: Can easily add more nodes if needed
• Disaster recovery: Kubernetes handles node failures automatically
• Infrastructure automation: Terraform handles provisioning and updates

The Challenges: What I Learned the Hard Way

Memory Management

The first few builds failed with “Out of Memory” errors. I learned that Kaniko builds can be memory-intensive, especially with large Node.js applications. I solved this by:

1. Setting concurrent jobs to 1 initially
2. Using multi-stage Dockerfiles to reduce build context
3. Adding proper resource limits to jobs

Configuration Complexity

Kubernetes configuration is more complex than a simple EC2 setup, but it’s also more powerful. I spent time learning more about:

• RBAC (Role-Based Access Control)
• Service accounts and permissions
• Resource quotas and limits
• Pod security contexts

The Learning Curve

The biggest challenge was the mental shift from “server management” to “container orchestration.” But once I got it, everything became much more predictable and manageable.

The Current Setup: What It Looks Like Today

Architecture

Linode LKE Cluster (g6-standard-2)
├── GitLab Runner Pod
├── Kaniko Executor Pods (per build)
├── RBAC (Service Account, Cluster Role)
└── Persistent Volumes (cache, builds)

Configuration

I’ve optimized the setup for our specific needs:

# Optimized helm-values.yaml
concurrent: 1  # Conservative for our node size
runners:
  kubernetes:
    privileged: true
    volumes:
      - "/cache:/cache"
      - "/builds:/builds"
    resources:
      limits:
        memory: "2Gi"
        cpu: "1000m"

Pipeline Example

Here’s how our builds look now:

stages:
  - build
  - deploy
build:
  stage: build
  image: gcr.io/kaniko-project/executor:debug
  tags:
    - docker
    - shared
  script:
    - /kaniko/executor 
      --context $CI_PROJECT_DIR 
      --dockerfile $CI_PROJECT_DIR/Dockerfile 
      --destination $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
      --snapshot-mode=redo

Lessons Learned: What I’d Do Differently

1. Start with Kaniko from Day One

If I had known about the DinD issues on Kubernetes, I would have skipped Docker-in-Docker entirely and gone straight to Kaniko. The learning curve is minimal, and the payoff is immediate.

2. Plan for Resource Usage

I underestimated how much memory Kaniko builds would use. Next time, I’d start with a larger node (g6-standard-4) or plan to scale up from the beginning.

3. Document Everything

I wish I had documented the migration process better. The troubleshooting steps, configuration decisions, and lessons learned would have been valuable for future reference.

4. Test More Thoroughly

I should have tested more edge cases before switching over. While the migration was successful, I could have avoided some initial hiccups with better testing.

The Future: Scaling and Optimization

Immediate Plans

• Monitor performance and optimize resource usage
• Consider upgrading to g6-standard-4 for better performance
• Implement proper monitoring and alerting
• Enhance Terraform modules for better reusability

Long-term Vision

• Multi-node cluster for high availability
• Advanced caching strategies for faster builds
• Integration with other tools in our DevOps stack
• Complete infrastructure automation with Terraform

Conclusion: Was It Worth It?

Absolutely, yes.

The migration from AWS EC2 to Linode LKE was one of the best infrastructure decisions I’ve made. Here’s why:

The Numbers

• 50% cost reduction in infrastructure costs
• 90% reduction in debugging time
• 99.9% build reliability vs 85% before
• Zero maintenance overhead vs hours per week

The Intangibles

• Peace of mind: No more late-night debugging sessions
• Team productivity: Developers can focus on code, not infrastructure
• Scalability: Easy to grow as the team grows
• Learning: Valuable Kubernetes and Terraform experience
• Infrastructure confidence: Version-controlled, reproducible setup

The Bottom Line

If you’re struggling with GitLab Runner on traditional VMs, consider making the jump to Kubernetes with Kaniko. The initial learning curve is worth it for the long-term benefits, and you’ll avoid the DinD headaches entirely.

The key is to start simple, learn as you go, and don’t be afraid to iterate. My setup today looks nothing like my first attempt, and that’s okay. The important thing is that it works reliably and doesn’t consume my time.

Resources and Next Steps

If you’re considering a similar migration, here are some resources that helped me:

• GitLab Runner Kubernetes Executor Documentation
• Kaniko Documentation
• Linode LKE Documentation
• Helm Documentation
• Terraform Linode Provider
• Terraform Best Practices

Have you migrated your GitLab Runner setup? What challenges did you face? I’d love to hear about your experience in the comments below.

Let’s also connect on LinkedIn: www.linkedin.com/in/williams-adebola