Self-Hosted Github Actions in EKS with Spot Instance

In this post, we will be exploring ways to self-host GitHub actions. Our major focus will be

• Runners should be able to autoscale
• It should be as cost-effective as possible (SPOT Instances)

Pre-requisite for the blogs

• Access to AWS account with ability to create VPC, EKS Cluster
• Helm Installation
• Kubectl
• Github Account
• Terraform

Step 1 : Lets create a EKS Cluster using Terraform

In this step, we provision a highly available Amazon EKS cluster using the terraform-aws-modules/eks/aws module. This setup includes VPC, subnets, NAT Gateway, and EKS node groups — optimized for scalability and cost with spot and on-demand instances.

We will be utilizing spot instance for the hosting runners required for the github actions. Sample terraform code looks like

locals {
  cluster_name = "${var.project}-eks"
  common_tags = {
    Project    = "runner"
    Enviroment = "dev"
    Owner      = "Bnay14"
  }
  eks_autoscaling_tags = merge(
    local.common_tags,
    {
      "k8s.io/cluster-autoscaler/enabled"               = "true"
      "k8s.io/cluster-autoscaler/${local.cluster_name}" = "owned"
    }
  )
}

data "aws_availability_zones" "available" {}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.21.0"
  name    = "${var.project}-vpc"
  cidr    = var.cidr
  azs     = slice(data.aws_availability_zones.available.names, 0, 3)

  public_subnets  = [for i in range(3) : cidrsubnet("${var.cidr}", 8, i)]
  private_subnets = [for i in range(3) : cidrsubnet("${var.cidr}", 8, i + 10)]

  enable_nat_gateway = true
  single_nat_gateway = true # Each AZ gets its own NAT Gateway

  enable_dns_hostnames = true
  enable_dns_support   = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = 1
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
  }

  tags = local.common_tags
}

module "eks" {
  source                                   = "terraform-aws-modules/eks/aws"
  version                                  = "~>20.31"
  cluster_name                             = "${var.project}-eks"
  cluster_version                          = "1.33"
  vpc_id                                   = module.vpc.vpc_id
  subnet_ids                               = module.vpc.private_subnets
  control_plane_subnet_ids                 = module.vpc.private_subnets
  cluster_endpoint_public_access           = true
  enable_cluster_creator_admin_permissions = true
  cluster_endpoint_public_access_cidrs     = ["X.X.X.X.X/32"] # Change this your IP 
  cluster_enabled_log_types                = []
  eks_managed_node_groups = {
    on_demand_nodes = {
      ami_type      = "BOTTLEROCKET_x86_64"
      instance_type = ["t3.small"]
      min_size      = 1
      max_size      = 2
      tags          = local.eks_autoscaling_tags

    }
    actions-nodes = {
      ami_type      = "BOTTLEROCKET_x86_64"
      instance_type = ["t3.small", "t3a.small"]
      desired_size = 0
      min_size      = 0
      max_size      = 3
      capacity_type = "SPOT"
      tags          = local.eks_autoscaling_tags
      taints = [{
        key    = "action-nodes"
        value  = "true"
        effect = "NO_SCHEDULE"
        }
      ]
      labels = {
        action-nodes = true
      }
    }
  }
  cluster_addons = {
    coredns                = {}
    eks-pod-identity-agent = {}
    kube-proxy             = {}
    vpc-cni                = {}

  }
  tags       = local.common_tags
  depends_on = [ module.vpc ]
}

Step 2: Create a pod identity for Cluster Autoscaler

In order to provide appropriate roles to cluster Autoscaler, we are going to use pod identity instead of IRSA.

More about pod identity. || Official Docs

We can do this using terraform. Sample terraform code

module "cluster_autoscaler_pod_identity" {
  source = "terraform-aws-modules/eks-pod-identity/aws"
  version = "1.11.0"
  name = "${var.project}-eks-clusterAutoscaler"
  attach_cluster_autoscaler_policy = true 
  cluster_autoscaler_cluster_names = ["module.eks.cluster_name"]
  associations = {
    eks = {
    cluster_name = module.eks.cluster_name
    namespace = "cluster-autoscaler"
    service_account = "cluster-autoscaler-sa"

    }
  }
  tags = local.common_tags
  depends_on = [module.eks]
}

From above, module we need to make sure two things.

• We need to deploy cluster autoscaler in namespace cluster-autoscaler.
• We need to have a service account named cluster-autoscaler-sa.

If you are deploying in different service account, and name space changes can be made inside association block.

Step 3: Accessing EKS cluster using kubectl

For this step, your IAM account should have required permission.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "eks:DescribeCluster"
      ],
      "Resource": "arn:aws:eks:<region>:<account-id>:cluster/<cluster-name>"
    }
  ]
}

More about permissions and other requirement can be found of official docs.

$ aws eks update-kubeconfig --region <region> --name <cluster_name>

Step 4: Install Cluster Autoscaler

In this step we will be installing cluster autoscaler. This will basically take care of scaling up nodes whenever there are pending pods in cluster.

$ helm repo add cluster-autoscaler https://kubernetes.github.io/autoscaler

Lets us create values.yaml for the custom values we will be using,

autoDiscovery:
  clusterName: runner-eks # Name of EKS cluster

awsRegion: us-east-1 # Region

extraArgs:
  balance-similar-node-groups: true
  scale-down-enabled: true
  scale-down-delay-after-add: 2m
  scale-down-unneeded-time: 3m
  scale-down-utilization-threshold: "0.4"
  skip-nodes-with-system-pods: true
  expander: least-waste
  max-node-provision-time: 15m
  scan-interval: 30s
  max-empty-bulk-delete: 10
  scale-down-delay-after-delete: 0s
  scale-down-delay-after-failure: 3m

rbac:
  create: true
  serviceAccount:
    create: true
    name: cluster-autoscaler-sa # Match this with defined in TF

resources:
  limits:
    cpu: 200m
    memory: 600Mi
  requests:
    cpu: 100m
    memory: 300Mi

priorityClassName: system-cluster-critical

nodeSelector:
  kubernetes.io/os: linux

Lets install cluster-autoscaler

$ helm upgrade --install cluster-autoscaler cluster-autoscaler/cluster-autoscaler \
  --namespace cluster-autoscaler \
  --create-namespace \
  -f values.yaml

After this you should be able to see pods up and running.

$ kubectl get pods -n cluster-autoscaler

NAME                                                         READY   STATUS    RESTARTS   AGE
cluster-autoscaler-aws-cluster-autoscaler-6c445fccfc-2rvsg   1/1     Running   0          28s

At this point we should have a cluster upa and running, also cluster autoscaling should also work fine. This means without the workload action-nodes should scale down to zero.

Step 5: Setup Authentication in Github

Before installing the actual actions-runner-controller we would be configuring authentication in github. This would enable us to connect runner to our desired github organization or repo.

There are couple of ways to manage authentication (Only can can be used at a time)

• Using Github App (Not supported for Enterprise)
• Using Personal Access Token (PAT)

We will be using PAT to configure authentication. Following is the permission required by token

# Required Scopes for Repository Runners
repo (Full control)

# Required Scopes for Organization Runners
repo (Full control)
admin:org (Full control)
admin:public_key (read:public_key)
admin:repo_hook (read:repo_hook)
admin:org_hook (Full control)
notifications (Full control)
workflow (Full control)

# Required Scopes for Enterprise Runners
admin:enterprise (manage_runners:enterprise)

Repository Runner: repo(Full Control)

Now lets keep the the generated token as a secrets object in k8s.

kubectl create namespace actions-runner-system && \
kubectl create secret generic controller-manager \
   --namespace=actions-runner-system \
   --from-literal=github_token='YOUR-PAT'

Step 6: Install Actions-Runner-Controller

We need to install cert-manager. This is required by arc.

$ helm repo add cert-manager https://charts.jetstack.io
$ helm install cert-manager cert-manager/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true \
  --wait

We will be using helm charts to install arc.

$ helm repo add actions-runner-controller https://actions-runner-controller.github.io/actions-runner-controller
$ helm upgrade --install --namespace actions-runner-system actions-runner-controller actions-runner-controller/actions-runner-controller -f values.yaml --wait

After the installation you should see a pod running in actions-runner-controller namespace.

Step 7: Configuring ARC Runner

Now we would install arc runner. We now have a CRDs where we can define the runner configuration. We will be deploying runners in seprate namespace.

$ kubectl create ns actions-runners

We want to make sure the actions nodes are configured in SPOT instance. So we would need to specify labels, and toleration in deployment. Our Runner deployment will look something like

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: github-runners
  namespace: actions-runner
spec:
  replicas: 2
  template:
    spec:
      repository: BS14/self-hosted-runner  # or use organization: your-org
      labels:
        - self-hosted
        - linux
        - x64
        - custom-runner
      env:
        - name: RUNNER_NAME_PREFIX
          value: "self-hosted-runner"
      resources:
        requests:
          cpu: "500m"
          memory: "512Mi"
        limits:
          cpu: "1"
          memory: "1Gi"
      dockerEnabled: false  # set to true if you want docker-in-docker
      nodeSelector:
        action-nodes: "true"
      tolerations:
        - key: "action-nodes"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"

Scaling Activity in SPOT Instance

On GitHub repository actions you could see two runner that would be attached.

Runners Registration in Repo Level

Step 8: Lets autoscale Runners

Instead of hardcoding number of replicas in runner, we can actually autoscaling policies. This make sure we have runner as required.

Manifest for autoscaling would be

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: github-runners
  namespace: actions-runner
spec:
  #replicas: 1  # will be defined in HorizontalRunnerAutoscaler
  template:
    spec:
      repository: BS14/self-hosted-runner  # change to your repo or use organization: your-org
      labels:
        - self-hosted
      resources:
        requests:
          cpu: "500m"
          memory: "512Mi"
        limits:
          cpu: "1"
          memory: "1Gi"
      nodeSelector:
        action-nodes: "true"
      tolerations:
        - key: "action-nodes"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: github-runners-autoscaler
  namespace: actions-runner
spec:
  scaleTargetRef:
    name: github-runners
  minReplicas: 1
  maxReplicas: 3
  metrics:
    - type: PercentageRunnersBusy
      scaleUpThreshold: "0.75"
      scaleDownThreshold: "0.25"
      scaleUpFactor: "2"
      scaleDownFactor: "0.5"

We are using PercentageRunnerBusy metrics for scaling. We can also other metrics More on Scaling

Scaled Runner

Step 9: Testing Workflow

Now lets us write a testing workflow which will run in the self-hosted runner. Sample workflow file would look something like

name: Self-Hosted Runner

on:
  push:
    branches:
      - main

jobs:
  test-runner:
    name: Verify Self-Hosted Runner
    runs-on: [self-hosted]  # Match the labels from your RunnerDeployment
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Print Runner Info
        run: |
          echo "Running on: $RUNNER_NAME"
          echo "Operating System: $RUNNER_OS"
          echo "Architecture: $RUNNER_ARCH"
          uname -a

      - name: Run Sample Task
        run: |
          echo "This is a test job running on a self-hosted GitHub Actions runner!"
          sleep 100

Workflow Output

This way we can leverage spot instances and self host the runner.

All the codes are in the repo https://github.com/BS14/self-hosted-runner

Another noteworthy mention apart from ARC would be runs-on. I have heard many good things about runs-on but haven’t tried yet. I will try this and let you know about the details.

HAPPY WORKFLOW fellow Engineers!!!