Building EKS with Terraform: Two Errors Worth Understanding

I spent this week building a Kubernetes cluster on AWS, then deploying Jenkins (a CI/CD tool) onto it, then tearing the whole thing down again. Here’s what that actually involved, explained without assuming you already know the jargon.

What even is a cluster

A Kubernetes cluster is a set of machines, called nodes, running your applications as containers. Kubernetes is the control layer sitting on top of those nodes.

It decides which node runs which container, watches for failures, and reschedules a container elsewhere the moment one goes down, without a human stepping in to do it manually.

EKS is Amazon’s managed version of this. AWS runs the hardest parts for you, and you bring your own computers.

Jenkins automates the boring parts of software development. Instead of a developer manually testing and publishing their code every time they make a change, Jenkins does it for them. I wasn’t using it for anything real here. It’s a good stand-in for “some real application” running on a cluster.

Building it with code instead of clicking

You could click through the AWS website to set all of this up by hand. The problem is that clicking doesn’t leave a record of what you did. Rebuild it next month and you’re clicking the same buttons from memory.

Terraform lets you write down what you want in a file, and it builds it for you. Here’s the actual block that defines the cluster:

module "eks_cluster" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0"

cluster_name    = var.cluster_name
  cluster_version = "1.32"  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets  enable_irsa = true
}

Change something, edit the file, run one command. Tear it down, one command does that too.

This matters because AWS charges by the hour. A cluster costs money every hour it exists, whether you’re using it or not. One command that reliably tears everything down is the difference between a learning project costing a few cents and it quietly costing real money because something kept running.

Here’s the backend config that stores Terraform’s own record of what it built:

backend "s3" {
  bucket       = "lydiah-eks-terraform-state"
  key          = "eks/terraform.tfstate"
  region       = "us-west-1"
  use_lockfile = true
  encrypt      = true
}

That use_lockfile line matters. Older Terraform setups needed a separate DynamoDB table just to prevent two people applying changes at the same time. That requirement is gone now. S3 can lock its own state file directly.

Where things actually went wrong

Things going smoothly on the first try teaches you very little. Things breaking, and figuring out why, teaches you almost everything.

The first failure was expected. Terraform needs to talk to the cluster to configure access to it, but the cluster’s address isn’t available until after the cluster is created. So the first attempt fails with a connection error. You add the cluster’s connection details once it exists, then run it again.

The second failure took real digging. Deploying Jenkins, it got stuck, waiting for something that never arrived. The problem was storage. Jenkins needs somewhere to save its data, and my cluster had no software installed for handing out storage on AWS. That doesn’t come by default on a self-managed set of nodes.

The fix was a driver called the EBS CSI driver, plus a scoped permission letting it act on the cluster’s behalf:

module "ebs_csi_irsa_role" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

role_name             = "${var.cluster_name}-ebs-csi-driver"
  attach_ebs_csi_policy = true  oidc_providers = {
    main = {
      provider_arn                = module.eks_cluster.oidc_provider_arn
      namespace_service_accounts  = ["kube-system:ebs-csi-controller-sa"]
    }
  }
}resource "aws_eks_addon" "ebs_csi_driver" {
  cluster_name             = module.eks_cluster.cluster_name
  addon_name               = "aws-ebs-csi-driver"
  service_account_role_arn = module.ebs_csi_irsa_role.iam_role_arn
}

Even after that, Jenkins was still stuck, for a different reason. Nothing had told the cluster which storage type to use by default, so its storage request just sat there unanswered. I had to explicitly create and mark one as default:

resource "kubernetes_storage_class" "ebs_gp3_default" {
  metadata {
    name = "gp3"
    annotations = {
      "storageclass.kubernetes.io/is-default-class" = "true"
    }
  }

storage_provisioner = "ebs.csi.aws.com"
  volume_binding_mode = "WaitForFirstConsumer"
}

Then I deleted the stuck pod and its storage request so they’d recreate properly against that new default. Neither of these two fixes was something I could have known upfront. Both came from reading the actual error Kubernetes gave me, kubectl describe pod and kubectl get pvc, and tracing each one back to a real cause rather than guessing.

Installing Helm and deploying Jenkins

Helm works like a package manager for Kubernetes. Instead of manually describing every piece Jenkins needs, someone has already packaged it, and installing it is close to one command:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

helm repo add jenkinsci https://charts.jenkins.io
helm install my-jenkins jenkinsci/jenkins --namespace jenkins-namespace

A few minutes later, Jenkins was running.

Reading its logs needed one extra thing to know. The pod runs more than one container, so kubectl has to be told which one, or it picks a sensible default automatically:

kubectl logs my-jenkins-0 --namespace jenkins-namespace -c jenkins

I also set up krew, a plugin manager for kubectl, to merge a separately generated cluster config into my default one:

kubectl krew install konfig
kubectl konfig import --save ./eks-kubeconfig

Taking it back down

Once everything worked, I tore it down, in order. Jenkins and its storage first, since deleting the cluster while storage is still attached can leave that storage behind, quietly costing money with nothing using it.

Then the cluster itself, and the network around it. Then I checked, using AWS’s own tools, that nothing was still running anywhere.

Why this is worth writing down

Most write-ups of projects like this make everything look smoother than it is in practice. Steps work, you move on, and you don’t build much intuition for what to do when the same steps don’t work for you, on a slightly different setup.

The parts of this project actually worth doing weren’t the parts that went to plan. They were the two times something broke for reasons nothing had flagged in advance, and I had to read an error message, form a guess about what it meant, and check whether I was right.

This project is part of my Cloud and DevOps Engineering apprenticeship at StegHub. The full Terraform code and README for this one are in my repo: https://github.com/LydiahLaw/Steghub-Devops-Cloud-Engineer/tree/main/Project-24-EKS-With-Terraform