Building a Secure EKS Cluster on AWS, and the RDS Bug That Almost Broke It

My name is Ochuko Edsam, and this article is about my Third Semester exam for the Tinyuka 2025 DevOps program. The task was Project Bedrock: provision a secure Amazon EKS cluster, deploy a full retail application against real managed AWS services instead of the in-cluster databases it ships with, lock down developer access, wire up observability, add an event-driven Lambda, and automate the whole thing with a CI/CD pipeline.

Ten days on paper. In practice it was a laptop that lost power more than once, a connection that dropped mid terraform apply more times than I want to count, and one database service that fought me harder than anything else in this project combined. But we got there.

Here’s what I built, what went sideways, and the one bug I want to spend the most time on because it will genuinely wreck your week if nobody warns you about it first.

What Did I Build?

The cluster runs the full retail-store-sample-app: ui, catalog, carts, orders, checkout, each in its own deployment. Catalog talks to RDS MySQL, orders talks to RDS PostgreSQL, carts talks to DynamoDB, all three swapped in for the in-cluster containers the app normally runs on. An IAM user gets console read-only plus an EKS Access Entry scoped to exactly one namespace and nothing else, view-only, verified against the real user’s own credentials, not just assumed from the policy JSON. Control plane logs and container logs both land in CloudWatch. An S3 bucket triggers a Lambda the moment something gets uploaded to it. GitHub Actions plans every Terraform change on pull request and applies it on merge.

All of that sounds tidy written out like this. Getting there was not tidy at all.

The Bug Nobody Warns You About: RDS Doesn’t Care About Your Teardown Schedule

Budget for this project was zero, so the plan was obvious: bring the VPC and EKS cluster up per session, tear everything down before closing the laptop, keep the AWS bill near nothing. That worked fine right up until I added RDS, at which point terraform destroy just stopped working.

Error: deleting ENIs for EC2 Subnet: 1 error occurred:
  * detaching RDS ENI: AuthFailure: You do not have permission
    to access the specified resource.

I stared at that for a while assuming it was a policy I’d forgotten to attach somewhere. It isn’t. AWS will not let anything except the RDS service itself detach a network interface RDS owns, and no amount of IAM tweaking gets around that. A VPC on a nightly destroy cycle and a database that needs to persist genuinely cannot live in the same Terraform state together. Full stop.

The fix, once I stopped fighting it, was to split the whole thing into two states sharing one S3 backend. One state holds the VPC, private subnets, RDS, and DynamoDB, and it never gets destroyed. The other holds everything actually expensive to leave running, the EKS cluster, the node group, the NAT Gateway, and gets rebuilt every session, reading the VPC’s IDs from the first state through a remote state reference. Two RDS instances left running around the clock cost less than a dollar a day. Worth every cent compared to what it saved me in nightly rebuild time.

I did not design that split going in. I backed into it after a failed destroy at midnight, on a laptop battery I did not trust to last another hour.

Read the Source Before You Guess the Schema

I built the DynamoDB table for the cart service assuming customerId as the primary key, because if you are reasoning about a shopping cart from scratch, that is the obvious key to reach for. It is wrong. AWS's own reference implementation for this exact app uses id as the hash key, with customerId sitting as a secondary index instead:

resource "aws_dynamodb_table" "carts" {
  hash_key = "id"

  attribute {
    name = "id"
    type = "S"
  }

  attribute {
    name = "customerId"
    type = "S"
  }

  global_secondary_index {
    name     = "idx_global_customerId"
    hash_key = "customerId"
  }
}

I only caught this because I went and pulled the real Terraform before writing a single Kubernetes manifest, on the theory that guessing a schema for a service I didn’t write myself is a great way to lose an evening later. This one would have shipped clean too. Pods green, health checks passing, every single cart write silently failing underneath the whole time.

The Part I Have to Own: Automation That Didn’t Go Far Enough

The CI/CD requirement was to automate infrastructure changes, and the pipeline does that properly, plan on pull request with the diff posted as a real comment, apply on merge, tested against an actual PR and not just assumed to work. That covers Terraform. It does not cover the application.

I made a call partway through the build to keep deploying the app itself, the Helm install, the manifest apply, the three secrets, as a manual step run live at the start of each session, instead of folding it into the same pipeline as the infrastructure. My reasoning at the time was that the nightly stack gets rebuilt from nothing every session anyway, and I was already sitting there to bring it up by hand. Looking back, that’s a narrower idea of “automated” than the project actually needed. A pipeline that only stands up empty infrastructure has automated the easy half and left the part that actually matters for someone to babysit. Fixing it isn’t complicated, the same apply job just needs to also run the Helm install and the manifest apply, with the handful of credentials it needs stored as repository secrets instead of typed in live. That upgrade is next on the list for this repo.

What I Learned

RDS’s ENI behavior, the real DynamoDB schema, a database password I almost committed to a public repo before catching it in a plain git status check, none of these were hard problems once I stopped assuming and actually went and checked. Every expensive mistake in this build happened at the exact moment I reached for the fast answer instead of the correct one, and you don't find out which one you picked until something's already live and someone's poking at it.

Full repository, including the pipeline upgrade in progress, is at github.com/Reallife-1/project-bedrock.

Thanks for reading.

Ochuko Edsam.