I Over-Engineered My University’s CI/CD Pipeline for Two Weeks. The Solution Might Have Been Just One Email.

A story about K8s runner tokens, SSH tunnels, a glorious golden era, three walls of failure, and the most anticlimactic plot twist in DevOps history.

📎 Context: This post is a deep-dive into one specific, chaotic chapter of our larger deployment journey for Inventory Gallery, a Next.js + Django REST Framework university project. If you want the full architectural picture — 3-tier environments, ephemeral pods, the HPA battle — check out the previous post first:

[embed]CSUI Gallery Inventory: A Deployment Story A war story about ephemeral pods, mixed content hacks, and learning to build resilient pipelines on shared…medium.com

There’s a specific kind of developer hubris that only university projects can produce. You’re building something real, on real infrastructure, with real deadlines — but you’re also a student, which means you have just enough knowledge to think you can outsmart the system and just enough free time to try. This is the story of how I spent two weeks engineering increasingly elaborate CI/CD workarounds, experienced a genuine golden era and watched it get shut down, hit every wall imaginable, surrendered in defeat — and then accidentally stumbled upon an answer that had been quietly taking shape the whole time.

The Inciting Incident: The Runner That Kept Crashing

Our project — a moderately complex full-stack web app with a Next.js frontend and a Django REST backend, deployed on the university’s Kubernetes cluster — started life with the most boring CI/CD setup imaginable. Push to GitLab, wait for the university’s shared runner to pick it up, hope for the best.

The shared runner was provided by our faculty, and it’s genuinely a generous piece of infrastructure to offer students. But it was shared across the entire faculty — every team, every project, every push from every student all competing for the same executor slots. During crunch periods, which in a semester-long software engineering course means basically always, the load was simply more than it could comfortably handle. Jobs didn’t just stall in a queue — they crashed outright mid-execution.

It wasn’t anyone’s fault in particular. It was the predictable reality of shared infrastructure under heavy concurrent use — a single runner trying to serve an entire faculty’s worth of simultaneous deployments. We wanted faster, more reliable feedback cycles. So we did what any student-developer does when faced with a bottleneck: we decided to engineer our way around it, which is always a great idea and never causes more problems than it solves.

Phase 1 & 2: The In-Cluster Runner — Our Golden Era

The first plan had a certain elegance to it. Instead of relying on the faculty’s shared runner, why not host our own GitLab Runner inside the same Kubernetes cluster our app was already running on? We had namespace access, we had kubectl, we had ambition. We registered a custom runner, pointed it at our namespace, and spun it up as a Pod.

There was one structural catch from day one: the campus Kubernetes API server is strictly internal — not reachable from outside the campus network, which is entirely reasonable from a security standpoint. To manage this runner from home — to register it, check its status, or touch the cluster at all — I needed to be on the campus network. So I set up an SSH tunnel using my campus SSH key, wrapped into a terminal alias I called proxy_kampus:

# Lives in .bashrc / .zshrc
# One command to open a SOCKS5 tunnel into the campus network
alias proxy_kampus='ssh -D 127.0.0.1:1080 -N -f -i ~/.ssh/campus_key your_username@campus-gateway.cs.ui.ac.id'

Type proxy_kampus, and a SOCKS5 proxy opens on 127.0.0.1:1080. Route your kubectl traffic through it, and your laptop can talk to the campus K8s API as if you were physically in the server room. This alias became the first thing I ran every morning before touching anything cluster-related — a small ritual that marked the start of the dev day.

Once we got it stable, this setup was genuinely spectacular. The in-cluster runner had everything going for it: it lived inside the cluster, giving it direct, low-latency access to every internal endpoint with no tunneling overhead for the actual CI jobs. It had the correct networking configuration already in place. Build times dropped dramatically, deploys were clean and fast, and the team’s development velocity improved noticeably. PRs got merged faster. The feedback loop from git push to deployed-on-staging shrank from a waiting game to something that actually felt snappy. For a few weeks, I genuinely felt like we had solved it.

The One Catch: Token Expiry and the Script That Tamed It

Here’s the thing about running a GitLab Runner inside a Kubernetes cluster: Pods are not permanent. They restart. They get evicted. And every single time our runner Pod came back up, its GitLab registration token had expired — and the runner showed as offline in GitLab.

The manual fix was tedious: navigate to GitLab settings, generate a new runner token, open a terminal, activate proxy_kampus, update the Kubernetes ConfigMap with the new token, then trigger a rollout restart of the runner Deployment. After the third time running through this ritual, I decided that if I was going to be doing DevOps, I was going to automate the annoying parts.

I wrote a Bash script that handled the entire token rotation flow in a single run:

#!/bin/bash
# GitLab Runner Token Auto-Renewer
# Make sure your campus network proxy is active before running this!

NAMESPACE="your-namespace-here"
CONFIGMAP_NAME="your-configmap-name"
DEPLOYMENT_NAME="your-runner-deployment"

echo "================================================="
echo "🔄 Auto-Renewer: GitLab Runner Token"
echo "================================================="
echo "Make sure your Campus Gateway proxy is active first!"
echo ""

# You still need to grab the new token from GitLab manually
read -p "Enter new Runner Token (glrt-...): " NEW_TOKEN

if [[ -z "$NEW_TOKEN" ]]; then
    echo "❌ Token cannot be empty. Operation cancelled."
    exit 1
fi

# Step 1: Pull the live ConfigMap from the cluster into a local temp file
echo "⏳ Fetching ConfigMap from cluster..."
kubectl get configmap $CONFIGMAP_NAME -n $NAMESPACE -o yaml > temp_config.yaml

if [ $? -ne 0 ]; then
    echo "❌ Cannot connect to Kubernetes. Is your KUBECONFIG valid and proxy active?"
    exit 1
fi

# Step 2: Use sed to surgically replace only the token value — nothing else in the config is touched
echo "🔧 Injecting new token into config..."
sed -i "s/token = \"[^\"]*\"/token = \"$NEW_TOKEN\"/g" temp_config.yaml

# Step 3: Push the modified ConfigMap back to the cluster and clean up the temp file
echo "🚀 Applying updated config to cluster..."
kubectl apply -f temp_config.yaml
rm temp_config.yaml

# Step 4: Force a Pod restart so the runner picks up the fresh token on startup
# Without this step, the running Pod keeps its old expired token in memory
echo "♻️  Restarting runner Deployment..."
kubectl rollout restart deployment $DEPLOYMENT_NAME -n $NAMESPACE

echo "✅ Done! Old runner Pod terminated, fresh Pod starting with new token."

The logic flows through four clear steps. First, it pulls the current ConfigMap out of the cluster into a local YAML file, since you need a local copy to safely edit it. Second, sed surgically targets the pattern token = "..." and replaces only the token string — nothing else in the config is touched. Third, kubectl apply pushes the modified YAML back to the cluster, updating the ConfigMap in place. Fourth — the part that actually matters — kubectl rollout restart forces Kubernetes to kill the old Pod and spin up a fresh one. Without that final step, the running Pod keeps its old, expired token in memory regardless of what the ConfigMap says. What used to be a five-minute manual ritual became a thirty-second script run.

With the automation in place, the golden era was sustainable. And it held — until it didn’t.

Phase 3: The Downfall — ITF Updates the Admission Policy

One morning, the runner Pod wouldn’t start. After some digging, the reason became clear: the cluster’s image admission policy had been updated, and our custom runner image was no longer on the permitted list. This is a completely understandable security measure for a shared, multi-tenant campus cluster — the infrastructure team has a responsibility to control what images run in the environment. We just hadn’t anticipated it, and it brought our golden era to an abrupt and unceremonious end.

We reached out to the campus IT department (ITF) with an email explaining our use case and requesting our runner image be reviewed for whitelisting. Then we waited. And while we waited, deadlines didn’t. We needed another plan immediately.

Phase 4: The Desperate Migration to Arch Linux

With the K8s runner offline and the shared runner still struggling under faculty-wide load, I moved the runner to my personal laptop. Arch Linux, because at this point we were fully committed to the bit.

The local machine had real CPU, real RAM, and a warm Docker layer cache. For the early pipeline stages, it was actually decent. The test jobs ran cleanly, and the SonarQube analysis stage executed without complaint. For that narrow window, it felt like progress.

But the proxy architecture introduced new fragility. To reach the campus K8s API from the laptop, every CI job needed the SOCKS5 tunnel to be active, and the tunnel needed to be threaded through the Docker executor itself — not just available in the host shell. The config.toml required two specific tweaks to make this work:

[[runners]]
  name = "local-arch-ppl-gallery"
  url = "https://gitlab.cs.ui.ac.id/"
  executor = "docker"
  [runners.docker]
    # Without host networking, 127.0.0.1 inside the container refers to the
    # container itself, not the host machine. Host networking shares the laptop's
    # full network stack with the container — so the SOCKS5 proxy at 127.0.0.1:1080
    # on the host becomes reachable from inside the CI job.
    network_mode = "host"
    environment = [
      "HTTP_PROXY=socks5://127.0.0.1:1080",
      "HTTPS_PROXY=socks5://127.0.0.1:1080",
      "NO_PROXY=localhost,127.0.0.1"
    ]

This held together through test and sonarqube. Then the pipeline reached build_image, and the whole thing fell apart.

Phase 5: Three Walls, One After Another

Wall One: The Kaniko DNS Deadlock. Kaniko — the tool we use to build Docker images inside CI without a Docker daemon — would start up, begin resolving DNS for the base image registry, and then silently freeze. No useful error message, just an indefinite hang until the job timeout killed it. The root cause was a compatibility gap: Kaniko makes low-level DNS resolution calls that don’t respect standard HTTP_PROXY environment variables the way most tools do. Routing it through SOCKS5 simply didn't work, and there was no clean fix without significantly restructuring the build stage.

Wall Two: The Registry Credential Masking. Pushing to the campus container registry requires credentials stored as masked GitLab CI variables — a correct and sensible security practice. Masked variables are only reliably injected into jobs running on infrastructure that GitLab trusts at a system level. On the local laptop runner, the registry password arrived in the job environment as the literal string [MASKED]. Not the password — the four characters [MASKED]. Every Kaniko --destination flag and every docker push failed because it was genuinely trying to authenticate with [MASKED] as the credential value. This is a deliberate security boundary, not a bug, and it exists for good reason — it just meant our external runner couldn't cross it.

Wall Three: The K8s Service Account Gap. Even if both of the above had been solvable, the deploy stage had its own constraint. The campus cluster expected deployments to use a specific Service Account with pre-configured image pull secrets for the internal registry. The shared faculty runner has these bindings in place as part of its campus-level setup. An external laptop runner has no mechanism to acquire those same bindings without cluster-admin access — permissions that, understandably, aren’t available to individual student teams.

Each wall individually was a reasonable piece of security or infrastructure design doing exactly what it was supposed to do. Together, they drew a clear boundary: the kind of deep integration we needed simply couldn’t be replicated from outside the campus ecosystem. That was a legitimate architectural reality, not a flaw to be debugged.

Phase 6: The White Flag

We reverted .gitlab-ci.yml back to the shared runner.

It took about fifteen minutes to undo weeks of effort. We stripped out the custom runner configuration, removed the proxy environment variables, and pushed a commit that was, at its core, an admission that we’d been trying to solve the wrong problem in the wrong direction. The pipeline went back to the shared queue. Builds went back to being slower. And we finally had a much clearer appreciation for why the shared runner works as reliably as it does: it lives inside the ecosystem it serves. The registry credentials are pre-wired. The Service Account bindings are in place. The network access is native. That’s not magic — it’s careful integration work, and it’s what makes the runner trustworthy for every team that depends on it.

The honest lesson: shared infrastructure that’s properly integrated with everything around it will outperform a cleverly improvised external workaround every single time. We’d been trying to replicate from the outside what had been carefully built from the inside. That gap isn’t something you can config.toml your way across.

The Plot Twist: I Checked My GitLab Settings on a Whim

Here’s where I have to admit something that made me laugh out loud at my own desk.

I was writing this blog post — somewhere around the three-walls section, feeling appropriately humbled — when I received an unrelated email reply from ITF about a completely different request. Nothing to do with runners at all. I opened it, handled it, and then, almost on a whim, navigated over to our GitLab group’s CI/CD runner settings page. I hadn’t checked it in a while. Just a casual glance.

There were new runners listed. Runners I had never registered. Runners with names and tags that clearly indicated they had been provisioned specifically for our project group.

At some point after I’d sent that original email to ITF — I still don’t know exactly when — the campus administrators had provisioned dedicated, group-specific GitLab Runners for our project. Properly configured, with registry access and the right cluster bindings, quietly ready to use. No announcement needed on their end; it was simply part of handling the request.

And I, meanwhile, had spent the intervening weeks tunneling through SSH, patching ConfigMaps with sed, wrestling with SOCKS5 proxies, watching Kaniko hang indefinitely, screaming at [MASKED] authentication errors, and eventually retreating back to the shared runner — without once thinking to check whether the original request had already been fulfilled.

The dedicated runner works exactly as you’d hope. Clean builds, clean deploys, fast feedback. It is precisely what we needed from the beginning, delivered by people who had the access and context to set it up correctly. All that’s left is a healthy appreciation for checking your GitLab settings more than once every few weeks.

The Actual Takeaway

I could end this with “follow up on your support requests” and that would technically be the most accurate lesson. But there’s something more genuinely useful here.

Everything I built during those weeks — the token automation script, the proxy_kampus alias, the host-network Docker configuration — none of it was wasted effort, even though the problem got resolved through a completely different path. Writing the token rotation script gave me a real, hands-on understanding of how Kubernetes ConfigMaps and rollout restarts work under the hood. Configuring the SOCKS5 proxy taught me how Docker's network isolation interacts with host services — a non-obvious edge case that comes up in real infrastructure work. Hitting the Kaniko DNS wall taught me that "route everything through the proxy" is not always the right abstraction, and that some tools make low-level syscalls that don't respect the proxy environment variables you've carefully set. These are things I now know from experience rather than documentation, and that kind of knowledge sticks differently.

None of it came from the dedicated runner appearing in my settings. It came from the weeks of trying.

The parallel lesson, though, is worth sitting with: sometimes the people responsible for your infrastructure are already quietly working on the problem you’re hacking around. A quick follow-up — or even just occasionally checking whether something has changed — can save a surprising amount of effort.

And honestly? The long way around taught me more. So I’m not entirely sorry.