Unlocking the Power of the Rails Console in GitLab on Kubernetes

When you run GitLab self-managed inside a Kubernetes cluster, you expect the web UI and APIs to be your main levers for change. But sometimes you hit a brick wall:

• A setting exists, but the UI throws a 500 error.
• An API mutation exists, but fails because the underlying database row is missing.
• A new project behaves differently to older ones, and you can’t explain why.

That’s exactly what happened to me with the “Keep artifacts from most recent successful jobs” toggle in the CI/CD settings. On older projects it worked fine. On new ones it failed with a GraphQL 500.

In the past, we had to mobilise a whole team just to change similar settings that weren’t available via GraphQL or the REST API. Six people spent hours clicking through repositories in the web UI because there was no programmatic way to do it. With GitLab on Kubernetes, opening the Rails console in the toolbox pod meant I could script the same change across every repo in a group (and its subgroups) in minutes — by myself. Whoa! I wish I had known that then! Steve!

I Smell Something Off

And not just the rotting smell of Banana Peels left in a green bin no one checks. We kept trying to click the toggle in Settings → CI/CD → General pipelines but it kept resulting in a 500 error.

• In browser DevTools, the failing request was POST /api/graphql.
• Logs from the gitlab-webservice pod showed a NoMethodError, pointing to ci_cd_settings being nil.

Translation: the project simply didn’t have a ci_cd_settings row yet, so the GraphQL mutation tried to call a method on nil. (Although I didn’t realise this at the time)

The Fix: Rails Console in Kubernetes

GitLab on K8s (Helm chart or Operator) ships with a toolbox pod that includes gitlab-rails. From there, you can open a Rails console:

kubectl exec -n gitlab -it deploy/gitlab-toolbox -- bash
gitlab-rails console

That drops you into a Ruby REPL with the entire GitLab application context loaded — models, migrations, settings, everything. It’s the same environment GitLab’s own code runs in.

What’s a REPL? REPL stands for Read–Eval–Print Loop. Read: It takes the command or expression you type. Eval (Evaluate): It runs that command in the context of the language or framework. Print: It shows you the result immediately. Loop: It goes back and waits for the next command. So a REPL is essentially an interactive shell for a programming language.

From there, it’s possible to inspect and repair project state directly:

p = Project.find_by_full_path("mygroup/myrepo")
s = p.ci_cd_settings || p.create_ci_cd_settings
s.update!(keep_latest_artifact: true)

This single snippet created the missing row and flipped the setting — no web UI, no API, no 500 error.

Scaling It: All Projects in a Group Tree

Doing this one-by-one is painful. The real power comes from using GitLab’s models to walk whole namespaces:

g = Namespace.find_by_full_path("your-new-group")

all_groups = g.self_and_descendants

all_groups.each do |ns|
  ns.projects.find_each do |p|
    s = p.ci_cd_settings || p.create_ci_cd_settings
    s.update!(keep_latest_artifact: true)
  end
end

That snippet:

• Starts at a top-level group.
• Recursively traverses all subgroups.
• Ensures every project has CI/CD settings.
• Sets the keep_latest_artifact flag to true everywhere.

What used to require a team-wide, repo-by-repo slog is now a one-liner in a console.

What the Console Actually Is

• It’s the Ruby on Rails console: a live REPL with your GitLab app code loaded.
• It connects directly to the production database and GitLab internals.
• Every ActiveRecord model (Project, Namespace, CiCdSettings, etc.) is available.
• You can inspect (p.attributes), update (p.update!), and create records (p.create_ci_cd_settings) exactly the way the app itself would.

In other words, it’s like sitting inside the GitLab app — bypassing web/API layers entirely. I love that! Wow!

The Power It Gives

• Repair broken state (missing rows, inconsistent data).
• Backfill defaults across thousands of projects.
• Run one-off scripts without waiting for a new GitLab release.
• Debug logic (you can call the same methods the app calls).

Limits and Risks

• It’s not guarded — you can break production data if you run destructive code.
• No history of what you ran — unless you log it yourself.
• Schema must match code — if migrations are incomplete, console commands may still fail.
• Support boundaries — GitLab Support may ask you not to use console hacks in production without guidance.

Takeaways

1. When the UI or API can’t set a project-level knob, the Rails console is your back door.
2. On Kubernetes, that means learning to exec into the toolbox pod and use gitlab-rails console.
3. You can fix state at scale (entire groups, subgroups, thousands of projects) with a short Ruby loop.
4. It turns what once required an entire coordination effort into a single operator’s action.

Fireside Chat

If you’re security-minded, you might be thinking: “Hang on, should people really be dropping into a Rails console in production and flipping settings?” Fair point. Most developers — and even many DevSecOps engineers — won’t normally have this level of access, nor should they by default.

But here’s the challenge: maybe they should, at least under a just-in-time permissions model. The irony is that in order to discover this fix, I had to already have console access. Without that, I couldn’t even build the case for why just-in-time access mattered.

Too often, security becomes a checkbox exercise — locked down by default, but disconnected from the reality of how engineers actually solve problems. The result? Costly delays, workarounds, or worse, no fix at all.

So what’s the solution? For me, it’s simple: access to do my job. Not blanket, permanent admin rights — but a model that recognises when engineers need elevated access, gives it safely, and then takes it away when the job is done.

Closing Thought

I used to think of GitLab as a sealed box where my only levers were the REST/GraphQL APIs. Now I know: if you really need to, you can climb inside the box. That power should be used carefully — but when you need it, it’s the difference between being blocked for weeks and fixing the problem in minutes.