How I Migrated Jenkins 2.401.2 to 2.555.1 on AWS Graviton

Credit : AI

In our organization, I manage CI/CD for ~115 pipelines, ~100 engineers and dozens of microservices across production, preprod and staging environments. When our Jenkins instance started showing its age - running 2.401.2 on an old x86_64 EC2. I decided it was time for a proper upgrade. Not just a version bump but a full migration to a cost-optimized AWS Graviton (ARM64) instance running Jenkins 2.555.1.

This is the story of how I did it in 2–3 hours, without taking down production.

Why I Migrated

The old Jenkins was running on old Docker container on an x86_64 EC2. It had accumulated years of manual fixes, undocumented tool installations and a 68GB agent workspace. A few things pushed me to act:

• Cost — Graviton instances offer ~40% better price-performance than equivalent x86_64
• Security — The old Jenkins had a public IP exposed directly. I wanted it behind nginx only
• Version — Jenkins 2.401.2 was well behind, missing security patches and modern plugin support
• Tech debt — There was no reproducible agent setup. Tools were installed ad-hoc directly into containers over years

The goal: migrate everything to a fresh Graviton instance, keep old Jenkins running in parallel until verified and not break a single production deployment.

The Architecture

Internet
    ↓
Cloudflare DNS (jenkins.x.x)
    ↓
Nginx (public subnet)
    ↓
Jenkins EC2 (private subnet)
Graviton
    ↓
Agent
3 Docker containers running JNLP agents

The new Jenkins has no public IP. All traffic goes through Nginx. This was a deliberate security improvement the old Jenkins was accessible directly from the internet.

Step 1 — Fresh Jenkins Install First

The biggest lesson I learned early: don’t copy config.xml from the old instance. I tried it once and Jenkins wouldn’t boot. The reason? config.xml contains references to plugins (like GoogleOAuth2SecurityRealm and RoleBasedAuthorizationStrategy) that need to be loaded before they can be parsed. On a fresh install, those plugins aren't ready at boot time.

My approach:

1. Start a fresh Jenkins 2.555.1 container
2. Install all ~101 plugins fresh (ARM64 compatible versions)
3. Then copy data — but not config.xml

Step 2 — Migrating the Data

I used rsync to copy from the old instance, with careful exclusions:

bash

rsync -avz --progress \
  --exclude='config.xml' \
  --exclude='plugins/' \
  --exclude='war/' \
  --exclude='workspace/' \
  /old/jenkins_home/ \
  user@ip:/opt/jenkins

What I migrated:

• 115 jobs — all pipeline and freestyle configurations
• 100 users — complete user accounts and preferences
• credentials.xml — all stored credentials
• secrets/ — Jenkins master key and encrypted secrets
• nodes/ — agent configuration files

What I intentionally skipped:

• config.xml (recreated fresh, then role strategy injected)
• plugins/ (installed fresh for ARM64 compatibility)
• workspace/ (rebuilds automatically)

Step 3 — Fixing Boot Failures

After copying data, Jenkins still wouldn’t start cleanly. I hit two issues:

Issue 1: Google SSO

The old config.xml had GoogleOAuth2SecurityRealm baked into it from years ago. Even on a fresh setup, any migrated config referencing the Google plugin would fail because the plugin isn't loaded at parse time.

Fix — replaced with HudsonPrivateSecurityRealm temporarily using Python:

import re
with open('/opt/jenkins/config.xml', 'r') as f:
    content = f.read()
old = re.compile(r'<securityRealm.*?>.*?</securityRealm>', re.DOTALL)
new = '<securityRealm class="hudson.security.HudsonPrivateSecurityRealm">...'
result = old.sub(new, content)

with open('/opt/jenkins/config.xml', 'w') as f:
    f.write(result)

Issue 2: Role Strategy Plugin

Same problem — RoleBasedAuthorizationStrategy referenced before the plugin loaded. I temporarily switched to Unsecured auth, let Jenkins boot, then injected the full role strategy XML from the old instance.

The injection approach worked cleanly — I extracted the complete <authorizationStrategy> block from the old config.xml using Python regex and injected it into the new one after Jenkins was stable. All 100 user-role assignments came back perfectly.

Step 4 — Nginx Routing

Adding jenkins.x.x to the existing Nginx setup was straightforward.

After adding the DNS A record and reloading Nginx, curl -I https://jenkins.x.x returned x-jenkins: 2.555.1 — my first confirmation the new instance was live.

Step 5 — Rebuilding the Agents Properly

This was the most interesting part. The old agents were a mess — tools installed manually over 2+ years, no reproducible setup, 68GB of accumulated workspace data.

For the new agents, I did it right: custom Dockerfiles for each agent type. This means every tool is explicitly declared, versioned, and reproducible. No more “I think something is installed somewhere on that box.”

The agents connect via JNLP/WebSocket — they initiate outbound connections to master on port 50000. No inbound SSH needed.

Step 6 — The Data I Had to Preserve

Three things don’t live in Jenkins itself but are critical for builds to work:

Maven settings.xml , I copied this from the old agent and mounted it into both root and non-root path — the old agent ran as root, the new one runs as jenkins, so both paths need to exist.

.m2/repository cache — This was the biggest surprise. One of our jobs downloads one library from jaspersoft.artifactoryonline.com — a domain that no longer exists. The old agent worked because it had a 4.5GB Maven cache built up over years. I had to copy the entire repository cache from the old agent to the new one.

Old Tools — Several legacy builds require Oracle JDK 8 specifically (not OpenJDK 8). Since it’s proprietary, I can’t just download it in a Dockerfile. I copied it directly from the old agent container and bind-mounted it into the new one.

The bind-mount approach for all of these is key — the data lives on the host, so it survives container rebuilds.

Step 7 — Running Both in Parallel

For the entire testing period, I ran old and new Jenkins simultaneously:

Old Jenkins: jenkinsold.x.x → old agents (still running)
New Jenkins: jenkins.x.x → new agents

On the agent EC2, both sets of containers ran side by side:

agent-old      → old master (SSH agent, still connected)
agent-new      → new master (JNLP agent, new)

This let me test every job on new Jenkins while old Jenkins continued handling production traffic. When I was confident a job worked correctly, I moved on to the next. The parallel period caught every issue before it could affect anyone.

Issues I Hit (And Fixed)

Real migrations don’t go perfectly. Here’s what I actually ran into:

| Issue                     | Root Cause                                                   | Fix                                                          |
|---------------------------|--------------------------------------------------------------|--------------------------------------------------------------|
| Jenkins won't boot        | Google SSO / role-strategy in config.xml before plugins load | Used Python regex to temporarily replace auth config         |
| Java class file 55.0 error| Agent had Java 11, Jenkins 2.555.1 requires Java 21          | Switched to inbound-agent:latest-jdk21                       |
| Docker permission denied  | jenkins user not in docker group (GID 992)                   | Added group with `groupadd -g 992` in Dockerfile             |
| Maven 401                 | Missing settings.xml on new agent                            | Mounted settings.xml to both .m2 paths                       |
| DNS fails in containers   | Docker DNS (127.0.0.11) vs AWS VPC DNS mismatch              | Added correct nameserver to /etc/resolv.conf                 |
| download fails            | Domain deprecated; old agent had cached dependencies         | Copied 4.5GB .m2/repository from old agent                   |
| Ansible 2.x.x breaks      | Python 3.x incompatibility (vendored six.moves issue)        | Switched to ansible-core 2.17.x in virtualenv                 |
| ec2_metadata_facts        | Collection not bundled in newer ansible-core                 | Installed via `ansible-galaxy collection install amazon.aws` |
| sudo not found            | Debian base image lacks sudo                                 | Installed sudo + added NOPASSWD in sudoers                   |

The Result

Everything running on new Jenkins after 3 hours:

✅ Jenkins 2.555.1 on Graviton (jenkins.x.x)
✅ 115 jobs migrated
✅ 100 users with full RBAC restored
✅ All credentials intact
✅ 3 new agents connected (Java 21)
✅ No public IP — all traffic via Nginx
✅ IST timezone throughout
✅ Production deployments running successfully

Switching to Graviton saves roughly 40% on compute cost. For a service that runs 24/7 and handles dozens of builds per day, that adds up.

Key Takeaways

1. Never copy config.xml directly Always start fresh and inject the auth/role strategy after Jenkins boots. Plugin class-loading order will cause mysterious boot failures. This cost me an hour before I figured it out.

2. Dockerfiles over ad-hoc installs Every tool must be in a Dockerfile. If it’s not in the Dockerfile, it doesn’t exist after the next container restart. I learned this the hard way with Ansible, jq and sudo all missing from the base image.

3. Agent home directories contain critical state Don’t assume agents are stateless. Years of builds accumulate caches, JDKs, and credentials that jobs depend on. Copy from old agents before decommissioning.

4. Run parallel before cutting over Keep old Jenkins running. Don’t cut DNS until every critical pipeline has run successfully on the new instance. Running both in parallel for a full day caught every issue before it could affect production.

5. Python version compatibility for Ansible pick ansible-core version carefully. 2.17.x is the sweet spot — modern enough for Python 3.13, backward-compatible enough for Python 3.7 targets. A virtualenv keeps it isolated cleanly.

6. Maven caches are not optional state If builds depend on libraries from deprecated or private repositories, the .m2/repository cache is not optional. It's state that needs to be preserved and migrated.

The migration took 2–3 hours end-to-end. Most of that time was debugging — boot failures, version incompatibilities, missing tools. The actual migration steps themselves are straightforward once you know what to expect.

Hopefully this saves you some of that debugging time.

Have questions or ran into something similar? Feel free to reach out.