GitHub Apps vs PATs: How to Secure Your ArgoCD Authentication at Scale

TL;DR

Personal Access Tokens are a fragile foundation for ArgoCD authentication. They expire, they’re tied to individual users, and when they fail, everything fails at once. This article walks through how we migrated to GitHub App authentication with zero downtime — and the unexpected pitfalls we hit along the way.

It Happened Twice

Around 4 AM on two separate occasions — once near the end of 2025 and again in early 2026 — our ArgoCD stopped syncing. Developers started flooding Slack. P1 incidents piled up. A war room opened. The root cause both times: a GitHub Personal Access Token had expired.

Our teams deploy via CircleCI and use a shared context to store environment variables. When that token expired, it wasn’t just one team affected — it was a company-wide outage. Every service that depended on ArgoCD to sync was frozen.

It was painful. And it was completely avoidable.

That experience prompted us to revisit how ArgoCD connected to GitHub in the first place.

The Problem with Personal Access Tokens

When we dug into our setup, we found a single PAT acting as a credential template covering our entire GitHub organization. It was stored in a Kubernetes secret like this:

apiVersion: v1
kind: Secret
metadata:
  name: argoproj-https-creds
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repo-creds
stringData:
  url: https://github.com/your-org
  type: git
  username: developer@your-org.com
  password: ghp_...

Beyond the expiration risk, we found a few other problems:

The token was user-bound. If the person who created it ever left the organization, the token would be revoked, and everything would break — silently.

No audit trail. There was no way to know which service accessed which repository and when.

The token was exposed in plaintext. Kubernetes stores the last-applied-configuration annotation on every resource. That annotation contained the raw PAT value, visible to anyone with read access to the namespace.

**insecure: true on some secrets.** A couple of per-repository secrets had TLS verification disabled, allowing credentials to be transmitted without certificate validation.

One token. No rotation. No scoping. No visibility. A single point of failure for the entire engineering organization.

Why GitHub Apps Are Different

A GitHub App authenticates as an installation on an organization — not as a user. It generates short-lived tokens automatically, scoped to only the repositories you explicitly grant access to.

The short token lifetime is worth emphasizing: the GitHub App never exposes a long-lived credential. Every time ArgoCD needs to authenticate, it generates a fresh token using the app’s private key. That token expires in one hour. There’s nothing to rotate manually, nothing to forget, and nothing to leak that will remain valid for long.

Architecture Overview

Before the migration, our setup looked like this

After the migration

Zero-downtime migration strategy

The diagrams above show the shift in authentication architecture.

One architectural decision is worth calling out: rather than creating one GitHub App per cluster, which would have polluted our shared CircleCI context with dozens of variables, we went with a single GitHub App for the organization, installed only on the repositories ArgoCD actually needs.

Step-by-Step Migration

1. Create the GitHub App

Go to your organization’s settings: [https://github.com/organizations/YOUR-ORG/settings/apps/new](https://github.com/organizations/YOUR-ORG/settings/apps/new)

Configure it with:

• Name: something descriptive like argocd-your-org
• Homepage URL: your org’s GitHub URL
• Permissions: Contents → Read-only, Metadata → Read-only (everything else: No access)
• Webhook: disabled
• OAuth / Device Flow: disabled

Generate and download the private key. Note the App ID from the General settings page.

2. Install the App

Go to Install App in the sidebar and install it on your organization. Choose Only select repositories and add only the repos ArgoCD needs to read. After installation, the URL will contain your Installation ID:

https://github.com/organizations/YOUR-ORG/settings/installations/INSTALLATION_ID

3. Create the Kubernetes Secret

apiVersion: v1
kind: Secret
metadata:
  name: argoproj-github-app-creds
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repo-creds
stringData:
  url: https://github.com/your-org
  type: git
  githubAppID: "<APP_ID>"
  githubAppInstallationID: "<INSTALLATION_ID>"
  githubAppPrivateKey: |
    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----

Or via kubectl:

kubectl create secret generic argoproj-github-app-creds \
  --namespace argocd \
  --from-literal=type=git \
  --from-literal=url=https://github.com/your-org \
  --from-literal=githubAppID="<APP_ID>" \
  --from-literal=githubAppInstallationID="<INSTALLATION_ID>" \
  --from-literal=githubAppPrivateKey="$(cat /path/to/private-key.pem)"

kubectl label secret argoproj-github-app-creds \
  -n argocd \
  argocd.argoproj.io/secret-type=repo-creds

4. Migrate Without Downtime

We kept the existing PAT secret in place while deploying the GitHub App secret alongside it. ArgoCD evaluates credential templates by specificity and recency — with two templates pointing to the same URL, it will use the GitHub App.

Once we confirmed the GitHub App was working, we disabled the PAT secret by removing its label:

kubectl label secret argoproj-https-creds -n argocd \
  argocd.argoproj.io/secret-type-

The - at the end of the label key removes it. The secret stays in the cluster as a fallback but ArgoCD no longer recognizes it. To re-enable it if needed:

kubectl label secret argoproj-https-creds -n argocd \
  argocd.argoproj.io/secret-type=repo-creds

The Private Key Pitfall Nobody Talks About

This is where things got interesting.

After deploying the GitHub App secret via Terraform — reading the private key from AWS Secrets Manager — ArgoCD started throwing this error:

could not parse private key: invalid key: Key must be a PEM encoded PKCS1 or PKCS8 key

The secret existed. The key looked right. But ArgoCD refused to use it.

Our first instinct was that the private key file itself was corrupted. To rule that out, we uploaded the original .pem file directly to AWS CloudShell and created the Kubernetes secret manually from it. That worked. The file was fine.

The real problem was in how the private key was stored in AWS Secrets Manager. When you store a multiline value as a JSON string, newlines need to be represented as \n. But there's a subtle difference between \n as a literal two-character sequence and \n as an actual newline character — and Terraform's templatefile function does not automatically convert between them.

We ran into two separate issues:

Issue 1: Leading spaces. When copying the formatted key into the Secrets Manager console, each line inadvertently had a leading space. A PEM key with leading spaces on each line is invalid.

Issue 2: Missing trailing newline. The last line of the key — -----END RSA PRIVATE KEY----- — had no newline after it. Some parsers are strict about this.

You can diagnose both issues with:

# Check line count (should be 27 for a standard 2048-bit RSA key)
kubectl get secret argoproj-github-app-creds -n argocd \
  -o jsonpath='{.data.githubAppPrivateKey}' | base64 -d | wc -l

# Check for leading spaces or Windows line endings
kubectl get secret argoproj-github-app-creds -n argocd \
  -o jsonpath='{.data.githubAppPrivateKey}' | base64 -d | cat -A | head -3

A clean key looks like this:

-----BEGIN RSA PRIVATE KEY-----$
MIIEpAIBAAKCAQEA...$

A $ at the end of each line is normal (it marks the newline). A space before the content is not:

-----BEGIN RSA PRIVATE KEY-----$
 MIIEpAIBAAKCAQEA...$   ← leading space, will fail

To generate the correctly formatted value for Secrets Manager, use:

python3 -c "
with open('/path/to/private-key.pem') as f:
    content = f.read()
print(content.replace('\n', '\\\\n'), end='')
"

This produces a single-line string with literal \n sequences that Secrets Manager and Terraform handle correctly.

Validating the Migration

After deploying, we confirmed the GitHub App was active in two ways.

First, we checked the ArgoCD repo-server logs for any authentication errors:

kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server \
  --tail=30 | grep -i "error\|fail\|auth"

A clean output means ArgoCD is authenticating successfully.

Second — and this was the most satisfying signal — the old PAT showed as never used on GitHub. ArgoCD had switched to the GitHub App entirely.

IaC Integration

If you manage your clusters with CDK or Terraform, you’ll want to provision this secret through your existing tooling rather than manually.

For Terraform, the pattern is straightforward:

data "aws_secretsmanager_secret_version" "github_secrets" {
  secret_id = var.argocd_github_secrets_arn
}

resource "kubectl_manifest" "argocd-github-app-secret" {
  yaml_body = templatefile("${path.module}/argocd-github-app-secret.yaml.tpl", {
    github_app_id              = jsondecode(data.aws_secretsmanager_secret_version.github_secrets.secret_string)["ARGOCD_GITHUB_APP_ID"]
    github_app_installation_id = jsondecode(data.aws_secretsmanager_secret_version.github_secrets.secret_string)["ARGOCD_GITHUB_APP_INSTALLATION_ID"]
    github_app_private_key     = jsondecode(data.aws_secretsmanager_secret_version.github_secrets.secret_string)["ARGOCD_GITHUB_APP_PRIVATE_KEY"]
  })
}

The key thing to get right is how the private key is stored in Secrets Manager. The value must be a single-line JSON string with \n representing each newline in the PEM file — not actual newline characters, and not leading spaces.

Lessons Learned

The migration itself was straightforward. The edge cases were not.

Private key formatting is where most people will get stuck. The error message ArgoCD gives you is not very helpful, and the cause is not obvious. If you hitcould not parse private key, check for leading spaces and missing trailing newlines before assuming the key is corrupted.

The zero-downtime migration pattern — keeping the PAT alongside the GitHub App during validation — gave us confidence to move quickly without risking a third 4 AM incident. Deploy both, validate, then remove the old credential. Never remove first.

Finally, the insecure: true flag on some of our secrets was a reminder that security debt tends to accumulate quietly. The GitHub App migration was a good opportunity to audit and clean up things that had been quietly wrong for a long time.

Final Thoughts

Two 4 AM incidents were enough. Moving to GitHub App authentication removed the fragility that caused both of them — no more manual rotation, no more user-bound tokens, no more single point of failure for the entire engineering organization.

The migration took less than a day. The peace of mind has been indefinite.

If you’re still relying on Personal Access Tokens for ArgoCD, this is worth doing sooner rather than later. Your future self — the one who would otherwise be in a war room at 4 AM — will thank you.

Have you gone through a similar migration? Run into different pitfalls? Share in the comments — these real-world details are what make this stuff actually useful.

Tags: ArgoCD · GitHub · Kubernetes · DevOps · Security · GitOps · AWS · IaC