IAM: The Invisible Architecture That Keeps Production From Burning Down

“Give a developer admin access and they’ll fix one bug. Give them least-privilege IAM and the whole cluster survives their worst day.”

Introduction: Why IAM Is Not a Setup-and-Forget Config

Most engineers treat IAM like a door lock — configure it once on day one, then forget it. That is the exact mindset that causes the majority of cloud and Kubernetes breaches.

Identity and Access Management (IAM) is the policy layer that answers two fundamental questions about every single request hitting your infrastructure:

• Authentication — Who are you?
• Authorization — What are you allowed to do?

These two concerns are architecturally separate, and conflating them is where most teams go wrong.

In 2024, the BeyondTrust breach happened because of a single overprivileged API key with static credentials. Attackers didn’t brute-force anything — they just walked in through an IAM door that was left open. In cloud environments, non-human identities (service accounts, CI/CD pipelines, automation scripts) outnumber human identities by 41:1 — and most teams have zero visibility into what those identities can actually do.

This is not a theoretical problem. It is production infrastructure sitting on a foundation of unaudited permissions.

The Mental Model: Three Layers of IAM

Before you write a single YAML file, you need to internalize the layered model:

┌─────────────────────────────────────────────┐
│           CLOUD IAM (Platform Level)         │
│  AWS IAM / GCP IAM / Azure RBAC              │
│  Controls: VMs, Storage, Clusters, APIs      │
├─────────────────────────────────────────────┤
│        KUBERNETES RBAC (Cluster Level)       │
│  Controls: Pods, Deployments, Secrets,       │
│            Namespaces, CRDs                  │
├─────────────────────────────────────────────┤
│     APPLICATION-LEVEL ACLs (App Level)       │
│  Controls: DB access, internal APIs,         │
│            service-to-service auth           │
└─────────────────────────────────────────────┘

Key rule: An identity must pass both cloud IAM and Kubernetes RBAC checks to act on cluster resources. Granting a GCP IAM role does not automatically grant Kubernetes permissions — and vice versa.

Part 1: Core Concepts with Production YAML

1.1 The Four RBAC Objects in Kubernetes

Role          → Namespaced permissions
ClusterRole   → Cluster-wide permissions
RoleBinding   → Binds a Role to a subject (User, Group, ServiceAccount)
ClusterRoleBinding → Binds a ClusterRole cluster-wide

Subjects can be:

• User (human, authenticated via kubeconfig or OIDC)
• Group (collection of users — very useful at scale)
• ServiceAccount (pods, CI/CD pipelines, operators)

1.2 Real Production Example: Read-Only Developer in a Namespace

This is the standard pattern for giving a developer visibility into production without the ability to destroy anything.

# 1. Define what the role CAN do
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: dev-readonly
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "pods/status", "services", "endpoints", "configmaps"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets", "statefulsets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources: ["jobs", "cronjobs"]
    verbs: ["get", "list", "watch"]

---
# 2. Bind it to a specific user
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: dev-readonly-binding
  namespace: production
subjects:
  - kind: User
    name: alice@company.com
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: dev-readonly
  apiGroup: rbac.authorization.k8s.io

Alice can now kubectl logs, kubectl get pods, and kubectl describe deployment in the production namespace. She cannot exec into pods, delete resources, or touch Secrets.

1.3 ServiceAccount for a CI/CD Pipeline (Tekton/ArgoCD Pattern)

Your CI/CD system is a non-human identity. It needs exactly the permissions to deploy — nothing more.

# Dedicated ServiceAccount — never use the 'default' SA for automation
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tekton-deployer
  namespace: cicd

---
# ClusterRole scoped to what a deployer actually needs
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: deployer-role
rules:
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets"]
    verbs: ["get", "list", "update", "patch"]
  - apiGroups: [""]
    resources: ["services", "configmaps"]
    verbs: ["get", "list", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]   # read only — never 'create' or 'delete'

---
# Bind it to the ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tekton-deployer-binding
subjects:
  - kind: ServiceAccount
    name: tekton-deployer
    namespace: cicd
roleRef:
  kind: ClusterRole
  name: deployer-role
  apiGroup: rbac.authorization.k8s.io

1.4 Verify Permissions with kubectl auth can-i

Don’t guess. Test.

# Can alice delete pods in production?
kubectl auth can-i delete pods --namespace=production --as=alice@company.com
# → no

# Can the tekton SA create deployments?
kubectl auth can-i create deployments \
  --namespace=production \
  --as=system:serviceaccount:cicd:tekton-deployer
# → yes

# Audit what a SA can do across all namespaces
kubectl auth can-i --list --as=system:serviceaccount:cicd:tekton-deployer

Part 2: IAM in Production — Real Scenarios

Scenario 1: Multi-Region, Bare-Metal Kubernetes

Running Kubernetes across multiple regions without cloud-managed IAM means you own the full identity stack. A common pattern in fintech and enterprise setups: namespace-per-environment combined with namespace-per-region.

Namespace structure:

payments-us-production
payments-eu-production
payments-ap-production
payments-us-staging

RoleBindings are replicated across namespaces, not shared:

# Reuse a ClusterRole, bind it namespace-by-namespace
# This limits blast radius — compromise in one region doesn't touch others

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sre-write-binding
  namespace: payments-us-production   # scoped to this namespace only
subjects:
  - kind: Group
    name: sre-team
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: sre-production-write
  apiGroup: rbac.authorization.k8s.io

The same ClusterRole is reused — but the RoleBinding scopes it. An SRE can restart pods in payments-us-production but not in payments-eu-production unless explicitly bound.

Scenario 2: Argo CD GitOps with Scoped Permissions

Argo CD is a privileged actor in your cluster. It should NOT run as cluster-admin.

# Argo CD Application Controller ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argocd-application-controller
  namespace: argocd

---
# Grant it deploy rights ONLY in the namespaces it manages
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argocd-controller-binding
  namespace: production
subjects:
  - kind: ServiceAccount
    name: argocd-application-controller
    namespace: argocd
roleRef:
  kind: ClusterRole
  name: admin       # scoped to namespace via RoleBinding
  apiGroup: rbac.authorization.k8s.io

Critical: Use a RoleBinding (not ClusterRoleBinding) even with the admin ClusterRole. This scopes cluster-admin-level power to only the namespace you specify.

Scenario 3: EKS / IRSA — Pod-Level AWS IAM

On AWS EKS, the best practice is IAM Roles for Service Accounts (IRSA). Each pod gets a cryptographically-signed OIDC JWT token that AWS STS validates to assume an IAM role — zero static credentials.

# ServiceAccount annotated with IAM role ARN
apiVersion: v1
kind: ServiceAccount
metadata:
  name: s3-reader
  namespace: payments
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/payments-s3-reader

---
# Pod picks it up automatically
apiVersion: v1
kind: Pod
metadata:
  name: payment-processor
  namespace: payments
spec:
  serviceAccountName: s3-reader
  containers:
    - name: app
      image: payments-app:v2.1.0
      # AWS SDK inside automatically uses the mounted OIDC token
      # No AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY needed

The AWS IAM policy on the role:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:ListBucket"],
    "Resource": [
      "arn:aws:s3:::payments-receipts",
      "arn:aws:s3:::payments-receipts/*"
    ]
  }]
}

Part 3: Scaling IAM Across a Massive Team

This is where most organizations fail. They start with 5 engineers hand-writing YAML, reach 50 engineers, and have 200 RoleBindings no one understands anymore.

3.1 The Golden Rule: Bind Groups, Not Individuals

Never bind individual users. Always bind Groups.

# Bad: binds a person
subjects:
  - kind: User
    name: john.doe@company.com

# Good: binds a team
subjects:
  - kind: Group
    name: backend-engineers

When John leaves, you remove him from the group in your IdP (LDAP/Okta/Google Workspace). The RoleBinding stays untouched. No RBAC cleanup required during offboarding.

3.2 Standardize Role Taxonomy Across Teams

Define a company-wide role taxonomy and enforce it. Example:

Role Name Can Do Cannot Do namespace-viewer get/list/watch all resources create, update, delete, exec namespace-developer CRUD on Deployments, ConfigMaps, Services touch Secrets, RBAC, cluster resources namespace-operator everything in developer + restart pods, exec modify RBAC, create namespaces namespace-admin full namespace control including RBAC cluster-wide access cluster-reader read across all namespaces (SRE / monitoring) write anything cluster-admin break-glass only, audited, time-limited permanent — this should not exist

Store these as ClusterRoles in a Git repo. Apply via CI. Never by hand.

3.3 GitOps for RBAC (The Only Acceptable Approach at Scale)

infrastructure-repo/
├── rbac/
│   ├── cluster-roles/
│   │   ├── namespace-viewer.yaml
│   │   ├── namespace-developer.yaml
│   │   ├── namespace-operator.yaml
│   │   └── cluster-reader.yaml
│   ├── bindings/
│   │   ├── production/
│   │   │   ├── backend-team-binding.yaml
│   │   │   └── sre-team-binding.yaml
│   │   └── staging/
│   │       └── all-engineers-binding.yaml
│   └── service-accounts/
│       ├── tekton-deployer.yaml
│       └── argocd-controller.yaml

PR = Access change. No PR → no access change. Every permission has a Git history. Every approval has a reviewer. This is your audit trail.

3.4 OPA Gatekeeper / Kyverno: Policy Enforcement at Admission

Don’t rely on humans to write correct RBAC. Use policy controllers to reject bad manifests before they land.

Kyverno policy: block wildcard verbs in any Role

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-wildcard-verbs
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-verbs
      match:
        resources:
          kinds:
            - Role
            - ClusterRole
      validate:
        message: "Wildcard verbs (*) are not allowed in roles."
        deny:
          conditions:
            - key: "{{ request.object.rules[].verbs[] | contains(@, '*') }}"
              operator: Equals
              value: true

What this catches: A developer who copy-pastes a role with verbs: ["*"] gets a hard rejection at kubectl apply time — not a security audit six months later.

3.5 Just-in-Time (JIT) Access for Production

For production-level sensitive operations, permanent access is a liability. The pattern:

Engineer requests access → Slack bot notifies on-call lead → Approval triggers
a time-boxed RoleBinding (e.g., 2-hour TTL) → Automatic cleanup after TTL expires

Tools: Apono, Teleport, Boundary (HashiCorp). Or roll your own with a simple operator.

Manual JIT script pattern (quick and dirty):

#!/bin/bash
# jit-access.sh — grant 2-hour production access

USER=$1
NAMESPACE=${2:-production}
DURATION=${3:-120}  # minutes

kubectl create rolebinding "jit-${USER}-$(date +%s)" \
  --clusterrole=namespace-operator \
  --user="${USER}" \
  --namespace="${NAMESPACE}"

echo "Access granted. Auto-revoke in ${DURATION} minutes."

sleep $((DURATION * 60))

kubectl delete rolebinding "jit-${USER}-$(date +%s)" \
  --namespace="${NAMESPACE}" 2>/dev/null || true

echo "Access revoked for ${USER}."

Part 4: Tips & Tricks That Save Hours

4.1 kubectl-who-can — Reverse Permission Lookup

Instead of reading every RoleBinding, ask: “who can delete secrets in production?”

# Install
kubectl krew install who-can

# Usage
kubectl who-can delete secrets -n production
# Returns: all users, groups, SAs that have this permission

kubectl who-can exec pods -n production
# Shows you exactly who can shell into your prod pods

Use case: Pre-audit sweep before a compliance review. Run this across all sensitive verbs (exec, delete, create on Secrets) and fix what you find.

4.2 Audit Logs: The RBAC Debug Trail

Kubernetes API server audit logs record every RBAC decision. Enable them and route to your log aggregator.

# audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log all secret access at RequestResponse level
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]
  # Log RBAC changes
  - level: Metadata
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "clusterroles", "rolebindings", "clusterrolebindings"]
  # Log everything else at Metadata level
  - level: Metadata

Add to your kube-apiserver flags:

--audit-log-path=/var/log/kubernetes/audit.log
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-maxage=30
--audit-log-maxbackup=10

Then query it:

# Who exec'd into pods in the last hour?
grep '"verb":"create"' /var/log/kubernetes/audit.log | \
  grep '"resource":"pods"' | \
  grep '"subresource":"exec"' | \
  jq '{user: .user.username, pod: .objectRef.name, time: .requestReceivedTimestamp}'

4.3 Simulate Access Before Applying

Before handing over a RoleBinding, simulate what the user will see:

# What can 'alice' do in production?
kubectl auth can-i --list \
  --namespace=production \
  --as=alice@company.com

# Test a ServiceAccount's permissions
kubectl auth can-i create deployments \
  --namespace=staging \
  --as=system:serviceaccount:cicd:tekton-deployer

4.4 Annotate Everything with Owner and Expiry

metadata:
  name: contractor-access-binding
  namespace: production
  annotations:
    iam.company.io/owner: "platform-team"
    iam.company.io/ticket: "INFRA-4421"
    iam.company.io/expires: "2025-09-01"    # script sweeps expired bindings
    iam.company.io/reviewed-by: "john.doe"

A cron job that sweeps for expired annotations and sends Slack alerts costs 30 minutes to write and saves your entire access hygiene.

4.5 Detect Over-Permissioned ServiceAccounts

# Find all SAs with ClusterRoleBindings (should be very few)
kubectl get clusterrolebindings -o json | \
  jq -r '.items[] | select(.subjects[]?.kind == "ServiceAccount") |
  "\(.metadata.name) → \(.roleRef.name) → \(.subjects[].name)"'

Any service account bound to cluster-admin or a wildcard ClusterRole is an incident waiting to happen.

4.6 Default ServiceAccount Lockdown

Kubernetes auto-mounts the default ServiceAccount token into every pod. Lock it down:

# At namespace level — set default SA to not auto-mount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: production
automountServiceAccountToken: false

# At pod level — override explicitly for pods that need it
spec:
  serviceAccountName: my-specific-sa
  automountServiceAccountToken: true

This prevents any arbitrary pod in your namespace from making Kubernetes API calls using the default SA — a common lateral movement vector.

Part 5: Red Pill Insights

These are the uncomfortable truths that nobody puts in the official docs.

⚠️ cluster-admin is a permanent backdoor

Most clusters have at least 3–5 humans bound to cluster-admin via ClusterRoleBinding. Every single one is a single compromised credential away from a full cluster wipe. The real answer is: no human should have a permanent ClusterRoleBinding to cluster-admin. Break-glass access only, with JIT, MFA re-auth, and full audit.

⚠️ Your CI/CD pipeline is your highest-risk identity

The entity with the broadest, most consistently-used permissions in your cluster is not an engineer — it’s your CI pipeline. If Tekton or Jenkins is compromised, it can push malicious container images, modify deployments silently, and pivot to secrets. Scope it like a hostile external actor that happens to need deploy access.

⚠️ RBAC gives you no “deny” rules

Kubernetes RBAC is purely additive. There is no Deny verb. If you bind a user to two roles, they get the union. The only way to restrict already-granted permissions is to remove bindings or use OPA/Kyverno admission webhooks — which operate at a different layer. Teams that don't know this spend hours wondering why their "restricted" user still has access.

⚠️ The default service account is mounted in every pod by default

Unless you explicitly disable it, every pod in your cluster has a token that can authenticate to the Kubernetes API. In older clusters (pre-1.24), those tokens never expire. That means any RCE in a pod is also a Kubernetes API access event. This is not theoretical — it’s how the 2023 Aqua Security research team found exposed clusters being actively cryptomined.

⚠️ IAM drift is silent and cumulative

Permissions added for an incident at 2 AM never get removed. Contractors finish their engagement and their bindings stay. Services get deprecated but their ServiceAccounts keep accumulating permissions. Six months later, your RBAC config is a graveyard of forgotten access. Automated quarterly RBAC audits are not optional — they are the minimum.

⚠️ Namespaces are not a security boundary

A common misconception: “I’ll put untrusted workloads in a separate namespace and they’re isolated.” Namespaces provide logical, not security, isolation. A pod with the right ServiceAccount permissions can still read secrets from other namespaces, exec into pods in other namespaces, and escalate via RBAC. Actual isolation requires Network Policies + PodSecurityAdmission + RBAC — all three together.

⚠️ Non-human identities are your actual attack surface

In 2025, the average cloud environment has 41 non-human identities per human. Your humans rotate passwords, use MFA, and get offboarded. Your service accounts use static tokens, never rotate, and nobody knows who owns them. The BeyondTrust breach in 2024 was a service account. The Codecov breach was a build pipeline. Treat every ServiceAccount as a first-class security principal with an owner, a scope, a TTL, and regular rotation.

Conclusion: IAM Is Operational Maturity

IAM is not a checkbox. It’s the operational discipline of knowing exactly who can do what to your infrastructure — and having evidence of it.

The teams that treat IAM as a first-class concern:

• Survive their engineers’ worst days without production incidents
• Pass compliance audits without scrambling
• Onboard contractors safely
• Offboard employees cleanly
• Detect intrusions by noticing permission anomalies before damage is done

The teams that don’t: they find out the hard way — usually at 3 AM with a Slack message that begins with “hey, something is deleting our pods.”

Start with least privilege. Git-track every binding. Audit quarterly. Automate the tedious parts. That’s the entire playbook.

Quick Reference Cheatsheet

# View all Role/ClusterRole bindings
kubectl get rolebindings,clusterrolebindings --all-namespaces -o wide

# What can a specific user do?
kubectl auth can-i --list --as=user@company.com -n production

# Who has access to secrets in production?
kubectl who-can get secrets -n production

# Check a ServiceAccount's effective permissions
kubectl auth can-i --list \
  --as=system:serviceaccount:NAMESPACE:SA_NAME \
  -n TARGET_NAMESPACE

# Find all ClusterRoleBindings to cluster-admin
kubectl get clusterrolebindings -o json | \
  jq -r '.items[] | select(.roleRef.name=="cluster-admin") |
  "\(.metadata.name): \(.subjects)"'

# Dry-run a RoleBinding before applying
kubectl apply --dry-run=server -f rolebinding.yaml

# Audit: show all exec events from audit log
grep '"subresource":"exec"' /var/log/kubernetes/audit.log | \
  jq '{user: .user.username, pod: .objectRef.name}'

Written for SRE, DevOps, and Platform Engineers running production Kubernetes clusters. Last updated: June 2025