My Kubernetes Journey #3: Bridging AWS Secrets Manager and SSM Parameter Store to Kubernetes

Introduction

When I first started working with Kubernetes on AWS EKS, I was juggling between Secrets Manager and SSM Parameter Store for managing sensitive data. But the bridge to Kubernetes secrets felt clunky. Either I had to bake secrets into manifests (eww), or resort to manual syncs. That’s when I stumbled upon External Secrets Chart — and everything changed.

This story is a walk through how I leveraged the External Secrets Chart to seamlessly connect AWS Secrets Manager and Parameter Store to Kubernetes, turning secret management from a headache into a delight.

Problem Discovery or Background

At the time, we were deploying workloads on EKS in a multi-namespace setup. Secrets were scattered across AWS Secrets Manager and Parameter Store, depending on who provisioned them. We had:

• SSM paths like /production/service-name/env
• Secrets Manager entries for PostgreSQL, API tokens, and SMTP creds

And all of this had to be reflected in Kubernetes Secrets.

The initial workaround was to either:

1. Manually sync and create secrets in each namespace, or
2. Use a bash script during deployment to read from AWS and create k8s secrets via kubectl

Neither was elegant. Rotation? Error-prone. Visibility? A mess.

We were already using EKS with IRSA (IAM Roles for Service Accounts), so there had to be a cleaner way.

What Went Wrong or What Was Tricky

Reading the docs on External Secrets Operator gave me hope… until I tried implementing it.

• The terminology was confusing: SecretStore, ClusterSecretStore, ExternalSecret
• Different backends had different formats (SSM path vs SecretsManager name)
• IAM permissions were tricky — initially, secrets weren’t syncing, and there was no clear error

Add to that, the CRDs had to be installed properly, and the documentation didn’t explain well how namespaces interact with the operator’s permissions.

Troubleshooting / Solution Process

Step 1: Install External Secrets Operator

We used Helm:

helm repo add external-secrets https://charts.external-secrets.io
helm upgrade --install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace

Step 2: Create IAM Role for External Secrets Access

We created a trust policy for IRSA and mapped it to a Kubernetes ServiceAccount in the external-secrets namespace.

⚠️ Caveat: You need to annotate the service account with the correct IAM role ARN. This is crucial for allowing External Secrets to fetch secrets using IRSA. As of the time of writing, pod identity is not supported by External Secrets Operator.

Step 3: Define the SecretStore

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-southeast-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets

For Parameter Store, just change service: SSM and define the path accordingly.

⚠️ Caveat: If you’re using SSM Parameter Store and planning to load values via envFrom, make sure the parameter value is in JSON format. Otherwise, the operator won't be able to map the keys into environment variables.

Step 4: Define the ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app-secret
  namespace: fluer-api
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: ClusterSecretStore
  target:
    name: app-secret
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: prod/database-url

Now, Kubernetes automatically syncs the secret value under prod/database-url into a secret named app-secret in the fluer-api namespace.

⚠️ Caveat: For AWS Secrets Manager, especially secrets like RDS credentials that are stored as JSON with lowercase keys, you may need to remap the keys to uppercase to follow environment variable naming conventions.

Wrap-up & Learnings

What I’d Do Differently

• Set proper RBAC and IAM mapping per namespace in advance
• Use a template to ensure consistency for multi-service setups

Watch Out For

• IRSA misconfigurations: check trust relationship and assume role policy
• SSM vs SecretsManager key naming — be explicit
• Sync errors usually mean missing permissions or invalid key path
• Parameter Store must return JSON for envFrom use case
• Secrets Manager values may need remapping to conform to ENV_VAR naming standards

If you’re managing secrets across AWS and Kubernetes, give External Secrets Chart a go. It’s saved us countless hours and reduced misconfigurations by a mile.

Got questions or want to share your setup? Let’s chat in the comments!