I Deleted My Production Namespace and Restored It in 3 Minutes — Here is Everything I Learned About Velero

A hands-on guide to Kubernetes backup, disaster recovery, and the PostgreSQL bug .

The Setup

I was exploring Velero for our production EKS cluster. The pitch sounded simple: install Velero, it backs up your Kubernetes objects and EBS volumes, and when disaster strikes you run one command to restore everything.

Reality was more interesting. Over several hours of live testing, I hit every gotcha in the book — a silent PostgreSQL data loss bug baked into the official Docker image, an IRSA trust policy misconfiguration that took three rounds to fix, and a restore that showed PartiallyFailed but actually worked perfectly.

This is everything I learned, written for the next engineer who doesn’t want to lose a weekend to the same traps.

What is Velero?

Velero is a CNCF open-source tool for backing up and restoring Kubernetes clusters. It does two things:

1. Kubernetes object backup — Velero queries the Kubernetes API and serializes every resource in your namespace (Deployments, StatefulSets, Services, ConfigMaps, Secrets, PVCs, etc.) to JSON files stored in S3.

2. Volume data backup — For each PersistentVolumeClaim, Velero triggers an EBS snapshot — a point-in-time block-level copy of the disk.

When you restore, Velero replays the K8s objects against the API server and creates new EBS volumes from the snapshots. Your pods come back up attached to their restored data.

Architecture at a glance

Inside your cluster, Velero runs as a single pod in the velero namespace. It watches for Backup and Restore CRDs, calls AWS APIs (via IRSA) to create EBS snapshots, and reads/writes Kubernetes object metadata to S3.

Two CRDs tell Velero where to store things:

• BackupStorageLocation points to your S3 bucket (stores K8s object definitions)
• VolumeSnapshotLocation points to AWS EBS snapshots (stores actual disk data)

Velero uses Custom Resource Definitions for everything. A Backup CRD triggers a backup run. A Schedule CRD creates recurring backups. A Restore CRD triggers a restore. This makes the entire backup lifecycle declarative and auditable in Git.

Prerequisites — Install These First

Before installing Velero, make sure the following tools are installed and configured.

1. AWS CLI

# Install (Linux/macOS)
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip && sudo ./aws/install

# Verify
aws --version
aws sts get-caller-identity   # confirms credentials are working

2. kubectl

# Install (Linux)
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl && sudo mv kubectl /usr/local/bin/

# macOS
brew install kubectl

# Verify — must point to your EKS cluster
kubectl version --client
kubectl cluster-info

3. eksctl

# Install (Linux/macOS)
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin

# Verify
eksctl version

4. Helm

# Install (Linux/macOS)
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Verify
helm version

5. Velero CLI

# Install (Linux)
VELERO_VERSION=v1.13.0
curl -Lo velero.tar.gz https://github.com/vmware-tanzu/velero/releases/download/${VELERO_VERSION}/velero-${VELERO_VERSION}-linux-amd64.tar.gz
tar -xzf velero.tar.gz
sudo mv velero-${VELERO_VERSION}-linux-amd64/velero /usr/local/bin/

# macOS
brew install velero

# Verify
velero version --client-only

Troubleshoot: Prerequisites

# Check all tools are installed
for tool in aws kubectl eksctl helm velero; do
  echo -n "$tool: "; command -v $tool && $tool version --client 2>/dev/null | head -1 || echo "NOT FOUND"
done

# Check kubeconfig is pointing to the right cluster
kubectl config current-context
kubectl config get-clusters

# Switch cluster context
aws eks update-kubeconfig --region us-east-1 --name prod-eks-cluster

Step 1: Create EKS Cluster with IRSA

Create the cluster

# Create cluster from eksctl config
eksctl create cluster -f eks_cluster_creation/eksctl.yml

# This takes 15-20 minutes. Watch progress:
eksctl utils describe-stacks --region=us-east-1 --cluster=prod-eks-cluster | grep StackStatus

eksctl config with Velero IRSA

# eks_cluster_creation/eksctl.yml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: prod-eks-cluster
  region: us-east-1
  version: "1.35"

iam:
  withOIDC: true
  serviceAccounts:
    - metadata:
        name: velero-server    
        namespace: velero
      attachPolicy:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action: ["ec2:*"]
            Resource: "*"
          - Effect: Allow
            Action: ["s3:*"]
            Resource: "*"

Critical detail: Helm’s Velero chart creates a ServiceAccount named velero-server, not velero. If your IRSA trust policy references system:serviceaccount:velero:velero, every S3 call will fail with AccessDenied. Match the name exactly.

Troubleshoot: Cluster Creation

# Check cluster status
eksctl get cluster --name prod-eks-cluster --region us-east-1

# Check OIDC provider was created
aws iam list-open-id-connect-providers | grep -i eks

# Check IRSA role was created
aws iam list-roles --query "Roles[?contains(RoleName, 'velero')].{Name:RoleName,Arn:Arn}" --output table

# Check trust policy on the IRSA role
ROLE_ARN=$(aws iam list-roles --query "Roles[?contains(RoleName, 'addon-iamserviceaccount-velero')].Arn" --output text)
aws iam get-role --role-name $(basename $ROLE_ARN) --query Role.AssumeRolePolicyDocument

# Check nodes are ready
kubectl get nodes -o wide
kubectl describe node <node-name> | grep -A5 "Conditions:"

Step 2: Create S3 Bucket for Velero

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
BUCKET="velero-backups-${AWS_ACCOUNT_ID}"
REGION="us-east-1"

# Create bucket
aws s3api create-bucket \
  --bucket $BUCKET \
  --region $REGION

# Enable versioning (important for recovery)
aws s3api put-bucket-versioning \
  --bucket $BUCKET \
  --versioning-configuration Status=Enabled

# Block public access
aws s3api put-public-access-block \
  --bucket $BUCKET \
  --public-access-block-configuration \
    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

echo "Bucket ready: $BUCKET"

Troubleshoot: S3 Bucket

# Verify bucket exists and versioning is on
aws s3api get-bucket-versioning --bucket $BUCKET

# Test Velero can write to bucket (simulate what Velero does)
aws s3 ls s3://$BUCKET
echo "test" | aws s3 cp - s3://$BUCKET/velero-test.txt
aws s3 rm s3://$BUCKET/velero-test.txt

# Check bucket region matches cluster region
aws s3api get-bucket-location --bucket $BUCKET

Step 3: Install Velero with Helm

Helm values file

# velero/values.yaml
credentials:
  useSecret: false    # IRSA handles auth

deployNodeAgent: false  # Not needed for EBS snapshots

configuration:
  backupStorageLocation:
    - name: default
      provider: aws
      bucket: ""#Add Bucket name here,to store the backups
      config:
        region: us-east-1
  volumeSnapshotLocation:
    - name: default
      provider: aws
      config:
        region: us-east-1

initContainers:
  - name: velero-plugin-for-aws
    image: velero/velero-plugin-for-aws:v1.9.0  #Use Latest imageid
    volumeMounts:
      - mountPath: /target
        name: plugins

schedules:
  myapp-backup:   # You can add critical namespace or full cluster
    schedule: "0 */6 * * *"   # Every 6 hours in production
    template:
      ttl: "720h"     # Keep 7 or 30 days based on the requirement
      includedNamespaces: ["myapp"] 
      snapshotVolumes: true
      defaultVolumesToFsBackup: false

Install

# Add Helm repo
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm repo update

# Get the IRSA role ARN eksctl created
VELERO_ROLE_ARN=$(aws iam list-roles \
  --query "Roles[?contains(RoleName, 'addon-iamserviceaccount-velero')].Arn" \
  --output text --region us-east-1 | head -1)

echo "Using role: $VELERO_ROLE_ARN"

# Create namespace
kubectl create namespace velero --dry-run=client -o yaml | kubectl apply -f -

# Install
helm install velero vmware-tanzu/velero \
  --namespace velero \
  -f velero/values.yaml \
  --set configuration.backupStorageLocation[0].bucket=$BUCKET \
  --set "serviceAccount.server.annotations.eks\.amazonaws\.com/role-arn=${VELERO_ROLE_ARN}" \
  --wait

# Force IRSA token injection
# (Kubernetes only injects the IRSA token at pod creation time.
#  If the annotation was set after the pod started, restart forces
#  a fresh pod with the token properly mounted.)
kubectl rollout restart deployment/velero -n velero
kubectl rollout status deployment/velero -n velero --timeout=60s

Upgrade (if already installed)

helm upgrade velero vmware-tanzu/velero \
  --namespace velero \
  -f velero/values.yaml \
  --set configuration.backupStorageLocation[0].bucket=$BUCKET \
  --set "serviceAccount.server.annotations.eks\.amazonaws\.com/role-arn=${VELERO_ROLE_ARN}" \
  --wait

After install/upgrade, verify the Helm release:

helm list -n velero

Output:

NAME    NAMESPACE  REVISION  UPDATED                               STATUS    CHART          APP VERSION
velero  velero     3         2026-03-11 18:55:07 +0530 IST         deployed  velero-11.4.0  1.17.1

• REVISION 3 — this is the 3rd helm install/upgrade (tracks history)
• APP VERSION 1.17.1 — the Velero version running

What does Helm actually install?

After helm install, run kubectl get all -n velero:

NAME                          READY   STATUS    RESTARTS   AGE
pod/velero-778b5c64d7-r8f9r   1/1     Running   0          5m

NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/velero   ClusterIP   172.20.224.87   <none>        8085/TCP   5m

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/velero   1/1     1            1           5m

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/velero-778b5c64d7   1         1         1       5m

Important: kubectl get all does NOT show the scheduler, backups, or storage locations. These are Velero Custom Resources (CRDs) — they are invisible to get all. You must query them by type.

Seeing ALL Velero components (the right command)

# This shows everything Velero manages
kubectl get schedules,backups,restores,backupstoragelocation,volumesnapshotlocation -n velero

Example output:

NAME                                              STATUS    SCHEDULE      LASTBACKUP   AGE
schedule.velero.io/velero-learning-backup         Enabled   */4 * * * *   2m           2h

NAME                                              STATUS      EXPIRES   AGE
backup.velero.io/myapp-backup-20260311130801      Completed   23h       2h
backup.velero.io/myapp-backup-20260311131201      Completed   23h       2h
backup.velero.io/dr-drill-001                     Completed   6d        1h

NAME                                              PHASE       LAST VALIDATED   AGE
backupstoragelocation.velero.io/default           Available   45s              2h

NAME                                              AGE
volumesnapshotlocation.velero.io/default          2h

 kubectl get volumesnapshotlocation -n velero     
>>                                          
NAME      AGE
default   3h6m
PS C:\Users\rishi_personal\OneDrive\Documents\eks-cluade> kubectl describe volumesnapshotlocation default -n velero
>> 
Name:         default
Namespace:    velero
Labels:       app.kubernetes.io/instance=velero
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=velero
              helm.sh/chart=velero-11.4.0
Annotations:  meta.helm.sh/release-name: velero
              meta.helm.sh/release-namespace: velero
API Version:  velero.io/v1
Kind:         VolumeSnapshotLocation
Metadata:
  Creation Timestamp:  2026-03-11T13:07:56Z
  Generation:          1
  Resource Version:    4993
  UID:                 aa1ca6ed-eac6-4877-8e37-9a52b5db1e06
Spec:
  Config:
    Region:  us-east-1
  Provider:  aws
Events:      <none>

The schedule is there — it just lives in its own CRD, not in the standard Kubernetes resource types.

Verify IRSA annotation on the ServiceAccount

kubectl get sa velero-server -n velero -o jsonpath='{.metadata.annotations}'

Output: { "eks.amazonaws.com/role-arn": "arn:aws:iam::047431531039:role/eksctl-prod-eks-cluster-addon-iamserviceaccou-Role1-fNvbWDoHJk4o", "meta.helm.sh/release-name": "velero", "meta.helm.sh/release-namespace": "velero" }

The eks.amazonaws.com/role-arn annotation is what enables IRSA. If this is missing, the Velero pod will use the node’s IAM role instead and S3 calls will fail.

Verify BackupStorageLocation and VolumeSnapshotLocation

kubectl get backupstoragelocation -n velero

Output:

NAME      PHASE       LAST VALIDATED   AGE    DEFAULT
default   Available   64s              147m   true

kubectl get volumesnapshotlocation -n velero

Output:

NAME      AGE
default   147m

kubectl get schedules -n velero

Output:

NAME                     STATUS    SCHEDULE      LASTBACKUP   AGE    PAUSED
velero-learning-backup   Enabled   */4 * * * *   3m24s        147m

All three together confirm Velero is fully operational:

• BackupStorageLocation Available → S3 reachable via IRSA
• VolumeSnapshotLocation exists → EBS snapshots configured
• Schedule Enabled with recent LASTBACKUP → automatic backups running

Troubleshoot: Velero Installation

# Check Velero pod is running
kubectl get pods -n velero
kubectl describe pod -n velero -l app.kubernetes.io/name=velero

# Check Velero logs
kubectl logs -n velero -l app.kubernetes.io/name=velero --tail=50

# Check the ServiceAccount has the IRSA annotation
kubectl get sa velero-server -n velero -o yaml 

# Check BackupStorageLocation (most important health check)
kubectl get backupstoragelocation -n velero
# Should show: Available

# If Unavailable — check what error Velero is seeing
kubectl describe backupstoragelocation default -n velero

Step 4: Verify IRSA is Working

kubectl get backupstoragelocation -n velero
# NAME      PHASE       LAST VALIDATED   AGE
# default   Available   10s              2m

Available means Velero successfully called s3:ListBucket. If you see Unavailable with AccessDenied, your IRSA is broken.

This was the hardest problem I hit. Before diving into commands, here’s the mental model that makes it click:

Three systems must agree on the same ServiceAccount name:

eksctl.yml          IAM Trust Policy         Helm (Kubernetes)
─────────────       ─────────────────        ──────────────────
name: velero-server → velero:velero-server ← SA name: velero-server

eksctl creates the IAM role and sets who can use it (the trust policy). Helm creates the Kubernetes ServiceAccount. These two systems don’t talk to each other — if the names don’t match, AWS silently rejects every S3 call with AccessDenied.

Step 1 — Check if IRSA token is even mounted in the pod:

kubectl exec -n velero deploy/velero -- env | grep AWS

You should see:

AWS_ROLE_ARN=arn:aws:iam::123456789:role/...
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

If these are missing, the IRSA token was never injected. The pod is using the node’s IAM role instead — meaning it has whatever permissions the EC2 node has, not Velero’s dedicated role.

Step 2 — Check the ServiceAccount has the IAM role annotation:

kubectl get sa velero-server -n velero -o jsonpath='{.metadata.annotations}'

Expected output:

{"eks.amazonaws.com/role-arn":"arn:aws:iam::ACCOUNT:role/..."}

If this annotation is missing, Helm didn’t wire up IRSA — check your --set serviceAccount.server.annotations in the helm install command.

Step 3 — Check the IAM trust policy has the right SA name:

ROLE_NAME=$(aws iam list-roles \
  --query "Roles[?contains(RoleName, 'addon-iamserviceaccount-velero')].RoleName" \
  --output text)

aws iam get-role --role-name $ROLE_NAME \
  --query "Role.AssumeRolePolicyDocument.Statement[0].Condition"

Look for the StringEquals condition. It should say velero:velero-server. If it says velero:velero — that’s the mismatch. The trust policy was created when your eksctl config had name: velero but Helm creates the SA as velero-server.

Step 4 — Fix the mismatch (two options):

Option A — Recreate the IRSA role with the correct name (clean, takes 2 min):

eksctl delete iamserviceaccount --name velero --namespace velero --cluster prod-eks-cluster
# Edit eksctl.yml: change name: velero  →  name: velero-server
eksctl create iamserviceaccount -f eks_cluster_creation/eksctl.yml --approve

Option B — Edit the trust policy condition to use a wildcard (faster):

Go to IAM → Roles → find the Velero role → Edit trust policy. Change:

"StringEquals": { "...:sub": "system:serviceaccount:velero:velero" }

to:

"StringLike": { "...:sub": "system:serviceaccount:velero:velero*" }

The wildcard covers both velero and velero-server — useful if you’re not sure which name Helm will use.

Step 5 — Restart Velero to pick up the new token:

kubectl rollout restart deployment/velero -n velero

IRSA tokens are only injected when a pod starts. If you fixed the IAM role after the pod was already running, it won’t pick up the new token until you restart it.

Step 6 — Verify it’s working:

kubectl get backupstoragelocation -n velero
# NAME      PHASE       LAST VALIDATED
# default   Available   10s

Available means Velero successfully called S3 using the IRSA token. You’re done.

The PostgreSQL Data Loss Bug You’ll Never See Coming

Debugging the mystery

After hours of investigation, I mounted the EBS volume directly to a debug pod to see what was actually on disk.

Create a file debug-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: pg-debug
  namespace: myapp
spec:
  containers:
  - name: debug
    image: busybox
    command: ["sh", "-c", "sleep 3600"]
    volumeMounts:
    - name: pg-data
      mountPath: /data
  volumes:
  - name: pg-data
    persistentVolumeClaim:
      claimName: postgres-storage-myapp-db-0   

kubectl apply -f debug-pod.yaml
kubectl exec -n myapp -it pg-debug -- sh

# Inside the pod — check if EBS has any data:
ls -la /data/
# (empty — this is the bug)
df -h /data

# Clean up
kubectl delete pod pg-debug -n myapp

The EBS volume was completely empty. PostgreSQL was running fine, writing data — but writing it somewhere else.

Root cause: the Docker VOLUME shadow

The official postgres:15 image contains this in its Dockerfile:

VOLUME /var/lib/postgresql/data

My original deployment had:

volumeMounts:
- name: postgres-storage
  mountPath: /var/lib/postgresql    # ← Parent of the VOLUME declaration

At runtime:

• EBS volume mounted at /var/lib/postgresql
• Docker sees VOLUME /var/lib/postgresql/data declared in the image
• Docker creates an anonymous (ephemeral) volume at /var/lib/postgresql/data
• PostgreSQL writes to /var/lib/postgresql/data — the anonymous volume
• EBS disk at /var/lib/postgresql stays empty
• On restore, EBS snapshot restores an empty disk
• Data is gone

The fix

containers:
- name: postgres
  image: postgres:15
  args: ["-c", "checkpoint_timeout=30s"]
  env:
  - name: PGDATA
    value: /var/lib/postgresql/data/pgdata   
  volumeMounts:
  - name: postgres-storage
    mountPath: /var/lib/postgresql/data     

Two changes:

1. Mount at /var/lib/postgresql/data — exactly where VOLUME is declared, so no shadow can form
2. Set PGDATA to a subdirectory — raw EBS volumes have a lost+found directory at root; PostgreSQL’s initdb refuses to initialize in a non-empty directory. Using a subdirectory avoids this.

After this fix, the EBS disk contained actual PostgreSQL data files. Backups and restores worked perfectly.

Troubleshoot: PostgreSQL + EBS

kubectl exec -n myapp -it pg-debug -- ls /data/pgdata/
# Bad:  empty or "no such file" — postgres is NOT writing to EBS

kubectl delete pod pg-debug -n myapp

# Step 2 — Get the postgres pod name (use this in the commands below)
kubectl get pods -n myapp
# copy the postgres pod name, e.g. myapp-db-0

# Step 3 — Confirm PGDATA is set to the right path
kubectl exec -n myapp -it myapp-db-0 -- env | grep PGDATA
# Should show: PGDATA=/var/lib/postgresql/data/pgdata

# Step 4 — Confirm postgres is using that path
kubectl exec -n myapp -it myapp-db-0 -- psql -U user -c "SHOW data_directory;"
# Should show: /var/lib/postgresql/data/pgdata

# Step 5 — Force a checkpoint before backup (flush RAM to disk)
kubectl exec -n myapp -it myapp-db-0 -- psql -U user -c "CHECKPOINT;"

Don’t forget checkpoint_timeout

PostgreSQL doesn’t write every commit straight to disk. It keeps recent changes in memory and flushes them periodically. By default that flush happens every 5 minutes.

EBS snapshots capture whatever is on disk at that moment. So if your backup runs just after a flush, great. If it runs just before, you could lose up to 5 minutes of data.

The easy fix — add this to your postgres deployment args:

args: ["-c", "checkpoint_timeout=30s"]

Now postgres flushes every 30 seconds. Worst case data loss drops from 5 minutes to 30 seconds.

Step 5: Automatic Scheduled Backups

Manual backups are good for DR drills. But in production you want backups running automatically without anyone having to remember.

Velero schedules are configured directly in values.yaml and deployed with Helm — no separate scripts needed.

Schedule in values.yaml

# velero/values.yaml
schedules:
  myapp-backup:
    disabled: false
    schedule: "0 */6 * * *"     # Every 6 hours in production
    template:
      ttl: "720h"               # Keep 30 days of backups
      storageLocation: default
      includedNamespaces:
        - myapp                 # Only backup your app namespace
      excludedNamespaces:
        - velero
        - kube-system
        - kube-public
        - kube-node-lease
      snapshotVolumes: true            # Take EBS snapshots
      defaultVolumesToFsBackup: false  # Don't use file system backup (use EBS snapshots)

For learning/testing you can set it to every 4 minutes:

schedule: "*/4 * * * *"    # Every 4 minutes — for learning only
ttl: "24h"                 # Keep only 24 hours

Switch back to "0 */6 * * *" before going to production.

Apply the schedule (Helm upgrade)

# After editing values.yaml, apply with helm upgrade
helm upgrade velero vmware-tanzu/velero \
  --namespace velero \
  -f velero/values.yaml \
  --set configuration.backupStorageLocation[0].bucket=$BUCKET \
  --set "serviceAccount.server.annotations.eks\.amazonaws\.com/role-arn=${VELERO_ROLE_ARN}"

The schedule is immediately active after the upgrade.

Verify the schedule is created and running

# List all schedules
kubectl get schedules -n velero
# NAME           STATUS    SCHEDULE      LAST BACKUP   AGE
# myapp-backup   Enabled   0 */6 * * *   2h ago        1d

# Describe schedule — shows next run time and last backup status
kubectl describe schedule myapp-backup -n velero

# See backups created by the schedule (they are named: <schedule-name>-<timestamp>)
kubectl get backups -n velero | grep myapp-backup
# NAME                          STATUS      EXPIRES   AGE
# myapp-backup-20240311060000   Completed   29d       6h
# myapp-backup-20240311120000   Completed   29d       1m

Manually trigger a scheduled backup now (without waiting)

# Useful for testing the schedule config before waiting 6 hours
velero backup create --from-schedule myapp-backup --wait

# Or trigger with a custom name
velero backup create manual-test --from-schedule myapp-backup --wait

Troubleshoot: Schedule not running

# Check if schedule is enabled (disabled: false)
kubectl get schedule myapp-backup -n velero -o yaml | grep -A2 "paused\|disabled"

# Check last backup status from the schedule

kubectl describe schedule myapp-backup -n velero | grep -A5 "Last Backup"

# Check if Velero controller is running and watching schedules

kubectl logs -n velero deploy/velero | grep -i schedule | tail -20

# If schedule shows errors, check the template is valid by doing a manual backup first
velero backup create test-manual --include-namespaces myapp --snapshot-volumes=true --wait
velero backup describe test-manual   # if this fails, the schedule template has the same issue

# Pause a schedule (stop automatic runs without deleting it)
kubectl patch schedule myapp-backup -n velero \
  --type merge -p '{"spec":{"paused":true}}'

# Resume
kubectl patch schedule myapp-backup -n velero \
  --type merge -p '{"spec":{"paused":false}}'

How automatic backups appear in S3

Each scheduled backup creates a folder in S3:

Each backup gets its own folder under s3://your-velero-bucket/backups/. So after a few runs you’d see something like:

backups/myapp-backup-20240311060000/    <- scheduled at 6am
backups/myapp-backup-20240311120000/    <- scheduled at 12pm
backups/myapp-backup-20240311180000/    <- scheduled at 6pm

Inside each folder: a .tar.gz with all the K8s objects and a logs file.

EBS snapshots are stored in your AWS account (EC2 → Snapshots), tagged with velero.io/backup=<backup-name>.

# See all EBS snapshots created by Velero
aws ec2 describe-snapshots \
  --owner-ids self \
  --filters "Name=tag-key,Values=velero.io/backup" \
  --query "Snapshots[*].{Backup:Tags[?Key=='velero.io/backup'].Value|[0],Id:SnapshotId,State:State,Size:VolumeSize}" \
  --output table

Where Are My Backups Stored? Retention, S3, and EBS Snapshots Explained

This is one of the most common questions. The answer is: in two places simultaneously, and you need to understand both.

The split: S3 + EBS Snapshots

When Velero backs up a namespace with PVCs, it stores data in two different AWS services:

In S3 — under backups/<backup-name>/ you’ll find:

• <backup-name>.tar.gz all K8s resources as JSON
• <backup-name>-resource-list.json.gz index of what was backed up
• <backup-name>-volumesnapshots.json.gz mapping of PVC → EBS snapshot ID
• <backup-name>-logs.gz
• velero-backup.json the backup manifest

In AWS EBS Snapshots (EC2 → Snapshots) the actual disk data from your PVCs. Each snapshot is tagged so you can trace it back:

snap-01067d180005793dd
  Size: 10 GB
  velero.io/backup = real-dr-test
  kubernetes.io/created-for/pvc/name = postgres-storage-myapp-db-0
  kubernetes.io/created-for/pvc/namespace = myapp

S3 stores the Kubernetes object definitions — Deployments, StatefulSets, Services, Secrets, PVCs, etc. Think of it as the “blueprint” of your namespace. Size is small (100KB–few MB).

EBS Snapshots store the actual disk data from your PersistentVolumeClaims — your PostgreSQL data files, uploaded files, anything on a PVC. Size matches your PVC (10GB, 100GB, etc.).

On restore, Velero reads the blueprint from S3 and creates new EBS volumes from the snapshots, then binds them to the restored PVCs.

Retention period (TTL)

Every backup has a TTL (Time To Live). When TTL expires, Velero automatically deletes both the S3 objects AND the EBS snapshots.

TTL is set in your schedule:

# velero/values.yaml
schedules:
  myapp-backup:
    template:
      ttl: "720h"    # 30 days — then auto-deleted

Or override per manual backup:

velero backup create my-backup \
  --include-namespaces myapp \
  --ttl 168h \
  --wait

Check expiration date of a specific backup:

velero backup describe real-dr-test

Real output (key fields):

Name:        real-dr-test
Namespace:   velero
Phase:       Completed

TTL:         720h0m0s
Started:     2026-03-11 20:20:50 +0530 IST
Completed:   2026-03-11 20:20:53 +0530 IST
Expiration:  2026-04-10 20:20:50 +0530 IST   Velero will delete this on April 10

List all backups and check their status:

velero backup get

Real output from our cluster:

NAME                                    STATUS           CREATED                   EXPIRES   STORAGE LOCATION
dr-drill-backup                         Completed        2026-03-11 19:18:09 IST   29d       default
dr-with-data                            Completed        2026-03-11 19:27:21 IST   29d       default
final-dr-test                           Completed        2026-03-11 19:54:49 IST   29d       default
hook-free-dr-test                       Completed        2026-03-11 20:04:31 IST   29d       default
prod-users-backup                       Completed        2026-03-11 19:43:21 IST   29d       default
real-dr-test                            Completed        2026-03-11 20:20:50 IST   29d       default
velero-learning-backup-20260311134420   PartiallyFailed  2026-03-11 13:44:20 IST   23h       default
velero-learning-backup-20260311153220   Completed        2026-03-11 15:32:20 IST   23h       default

You can immediately see two TTL strategies at a glance from the EXPIRES column:

• Manual backups (real-dr-test, dr-drill-backup) → 29d remaining = 30-day TTL
• Scheduled backups (velero-learning-backup-*) → 23h remaining = 24-hour TTL, deleted tomorrow

Where to see the backups in AWS

In S3 — list all backup folders:

aws s3 ls s3://velero-backups-047431531039/backups/

Output:

PRE dr-drill-backup/
                           PRE dr-with-data/
                           PRE final-dr-test/
                           PRE hook-free-dr-test/
                           PRE prod-users-backup/
                           PRE real-dr-test/
                           PRE velero-learning-backup-20260311134420/
                           PRE velero-learning-backup-20260311153220/

Each folder = one backup. Now look inside one:

aws s3 ls s3://velero-backups-047431531039/backups/real-dr-test/

Output:

2026-03-11    29  real-dr-test-csi-volumesnapshotclasses.json.gz
2026-03-11    27  real-dr-test-itemoperations.json.gz
2026-03-11  9261  real-dr-test-logs.gz
2026-03-11    29  real-dr-test-podvolumebackups.json.gz
2026-03-11  1057  real-dr-test-resource-list.json.gz      what was backed up
2026-03-11    49  real-dr-test-results.gz
2026-03-11   304  real-dr-test-volumeinfo.json.gz
2026-03-11   270  real-dr-test-volumesnapshots.json.gz    EBS snapshot IDs stored here
2026-03-11 99711  real-dr-test.tar.gz                     all K8s objects (Deployments, Secrets, etc.)
2026-03-11  3665  velero-backup.json                      backup metadata

The whole namespace — 86 Kubernetes objects — fits in a 99KB tar.gz file. The actual data (PostgreSQL) is in the EBS snapshot, not here.

In EBS Snapshots (EC2 console or CLI):

# List all EBS snapshots created by Velero
aws ec2 describe-snapshots \
  --owner-ids self \
  --filters "Name=tag-key,Values=velero.io/backup" \
  --query "Snapshots[*].{
    BackupName:Tags[?Key=='velero.io/backup'].Value|[0],
    PVC:Tags[?Key=='kubernetes.io/created-for/pvc/name'].Value|[0],
    SnapshotId:SnapshotId,
    State:State,
    SizeGB:VolumeSize,
    Created:StartTime
  }" \
  --output table

Real output from our cluster:

-------------------------------------------------------------------------------------------------------------------------------------------
|                                                       DescribeSnapshots                                                                 |
+-------------------+------------------------------------+------------------------------+---------+-------------------------+-------------+
|    BackupName     |            Created                 |            PVC               | SizeGB  |       SnapshotId        |    State    |
+-------------------+------------------------------------+------------------------------+---------+-------------------------+-------------+
|  real-dr-test     |  2026-03-11T14:50:51.677000+00:00  |  postgres-storage-myapp-db-0 |  10     |  snap-01067d180005793dd |  completed  |
|  prod-users-backup|  2026-03-11T14:13:23.998000+00:00  |  postgres-storage-myapp-db-0 |  10     |  snap-069b818cbb342d112 |  completed  |
|  dr-drill-backup  |  2026-03-11T13:48:15.963000+00:00  |  postgres-storage-myapp-db-0 |  10     |  snap-0ff51cc6d86d7d92b |  completed  |
|  hook-free-dr-test|  2026-03-11T14:34:33.458000+00:00  |  postgres-storage-myapp-db-0 |  10     |  snap-00e6230ef3123fab4 |  completed  |
|  final-dr-test    |  2026-03-11T14:24:50.785000+00:00  |  postgres-storage-myapp-db-0 |  10     |  snap-0cb05ff8147131fad |  completed  |
|  dr-with-data     |  2026-03-11T13:57:25.486000+00:00  |  postgres-storage-myapp-db-0 |  10     |  snap-04cbb0bff76cca127 |  completed  |
+-------------------+------------------------------------+------------------------------+---------+-------------------------+-------------+

Each row is one PVC backup. Every manual backup (real-dr-test, dr-drill-backup, etc.) created its own EBS snapshot. The postgres-storage-myapp-db-0 PVC is the PostgreSQL data disk — 10GB, snapshot taken in seconds, stored in AWS independently.

In the AWS Console:

• S3 → your bucket → backups/ folder
• EC2 → Snapshots → filter by tag velero.io/backup

See which PVC was snapshotted and what EBS snapshot was created

velero backup describe real-dr-test --details

Scroll to the bottom of the output — you’ll see:

Backup Volumes:
  Velero-Native Snapshots:
    pvc-2da23eb0-c5c0-45f3-a1d0-1d887ff30762:
      Snapshot ID:        snap-01067d180005793dd
      Type:               gp3
      Availability Zone:  us-east-1f
      Result:             succeeded

This tells you exactly which PVC was snapshotted and which EBS snapshot holds the data. If you ever need to manually verify the snapshot exists, take that snap-01067d180005793dd ID and check it in EC2 → Snapshots.

What happens when TTL expires

Velero’s garbage collection controller runs periodically and:

1. Finds backups past their expiration date
2. Deletes the S3 folder for that backup
3. Calls ec2:DeleteSnapshot to delete the EBS snapshot
4. Deletes the Backup CRD object from Kubernetes

Both S3 and EBS snapshots are cleaned up together — you don’t need to manage this manually.

Warning: If you manually delete the Backup CRD object in Kubernetes without going through Velero, the EBS snapshots will become orphaned and you’ll keep paying for them. Always delete backups via Velero: velero backup delete <name>.

Does restore auto-pick the latest backup?

No. You always pick the backup yourself. Run this first to see what’s available:

velero backup get

Real output:

NAME                                    STATUS           CREATED                   EXPIRES   STORAGE LOCATION
dr-drill-backup                         Completed        2026-03-11 19:18:09 IST   29d       default
dr-with-data                            Completed        2026-03-11 19:27:21 IST   29d       default
final-dr-test                           Completed        2026-03-11 19:54:49 IST   29d       default
hook-free-dr-test                       Completed        2026-03-11 20:04:31 IST   29d       default
prod-users-backup                       Completed        2026-03-11 19:43:21 IST   29d       default
real-dr-test                            Completed        2026-03-11 20:20:50 IST   29d       default
velero-learning-backup-20260311134420   PartiallyFailed  2026-03-11 13:44:20 IST   23h       default
velero-learning-backup-20260311153220   Completed        2026-03-11 15:32:20 IST   23h       default

Look at the CREATED and EXPIRES columns:

• Scheduled backups expire in 23h — these are short lived, not for DR
• Manual backups expire in 29d — these are the ones to restore from

Pick the backup you want and restore:

velero restore create my-restore --from-backup real-dr-test --wait

That’s it. No scripts, no jq. Just read the list and pick the name.

What happens if AWS Snapshot Lifecycle (DLM) deletes snapshots Velero still references?

This is a silent and dangerous production trap.

The scenario: Your team adds an AWS Data Lifecycle Manager (DLM) policy to clean up all EBS snapshots older than 7 days. Velero’s TTL is 30 days. After day 7, AWS deletes the snapshots. Velero still shows those backups as Completed — but the actual disk data is gone.

What Velero thinks (day 15):     What actually exists in AWS:
────────────────────────────     ────────────────────────────
Backup CRD: Completed          S3 folder: exists 
TTL: 30 days remaining         EBS Snapshot: DELETED by DLM 

Result: Silent restore failure. When you run a restore:

• K8s objects (Deployments, Services, Secrets) restore from S3 successfully
• PVC restore fails Velero calls ec2:CreateVolume from the snapshot ID, gets InvalidSnapshot.NotFound
• PVC stays in Pending state forever
• PostgreSQL pod never starts no storage
• Data is permanently lost, no recovery possible

The same thing happens if you manually delete an EBS snapshot from the AWS console.

How to detect this before it’s too late:

# Check if any Velero-referenced EBS snapshots are missing from AWS
kubectl get backups -n velero -o json | \
  jq -r '.items[].metadata.name' | while read backup; do
    SNAP=$(velero backup describe $backup --details 2>/dev/null | grep "Snapshot ID" | awk '{print $NF}')
    if [ -n "$SNAP" ]; then
      STATE=$(aws ec2 describe-snapshots --snapshot-ids $SNAP \
        --query "Snapshots[0].State" --output text 2>/dev/null)
      echo "$backup → $SNAP → ${STATE:-MISSING}"
    fi
  done
# Output:
# real-dr-test → snap-01067d180005793dd → completed    ← safe
# old-backup   → snap-0deadbeef1234567 → MISSING       ← cannot restore PVCs!

The fix: Never apply blanket AWS lifecycle policies to EBS snapshots. Let Velero manage its own snapshot lifecycle via TTL. If you must use DLM, exclude snapshots tagged with velero.io/backup:

DLM Policy filter: exclude tag key = velero.io/backup

This ensures AWS lifecycle policies skip Velero-managed snapshots entirely.

The Live Disaster Recovery Drill

With the PostgreSQL fix in place, I ran a full DR drill.

Setup create test data

POSTGRES_POD=$(kubectl get pod -n myapp -l app=postgres -o name | head -1)

# Create table and insert rows
kubectl exec -n myapp -it $POSTGRES_POD -- \
  psql -U user myappdb -c "CREATE TABLE users (id SERIAL, name TEXT);"

kubectl exec -n myapp -it $POSTGRES_POD -- \
  psql -U user myappdb -c "INSERT INTO users (name) VALUES ('Alice'), ('Bob'), ('Charlie');"

# Confirm data is there
kubectl exec -n myapp -it $POSTGRES_POD -- \
  psql -U user myappdb -c "SELECT * FROM users;"

# Force checkpoint so data is on disk
kubectl exec -n myapp -it $POSTGRES_POD -- psql -U user -c "CHECKPOINT;"

Trigger backup

velero backup create dr-drill-001 \
  --include-namespaces myapp \
  --snapshot-volumes=true \
  --wait

# Verify backup completed successfully
velero backup describe dr-drill-001
velero backup logs dr-drill-001 | tail -20

# Check EBS snapshot was created
velero backup describe dr-drill-001 --details | grep "Snapshot"

Disaster

kubectl delete namespace myapp
# namespace "myapp" deleted

kubectl get pods -n myapp
# Error from server (NotFound): namespaces "myapp" not found

Recovery

velero restore create dr-drill-001-restore \
  --from-backup dr-drill-001 \
  --wait

# Watch restore progress
velero restore describe dr-drill-001-restore
kubectl get pods -n myapp -w

Verify data survived restore

POSTGRES_POD=$(kubectl get pod -n myapp -l app=postgres -o name | head -1)

kubectl exec -n myapp -it $POSTGRES_POD -- \
  psql -U user myappdb -c "SELECT * FROM users;"

#  id | name
# ----+-------
#   1 | Alice
#   2 | Bob
#   3 | Charlie

All three rows survived the namespace deletion and restore.

Troubleshoot: Backup Failures

# Backup stuck in InProgress
velero backup logs my-backup --follow

# Check Velero pod health during backup
kubectl logs -n velero deploy/velero -f

# Check EBS snapshot status in AWS
aws ec2 describe-snapshots \
  --filters "Name=tag:velero.io/backup,Values=my-backup" \
  --query "Snapshots[*].{Id:SnapshotId,State:State,Progress:Progress}" \
  --output table

# Backup shows PartiallyFailed (not just restore)
velero backup describe my-backup --details 2>&1 | grep -A5 "Errors"

# Check if there are resources Velero can't access
kubectl get events -n velero --sort-by='.lastTimestamp' | tail -20

Troubleshoot: Restore Failures

# See what specifically failed in a PartiallyFailed restore
velero restore describe my-restore --details 2>&1 | grep -A20 "Errors"

# Check all restored resources came back
kubectl get all -n myapp

# Check PVCs are bound (means EBS snapshots restored successfully)
kubectl get pvc -n myapp
# All should show STATUS=Bound

# Check pods are not in CrashLoopBackOff
kubectl get pods -n myapp
kubectl describe pod -n myapp <pod-name>
kubectl logs -n myapp <pod-name> --previous

# If pods are running but app not accessible — check ALB is ready (takes 2-3 min)
kubectl get ingress -n myapp
kubectl describe ingress -n myapp <ingress-name> | grep -A5 "Events"

Understanding PartiallyFailed Restore Status

Your restore will likely show PartiallyFailed. Don’t panic.

kubectl get restore -n velero
# NAME                   AGE   PHASE
# dr-drill-001-restore   2m    PartiallyFailed

velero restore describe dr-drill-001-restore --details
# Warnings:
#   Velero:    <none>
# Errors:
#   Velero:
#     error restoring TargetGroupBinding/myapp/...:
#     the server could not find the requested resource

What’s happening: When you backed up, your namespace had TargetGroupBinding resources (created by the AWS Load Balancer Controller to link ALB target groups to pods). These resources reference specific AWS ALB target groups by ARN.

When you delete the namespace, the ALB Controller deletes those target groups from AWS. On restore, Velero tries to recreate the TargetGroupBinding objects pointing to non-existent target groups — this fails.

The resolution: This is expected and self-healing. The ALB Controller watches Ingress objects. When Velero restores your Ingress, the controller provisions a new ALB and new target groups, then creates new TargetGroupBinding objects. Your application becomes accessible within 2-3 minutes of the restore completing.

The actual application data, pods, and services are all restored successfully. The PartiallyFailed label refers only to these AWS-managed binding objects.

Production Checklist

Before you rely on Velero in production:

Storage

• S3 bucket with versioning enabled
• S3 lifecycle policy (move old backups to Glacier after 30 days)
• Bucket in same region as cluster (avoid cross-region data transfer costs)
• BackupStorageLocation shows
• Available phase

IAM

• IRSA configured — no static credentials in Secrets
• Velero SA annotation matches Helm’s SA name (
• velero-server)
• IAM policy has
• ec2:* and s3:* on appropriate resources
• Trust policy principal matches actual SA name (not a guess)

Database backups

• Mount EBS at
• VOLUME declaration path (not parent)
• PGDATA set to subdirectory inside mount
• checkpoint_timeout set to acceptable window (30s–60s)
• Pre-backup hooks for CHECKPOINT if zero data loss required

Schedule

• Backups scheduled at appropriate frequency (every 6 hours for most workloads)
• TTL set (720h = 30 days recommended)
• snapshotVolumes: true confirmed
• Critical namespaces explicitly included

Testing

• DR drill performed in staging before production
• RTO measured and acceptable
• Data verified after restore (not just pods Running)
• Backup notifications/alerting configured

5 Lessons Learned

1. The Docker VOLUME declaration is a silent killer. The postgres:15 image’s VOLUME /var/lib/postgresql/data will intercept your EBS mount if you’re not careful. The bug is completely invisible during normal operation — Postgres works fine, it’s just writing to ephemeral storage. You only discover it when you try to restore. Always mount at the exact path declared in VOLUME and use a PGDATA subdirectory.

2. IRSA requires exact name matching across three systems. eksctl creates the trust policy. Helm creates the ServiceAccount. The trust policy must reference the exact SA name that Helm uses. These systems don’t talk to each other — you have to verify the chain manually. A single character difference means AccessDenied with no helpful error message.

3. Checkpoints matter for database backups. EBS snapshots are not application-consistent. For databases, either set checkpoint_timeout to a short value or use pre-backup hooks to force a checkpoint. The difference between a 5-minute and 30-second checkpoint timeout is the difference between “we lost the last 5 minutes” and “we lost the last 30 seconds.”

4. PartiallyFailed is the expected success state. Don’t spend time debugging PartiallyFailed restore status before verifying your application actually works. In most EKS setups with ALB, the partial failure is always the TargetGroupBinding resources — which auto-heal. Check your pods and your data first.

5. Velero is not a replacement for database-native backups. Velero + EBS snapshots give you crash-consistent backups. For production databases, you should also run database-native backups (pg_dump, WAL archiving with pgWAL/Barman) for point-in-time recovery. Use Velero for namespace-level DR drills. Use native tools for row-level or transaction-level recovery.

Full Commands Reference

Velero Backup

# Trigger manual backup
velero backup create my-backup \
  --include-namespaces myapp \
  --snapshot-volumes=true \
  --wait

# Backup with TTL override
velero backup create my-backup \
  --include-namespaces myapp \
  --ttl 24h \
  --wait

# Backup multiple namespaces
velero backup create full-backup \
  --include-namespaces myapp,monitoring \
  --wait

# Check backup status
velero backup describe my-backup
velero backup describe my-backup --details   # shows per-resource results
velero backup logs my-backup

# List all backups
velero backup get
kubectl get backups -n velero -o wide

Velero Restore

# Restore full backup
velero restore create my-restore \
  --from-backup my-backup \
  --wait

# Restore only specific namespace
velero restore create my-restore \
  --from-backup my-backup \
  --include-namespaces myapp \
  --wait

# Restore to a different namespace
velero restore create my-restore \
  --from-backup my-backup \
  --namespace-mappings myapp:myapp-restored \
  --wait

# Check restore status
velero restore describe my-restore
velero restore describe my-restore --details   # shows errors
velero restore logs my-restore

# List all restores
velero restore get
kubectl get restores -n velero

Schedule Management

# List schedules
kubectl get schedules -n velero
velero schedule get

# Manually trigger a scheduled backup now
velero backup create --from-schedule myapp-backup

# Pause a schedule
kubectl patch schedule myapp-backup -n velero \
  --type merge -p '{"spec":{"paused":true}}'

# Resume a schedule
kubectl patch schedule myapp-backup -n velero \
  --type merge -p '{"spec":{"paused":false}}'

Health Checks

# Overall Velero health
kubectl get backupstoragelocation -n velero
kubectl get volumesnapshotlocation -n velero
kubectl get pods -n velero

# Velero version (client + server)
velero version

# Check Velero logs for errors
kubectl logs -n velero deploy/velero --tail=100 | grep -i error

# Check recent backup failures
kubectl get backups -n velero -o json | \
  jq -r '.items[] | select(.status.phase != "Completed") | [.metadata.name, .status.phase] | @csv'

AWS Verification

# List EBS snapshots created by Velero
aws ec2 describe-snapshots \
  --owner-ids self \
  --filters "Name=tag-key,Values=velero.io/backup" \
  --query "Snapshots[*].{Id:SnapshotId,Backup:Tags[?Key=='velero.io/backup'].Value|[0],State:State,Size:VolumeSize}" \
  --output table

# Check S3 backup objects
aws s3 ls s3://$BUCKET/backups/ --recursive --human-readable | sort -k1,2

# Check S3 bucket size
aws s3api list-objects-v2 \
  --bucket $BUCKET \
  --query "sum(Contents[].Size)" \
  --output text | awk '{print $1/1024/1024/1024 " GB"}'

Conclusion

Velero is a mature, production-ready tool for Kubernetes disaster recovery. The three minute RTO for a full namespace restore including a live PostgreSQL database with real data is genuinely impressive.

The gotchas are real but solvable:

• The Docker VOLUME shadow bug is the biggest trap for stateful workloads
• IRSA configuration requires careful name alignment across eksctl and Helm
• Checkpoint timing affects data durability

Once you’ve run a successful DR drill in staging and confirmed your RTO is acceptable, Velero gives you meaningful confidence that a namespace-level disaster is recoverable.

The only remaining question is how often you’re running those drills. A backup you’ve never tested is a backup you don’t actually have.