Kubernetes Resource Requests Are a Lie. How to Fix Them

How to use VPA and Goldilocks to right-size production workloads without breaking SLOs

Open your cluster right now and run this:

kubectl top pods -A | sort -k3 -rn | head -20

Then run this:

kubectl get pods -A -o json | jq -r '.items[] | 
  .metadata.namespace + "/" + .metadata.name + " requests: " + 
  (.spec.containers[0].resources.requests.cpu // "none") + 
  " / " + (.spec.containers[0].resources.requests.memory // "none")'

Compare the two outputs. In most clusters, the gap between what pods request and what they actually use is enormous. A pod requesting 2 CPU cores using 0.08. A pod requesting 4Gi memory sitting at 400Mi. Multiply that across hundreds of pods and you have a cluster that’s simultaneously over-provisioned on paper and perpetually short on schedulable capacity.

Resource requests aren’t a technical problem. They’re a social one. Engineers set them high at deployment time because nobody wants to be the person who caused an OOMKill. They never get revisited because there’s no forcing function. They accumulate until a FinOps audit finds them or the compute bill forces the conversation.

Here’s how to make them accurate — without breaking anything that’s already running.

Why wrong requests cause real problems

Resource requests in Kubernetes do two things: they influence scheduling decisions and they affect quality of service. The scheduler uses requests — not limits — to decide which node has room for a pod. If you request 2 CPU and use 0.08, you’re holding 1.92 CPU hostage from the scheduler. That node shows as full when it has real capacity available.

Over-provisioning requests — nodes fill up on paper before they fill up in reality. Cluster autoscaler adds nodes that aren’t needed. You’re paying for compute headroom that exists only in your YAML.

Under-provisioning requests — the scheduler places too many pods on a node. When real load hits, pods compete for CPU and memory. The ones with low requests get throttled first. SLOs break.

Neither extreme is safe. The only answer is accuracy — and accuracy requires data.

The Vertical Pod Autoscaler — recommendations without enforcement

VPA observes actual resource usage over time and generates recommendations. Optionally it applies those recommendations by restarting pods — but most teams should start with recommendation mode only. You get the data without the risk of VPA restarting production pods at an inconvenient moment.

Install via Helm:

helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm install vpa fairwinds-stable/vpa \
  --namespace vpa \
  --create-namespace \
  --set recommender.enabled=true \
  --set updater.enabled=false \
  --set admissionController.enabled=false

updater.enabled=false and admissionController.enabled=false puts VPA in pure recommendation mode — it watches and suggests, never acts.

Create a VPA object per workload:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-server-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  updatePolicy:
    updateMode: "Off"  # recommendations only, no automatic updates
  resourcePolicy:
    containerPolicies:
      - containerName: api-server
        minAllowed:
          cpu: 50m
          memory: 64Mi
        maxAllowed:
          cpu: 2000m
          memory: 2Gi

After 24–48 hours, read the recommendations:

kubectl get vpa api-server-vpa -n production -o yaml

The status.recommendation section is what you're after:

status:
  recommendation:
    containerRecommendations:
    - containerName: api-server
      lowerBound:
        cpu: 80m
        memory: 120Mi
      target:
        cpu: 150m
        memory: 256Mi
      upperBound:
        cpu: 400m
        memory: 512Mi

target is what VPA recommends for normal operation. upperBound is a buffer for traffic spikes. Use target as your request value and upperBound as your limit — not the other way around.

The sharp edge: VPA recommendations are only as good as the observation window. 18 hours of data misses weekend patterns, batch jobs, and month-end load. Give it at least 7 days, ideally 14, before trusting numbers for production workloads.

Goldilocks — recommendations at scale

VPA per-workload works for a handful of deployments. For a cluster with 50 or 200 workloads, Goldilocks creates VPA objects for every Deployment in a namespace and surfaces everything through a single dashboard.

helm install goldilocks fairwinds-stable/goldilocks \
  --namespace goldilocks \
  --create-namespace

kubectl label namespace production goldilocks.fairwinds.com/enabled=true
kubectl label namespace staging goldilocks.fairwinds.com/enabled=true

Access the dashboard:

kubectl port-forward svc/goldilocks-dashboard 8080:80 -n goldilocks

The dashboard shows current requests alongside VPA’s lowerBound, target, and upperBound for every workload — and generates the Helm values diff to apply each recommendation. For a workload running with inflated requests, it looks like this:

# Goldilocks suggested values for api-server
resources:
  requests:
    cpu: 150m      # was 2000m
    memory: 256Mi  # was 2Gi
  limits:
    cpu: 400m      # was 2000m
    memory: 512Mi  # was 2Gi

What to look for in the dashboard:

• Workloads where current CPU request is 10x the target — biggest savings, lowest risk
• Workloads where current memory request is below lowerBound — OOMKill risk hiding in plain sight
• Workloads with wide lowerBound-to-upperBound gaps — spiky traffic, needs careful limits

Applying recommendations without breaking SLOs

The wrong way to apply right-sizing is the common way: copy the VPA target directly into your Deployment manifests, apply everything at once, and wait to see what breaks.

# Wrong — limits equal to requests, applied all at once
resources:
  requests:
    cpu: 150m
    memory: 256Mi
  limits:
    cpu: 150m      # same as request — any spike gets throttled immediately
    memory: 256Mi  # same as request — any spike OOMKills the pod

Setting limits equal to requests is the most common right-sizing mistake. It feels precise. In practice, any traffic spike that pushes usage above the request immediately hits the limit — CPU gets throttled, memory triggers an OOMKill. You’ve traded over-provisioning for brittleness.

# Right — limits above requests, applied in stages
resources:
  requests:
    cpu: 150m      # VPA target
    memory: 256Mi  # VPA target
  limits:
    cpu: 400m      # VPA upperBound — headroom for spikes
    memory: 512Mi  # VPA upperBound — headroom for spikes

The staged rollout pattern:

1. Start with non-critical workloads. Background workers, batch jobs, internal tooling. Apply recommendations directly. Watch for two weeks. Build confidence before touching customer-facing services.

2. Step down in stages for production workloads. Don’t jump from 2000m to 150m in one change. Go to 800m first, observe for a week, then to 400m, then to target. Each step is a smaller blast radius.

3. Watch these for 48 hours after each change:

# OOMKill events
kubectl get events -n production --field-selector reason=OOMKilling

# HPA unexpected scaling — signals resource pressure
kubectl get hpa -n production -w

# CPU throttling visible in metrics-server
kubectl top pods -n production

The VPA and HPA conflict: never enable VPA’s Auto mode on a workload that uses HPA. They fight — VPA resizes pods while HPA tries to scale them horizontally. Keep VPA in Off mode for any HPA-managed workload and apply recommendations manually.

The things that will trip you up

Memory recommendations lag real spikes. VPA’s memory recommender uses a percentile model that dampens short spikes. A monthly batch job that doubles memory usage for two hours may not appear in a 7-day window. Check usage graphs for the full billing cycle before trusting memory numbers.

Namespace ResourceQuotas block VPA updates. If a namespace quota caps memory at 4Gi and VPA tries to schedule a pod with 512Mi — down from 2Gi — the math still works. But if VPA recommends up (catching an under-provisioned workload), the pod can fail to start if the quota is already tight. Audit quotas before enabling Auto mode.

VPA needs metrics-server access. In clusters with restrictive RBAC, recommendations stay empty indefinitely. If nothing appears after 48 hours, check VPA recommender logs before assuming the tool doesn’t work.

Before and after

Metric Before After Total requested CPU (47 deployments) 94 cores 31 cores Node count (m5.4xlarge) 6 4 SLO breaches during transition — 0 Monthly compute cost baseline −33%

The cluster wasn’t under-engineered. The resource requests were over-engineered — set once, never revisited, accumulated across two years of deployments. Fixing them didn’t require more monitoring, more engineering, or more risk tolerance. It required looking at data VPA had already collected and acting on it.

Run kubectl top pods. Deploy VPA in recommendation mode. Open Goldilocks in a week.

The numbers will be uncomfortable. That’s the point.

Tom Jose is a DevOps Engineer at Kotaicode, where the team helps companies get their Kubernetes and AWS infrastructure production-ready.