Solving the Karpenter Price-Performance Trap with NodeOverlays

When we migrated our fleet from AWS Auto Scaling Groups (ASGs) and Cluster-autoscaler to **Karpenter**, we expected better efficiency, faster scaling, and lower costs. And we got exactly all that. (I’ve even talked about the “Karpenter Effect” here).

My past talk about Karpenter effect

But shortly after the migration, we hit a roadbump. A customer reported a significant degradation in performance.

Their application became sluggish, and their Horizontal Pod Autoscaler (HPA) was spiking, spinning up more pods than usual. When we looked under the hood, we found the culprit: Karpenter was being “too good” about money.

The Problem: The “price-performance” Trap

Karpenter’s default mission is to find the cheapest instance type that satisfies your requirements. In our case, at that time, for an x86 workload, Karpenter was consistently choosing m5a.4xlarge over the newer m6i.4xlarge.

On paper, the m5a appears cheaper. However, 6th generation instances are roughly 15% more efficient than the 5th generation. This pattern generally holds across instance generations: newer generations tend to be slightly more expensive, but the performance improvements typically outweigh the price difference. Because the older nodes were less performant, the customer’s application had to scale to more pods to handle the same traffic, effectively eliminating any savings from the lower instance cost. It also degraded performance, since their current scaling strategy assumes the performance characteristics of 6th generation instances.

I opened an issue in the upstream Karpenter repo in August 2024 to discuss “price-performance” vs. “pure price.”

The issue I opened regarding price-performance vs price

The Bandage: Layered NodePool Architecture

To workaround this, we implemented what we called a Layered NodePool Architecture. We defined two NodePools:

1. High Priority NodePool (Weight: 50): Forced newer generations (e.g., Newest Generation -1 ).
2. Fallback NodePool (Weight: 10): Allowed older generations (Newest Generation — 2)

# Simplified Layered Architecture example
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: preferred-gen
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.k8s.aws/instance-generation"
          operator: Gt
          values: ["6"]
  weight: 50 # Karpenter tries this first
---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: fallback-gen
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.k8s.aws/instance-generation"
          operator: Gt
          values: ["5"]
  weight: 10 # Only used if the first one fails due to capacity shortage
# the fallback will be latest -2 to prevent having too big of a gap
# (m7 and m5 instead of m7 and m6 for example)

You might wonder: “If newer Generation is better, why not just hard-restrict the NodePool to only allow the newer Generation?” The reality of AWS at scale is that capacity is never guaranteed. Newer instance generations are often the first to hit “Insufficient Instance Capacity” (ICE) errors, especially during regional spikes or when a generation is still being rolled out across all Availability Zones. If we had locked ourselves into only the newest generation, a capacity shortage would have prevented our clusters from scaling at all, leading to a “Pending pods” issue.

Launching a new EC2 instance. Status Reason: We currently do not have sufficient xxxxx capacity in the Availability Zone you requested (eu-west-1b). 
Our system will be working on provisioning additional capacity. 
You can currently get xxxxx capacity by not specifying an Availability Zone in your request or choosing eu-west-1a, eu-west-1c. 
Launching EC2 instance failed.

The layered architecture was our safety net — it told Karpenter: “Please give us the high-performance generation if you can, but if it’s not available, we’ll take the cheaper older to keep the lights on.”

The Downside: Operational Toil

This worked, but it created operational toil. Every time AWS released a new generation, we had to manually bump the version numbers in multiple NodePools across all our clusters to keep the price-performance optimal.

The New Way: Enter NodeOverlays 🚀

Finally, there is a better way. The Karpenter team introduced NodeOverlays (currently in Alpha).

I found this out from a Linkedin post from Christian Melendez and it immediately reminded me of this issue.

NodeOverlays allow you to “tilt” Karpenter’s decision-making by injecting alternative metadata — like price adjustments — into its scheduling simulation. Instead of hard-coding NodePools, we can now penalize older generations to make newer ones look more attractive.

The example specifically for this usecase can be found here: https://github.com/aws-samples/karpenter-blueprints/tree/main/blueprints/node-overlay#scenario-1-prioritizing-latest-generation-instances

How it Works

When Karpenter evaluates which instance to provision, it applies the priceAdjustment from the NodeOverlay CR. By adding a "virtual tax" to older nodes, we guide Karpenter toward the latest hardware without losing the ability to fall back if capacity is tight.

Scenario: Prioritizing the Latest Generation

1. The Baseline

Before applying overlays, our test NodePool (configured with generation > 6) naturally gravitated toward the cheapest Gen 7 options:

# simplified version
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: worker-nodepool-test
spec:
  template:
    spec:
      requirements:
      - key: karpenter.k8s.aws/instance-generation
        operator: Gt
        values:
        - "6"
      - key: karpenter.sh/capacity-type
        operator: In
        values:
        - on-demand
      - key: kubernetes.io/os
        operator: In
        values:
        - linux

# Current state: Karpenter picks the cheapest available (Gen 7)
$ kubectl get nodeclaims
NAME                        TYPE               CAPACITY    ZONE
worker-nodepool-test-dgxfn   c7i-flex.xlarge    on-demand   eu-west-1b
worker-nodepool-test-jn6ml   c7i-flex.4xlarge   on-demand   eu-west-1a

2. Applying the “Virtual Tax”

We want Karpenter to prefer Generation 8. To do this, we apply a priceAdjustment to older generations. It only changes how Karpenter perceives the cost during its decision-making.

Note: Ensure the NodeOverlay feature gate is enabled in your Karpenter controller settings since it’s still an alpha feature!

This needs to be enabled otherwise it won’t work

# penalize-generations.yaml
apiVersion: karpenter.sh/v1alpha1
kind: NodeOverlay
metadata:
  name: penalize-gen7
spec:
  weight: 10
  requirements:
    - key: karpenter.k8s.aws/instance-generation
      operator: In
      values: ["7"]
  priceAdjustment: "+20%" # Makes Gen 7 20% more expensive

Once applied, verify they are active:

$ kubectl get nodeoverlay
NAME             READY     
penalize-gen7    True

The Results

After deleting the old Gen 7 nodeclaims to trigger a re-provisioning, Karpenter’s simulation now sees Gen 8 as the "cheapest" option due to the penalties on Gen 7.

$ kubectl get nodeclaims -o wide
worker-nodepool-test-2xlkf    c8a.2xlarge        on-demand   eu-west-1b   True    
worker-nodepool-test-7jgtd    c8a.4xlarge        on-demand   eu-west-1b   True  
worker-nodepool-test-k5xbq    c8a.2xlarge        on-demand   eu-west-1b   True 
worker-nodepool-test-ncd6z    c8a.xlarge         on-demand   eu-west-1c   True 

Even though a c7g might technically cost less in real dollars, Karpenter saw the c8g as the better deal because of our overlay.

A Note on Spot Instances ⚠

While this works deterministically for On-Demand instances, Spot instances behave slightly differently.

When you apply NodeOverlays to Spot, Karpenter switches its allocation strategy from price-capacity-optimized to capacity-optimized-prioritized. The price adjustments are passed to the EC2 Fleet API as priorities.

Because priorities are relative, EC2 will still prioritize capacity availability to reduce the risk of interruptions.

Conclusion

The move from Pure Price to Price-Performance is a huge win for platform teams. With NodeOverlays, we:

• Improve performance by staying on the latest hardware.
• Reduce toil by managing preferences globally rather than via complex “layered” NodePools.
• Maintain reliability because if Gen 8 is unavailable, Karpenter will still fall back to a lower available generation.

If you’ve been struggling with “cheap but slow” nodes in your Karpenter fleet, it’s time to look at the NodeOverlay.