Migrating from AWS VPC CNI & Kube-Proxy to Cilium on EKS

If you are running Kubernetes in production, network security and visibility eventually become top priorities. Recently, I decided to migrate our Amazon EKS clusters (running version 1.35 on Bottlerocket OS) from the default AWS VPC CNI and kube-proxy over to Cilium.

Why the Move?

The migration was driven by three main objectives:

1. SOC 2 Compliance: We needed to achieve transparent, in-transit pod-to-pod encryption
2. Deep Observability: We wanted granular visibility into our traffic flows, allowing us to visualize pod-to-pod communication using Hubble.
3. No SideCar: By leveraging Cilium’s eBPF-based architecture, we could achieve advanced network routing, security, and observability without injecting resource-heavy sidecar proxies (like Istio) into every single pod. This sidecar-less approach significantly reduces compute overhead, lowers network latency, and simplifies pod lifecycle management across the cluster.

Migrating CNIs on a live cluster can be daunting, but by using a blue/green node pool strategy, the transition was seamless with zero disruption to our internal or external services. Here is the step-by-step breakdown of how I achieved it.

Phase 1: Preparing the Cilium Configuration

Instead of running Cilium alongside kube-proxy, I opted for a strict replacement to let Cilium manage routing natively via eBPF.

1. IPAM & Native Routing

I configured Cilium to use ENI IPAM mode, leveraging the existing VPC CIDR for native routing. Since we are using Bottlerocket OS, the network interfaces start with eth*, which we specified in the device list. Installing Cilium in ENI mode is the recommended installation method for Amazon EKS clusters.

ipam:
  mode: "eni"
  ciliumNodeUpdateRate: "15s"

eni:
  enabled: true
  awsReleaseExcessIPs: true
  awsEnablePrefixDelegation: true
  nodeSpec:
    firstInterfaceIndex: 1

2. Strict Kube-Proxy Replacement

I have disabled legacy host routing and enabled native routing and BPF masquerading. We also pointed Cilium to our specific EKS API endpoint (because the standard kubernetes.default.svc ClusterIP will not be resolvable or routable until the Cilium agent is actually running) to ensure it could communicate with the control plane directly.

routingMode: native
ipv4NativeRoutingCIDR: "10.50.0.0/16"

endpointRoutes:
  enabled: false

bpf:
  hostLegacyRouting: false
  masquerade: true

kubeProxyReplacement: true
kubeProxyReplacementHealthzBindAddr: 0.0.0.0:10256

devices:
  - "eth+" # Specifically for Bottlerocket OS

# Specify EKS endpoint & port
namespaceOverride: kube-system
k8sServiceHost: "REDACTED.gr7.ap-southeast-3.eks.amazonaws.com"
k8sServicePort: 443

3. Enabling WireGuard Encryption

Enabling encryption for SOC 2 compliance is remarkably simple in Cilium. I chose WireGuard over IPsec for its performance and modern architecture. more on Wireguard encryption in cilium: https://docs.cilium.io/en/stable/security/network/encryption-wireguard/

encryption:
  enabled: true
  type: wireguard

4. Node Affinity & Taints

To ensure a smooth transition, we need to instruct Cilium to only schedule on our new node pool (labeled cni=cilium) and configured it to remove the agent-not-ready taint once it successfully starts.

agentNotReadyTaintKey: node.cilium.io/agent-not-ready
removeNodeTaints: true
priorityClassName: system-node-critical

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: cni
              operator: In
              values: ["cilium"]

5. Taming the Cilium Operator (Avoiding CoreDNS Restarts)

By default, the Cilium operator will aggressively restart unmanaged CoreDNS before the Cilium pod even starts so that endpoints get created. We experienced continuous CoreDNS restart loops on nodes that weren’t even running Cilium yet. To stop this, I disabled the restart behavior and locked the operator to the new nodes.

operator:
  priorityClassName: system-node-critical
  unmanagedPodWatcher:
    restart: false # disable restrating unmanaged pods
  tolerations:
    - operator: "Exists"
      effect: "NoSchedule"
    - operator: "Exists"
      effect: "NoExecute"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: cni
                operator: In
                values:
                  - cilium

6. Enabling Hubble for Observability

Finally, we enabled Hubble UI, relay, and Prometheus metrics to get the deep visual insights we needed.

hubble:
  enabled: true
  metrics:
    enabled:
    - dns:query;ignoreAAAA
    - drop
    - tcp
    - flow # this gives the flow metrics between
    - icmp
    - http
  relay:
    enabled: true
    replicas: 1
    # ... [Add resource limits & tolerations here]
  ui:
    enabled: true
    # ... [Add resource limits & tolerations here]

Phase 2: Preparing the EKS Nodes

To prevent a sudden cluster-wide CNI swap, we used a Blue/Green node approach.

• Fence off the old nodes: We labeled existing nodes with cni=aws-vpc. We then updated the DaemonSet for both aws-node (VPC CNI) and kube-proxy with a nodeAffinity rule targeting cni: aws-vpc. This ensures they won't try to run on our incoming Cilium nodes.
• Provision the new nodes: We rolled out a new Terraform node group configured with the cni=cilium label and a specific initialization taint.

module "green-app-nodes" {
  source                 = "git@bitbucket.org:REDACTED/terraform-modules.git//eks-data-asg"
  cluster_name           = "dev"
  cluster_version        = "1.35"
  # ...
  node-labels = {
    "nodesrole" = "app"
    "cni"       = "cilium"
  }
  node-taints = {
    "noderole"                       = "app:NoSchedule"
    "node.cilium.io/agent-not-ready" = "true:NoSchedule" 
  }
}

When the nodes came up, they correctly sat in a NotReady state waiting for the CNI.

10538 ◯  k get node -l nodesrole=app
NAME                                              STATUS     ROLES    AGE     VERSION
ip-10-50-72-142.ap-southeast-3.compute.internal   NotReady   <none>   3s      v1.35.2-eks-f69f56f
ip-10-50-74-33.ap-southeast-3.compute.internal    Ready      <none>   6d20h   v1.35.2-eks-f69f56f
ip-10-50-75-232.ap-southeast-3.compute.internal   NotReady   <none>   4s      v1.35.2-eks-f69f56f
ip-10-50-77-10.ap-southeast-3.compute.internal    Ready      <none>   6d20h   v1.35.2-eks-f69f56f
ip-10-50-78-3.ap-southeast-3.compute.internal     NotReady   <none>   4s      v1.35.2-eks-f69f56f
ip-10-50-80-93.ap-southeast-3.compute.internal    NotReady   <none>   4s      v1.35.2-eks-f69f56f
ip-10-50-81-120.ap-southeast-3.compute.internal   Ready      <none>   6d20h   v1.35.2-eks-f69f56f
ip-10-50-93-88.ap-southeast-3.compute.internal    Ready      <none>   6d20h   v1.35.2-eks-f69f56f

Phase 3: Rollout Cilium

With our blue/green infrastructure set, we applied the Cilium Helm chart. As soon as the cilium pods initialized on the new nodes, they automatically removed the agent-not-ready taints, bringing the nodes to a Ready state.

10541 ◯  k get po -A --sort-by .metadata.creationTimestamp | grep cilium;
kube-system                cilium-q7cwp                                             1/1     Running             0               24s
kube-system                cilium-ltclf                                             1/1     Running             0               24s
kube-system                cilium-f9cjx                                             1/1     Running             0               24s
kube-system                cilium-zx4fc                                             1/1     Running             0               23s

10542 ◯  k get node -l nodesrole=app                                                                                                                                                                                                               ⏎
NAME                                              STATUS                     ROLES    AGE     VERSION
ip-10-50-72-142.ap-southeast-3.compute.internal   Ready                      <none>   48s     v1.35.2-eks-f69f56f
ip-10-50-74-33.ap-southeast-3.compute.internal    Ready,SchedulingDisabled   <none>   6d20h   v1.35.2-eks-f69f56f
ip-10-50-75-232.ap-southeast-3.compute.internal   Ready                      <none>   49s     v1.35.2-eks-f69f56f
ip-10-50-77-10.ap-southeast-3.compute.internal    Ready,SchedulingDisabled   <none>   6d20h   v1.35.2-eks-f69f56f
ip-10-50-78-3.ap-southeast-3.compute.internal     Ready                      <none>   49s     v1.35.2-eks-f69f56f
ip-10-50-80-93.ap-southeast-3.compute.internal    Ready                      <none>   49s     v1.35.2-eks-f69f56f
ip-10-50-81-120.ap-southeast-3.compute.internal   Ready,SchedulingDisabled   <none>   6d20h   v1.35.2-eks-f69f56f
ip-10-50-93-88.ap-southeast-3.compute.internal    Ready,SchedulingDisabled   <none>   6d20h   v1.35.2-eks-f69f56f

Cordon, Drain, and Verify

10543 ◯ alias drain='k drain --delete-emptydir-data --ignore-daemonsets'
10544 ◯  drain ip-10-50-74-33.ap-southeast-3.compute.internal
node/ip-10-50-74-33.ap-southeast-3.compute.internal already cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/aws-node-m6ngm, kube-system/ebs-csi-node-hspq4, kube-system/kube-proxy-4rqxp, kube-system/s3-csi-node-vkczb, observability/grafana-k8s-monitoring-alloy-logs-x5hj8, observability/grafana-k8s-monitoring-alloy-receiver-qvgmj, ops/prometheus-node-exporter-6zrxj
evicting pod ...
...
...

Proving SOC 2 Encryption Compliance

To verify the WireGuard encryption was actively securing our traffic, we exec’d into a Cilium pod:

kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep -A5 Encryption
Encryption:              Wireguard         [NodeEncryption: Disabled, cilium_wg0 (Pubkey: z7DPaEWcp7Ph8zUJVgmoRJyDnr5F7NX3Sk965LnLKkc=, Port: 51871, Peers: 9)]
Cluster health:          10/10 reachable   (2026-05-20T05:38:27Z)   (Probe interval: 3m2.671346263s)
Name                     IP                Node                     Endpoints
Modules Health:          Stopped(24) Degraded(0) OK(106)

-----------------------------------------------------------------------------
which means
1. Cilium WireGuard encryption is enabled
2. the WireGuard interface is active
3. cross-node pod traffic is traversing encrypted tunnels
"NodeEncryption: Disabled" - does NOT mean encryption is off.
it means,
- pod-to-pod traffic across nodes IS encrypted
- host-network/node traffic itself is NOT encrypted

By default, Cilium automatically disables node-to-node encryption from and to Kubernetes control-plane nodes, i.e. any node with the node-role.kubernetes.io/control-plane label will opt-out of node-to-node encryption to prevent worker nodes from being locked out if they cannot update their keys due to a lost connection to the API server (

WireGuard creates a dedicated interface (cilium_wg0) and communicates strictly over **UDP Port 51871**. To definitively prove traffic was encrypted over the wire, we ran tcpdump on the node's eth0 interface:

root@ip-10-50-78-3:/home/cilium# tcpdump -ni eth0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
06:24:34.424254 IP 10.50.78.3.10250 > 10.50.198.120.47334: Flags [P.], seq 1350630264:1350630294,
ack 1497774172, win 112, length 30
06:24:34.424290 IP 10.50.78.3.10250 > 10.50.198.120.47334: Flags [P.], seq 30:205, ack 1, win 112,
length 175
06:24:34.424732 IP 10.50.198.120.47334 > 10.50.78.3.10250: Flags [.], ack 30, win 1880, length 0
06:24:34.424776 IP 10.50.198.120.47334 > 10.50.78.3.10250: Flags [.], ack 205, win 1903, length 0
06:24:34.463848 IP 10.50.78.3.10250 > 10.50.198.120.47334: Flags [P.], seq 205:235, ack 1, win 112,
length 30
06:24:34.463867 IP 10.50.78.3.10250 > 10.50.198.120.47334: Flags [P.], seq 235:907, ack 1, win 112,
length 672
06:24:34.464314 IP 10.50.198.120.47334 > 10.50.78.3.10250: Flags [.], ack 235, win 1903, length 0
06:24:34.464314 IP 10.50.198.120.47334 > 10.50.78.3.10250: Flags [.], ack 907, win 1926, length 0
06:24:34.475654 IP 10.50.194.106.4240 > 10.50.78.3.44874: Flags [.], ack 364212217, win 70, length 0
06:24:34.475695 IP 10.50.78.3.44874 > 10.50.194.106.4240: Flags [.], ack 1, win 71, length 0
06:24:34.573830 IP 10.50.78.3.10250 > 10.50.198.120.47334: Flags [P.], seq 907:937, ack 1, win 112,
length 30
06:24:34.573849 IP 10.50.78.3.10250 > 10.50.198.120.47334: Flags [P.], seq 937:1888, ack 1, win 112,
length 951
06:24:34.574316 IP 10.50.198.120.47334 > 10.50.78.3.10250: Flags [.], ack 937, win 1926, length 0
06:24:34.574316 IP 10.50.198.120.47334 > 10.50.78.3.10250: Flags [.], ack 1888, win 1948, length 0
06:24:34.909027 IP 10.50.78.3.51871 > 10.50.200.138.51871: UDP, length 128
06:24:34.909565 IP 10.50.200.138.51871 > 10.50.78.3.51871: UDP, length 224
06:24:34.909721 IP 10.50.78.3.51871 > 10.50.200.182.51871: UDP, length 128
06:24:34.910428 IP 10.50.200.182.51871 > 10.50.78.3.51871: UDP, length 304
06:24:34.913857 IP 10.50.78.3.10250 > 10.50.198.120.47334: Flags [P.], seq 3258:4151, ack 1, win
112, length 893
06:24:35.167178 IP 10.50.72.142.51871 > 10.50.78.3.51871: UDP, length 2256
06:24:35.183742 IP 108.137.60.165.443 > 10.50.78.3.54038: Flags [P.], seq 4276399503:4276399644, ack
1783024787, win 443, length 141
06:24:35.183799 IP 10.50.78.3.54038 > 108.137.60.165.443: Flags [.], ack 141, win 2640, length 0
06:24:35.539689 IP 10.50.80.93.51871 > 10.50.78.3.51871: UDP, length 2752
06:24:36.283539 ARP, Request who-has 10.50.93.155 tell 10.50.78.3, length 28
06:24:36.283567 ARP, Reply 10.50.93.155 is-at 0e:b7:8f:96:d7:77, length 28
06:24:37.535835 IP 10.50.78.3.39382 > 108.137.60.165.443: Flags [P.], seq 3748905810:3748905849, ack
741212595, win 1188, length 39
06:24:37.614016 IP 10.50.78.3.51871 > 10.50.94.106.51871: UDP, length 2048
06:24:37.937039 IP 10.50.78.3.51871 > 10.50.194.57.51871: UDP, length 2672
06:24:38.867098 IP 10.50.78.3.51871 > 10.50.75.232.51871: UDP, length 2672
... [Truncated for layout readability - 274 packets logged successfully] ...
^C
274 packets captured
279 packets received by filter
0 packets dropped by kernel
Internal Network Engineering Report • Reference Code Added 3

1. The WireGuard Pod-to-Pod Traffic (UDP Port 51871) Every time you see UDP, length [X] on port 51871 (e.g., communicating with 10.50.200.138 or 10.50.75.232), you are looking at encrypted pod payloads traversing your network. You cannot see the inner IPs or protocols here because eth0 only sees the outer encrypted envelope.
2. Unencrypted Node-Level Traffic (TCP Ports 10250 & 4240) Because node-to-node encryption is disabled, we expect to see host-level Kubernetes communications traversing eth0 in cleartext. Your dump confirms this perfectly:

• Port 10250 (Kubelet API): You have a lot of TCP traffic between your node (10.50.78.3) and 10.50.198.120. Port 10250 is the Kubelet. This is likely your Kubernetes API server, metrics-server, or Prometheus scraping logs/metrics from this node's Kubelet.
• Port 4240 (Cilium Health): At 06:24:34.475654, we see a ping from 10.50.194.106 to your node on port 4240. This is the cilium-health agent checking node-to-node connectivity.

How to Analyze Further

Right now, we are sniffing eth0. This is great for verifying that WireGuard is working (by seeing port 51871) and that node traffic is unencrypted.

However, if you are trying to troubleshoot a Pod-to-Pod connectivity issue, sniffing eth0 is a dead end because the payload is encrypted.

To see inside the WireGuard tunnel and analyze the actual pod traffic (the inner IP headers, HTTP requests, DNS lookups, etc.), you need to run tcpdump against the decrypted WireGuard interface cilium_wg0 created by Cilium:

root@ip-10-50-78-3:/home/cilium# tcpdump -ni cilium_wg0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on cilium_wg0, link-type RAW (Raw IP), snapshot length 262144 bytes
06:22:29.757950 IP 10.50.77.69.42844 > 10.50.194.47.4317: Flags [P.], seq 512615461:512618006, ack 2027503549, win 70, options [nop,nop,TS val 3290570405 ecr 836288998], length 2545
06:22:29.758498 IP 10.50.194.47.4317 > 10.50.77.69.42844: Flags [P.], seq 1:31, ack 2545, win 548, options [nop,nop,TS val 836299000 ecr 3290570405], length 30
06:22:29.758529 IP 10.50.77.69.42844 > 10.50.194.47.4317: Flags [.], ack 31, win 70, options [nop,nop,TS val 3290570406 ecr 836299000], length 0
06:22:29.758653 IP 10.50.77.69.42844 > 10.50.194.47.4317: Flags [P.], seq 2545:2562, ack 31, win 70, options [nop,nop,TS val 3290570406 ecr 836299000], length 17
06:22:29.758707 IP 10.50.194.47.4317 > 10.50.77.69.42844: Flags [P.], seq 31:69, ack 2545, win 548, options [nop,nop,TS val 836299000 ecr 3290570405], length 38
06:22:29.805429 IP 10.50.194.47.4317 > 10.50.77.69.42844: Flags [.], ack 2562, win 548, options [nop,nop,TS val 836299047 ecr 3290570406], length 0
06:22:29.813562 IP 10.50.77.69.42844 > 10.50.194.47.4317: Flags [.], ack 69, win 70, options [nop,nop,TS val 3290570461 ecr 836299000], length 0
06:22:30.071898 IP 10.50.77.188.43018 > 10.50.92.86.4317: Flags [P.], seq 3451821091:3451823127, ack 2879682561, win 70, options [nop,nop,TS val 3836044087 ecr 1119428115], length 2036
06:22:30.071949 IP 10.50.92.86.4317 > 10.50.77.188.43018: Flags [.], ack 2036, win 2396, options [nop,nop,TS val 1119433116 ecr 3836044087], length 0
06:22:30.072069 IP 10.50.92.86.4317 > 10.50.77.188.43018: Flags [P.], seq 1:31, ack 2036, win 2397, options [nop,nop,TS val 1119433116 ecr 3836044087], length 30
06:22:30.072277 IP 10.50.77.188.43018 > 10.50.92.86.4317: Flags [.], ack 31, win 70, options [nop,nop,TS val 3836044088 ecr 1119433116], length 0
06:22:30.072349 IP 10.50.92.86.4317 > 10.50.77.188.43018: Flags [P.], seq 31:69, ack 2036, win 2397, options [nop,nop,TS val 1119433116 ecr 3836044088], length 38
06:22:30.072363 IP 10.50.77.188.43018 > 10.50.92.86.4317: Flags [P.], seq 2036:2053, ack 31, win 70, options [nop,nop,TS val 3836044088 ecr 1119433116], length 17
06:22:30.103429 IP 10.50.77.180.33472 > 10.50.92.86.4317: Flags [P.], seq 113065905:113068070, ack 2840479140, win 70, options [nop,nop,TS val 3302910205 ecr 3064157493], length 2165
06:22:30.103464 IP 10.50.92.86.4317 > 10.50.77.180.33472: Flags [.], ack 2165, win 476, options [nop,nop,TS val 3064162494 ecr 3302910205], length 0
06:22:30.103577 IP 10.50.92.86.4317 > 10.50.77.180.33472: Flags [P.], seq 1:31, ack 2165, win 480, options [nop,nop,TS val 3064162495 ecr 3302910205], length 30
06:22:30.103845 IP 10.50.92.86.4317 > 10.50.77.180.33472: Flags [P.], seq 31:69, ack 2165, win 480, options [nop,nop,TS val 3064162495 ecr 3302910205], length 38
06:22:30.103877 IP 10.50.77.180.33472 > 10.50.92.86.4317: Flags [.], ack 31, win 70, options [nop,nop,TS val 3302910205 ecr 3064162495], length 0
06:22:30.103946 IP 10.50.77.180.33472 > 10.50.92.86.4317: Flags [P.], seq 2165:2182, ack 31, win 70, options [nop,nop,TS val 3302910205 ecr 3064162495], length 17
06:22:30.119035 IP 10.50.77.188.43018 > 10.50.92.86.4317: Flags [.], ack 69, win 70, options [nop,nop,TS val 3836044135 ecr 1119433116], length 0
06:22:30.123553 IP 10.50.92.86.4317 > 10.50.77.188.43018: Flags [.], ack 2053, win 2397, options [nop,nop,TS val 1119433168 ecr 3836044088], length 0
06:22:30.149032 IP 10.50.77.180.33472 > 10.50.92.86.4317: Flags [.], ack 69, win 70, options [nop,nop,TS val 3302910251 ecr 3064162495], length 0
06:22:30.153541 IP 10.50.92.86.4317 > 10.50.77.180.33472: Flags [.], ack 2182, win 480, options [nop,nop,TS val 3064162545 ecr 3302910205], length 0
06:22:30.470892 IP 10.50.81.60.35960 > 10.50.92.86.4317: Flags [P.], seq 1809322773:1809325439, ack 3356280991, win 70, options [nop,nop,TS val 502399374 ecr 493412681], length 2666
06:22:30.470950 IP 10.50.92.86.4317 > 10.50.81.60.35960: Flags [.], ack 2666, win 618, options [nop,nop,TS val 493417628 ecr 502399374], length 0
06:22:30.471080 IP 10.50.92.86.4317 > 10.50.81.60.35960: Flags [P.], seq 1:31, ack 2666, win 620, options [nop,nop,TS val 493417628 ecr 502399374], length 30
06:22:30.471410 IP 10.50.81.60.35960 > 10.50.92.86.4317: Flags [.], ack 31, win 70, options [nop,nop,TS val 502399374 ecr 493417628], length 0
06:22:30.471417 IP 10.50.92.86.4317 > 10.50.81.60.35960: Flags [P.], seq 31:69, ack 2666, win 620, options [nop,nop,TS val 493417628 ecr 502399374], length 38
06:22:30.471425 IP 10.50.81.60.35960 > 10.50.92.86.4317: Flags [P.], seq 2666:2683, ack 31, win 70, options [nop,nop,TS val 502399374 ecr 493417628], length 17
06:22:30.513558 IP 10.50.92.86.4317 > 10.50.81.60.35960: Flags [.], ack 2683, win 620, options [nop,nop,TS val 493417671 ecr 502399374], length 0
...
..
.
^C
695 packets captured
703 packets received by filter
0 packets dropped by kernel

The Verdict: While interaction between the local kubelet socket and the control platform remained unencrypted (expected), all pod-to-pod traffic was fully encapsulated. No raw HTTP 1.1 packets were visible on the wire; everything was securely wrapped in WireGuard UDP packets on port 51871.

Latency Comparison:

Test details
- tool used: qperf (run in a pod)
- qperf client command: "qperf","-t", "60","-v" ,"<QPERF_SERVER_POD_IP>", "tcp_lat", "tcp_bw", "udp_lat", "udp_bw", "-m", "128k"
both qperf server & client scheduled on same AZ but on different nodes

10937 ± k get po -owide                                                                                                                                                                                                                        ⏎ ✹ ✭
NAME           READY   STATUS      RESTARTS   AGE    IP             NODE                                              NOMINATED NODE   READINESS GATES
qperf-client   0/1     Completed   0          112m   10.50.77.252   ip-10-50-80-93.ap-southeast-3.compute.internal    <none>           <none>
qperf-server   1/1     Running     0          133m   10.50.77.181   ip-10-50-72-142.ap-southeast-3.compute.internal   <none>           <none>

Thanks for reading :)

Full helm values.yaml https://github.com/pavankumar-go/home/tree/main/cilium-helm