How to Migrate Existing Node Pools to OKE Generic VNIC Attachments?

Generic VNIC Attachment (GVA) provides OKE with more flexible and scalable pod networking. Compared to the traditional VCN-native CNI model, GVA uses CIDR block allocations for pods instead of individual IPs, which speeds up worker node provisioning and cluster join time, increases the maximum pod density per worker node, and improves scalability for large clusters.

GVA requires the pod subnet to contain at least two CIDR blocks:

• A CIDR block for individual IP allocations (primary VNIC IPs)
• A CIDR block for pod block allocations

Both CIDR blocks must implement mandatory security list rules for pod communication, as described here.

This guide explains how to migrate an existing node pool and pod subnet to a GVA-enabled node pool.

Important: This procedure requires downtime because existing worker nodes and pod IP allocations must be removed before the subnet can be reconfigured.

Migration procedure

A. Remove existing pod IP allocations

Before adding multiple CIDR blocks to the target pod subnet, all existing IP allocations must be released.

1. Scale down the targeted node pools to 0, which will release and remove all previous IP allocations from the target pod subnet, along with the worker nodes.
2. Ensure that the list of allocated IP addresses for the target pod subnet is empty by checking it.
3. If there are orphaned IPs or VNICs, raise an SR to enforce deletion.

B. Add new CIDR block for individual IP allocations

Although the GVA feature focuses on block-level allocations, we still need individual IP address allocations, such as primary IP addresses for each VNIC within a worker node.

1. Add a CIDR block that will host individual IP leases. Worker node VNICs will use it as a mandatory primary IP address.

Pay attention to the maximum number of worker nodes the subnet can host, as that number corresponds to the number of primary IP addresses. E.g., if you plan to host 100 worker nodes, make sure to use at least */25* CIDR block with 126 usable IPs for the individual IP block.

The new CIDR block (e.g. */25*) should appear as the second CIDR block currently attached to the subnet.

C. Reconfigure the original CIDR block for pod block allocations

Since the target subnet already used the original CIDR block (which should be empty now) for pod communication in a regular VCN Native CNI setup, we need to transform it for block allocation.

1. Remove the first original CIDR block from the target subnet, leaving only the new CIDR block from Step B as the first and only entry in the list of CIDR blocks.
2. Recreate the original block and adjust the subnet mask to the forecast below, this time as a second item in the list of CIDRs.

Forecast usable pod IP addresses across all worker nodes. E.g., if you plan to host 100 worker nodes, each with 128 pods per node, that requires 12,800 IP addresses; therefore, make sure to reserve at least a */18 block for the new pod block allocations. Each worker node will then lease `/25* from that/18` block.

The subnet CIDR order should now be:

• Smaller CIDR (e.g. /25) for worker node primary IPs
• Larger CIDR (e.g./18) for pod block allocations

D. Scale the node pool up

Once the subnet setup is ready to host individual IPs and block allocations, you may proceed to scale up the worker nodes.

1. Edit the target node pool and scale to the desired number. Primary IPs from VNIC should appear in the first, smaller CIDR block, while the pod-allocated block should appear in the second, larger CIDR block.

After worker nodes have been provisioned:

• Primary VNIC IPs should be allocated from the smaller CIDR (e.g. /25)
• Pod allocation blocks should be leased from the larger CIDR (e.g./18)
• Pods should receive individual IPs from the pod allocation CIDR

Additional recommendations

• Perform the migration during a maintenance window.
• Carefully forecast pod density before sizing the pod allocation CIDR.
• Test the procedure in a non-production cluster first.