vPC & HSRP Nexus 9K

A few years back I was replacing old core switches with a fresh set of Nexus 9K’s. For me this was the first time experiencing NX-OS instead of IOS. I wrote up some scribbles somewhere found some things a long the way and now I thought “Let’s write these down for future reference.”

For redundancy and such we chose to implement vPC and HSRP. Why HSRP on the core you ask? Design choice was to keep the Core as the primary L3 routing layer.

Why not GLBP because Active/Standby would be a better option for a failover situation. With vPC and HSRP on Nexus 9K, both switches can send and receive traffic for the VIP at the same time (active/active) at the data plane for better bandwidth and resilience. But only one switch actually answers ARP queries for the gateway (the HSRP active), so the control plane stays “active/standby.” GLBP, which would do real load sharing with multiple ARP answers, just isn’t an option on Nexus 9K. So vPC plus HSRP is the next best thing.

HSRP (Hot standby Routing Protocol)

First off let’s start configuring HSRP. HSRP is Cisco’s proprietary redundancy protocol that allows a standby router to take over active control of the packet forwarding process in the event of a failure. In IOS, the HSRP process is running all the time without user intervention. With NX-OS, you must enable this feature (and others) using the “feature hsrp”” command. Once HSRP is configured, if you remove the “feature hsrp” command, all the HSRP configuration will be deleted.

Keynotes;

• HSRPv1 is enabled by default which is not compatible with HSRPv2 BUT can coexist in an HSRPv2 environment (both versions can run on the same device).
• HSRPv1 supports 256 group numbers (0–255) while HSRPv2 supports 4096 (0–4095).
• HSRPv2 is enabled on a per-interface basis.
• NX-OS uses the “hsrp #” command versus the IOS “standby ip” command.
• In a L3 switch environment this is normally set up in the VLAN SVI’s configuration or turning a switchport into a L3 port using the “no switchport” command and assigning that port an IP address.

HSRP Configuration examples

Here are some configuration examples for a minimal HSRP configuration. Below you can see the basic difference in the configuration between NX-OS and IOS. Nothing special but the NX-OS does simplify things a little and improves readability.

HSRP on IOS

SWITCH 1

interface vlan 50 ip address 10.50.50.2 255.255.255.0 standby 50 ip 10.50.50.1 standby 50 preempt standby 50 priority 110

SWITCH 2

interface vlan 50 ip address 10.50.50.3 255.255.255.0 standby 50 ip 10.50.50.1

HSRP on NX-OS

You need to make the vPC Primary the HSRP Active. On Cisco NX-OS the switch with the highest priority becomes active. Ensure your vPC primary has a higher HSRP priority than the secondary peer.

SWITCH 1

interface vlan 50 ip address 10.50.50.2/24 hsrp 0 ip 10.50.50.1 preempt priority 110

SWITCH 2

interface vlan 50 ip address 10.50.50.3/24 hsrp 0 ip 10.50.50.1 preempt priority 100

vPC (Virtual Port-Channel) on NX-OS

Above is the vPC connections between the core and a set of distribution switches. Due to other configuration limitations, all routing and HSRP is configured on the core. The 3560X switches are L2 switches to provide out-of-band management for all switches in the racks using VLAN 99. By moving the keep-alive links to normal interfaces, we were able to provide redundancy with two links vs one.

STP Root Placement

Even though vPC is a loop-free topology you should not disable STP to prevent user error by someone else it’s recommended to configure the STP on your Peer-Links and to configure your root on your vPC domain.

vPC Configuration

VPC Peer-Link on the core. On all the switches with NX-OS, the following needs to be configured For role priority in vPC lower wins from higher.

feature vpc feature hsrp feature lacp

VPC Domain and Switch Configuration

SW-CORE-A

vpc domain 10 peer-switch spanning-tree vlan 1–499,501–4094 priority 4096 role priority 120 system-priority 1024 peer-keepalive destination 10.199.199.2 source 10.199.199.1 vrf vpc-keepalive delay restore 10 peer-gateway auto-recovery reload-delay 300 ip arp synchronize

SW-CORE-B

vpc domain 10 peer-switch spanning-tree vlan 1–499,501–4094 priority 4096 role priority 150 system-priority 1024 peer-keepalive destination 10.199.199.1 source 10.199.199.2 vrf vpc-keepalive delay restore 10 peer-gateway auto-recovery reload-delay 300 ip arp synchronize

SW-CORE-A

interface port-channel99 description VPC Peer Link — SW-CORE-B switchport mode trunk switchport trunk allowed vlan 1–499,501–4094 spanning-tree port type network vpc peer-link

SW-CORE-B

interface port-channel99 description VPC Peer Link-SW-CORE-A switchport mode trunk switchport trunk allowed vlan 1–499,501–4094 spanning-tree port type network vpc peer-link

Port-Channel Members (Same on both switches)

99 Po99(SU) Eth LACP Eth1/47(P) Eth1/48(P)

VPC Peer Link on the Distribution Layer

SW-DIST-A

interface port-channel1 description VPC Peer Link — SW-DIST-B switchport mode trunk spanning-tree port type network vpc peer-link

SW-DIST-B

interface port-channel1 description VPC Peer Link — SW-DIST-A switchport mode trunk spanning-tree port type network vpc peer-link

Port-Channel Members (Same on both switches) 1 Po1(SU) Eth LACP Eth1/29(P) Eth1/30(P)

VPC Peer-Keepalive Links on the Core

These links need to be in their own VRF table and the interfaces need to be Layer3. Also it’s really a pleasure to be able to use CIDR notation rather than type out the subnet mask in NX-OS commands.

SW-CORE-A

interface port-channel910 no switchport speed 1000 vrf member vpc-keepalive ip address 10.199.199.1/30

SW-CORE-B

interface port-channel910 no switchport speed 1000 vrf member vpc-keepalive ip address 10.199.199.2/30

Port-Channel Members for Keepalive

910 Po910(RU) Eth NONE Eth1/39(P) Eth1/40(P)

Member Interface Configuration

interface Ethernet#/# description keep-alive: SW-CORE-# <> SW-CORE-# no switchport vrf member vpc-keepalive channel-group 910 lacp no shutdown

Peer Keepalives for Distribution

SW-DIST-A

interface port-channel920 description keep-alive: SW-DIST-A <> SW-DIST-B no switchport vrf member vpc-keepalive ip address 10.199.199.13/30

SW-DIST-B

interface port-channel920 description keep-alive: SW-DIST-B <> SW-DIST-A no switchport vrf member vpc-keepalive ip address 10.199.199.12/30

Port-Channel Members for Keepalive

920 Po920(RU) Eth NONE Eth1/2(P) Eth1/4(P)

Member Interface (Same for all)

interface Ethernet#/# no switchport vrf member vpc-keepalive channel-group 920 no shutdown

So this would be the basic configuration for this kind of setup. Yes you’d need peer-routing for the more advanced L3 routing over vPC, but that’s going into more specifics.