The Challenge: Unexplained Network Outage on a Newly Configured Exadata LACP Bond

A customer had recently deployed an Oracle Exadata Database Machine and aimed to enhance their public network resilience and throughput by implementing Link Aggregation Control Protocol (LACP) on one of their critical network bonds (e.g., bondeth0, which connects to their client and application network). The network team had already configured the corresponding switch ports for LACP, creating a port-channel and enabling LACP on the physical member interfaces.

Following this, the Exadata system administrators proceeded to configure the bondeth0 interface on the Exadata database nodes, believing they had set it up for LACP. However, upon completion, they experienced a complete loss of network communication over this bondeth0 interface. Pings to the Exadata IPs on this bond failed, and application connectivity was severed. The interface appeared "up" on the OS level, but no traffic would pass.

The team was perplexed. The physical links were confirmed to be active, and the switch team insisted their LACP configuration was standard and correct. The immediate suspicion fell on a misconfiguration on the Exadata side, despite the interface showing as active.

The Core Problem as Inferred from Document 2198475.1:

The document “Link Aggregation Control Protocol (LACP) on Exadata (Doc ID 2198475.1)” provides a crucial insight that directly explains this scenario:

“It must be pointed out that when LACP is configured on the switch, it must be configured in the bonding driver also. With the switch configured with LACP, no communication was possible until the bonding driver was configured for LACP. LACP is a protocol that both the switch and network interfaces use to gain membership in the aggregationgroup. Without that membership negotiation, the switch will not forward traffic to the network interface.”

This statement pinpoints the likely cause: while the switch was configured for LACP, the Exadata’s bonding driver for bondeth0 was either not configured for LACP at all, or was configured incorrectly, preventing the necessary LACPDU (Link Aggregation Control Protocol Data Unit) negotiation. Without successful negotiation, the switch would not include the Exadata's ports in the active LACP group and therefore would not forward traffic to them.

How the Problem Can Be Resolved

The resolution involves ensuring that both the Exadata host-side bonding configuration and the network switch configuration are correctly set up for LACP and are compatible.

1. Host-Side Configuration (Exadata Linux)

The primary step is to correctly configure the bonding options on the Exadata database node(s) for the affected bond interface (e.g., bondeth0). File to Modify: /etc/sysconfig/network-scripts/ifcfg-bondeth0 (or the relevant bond interface name) Parameter to Set: BONDING_OPTS The document provides an example of a correctly configured BONDING_OPTS line: BONDING_OPTS="mode=4 miimon=100 downdelay=200 updelay=200 num_grat_arp=100 lacp_rate=1 xmit_hash_policy=layer3+4"

Let’s break down the critical components from this example as per the document:

• mode=4: This is essential. It specifies IEEE 802.3ad dynamic link aggregation, which is LACP. If this was missing or set to another mode, LACP would not function.
• lacp_rate=1: This sets the LACPDU packet exchange rate to "fast" (every 1 second). The document states: "This interval needs to be set on the switch too." So, this must match the switch configuration. If the switch is set to slow (0, every 30 seconds) and the host to fast (1), or vice-versa, it could lead to issues or slower detection of link failures.
• xmit_hash_policy=layer3+4: This parameter determines how outbound traffic is distributed across the member NICs in the bond. . The document details three options: . layer2: "Uses XOR of hardware MAC addresses... This algorithm will place all traffic to a particular network peer on the same slave. This algorithm is 802.3ad compliant." (Default if not specified in older bonding versions). . layer2+3: "Uses XOR of hardware MAC addresses and IP addresses... This algorithm will place all traffic to a particular network peer on the same slave... This policy is intended to provide a more balanced distribution of traffic than layer2 alone... This algorithm is 802.3ad compliant." . layer3+4: "Uses OSI layer 3 and layer 4 protocol information... This allows for traffic to a particular network peer to span multiple slaves, although a single connection will not span multiple slaves... This algorithm is not fully 802.3ad compliant. A single TCP or UDP conversation containing both fragmented and un-fragmented packets will see packets striped across two interfaces. This may result in out of order delivery."
• The choice of xmit_hash_policy should ideally align with the load-balancing strategy on the switch for inbound traffic to ensure consistent behavior, though they are configured independently.

Utility for Configuration (as mentioned in the document): The document states:

“To configure this, use the utility /opt/oracle.cellos/ipconf.pl. This utility will properly configure the Exadata Engineered System's network configuration files."

While manual editing is possible, using the provided Exadata utility is the recommended method to ensure all related configurations are handled correctly.

2. Switch-Side LACP Configuration (Generic Cisco Example)

The document emphasizes that LACP requires configuration on both ends. It provides a generic example for a Cisco switch. The customer’s network team would need to ensure their switch configuration is compatible.

Key Switch Configuration Aspects (from the Cisco example in the document):

. Create a Port-Channel Interface:

Cisco> (config)# interface port-channel 1
Cisco> (config-if)# description MySwitch LACP
Cisco> (config-if)# switchport access vlan 3004
Cisco> (config-if)# switchport mode access

This creates the logical aggregated interface (port-channel 1), assigns it a description, and configures VLAN and access mode.

. Set Switch Load Balancing:

Cisco>(config)# port-channel load-balance Ethernet source-dest-port

This command sets the load-balancing algorithm for traffic inbound to the Exadata. The document lists various options for Cisco switches like destination-ip, source-destination-ip, source-dest-port, etc. This should ideally complement the Exadata's xmit_hash_policy. For example, if the Exadata uses layer3+4 (which considers IP addresses and ports), the switch using source-dest-port or source-destination-ip would be a common complementary setup.

. Configure Physical Member Interfaces:

Cisco> (config)# interface Ethernet 1/1
Cisco> (config-if)# switchport access vlan 3004
Cisco> (config-if)# switchport mode access
Cisco> (config-if)# channel-group 1 mode passive
Cisco> (config-if)# lacp rate fast
Cisco> (config-if)# exit

Cisco> (config)# interface Ethernet 1/2
Cisco> (config-if)# switchport access vlan 3004
Cisco> (config-if)# switchport mode access
Cisco>(config-if)# channel-group 1 mode passive
Cisco>(config-if)# lacp rate fast
Cisco> (config-if)# exit

. channel-group 1 mode passive: This assigns the physical Ethernet interface to port-channel 1 and sets the LACP mode. passive means the switch will only enable LACP if it detects LACPDU from the connected device (Exadata). Alternatively, active mode would have the switch actively try to negotiate LACP. Often, one side is active and the other passive, or both are active.

• lacp rate fast: This configures the switch to send LACPDU packets every second. This must match the lacp_rate=1 setting on the Exadata host.

3. Verification Steps:

1. Apply Configurations: Apply the correct BONDING_OPTS on the Exadata (preferably using ipconf.pl or by carefully editing the ifcfg-bondeth0 file and restarting the network service or interface). Ensure the switch configuration is finalized by the network team.
2. Check LACP Status on Switch: The network team can verify if the port-channel is up and if the member ports from Exadata have successfully joined the LACP group.
3. Check LACP Status on Exadata:

• Examine /proc/net/bonding/bondeth0. This file will show the bonding mode, LACP rate, status of slave interfaces, and partner (switch) LACP information if the negotiation is successful.

4. Test Connectivity: Perform ping tests, application connectivity tests, etc.

By ensuring that the Exadata’s bonding interface is explicitly configured for mode=4 (LACP), and that parameters like lacp_rate are synchronized with the switch, and that an appropriate xmit_hash_policy is chosen, the LACP negotiation would succeed. The switch would then correctly forward traffic to the Exadata host over the aggregated link, resolving the network outage.