BGP best path selection in L3VPN: hidden pitfall

If you have ever configured MPLS L3VPN, it should raise no doubt that BGP is the tool the whole setup revolves around. As a protocol with a strong sense of dignity (after all, Internet is built with it), it has a fairly long list of decision-making points called best path selection algorithm. Despite the horrifying length of the list, most of the items are mere tie-breakers rather than knobs used for traffic engineering. Sometimes, however, the most ubiquitous attributes are not the right ones for the job. If you see EIGRP in L3VPN environment — beware, it might be the case we are going to discuss in this article.

Let’s start with a sample topology:

R1-R3 are PE-routers that run LDP in the core along with OSPF. They are also providing connectivity between sites using various IGPs: R1↔R4 ≡ OSPF, R2↔R5 ≡ EIGRP, R3↔R6 ≡ eBGP. Client devices R4-R6 reside in a single VRF. R4 and R5 provide R6 with an access to some service, reachable via 8.8.8.8/32.

Sample config from R1:

R1(config)#vrf definition A
R1(config-vrf)# rd 1:1
R1(config-vrf)# route-target export 1:1
R1(config-vrf)# route-target import 1:1
R1(config-vrf)# address-family ipv4
R1(config)#interface Loopback0
R1(config-if)# ip address 1.1.1.1 255.255.255.255
R1(config)#interface FastEthernet0/0
R1(config-if)# vrf forwarding A
R1(config-if)# ip address 192.168.14.1 255.255.255.0        
R1(config)#interface FastEthernet1/0
R1(config-if)# ip address 192.168.13.1 255.255.255.0
R1(config)#interface FastEthernet1/1
R1(config-if)# ip address 192.168.12.1 255.255.255.0
R1(config)#router ospf 2 vrf A
R1(config-router)# redistribute bgp 123 subnets
R1(config-router)# network 0.0.0.0 255.255.255.255 area 0
R1(config)#router ospf 1
R1(config-router)# mpls ldp autoconfig
R1(config-router)# router-id 1.1.1.1
R1(config-router)# network 0.0.0.0 255.255.255.255 area 0
R1(config)#router bgp 123
R1(config-router)# bgp router-id 1.1.1.1
R1(config-router)# no bgp default ipv4-unicast
R1(config-router)# neighbor L3VPN peer-group
R1(config-router)# neighbor L3VPN remote-as 123
R1(config-router)# neighbor L3VPN update-source Loopback0
R1(config-router)# neighbor 2.2.2.2 peer-group L3VPN
R1(config-router)# neighbor 3.3.3.3 peer-group L3VPN
R1(config-router)# address-family vpnv4
R1(config-router-af)#  neighbor L3VPN send-community both
R1(config-router-af)#  neighbor 2.2.2.2 activate
R1(config-router-af)#  neighbor 3.3.3.3 activate
R1(config-router-af)# exit-address-family
R1(config-router)# address-family ipv4 vrf A
R1(config-router-af)#  redistribute ospf 2

As for CE routers, their config is even more ascetic:

R6(config)#interface Loopback0
R6(config-if)# ip address 6.6.6.6 255.255.255.255
R6(config)#interface FastEthernet0/0
R6(config-if)# ip address 192.168.36.6 255.255.255.0
R6(config)#router bgp 6
R6(config-router)# bgp router-id 6.6.6.6
R6(config-router)# no bgp default ipv4-unicast
R6(config-router)# neighbor 192.168.36.3 remote-as 123
R6(config-router)# address-family ipv4
R6(config-router-af)#  network 6.6.6.6 mask 255.255.255.255
R6(config-router-af)#  neighbor 192.168.36.3 activate

This lab uses loopbacks to emulate the service:

R4(config)#interface Loopback1
R4(config-if)# ip address 8.8.8.8 255.255.255.255

R5(config)#interface Loopback1
R5(config-if)# ip address 8.8.8.8 255.255.255.255

Let’s see first whether R6 is able to reach 8.8.8.8 at all:

R6#ping 8.8.8.8 source loopback 0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 6.6.6.6 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 52/60/80 ms

The key point of this setup is to have redundant paths from R6 to 8.8.8.8/32 via R4 and R5, that’s why there are direct BGP sessions between R1-R3 and R2-R3. Both R1 and R2 should consider IGP prefixes to be the best in BGP RIB (they are locally originated routes) thus R3 should have 2 prefixes in BGP RIB, one for each peer.

R3#sho bgp vpnv4 unicast all 
BGP table version is 13, local router ID is 3.3.3.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf A)
 *>i 4.4.4.4/32       1.1.1.1                  2    100      0 ?
 *>i 5.5.5.5/32       2.2.2.2             103040    100      0 ?
 *>  6.6.6.6/32       192.168.36.6             0             0 6 i
 *>i 8.8.8.8/32       2.2.2.2             103040    100      0 ?
 *>i 192.168.14.0     1.1.1.1                  0    100      0 ?
 *>i 192.168.25.0     2.2.2.2                  0    100      0 ?

Somehow R3 receives only a single prefix instead of two. It might be possible that R1 does not send any update regarding 8.8.8.8/32:

R1#sho bgp vpnv4 unicast all
BGP table version is 13, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf A)
 *>  4.4.4.4/32       192.168.14.4             2         32768 ?
 *>i 5.5.5.5/32       2.2.2.2             103040    100      0 ?
 *>i 6.6.6.6/32       3.3.3.3                  0    100      0 6 i
 r>i 8.8.8.8/32       2.2.2.2             103040    100      0 ?
 r                    192.168.14.4             2         32768 ?
 *>  192.168.14.0     0.0.0.0                  0         32768 ?
 *>i 192.168.25.0     2.2.2.2                  0    100      0 ?

R1 considers 8.8.8.8/32 from R2 to be the best route (symbol ‘>’). Since aforementioned prefix is received via iBGP, it’s not forwarded to any normal iBGP peer. Why does not R1 pick the local route though? According to the best path selection, only weight and local preference have precedence higher than the local routes. Besides, local routes also have a weight set to 32768 by default so 8.8.8.8/32 from OSPF must be selected the best (32768 vs 0), not the one from R2!

R1#sho bgp vpnv4 unicast all 8.8.8.8/32
BGP routing table entry for 1:1:8.8.8.8/32, version 13
Paths: (2 available, best #1, table A, RIB-failure(17) - next-hop mismatch)
  Not advertised to any peer
  Refresh Epoch 1
  Local
    2.2.2.2 (metric 2) from 2.2.2.2 (2.2.2.2)
      Origin incomplete, metric 103040, localpref 100, valid, internal, best
      Extended Community: RT:1:1 Cost:pre-bestpath:128:103040 0x8800:32768:0 
        0x8801:1:2560 0x8802:65281:25600 0x8803:65281:1500 0x8806:0:134744072
      mpls labels in/out nolabel/22
  Refresh Epoch 1
  Local
    192.168.14.4 from 0.0.0.0 (1.1.1.1)
      Origin incomplete, metric 2, localpref 100, weight 32768, valid, sourced
      Extended Community: RT:1:1 OSPF DOMAIN ID:0x0005:0x000000020200 
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:192.168.14.1:0

Since it’s quite tedious to remember numbers by heart, it’s nice to have some cheat sheet with community format for OSPF and EIGRP at hand. However, the culprit we are looking for has a self-explanatory name — Cost:pre-bestpath:128:103040.

The purpose of cost community with EIGRP in L3VPN is relatively simple. For example, consider two sites, A and B, that are connected with fast L3VPN as a primary path and a slow leased line as a backup path. At some point Prefix-A emerges within Site-A and propagates across these paths. In such a case there is a race condition present:

1. Update about Prefix-A reaches Site-B via MPLS backbone first. Site-B PE installs the prefix into BGP RIB, redistributes it into EIGRP (reconstructing metric with communities) and announces the prefix within Site-B. Since leased line has an unfavourable metric, the path through MPLS backbone would be selected by site-B as expected.
2. Update about Prefix-A reaches Site-B via leased line first. Site-B PE imports the prefix into BGP RIB. When BGP update from Site-A finally reaches PE on Site-B, this new prefix would not be considered the best because there would be a better, locally originated, one. In such a case, Site-B would use leased line as a primary path towards Prefix-A.

Cost community remedies such a problem. It allows BGP to consider a certain metric at a specific point in best path selection algorithm (in our case, before the whole process). EIGRP cost is used as a value for such a metric thus the shortest path is always selected.

Although aforementioned approach works well in a pure EIGRP L3VPN environment, it has an obvious side effect — it breaks the usual path selection process. As you could imagine, OSPF does not use cost community at all so there is a default value of 0x7FFFFFFF (214748347) that is much higher than a typical EIGRP metric.

At first glance, the problem could be fixed by changing EIGRP metric for 8.8.8.8/32 to be higher than 0x7FFFFFFF; however, it would only reverse the issue because at this point R2 would compare OSPF default cost community metric with an awful cost of the prefix imported from EIGRP.

A possible solution could be to give up cost community at all, however, there is no direct command for that yet. The available knob though allows to completely ignore cost community:

R1(config)#router bgp 123
R1(config-router)#bgp bestpath cost-community ignore

R3#sho bgp vpnv4 unicast all 
BGP table version is 26, local router ID is 3.3.3.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf A)
 *>i 4.4.4.4/32       1.1.1.1                  2    100      0 ?
 *>i 5.5.5.5/32       2.2.2.2             103040    100      0 ?
 *>  6.6.6.6/32       192.168.36.6             0             0 6 i
 * i 8.8.8.8/32       1.1.1.1                  2    100      0 ?
 *>i                  2.2.2.2             103040    100      0 ?
 *>i 192.168.14.0     1.1.1.1                  0    100      0 ?
 *>i 192.168.25.0     2.2.2.2                  0    100      0 ?

Although we managed to make R1 ignore cost community, R3 requires the same config; otherwise it would always select EIGRP-originated route despite other BGP attributes.

R3(config)#router bgp 123
R3(config-router)#bgp bestpath cost-community ignore

R3#sho bgp vpnv4 unicast all 
BGP table version is 29, local router ID is 3.3.3.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf A)
 *>i 4.4.4.4/32       1.1.1.1                  2    100      0 ?
 *>i 5.5.5.5/32       2.2.2.2             103040    100      0 ?
 *>  6.6.6.6/32       192.168.36.6             0             0 6 i
 *>i 8.8.8.8/32       1.1.1.1                  2    100      0 ?
 * i                  2.2.2.2             103040    100      0 ?
 *>i 192.168.14.0     1.1.1.1                  0    100      0 ?
 *>i 192.168.25.0     2.2.2.2                  0    100      0 ?

At last BGP RIB selection adheres to the common path selection rules: lower MED wins as expected. An alternative approach would be to remove cost community manually from announced updates:

R2(config)#ip extcommunity-list 100 permit pre-bestpath
R2(config)#route-map COSTCOMM
R2(config-route-map)#set extcomm-list 100 delete
R2(config)#router bgp 123
R2(config-router)#address-family vpnv4
R2(config-router-af)#neighbor L3VPN route-map COSTCOMM out

R2#sho bgp vpnv4 unicast all extcommunity-list 100
BGP table version is 23, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf A)
 *>  5.5.5.5/32       192.168.25.5        103040         32768 ?
 *>  8.8.8.8/32       192.168.25.5        103040         32768 ?
 *>  192.168.25.0     0.0.0.0                  0         32768 ?

R3#sho bgp vpnv4 unicast all 8.8.8.8/32
BGP routing table entry for 1:1:8.8.8.8/32, version 46
Paths: (2 available, best #1, table A)
  Advertised to update-groups:
     1         
  Refresh Epoch 5
  Local
    1.1.1.1 (metric 2) from 1.1.1.1 (1.1.1.1)
      Origin incomplete, metric 2, localpref 100, valid, internal, best
      Extended Community: RT:1:1 OSPF DOMAIN ID:0x0005:0x000000020200 
        OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:192.168.14.1:0
      mpls labels in/out nolabel/23
  Refresh Epoch 1
  Local
    2.2.2.2 (metric 2) from 2.2.2.2 (2.2.2.2)
      Origin incomplete, metric 103040, localpref 100, valid, internal
      Extended Community: RT:1:1 0x8800:32768:0 0x8801:1:2560 
        0x8802:65281:25600 0x8803:65281:1500 0x8806:0:134744072
      mpls labels in/out nolabel/23

The purpose of this article is twofold: to introduce cost community and to practice hands-on troubleshooting. Remember, there is no magic in technology but defaults.

Kudos for review: Anastasiia Kuraleva