Stop Designing Route Reflectors Like It’s 2005 (Part 2: The Fast-Twitch Muscle)

In Part 1 we structurally isolated the Route Reflector onto its own compute. The brain was protected. But protecting the brain does not protect the body. That is exactly what this part is about.

The natural next step is optimization: tuning Optimal Route Reflection (ORR) and maximizing RIB Sharding to squeeze every possible millisecond of control-plane performance out of your x86 instances.

It is a logical move. But relying purely on a smarter brain for survivability is a dangerous trap. A network with a faster brain and the same legacy nervous system will still watch its own outages happen in slow motion.

Consider what happens during a standard topology change. A PE-facing link goes down, removing an active BGP next-hop. Even a highly optimized Route Reflector processing the WITHDRAW exactly as designed (best-path recompute, RIB-OUT serialization, TCP transmission to every client) takes time.

In a legacy RR design lacking a pre-loaded alternate, the operational math is unforgiving. Every millisecond your Route Reflector spends processing a BGP failure is a millisecond your edge routers spend firing packets into a black hole.

Let’s call this the Propagation Tax. It is the time delta between the physical reality changing and your control plane finishing its update cycle. It manifests as:

• Packet loss during routine topology changes.
• Customer escalations for micro-outages.
• SLA breaches that defy explanation because “we have full redundancy.”

This tax is structurally unavoidable as long as your PEs remain blind to anything except the single path the RR has chosen for them. This exposes the core flaw in legacy design.

When a traditional Route Reflector receives multiple paths to a prefix it runs its best-path algorithm and normally advertises only the selected best path to each client. The alternates may still exist inside the RR but they are hidden from the PEs. That is path hiding.

Path hiding once made sense. Memory was expensive and FIBs were small. But more importantly the early internet was built for best-effort delivery. In a mostly text-based world, if packets stopped flowing for a few seconds while the control plane caught up it was perfectly acceptable.

None of those things are true anymore….

When the single advertised path goes down, the PE blackholes traffic until the RR detects the change, recomputes, serialises the update, and pushes it. With full internet tables, policy-heavy RIB-OUT, and many RR clients, this can easily move from milliseconds into seconds under real failure conditions. In modern networking a second is a geological era.

BGP Add-Paths (RFC 7911) gives you a way to end the lie. It changes the conversation between the RR and the PE from a monologue into a briefing. The RR can now send multiple paths for the same prefix. The PE can learn the alternates in advance, and on PIC-capable platforms those alternates can be pre-installed into the forwarding chain before the failure occurs.

But there is a way to do this very, very badly.

The mechanism is pretty straightforward . When you enable Add-Paths “send all” or a high path count every RR client receives N times the BGP UPDATE volume on every churn event. The RIB-OUT generation work on the RR scales with (paths × clients). The BGP-IN parsing work on each PE scales with paths × prefixes touched. On a modern PE with a dedicated BGP control-plane CPU, this is absorbed. On a five-year-old PE running BGP on a shared route processor, it is not.

The lesson is to treat Add-Paths advertisement policy as a per-platform contract. Use add-path send-count, per-neighbor filters, or AFI-specific scoping. The goal is the minimum path diversity required for PIC Edge to function (typically two) not the maximum the protocol allows.

But Add-Paths alone is just data. The reflex still has to fire.

To understand how the reflex actually fires, you have to look at the data plane. Prefix Independent Convergence is structurally impossible without a Hierarchical Forwarding Information Base (FIB).

In a legacy flat FIB every BGP prefix is mapped directly to a physical outgoing interface and MAC address. If a link fails, the router CPU must individually rewrite every single prefix entry in the hardware. This takes seconds.

In a Hierarchical FIB the architecture is decoupled into a strict chain of pointers: a BGP Prefix points to a Protocol Next-Hop, the Next-Hop points to an IGP Adjacency and the Adjacency points to a Physical Interface.

This decoupling changes the physics of a network failure. If a physical core link dies the silicon only updates the Adjacency pointer. The million BGP prefixes sitting at the top of the chain are never touched. They remain completely unaware that the underlying physical exit changed. If an eBGP peer dies, the silicon simply flips the NextHop pointer to a pre-programmed backup. Every prefix pointing to that NextHop instantly rides the new path.

The control plane CPU is completely bypassed. Updating a single parent pointer instantly reroutes millions of children in the hardware. This pointer math is the actual fast-twitch muscle that makes PIC possible.

PIC Core protects against failures inside your IGP domain. It relies entirely on BGP next-hop recursion using this pointer chain.

Millions of BGP prefixes point to a single BGP Next-Hop IP address. That BGP Next-Hop is resolved by the IGP. When a core link dies, the IGP detects the topology change and calculates a new shortest path. The router then updates the single IGP pointer for that specific BGP Next-Hop in the FIB.

Because all the BGP prefixes are just pointing to the Next-Hop, they do not need to be touched. They instantly ride the new IGP path with zero per-prefix churn. The IGP handles the crisis, removing BGP from the synchronous repair path entirely.

But PIC Core only handles failures between PEs. It cannot save you when the PE itself, or its external eBGP peering, is what died. The IGP cannot route around an exit point that no longer exists.

PIC Edge: The Pre-Loaded Reflex

That is what PIC Edge is for. It protects against the failure of the exit point itself.

Instead of relying on the IGP to find a new path, the router pre-programs a special pointer structure (often called a PathList) into the FIB. Millions of BGP prefixes point to this single PathList.

Inside this PathList, there is a Primary Next-Hop and a Backup Next-Hop. When the primary eBGP peer or remote PE goes offline, the hardware detects the failure and flips the pointer inside the PathList from Primary to Backup. Every BGP prefix pointing to that PathList instantly fails over in the data plane without waiting for BGP to withdraw and re-advertise routes.

This acts as a pre-loaded reflex. Conceptually, it is the exact same philosophy as an EIGRP Feasible Successor, but executed directly in the hardware silicon. The router handles the failure instantly on its own, without waiting for the control plane.

Here is the dependency that makes or breaks the whole architecture

You cannot have a Pre-Loaded Reflex if the PE does not know a backup path exists.

If your Route Reflector is still hiding the independent alternate, PIC Edge cannot protect that failure domain properly. The PE needs path diversity before the failure, whether delivered through Add-Paths, diverse-path, per-PE RD design, or an equivalent mechanism. Without it, the hardware has nothing to put in the backup pointer. The reflex has no muscle to fire.

If you have been optimising your RRs, you have heard of BGP Optimal Route Reflection (ORR). I will be completely transparent here: while I have the operational experience for Add-Paths and PIC Edge, I have never deployed ORR in a live production backbone. But theoretically, it fits into this architecture perfectly.

ORR is sometimes confused with Add-Paths because both involve the RR being smarter about what it sends. They solve completely different problems.

A legacy RR calculates the best path from its own physical perspective in the topology. This forces remote PEs into sub-optimal routing. They exit the network from wherever the RR thinks is closest which is rarely where they actually are. Hot-potato routing breaks.

ORR makes the Route Reflector empathetic. It creates a virtual topology centered on the client PE, calculating the absolute optimal single path for that specific location. RR-induced sub-optimal routing is solved. But if that one perfect path dies, the PE is still blind. ORR does nothing for failover.

Here is the cleanest way to think about the relationship: ORR optimises the steady state. Add-Paths armours the transient.

ORR makes your reflected best path topologically correct when nothing is wrong. Add-Paths keeps your network alive when something is. They operate in entirely different time domains, and a serious backbone should evaluate both deliberately.

The modern design is not “ORR or Add-Paths.” In a mature design, ORR and Add-Paths can be combined. ORR makes the advertised paths topology-aware from the client’s perspective while Add-Paths allows more than one of those paths to be delivered to the edge.

To bring the entire architecture into focus, it forms a triad:

• ORR provides topological correctness.
• Add-Paths (or diverse-path / per-PE RD) provides path visibility.
• PIC Edge / Core provides forwarding-time survivability.

To understand why all of this matters and why we route around the control plane instead of trying to make it faster, walk through what actually happens during a remote failure in a network that lacks a Pre-Loaded Reflex:

1. Detection. A PE notices its peering is down. (Fast, milliseconds.)
2. Transmission. The PE generates a BGP WITHDRAW and sends it to the RR.
3. Processing. The RR queues the update, ingests it into RIB-IN, runs best-path across potentially millions of paths, and writes RIB-OUT.
4. Propagation. The RR serialises updates into TCP sockets and transmits them to every other client PE.
5. Execution. Remote PEs receive, parse, and finally update their FIBs.

In an idle network, this completes in milliseconds. During a real topology change, when the RR is already churning through a BGP storm, it can stretch into full seconds. I have seen it stretch further than that, on networks I will not name, during incidents I am still asked not to discuss.

During every millisecond of that window, the rest of the network is forwarding traffic into a route that no longer exists. The remote PEs are not negligent. They are simply operating on the last truth the RR sent them. The RR has not yet tapped them on the shoulder.

This is the Propagation Tax, paid in dropped packets. And it is paid by your customers, not by your control plane.

You cannot make BGP fast enough to outrun this. BGP is stateful, serialised, and gated by TCP. No amount of compute on your vRR fundamentally changes its character. Throwing CPU at the control plane is like making a courier run faster: it helps, but the courier is still slower than the radio.

If your RRs are still sending only the single best path and your PEs have no independent pre-loaded alternate, three things are true at once:

1. You are paying the Propagation Tax in every failure event.
2. You cannot properly protect that failure domain with PIC Edge,no matter how modern your hardware is.
3. Your SLA is being defended by a control plane that fundamentally cannot meet it.

The fix is not exotic. Add-Paths is widely available in major BGP implementations but support must be validated per platform, AFI/SAFI, direction, and scale profile. PIC Edge has been shipping in major NOSes for years.

ORR may look like a configuration knob but it is really a topology contract: it only works if the RR has the right IGP view, the right client-location model and enough path visibility.

The blocker is almost never the technology. It is the design assumption inherited from earlier architectures that survivability is something the control plane delivers in real time.

BGP was designed in an era when networks had time to think. Modern networks do not have that luxury.

The design goal is not to make the memo faster.

The design goal is to stop waiting for the memo.

Design for the reflex.

BGP #NetworkEngineering #Routing #NetworkArchitecture

Originally published at https://www.linkedin.com.