The Scaling Wall: Why Trendyol Migrated Data Center to Multi-Site EVPN

Created by Gemini

When it comes to data center architectures, the idea of a “single large pool” (Single Fabric) often sounds very appealing. Everything is within a single BGP domain, every Leaf sees every Leaf, and every VLAN is everywhere…

If your leaf-spine traffic has reached the levels shown in the graph, it’s worth taking a closer look at the number of routes. It might be time to consider a multi-site setup.

Leaf-Spine Traffic for One Fabric

However, as your network grows, this “peaceful pool” slowly transforms into a vast ocean that is difficult to control. In this article, we’ll discuss the challenges Trendyol faced with a single fabric and the benefits it gained from implementing a multi-site EVPN architecture.

Created by Gemini

Why did we get stuck in one VXLAN EVPN Domain?

In a fabric architecture, EVPN answers the question “Where is the destination?”, while VXLAN answers the question “How do I get to the destination?”

EVPN is the control plane of the fabric. It finds the physical device (VTEP/Leaf) on which a packet’s destination (MAC or IP address) is located.

VXLAN is the data plane of the fabric. Once the destination is known, VXLAN determines how the packet should be encapsulated and transported to reach it.

VTEPs (VXLAN Tunnel Endpoints), also known as leaf nodes, establish tunnels with each other. However, if the number of VTEPs exceeds the limit (which varies by vendor), the devices’ tunnel tables begin to fill up and cannot encapsulate packets for new VTEPs. For example, a province does not have 500 districts because that would complicate management and service delivery. Instead, these districts can be distributed among many provinces.

In an EVPN domain, scalability is not only about the capacity of a single device, but also about the total load of the entire fabric and the health of the control plane.

This is where BGP limits become important. As the number of EVPN Type-2 (MAC and IP) and Type-5 (IP Prefix) routes increases, the BGP process on leaf devices begins to tire and slow down. It is crucial to monitor the MAC address table and the host route table in the forwarding tables. Once the limits exceed 80%, it’s time to take action. We can compare this to a bus: If a bus has 40 seats, it should carry maximum 40 passengers. Carrying more passengers reduces comfort, so the best solution is to use a second bus.

Created by Gemini

What are the benefits of Multi-Site, and what were the key considerations on Trendyol?

The Multi-Site (EVPN Border Gateway) architecture encapsulates each site within its own independent “limit area” and provides a structure that can support a total of millions of devices.

In the EVPN/VXLAN architecture, the Border Gateway (BGW) strategically connects different EVPN domains (sites), carries traffic between domains, and maintains scalability limits.

As Trendyol’s data centers expanded, we began to require a multi-site design. With this design, we divided the data centers into sites and deployed Border Gateway devices. By connecting the BGWs to devices in the core layer, we enabled communication between sites. These BGWs provided us with the following flexibility.

VXLAN EVPN Multi-Site Design

1. Control Plane Gateway There are thousands of MAC and IP routes (Type-2 and Type-5) within a fabric. If you connect two fabrics directly, all routes on both sides become mixed, and the table limits are reached. When transferring EVPN routes received from one site to another, the BGW changes the Next-Hop information to its own IP address. Leaf switches do not need to recognize the thousands of VTEPs at the remote site. The only “Next-Hop” address they need to know to reach remote destinations is the Border Gateway.

2. Data Plane Gateway VXLAN tunnels are typically set up from VTEP to VTEP (leaf-to-leaf). However, this changes in a multi-site architecture. In this architecture, a leaf in site A sends a packet to the BGW via a VXLAN tunnel. The BGW then de-encapsulates the packet, determines its destination, and re-encapsulates it in a new VXLAN tunnel for Site B. This “segmentation” ensures that the lifetime of the tunnels ends at the site boundary. This makes it easier to manage tunnel complexity in the network.

3. Anycast Border Gateway In large structures, a single BGW can create a physical bottleneck. Therefore, two or more devices typically operate as a single virtual BGW using Anycast logic. This means that all BGWs have the same IP address (EVPN multi-site anycast IP). Traffic is balanced between these devices using ECMP (Equal-Cost Multi-Path). If one BGW fails, traffic continues to flow through the other BGW within milliseconds.

We have a control mechanism for external (DCI) and internal (fabric) interfaces. If all external or internal interfaces go down, the EVPN Multi-Site anycast IP announcement is stopped. This enables traffic to switch to another BGW seamlessly. When the interfaces come back up, the announcement is not passed until the BGW neighbors/tunnels stabilize, thanks to the delay restore time. Once this period ends, the announcement is sent, and traffic begins to flow through the BGW.

4. BUM Traffic BUM (Broadcast, Unknown Unicast, Multicast) traffic is the biggest enemy of a fabric. A BGW prevents broadcast traffic within a site from flowing uncontrollably to another site. BGWs typically use the “Designated Forwarder” mechanism to prevent BUM traffic from remote sites from being copied and causing loops. A “Broadcast Storm” at one site cannot bypass the BGW and crash your other data center.

5. RT Design VRFs requiring L3 extension should be created in BGW. There is no need to maintain VRF leaks in Leafs in BGW. In BGW, we only import/export their own RT (Route Target) values to ensure routes flow through the correct VRFs.

Scaling Without Limits

One of our key achievements with Multi-Site was accelerating Trendyol’s growth. As our platform and services expanded, we continued to provide seamless service to our customers.

To meet this scale, Trendyol Facility team laid 1000 km of fiber-optic cable and 400 km of copper cable in our data centers. We achieved a power capacity of 10 MW. With the transition to a multi-site architecture, our number of hosts increased from approximately 2500 to 8500 across three regions.

Created by Gemini

Ensuring a seamless transition was critical for Trendyol, so we spread nearly 150 hours of night work over a three-month period. As the Network team, we planned all the work. We successfully completed all the transitions in collaboration with the Facility, Traffic, and Network Security teams. Behind the scenes, we carried out a project process spanning 1.5 years, filled with planning and testing, in collaboration with our Product team.

In conclusion, you may face the same challenges that Trendyol experienced when using a single fabric. You have read the most important points to consider in multi-site design. Trendyol overcame significant scalability challenges by transitioning to a multi-site EVPN architecture. If you find it suitable for your structure, you can implement the multi-site design that we use at Trendyol. You can also review the relevant vendor documentation for configurations and access more in-depth information from IETF documents related to multi-site design.

Careers at Trendyol Tech

Would you like to build scalable technology that powers millions of users daily? Join the Trendyol Tech team and be part of our mission to redefine digital excellence.

Explore career opportunities here: https://careers.trendyol.com/