The Case for Immutable Network Configuration

Overview

For over a decade, a powerful idea has reshaped how we manage applications and servers: never update, always replace.

This is the principle of immutable infrastructure that powers the modern DevOps landscape. We stopped treating servers like pets to be nursed back to health and started treating them differently — disposable and easily replaceable. This mindset gave us the speed and reliability of containers, Kubernetes, and serverless architectures.

But what about the network? Our routers, switches, and firewalls have largely remained in the “pet” category, with configurations that are carefully modified, tweaked, and updated over years.

This post poses a question: Can we apply the same immutable mindset to network configuration? What if, instead of updating a device’s config, we always created a new one from scratch?

The Theory Meets Reality: A Spine-Tingling Discovery

For the last couple of years I will be using Arista Validated Design during my deployments with an excellent experience so far. And I often expect AVD’s changes to be surgical and minimal. If you add a new device, you expect the automation to generate a new configuration for that device, perhaps with minor additions to its immediate neighbors. But what happens when the automation decides that a small change requires a complete rewrite?

We ran into this exact scenario during a routine capacity upgrade in a leaf-spine fabric. Our task was simple: I already have added multiple times new leaf(s), but now was the time for a new spine.

• Adding a new leaf produced the expected result: a small, additive, and predictable configuration change.
• Adding a new spine, however, was completely different. AVD re-calculated and replaced nearly the entire network underlay. “The change was so disruptive it was akin to rebuilding the network from scratch.” All point to point connections between leaf and spines changed IP and almost all the underlay BGP neighbors.

Don’t just take my word for it — see for yourself. I’ve created a sample Git repository that demonstrates these two scenarios. You can check out the branches and compare the diffs yourself.

• Check out the repository here: https://github.com/rogerscuall/network-immutability

Inside, you’ll find branches that represent the network’s initial state, the state after adding a leaf, and the dramatically different state after adding a spine.

The View from the CLI: “Are You Insane?”

Let’s be honest. If I had proposed this to any traditional network team without context, I would have been booed out of the room. The first unspoken question from any network engineer would be, “Are you completely insane?”

From the perspective of someone who lives in the command-line interface (CLI), network changes are about precision and minimizing risk. The idea of completely rewriting the underlay to add a single device is not just counter-intuitive; it’s terrifying. No one in their right mind would do this to a live fabric.

The core principle is to touch as little as possible.

What AVD did was the equivalent of tearing down an entire house just to replace the flooring in one room.

It felt reckless.

Fortunately, I work with an awesome and open-minded network team. Their initial skepticism was completely justified, but they were willing to hear me out when I tried to explain why this “crazy” approach might not be a bug, but a powerful feature.

But first, they sent me away to do my homework and come back with a very good reason for why we should even consider such a disruptive-looking operational model. And so I did.

Doing the Homework: Testing, Tickets, and a Moment of Truth

So, I did my homework. My first step was the standard one for any network engineer facing unexpected behavior: I opened a case with Arista support. With the options provided I moved to the next phase.

Then was time for Arista’s Cloud Test (ACT) platform. I spun up a digital twin of our fabric and went to town. I didn’t just test the change once; I rehearsed the entire lifecycle 20 times:

• Deploy the initial fabric.
• Add a new spine and observe the “disruptive” change.
• Perform CVP snapshots before and after the change and the same for ANTA tests.
• Perform a roll-back by removing spine and back to step 1.

If you want to follow alone you can build and deploy in main and again build and deploy in adding-spine-switch.

After hours of rigorous testing, one thing became crystal clear: while the change was huge in scope, it was also 100% predictable and repeatable. The “chaos” was, in fact, highly ordered.

In Defense of AVD

The testing and support case also revealed a critical piece of information: I was partly to blame.

AVD is an incredibly powerful and flexible framework. In my initial setup, I had missed max_uplink_switches that could have reserved the ranges of a future spine. There are multiple other ways to control this behavior, like creating the future switches and setting the is_deployed=falsebut I hadn't used any of them. In essence, I had cornered myself into this situation.

This led to a crucial decision. Should I add complexity and new logic to our AVD project just to preserve the existing IP schema? Or should I embrace the simpler, deterministic model that AVD defaulted to, even if it required a planned, one-time network event?

My preference was the later. For me, the long-term simplicity and maintainability of the project weighed more heavily than avoiding a momentary disruption.

It was important to me to keep our A-V-D codebase clean and simple, making it easier for anyone on the team to manage for years to come. It was a conscious trade-off: a small, planned operational cost today for a huge reduction in complexity tomorrow. 💡

From Plan to Production: Go no go meeting

With all my test results, it was time to move forward. This meant getting approval from the teams who live on top of the network I was proposing to change.

The network team was the first conversation. Having seen the extensive testing in ACT and understanding the long-term benefits of a simpler AVD codebase, they quickly saw the logic. They appreciated the rigor behind the plan and gave their sign-off.

The server and application teams, however, had a different and completely valid concern. From their point of view again, we weren’t just adding a spine; we were “pretty much replacing the entire network” underneath their running applications. After several discussions explaining the controlled nature of the change, we got the final green light, but it came with one important condition. On migration night, we would have an “all hands on deck” war room — a full crew of network, server, and application engineers ready to monitor every critical service to see exactly how they behaved during the event and test after.

The stage was set. The plan was approved. Now, it was time to execute.

Migration Night: The Three-Second Disruption

The night of the change, we went old-fashioned. The maintenance bridge was packed, with around 15 people joining from Network Engineering, NetDevOps, and the Server and Application teams. The application owners had their own separate room, standing by to run a battery of tests, fully expecting something to happen.

As a nod to tradition, we started an extended ping between several leaf switches and the network border. With all eyes on the monitor, we executed the change. We watched the steady stream of successful replies… until:

...
Reply from 10.10.31.10: bytes=32 time<1ms TTL=64
Reply from 10.10.31.10: bytes=32 time<1ms TTL=64
Request timed out.
Request timed out.
Request timed out.
Reply from 10.10.31.10: bytes=32 time<1ms TTL=64
Reply from 10.10.31.10: bytes=32 time<1ms TTL=64
...

Three lost packets. That was it. The entire network underlay and part of the overlay had been re-provisioned, and the total disruption was less than three seconds.

And lest you think this was a fluke, as of this writing, we have performed this exact migration, with minor variations, three more times on different EVPN/VXLAN production fabrics (around 20 switches per fabric) with around nearly identical, near-hitless results.

Let’s pause for a moment to give credit where it’s due. This is a massive feat of engineering by the Arista team. The stability of EOS, the quality of the hardware, and the countless details hidden under the hood are what allow an entire network to be reconfigured from scratch with an impact so small it’s barely measurable.

Immediately after the change, our dashboards in CVP lit up green. Automated validation tests run by ANTA showed no signs of trouble. It took less time to replace the entire network than it took you to read these last two paragraphs. We called the server team and gave them the green light for testing. Their immediate reply summarized it all:

“Are you sure you did the change? We didn’t see anything on our end.”

A New Mindset: Treating Configuration Like a Container

Was all that homework necessary? The hours spent simulating the change in ACT, the cross-team coordination, the support tickets… for a change that took one minute? The answer is an emphatic and unquestionable yes. Due diligence, testing, and communication are never optional.

The success of our three-second migration wasn’t just a technical victory; it was a validation of a different way of thinking. It brings us back to the idea we started with: managing network device configurations in the same way we manage containers — where we never update, we always create new.

However, this process forced me to look deeper at the philosophy of tools like AVD. An interesting fact about its operation is that AVD doesn’t look at the existing running configuration to decide what to build next. Its only takes as input is a set of YAML files describing the desired state. From this, it generates the entire configuration from scratch, every single time.

Think about that. AVD generates the configuration in full regardless of whether you’re changing a single character in an interface description or adding a new spine. The underlying process is identical.

This led me to the most important realization. By the time this “disruptive” problem appeared, we had already performed over 2,000 successful changes using AVD on these fabrics. This means the tool had already generated a complete, valid configuration from scratch over 2,000 times.

Is that a large enough number to trust the process? If we believe in that process, was this one change truly that different from the rest? AVD did more or less work, no. One more time generated a complete VALID configuration.

This entire experience leaves us with a final, critical question.

Are we ready to manage network configurations like we manage containers? Always creating new?