A network inside the network: virtual networking explained

Shared hardware, private networks

If you have ever dipped your toes into sysadmin cosplay, homelabbing, or virtualization, you have probably met hypervisors, virtual machines, and the strange joy of watching a guest OS boot inside another machine.

Getting VMs to run is already a project.

But it is only half of the story.

Once a VM exists, it usually needs to talk to something. It may need to reach a database, install packages from the internet, expose a website, call an API, or communicate with another VM. A virtual machine without networking is not very useful, unless your goal is to create a very expensive digital paperweight.

Things become more interesting when several users share the same physical infrastructure.

Imagine Alice wants one private network for her application and another one for her database. Bob wants his own private network too. Both of them may choose very common private IP ranges, such as 10.0.0.0/24 or 192.168.1.0/24.

On a normal home network, if two machines use the same IP address in the same place, things break. There is only one 10.0.0.5 on that network. If two devices claim it, the rest of the network gets confused very quickly.

So how can a cloud platform let several users create their own networks freely, sometimes using the same private IP ranges, without everything colliding?

That is where virtual networking comes in.

The problem is not only that packets need to move. Packets always need to move. The real problem is that they need to move while keeping each user’s environment separated from everyone else’s.

Alice’s network should not become Bob’s network just because their VMs happen to run on the same physical cluster. If Alice exposes a database inside her private network, Bob should not automatically be able to see it. If Bob creates a network with the same private IP range as Alice, it should not conflict with Alice’s network.

From the user’s point of view, the model should be simple:

They create a network, attach machines to it, and use it.

From the cloud operator’s point of view, this should not require manually reconfiguring the physical switches every time someone wants a subnet.

Before looking at how cloud platforms solve this, it helps to start with the traditional way of separating networks: VLANs.

VLANs help, but are too tied to the physical network

Most homelabbers have at least heard of VLANs.

The idea is simple: one physical network can carry several logical networks. Instead of using a separate switch for every network, Ethernet frames can be tagged with a VLAN ID. A switch that understands VLANs uses that tag to know which logical network the frame belongs to.

For example, imagine two devices plugged into the same physical switch. One is in VLAN 10, the other is in VLAN 20. Even though they share the same switch, they are not in the same Layer 2 network. They cannot directly exchange Ethernet frames unless a router or firewall is configured to route traffic between those VLANs.

This is very useful.

At home, you can put your personal devices in one VLAN and your IoT devices in another. This way, if your smart light bulb suddenly decides to become a network security researcher, it does not get direct access to your laptop.

When my suspicious connected toothbrush tries to reach my beloved workstation VLAN.

You can also configure trunk ports. A trunk carries several VLANs over the same physical link. On Linux, this often appears as subinterfaces such as eth0.10 and eth0.20, where the number at the end is the VLAN ID.

So we could imagine using VLANs for tenant isolation.

Alice gets VLAN 10. Bob gets VLAN 20. Their traffic is separated.

That works, but it is still very tied to the physical network.

If I create a new VLAN, I usually need to make sure my switches carry it on the right ports. I may need to update trunks, routing, firewall rules, host networking configuration, or all of those at once. For a few stable networks, this is perfectly fine. For a cloud-like platform where users are supposed to create networks themselves, it becomes painful very quickly.

Imagine Alice wants a temporary private network for a test environment. Then Bob wants two networks. Then another tenant wants a network using the same private IP range as Alice. Then a VM moves from one physical node to another.

If every change requires someone to touch the physical switch or manually update host configuration, users do not really have self-service networking. They have a human standing between them and the network.

There is also a scale limit. Traditional VLAN IDs are 12 bits, which gives 4094 usable VLANs. In a homelab, that is probably not the first problem you would hit. In a real cloud, where many tenants can create many networks, it becomes a real limit.

VLANs can isolate traffic, but they do not fully give us the model we want.

We want users to create networks through an API. We want those networks to be isolated. We want overlapping private IP ranges to work. And we do not want to reconfigure the physical switches every time a tenant creates a new network.

This is where cloud networking services come in.

Neutron turns networks into logical objects

In OpenStack, the networking service is called Neutron.

Neutron lets users create networking resources such as networks, subnets, ports, routers, security groups, and floating IPs.

For example, Alice can create a private network in her OpenStack project. Then she can create a subnet inside it, attach a VM to that network, and define security rules to allow SSH or HTTP traffic.

From her point of view, she is creating networking objects in OpenStack. She does not need to know which Linux bridge exists on which host, which tunnel carries the traffic, or which low-level rules are installed underneath. She does not need to ask someone to create a VLAN on a switch. She does not need to know which physical node will run her VM.

That is the useful part.

The network becomes something the platform can model and manage.

Of course, packets still have to move for real. OpenStack cannot just declare that a network exists and hope the universe politely agrees. At some point, something on the hosts has to forward packets, apply security rules, route traffic, create tunnels, or drop traffic that should not pass.

In many OpenStack deployments, this is done with OVN and Open vSwitch.

But before talking about those two, we need to look at the packet itself, because overlay networking becomes easier to understand once we remember that network traffic is already built in layers.

When a client sends a request to a server, it does not simply put “hello website” on the cable.

An HTTP request is carried by TCP. TCP is carried by IP. IP is carried by Ethernet on the local network.

Each layer adds the information it needs: TCP ports and sequence numbers, IP addresses, MAC addresses, and so on.

As you can see, a TCP request may span multiple Ethernet frames.

A single HTTP request may even span multiple Ethernet frames. The application does not usually care about that detail. It sends data, and the networking stack splits, wraps, forwards, reassembles, and delivers it.

That layered model matters because overlay networking uses a similar idea.

Instead of only wrapping application data inside TCP, IP, and Ethernet, we wrap a tenant network packet inside another packet that the physical network can carry.

Overlay networking carries tenant traffic between hosts

I know that’s how it sounds, but I swear, there’s more to it.

The basic idea of an overlay network is simple: create a virtual network on top of an existing physical network.

The physical network is still there. Packets still cross real cables, real switches, and real network cards. But tenant networks are not directly mapped to physical switch ports, dedicated cables, or manually configured VLANs.

Let’s say Alice has two VMs on the same OpenStack private network.

VM A runs on compute node 1. VM B runs on compute node 2.

From Alice’s point of view, both VMs are simply attached to the same private network. VM A sends traffic to VM B as if they were on the same local network.

Physically, however, they are not on the same machine.

The frame sent by VM A reaches the virtual switching layer on node 1. That virtual switch sees that the destination VM is not local. It then wraps the original tenant frame inside another packet and sends it across the physical network to node 2.

On node 2, the receiving virtual switch removes the outer tunnel headers, recovers the original tenant frame, and forwards it to VM B’s virtual interface.

The physical network is still doing the actual transport. It carries traffic between OpenStack nodes. The overlay layer keeps track of which tenant network the traffic belongs to.

The inner frame belongs to Alice’s private network. The outer packet belongs to the physical network between the OpenStack hosts.

The underlay physical network still exists, it just carries our network abstraction.

So the same traffic has two contexts:

The tenant context, where Alice’s VMs communicate inside their private network.

The transport context, where physical nodes send packets to each other across the real infrastructure.

The physical network does not need to know Alice’s full virtual topology. It does not need to know which VM is in which tenant network. It only needs to move an encapsulated packet from one host to another.

That is enough to let Alice’s network span several physical machines without creating a dedicated physical network for her.

It is worth noting that not every packet needs to cross a tunnel. If two VMs are on the same compute node, the traffic may stay local inside that host. Encapsulation becomes necessary when traffic has to cross the physical network between nodes.

The underlay still exists.

It just carries the overlay.

Geneve is the envelope

In OpenStack deployments using OVN, this encapsulation is commonly done with Geneve.

Geneve is not a virtual switch. It is not the service that decides where packets should go. It is the tunnel format used to carry tenant traffic across the physical network.

You can think of Geneve as the envelope.

The original tenant frame goes inside. Then the system adds outer headers so the physical network knows how to carry the packet from one OpenStack node to another.

Geneve uses UDP, commonly on port 6081.

The overlay pictured.

At first, UDP may sound like a strange choice. TCP feels reassuring. It provides a reliable byte stream, retransmissions, ordering, and congestion control.

But putting tenant packets inside TCP can create awkward behavior, especially when the inner traffic is already TCP. One lost segment in the outer TCP connection could delay unrelated inner traffic behind it. The outer TCP connection would treat the tunnel as one ordered byte stream, even though the tunnel may be carrying many independent flows.

UDP is simpler for this job.

One UDP datagram can carry one encapsulated overlay packet. The overlay system keeps packet boundaries clear, and the inner traffic keeps its own behavior. If the inner flow is TCP, then the inner TCP connection can handle retransmissions and ordering itself.

The tunnel also carries context. The receiving side must know which logical network the packet belongs to. In overlay networking, this is typically done with tunnel metadata, such as a virtual network identifier or other information carried with the encapsulated packet.

That context is essential.

Without it, the receiving node would only see an inner frame with addresses such as 10.0.0.5. But 10.0.0.5 may exist in Alice’s network and Bob’s network at the same time. The address alone is not enough. The packet must also be interpreted inside the right logical network.

When a packet arrives on the destination node, Open vSwitch receives it through its Geneve tunnel port. It removes the outer Ethernet, IP, UDP, and Geneve headers. Then it recovers the original tenant Ethernet frame and forwards that frame to the correct VM interface.

The host does not extract the TCP segment and hand it directly to the VM. The VM receives normal-looking network traffic on its virtual NIC, and its own network stack handles Ethernet, IP, TCP, and the application data from there.

That detail matters.

The VM does not know about the whole tunnel machinery. From its point of view, it simply has a network interface and receives packets.

The simplified decapsulation process.

The complicated part happens around the VM, not inside it.

Which is probably for the best.

There is one practical detail to keep in mind: MTU. The MTU is the largest packet size a network link can carry without fragmentation. Because Geneve adds an extra envelope around the original tenant frame, the resulting packet is bigger than the packet the VM originally sent. If the physical network MTU is too small, large packets may be fragmented or dropped, which can lead to very confusing network issues.

Overlay networking gives us flexibility, but it is not magic. It still has to obey physics, packet sizes, and the occasional unpleasant network debugging session.

OVN describes the virtual network, OVS applies it locally

Now we can talk about OVN and Open vSwitch.

Open vSwitch, often shortened to OVS, is the virtual switch running on each compute node. It moves packets between VM interfaces, local bridges, router interfaces, and tunnel ports.

OVN, for Open Virtual Network, is the system that describes what the virtual network should look like and distributes that information to the nodes.

A tenant network is not a physical switch sitting somewhere in a rack. It is represented by logical objects: logical switches, logical routers, logical ports, security rules, NAT rules, and port bindings.

These objects describe the virtual network.

They say:

1. Which VM port belongs to which network.
2. Which MAC and IP addresses are attached to that port.
3. Which router connects which networks.
4. Which traffic should be allowed or blocked.
5. Which physical node currently hosts a VM.

OVN stores the logical network state and distributes the relevant information to each node. The local OVN controller then programs OVS so packets are handled according to that logical model.

Put differently:

Neutron is the API layer users interact with. OVN is the logical network control plane. OVS is the local virtual switch that applies the rules. Geneve is the tunnel format used when traffic must cross the physical network.

For example, if Alice’s VM on node 1 sends traffic to another VM on node 2, OVS on node 1 needs to know what to do with the packet.

Is the destination local? Should the packet be sent through a Geneve tunnel? Should it be routed? Should it be dropped because of a security rule? Should NAT be applied?

OVN provides the logical network information, and OVS uses that information to handle the packet locally.

A metaphor helps here.

Imagine a large building hosting several events at the same time. In a traditional setup, people are guided by the physical layout: corridors, barriers, locked doors, gates, signs, and security checkpoints. If you want to separate two groups, you create separate paths. If you want to change how people move, you move barriers or open and close doors.

That is similar to traditional networking, where cables, switch ports, VLANs, routers, and firewall interfaces define where traffic can go.

Now imagine the building changes all the time.

New groups arrive. Some groups leave. Some need private rooms. Some need access to a shared exit. Some may even use the same room numbers internally, but those room numbers do not mean the same thing for everyone.

You do not want to rebuild the building every time, so you place agents at checkpoints.

Each person carries a badge. The badge says which group they belong to and where they want to go. The agent reads the badge, checks the instructions, and sends the person in the right direction.

Someone from group A going to room 10 may go left. Someone from group B also going to room 10 may go right. The room number is the same, but the context is different.

Someone trying to reach the public exit may be sent toward another corridor. Someone without the right badge may be stopped.

That is close to what happens in an overlay network.

Packets are like the people. OVS is like the local checkpoint agent on each node. OVN is the system that prepares and distributes the instruction sheets.

The physical building still exists, just like the physical network still exists. People still walk through real corridors, and packets still cross real cables, switches, and network interfaces.

But separation is no longer created only by the physical layout. It is also created by rules applied at each checkpoint.

This is what allows several tenants to share the same physical infrastructure while still having separate networks.

Overlapping IPs work because the IP is interpreted inside a logical network

This also explains how overlapping IP ranges can work.

Alice can have a VM using 10.0.0.5 inside her private network. Bob can also have a VM using 10.0.0.5 inside his own private network.

In a normal flat network, this would be a conflict.

In OpenStack tenant networking, those addresses exist inside different logical networks.

Back to the building metaphor: two people can both say they are going to room 10, but if they belong to different events, room 10 may refer to two different places. The checkpoint agent does not only look at the room number. It also looks at the group.

OVS does something similar with packets.

It does not only care about the destination IP address. It also knows which logical network the packet belongs to, because OVN gave it the rules for that network and the tunnel carries the context needed when traffic moves between nodes.

So when Alice creates a network in OpenStack, she is not creating a physical switch and she is not asking someone to configure a VLAN manually. She creates a Neutron network.

Neutron turns that request into the objects needed by the backend networking system. In an OVN-based deployment, those objects are represented in OVN’s logical model. OVN distributes the relevant state, and OVS applies the corresponding behavior on each compute node.

If Alice then creates a VM and attaches it to that network, OpenStack creates the needed port and connects the VM to the virtual switching layer.

From Alice’s point of view, she has a private network.

From the operator’s point of view, the physical network does not need to be redesigned every time a tenant creates one.

This is what makes the model useful. Users can create their own private networks. Those networks are isolated. Overlapping private IP ranges can coexist. The physical switches do not need to be manually reconfigured for every new project.

There is one important nuance, though.

Overlapping IP ranges work as long as those networks remain separate logical contexts. If you try to directly route Alice’s 10.0.0.0/24 network into Bob’s 10.0.0.0/24 network without NAT, policy, or some other form of separation, the ambiguity comes back.

The platform can keep separate networks separate.

It cannot make the same address mean two different things inside the same routing context.

That solves a large part of the tenant networking problem. It mostly covers communication inside the platform: VM to VM, application to database, backend to internal service, and so on.

If a tenant wants to expose a website to the internet, there is another problem to solve. The private network eventually has to connect to something outside itself. That is where external networks, floating IPs, NAT, load balancers, or reverse proxies start to matter.

But that is another subject.

The point here is how several private networks can exist on the same shared infrastructure without becoming the same network.

The physical network is still there

Virtual networking can sound like the physical network has somehow been replaced by pure abstraction.

It has not.

The physical network is still doing the boring but very real job of moving packets between machines. Packets still cross real cables, real switches, and real network cards. If the physical network breaks, the overlay does not magically levitate frames across the room.

The difference is that tenant networks are no longer directly represented by switch ports, cables, or dedicated VLANs.

They exist as logical networks.

The physical network is used as a transport layer between the hosts that run them.

When Alice’s VM talks to another VM on the same tenant network, Alice does not need to care whether those VMs are on the same physical node or not. If they are local, the host can handle the traffic locally. If they are on different nodes, the traffic can be encapsulated and carried across the physical network.

OVN describes the logical network. OVS applies the rules locally. Geneve carries tenant frames between hosts when they need to cross the physical network. Neutron gives users the API to create and manage the whole thing.

Once that clicks, cloud networking becomes a little less magical.

Not simple, exactly. Let’s not get carried away.

But at least the weirdness has a shape.

Alice can create a private network. Bob can create another one. They can even both use 10.0.0.5.

The physical infrastructure remains mostly the same, while the virtual networking layer keeps track of which packet belongs to which logical network.

Still real packets. Still real cables. Just more context !