Multi-VLAN MetalLB on OpenShift4 Without BGP

A Production-Grade L2 Approach for homogeneous Telco and Financial Environments

How to make MetalLB’s Layer 2 mode work across multiple VLANs on bare-metal and virtualized OpenShift 4.x workers without touching a single BGP session.

Why This Article Exists

In telecom and financial-services data centers alike, BGP is often the “textbook” answer for exposing LoadBalancer IPs from a Kubernetes cluster. But real networks rarely match textbooks. Both industries tend to run highly segmented, VLAN-based access networks, for very similar reasons:

• Telco operators segment traffic by function — OAM, signaling, media, subscriber data — often across CNF workloads with strict isolation requirements.
• Financial institutions segment traffic for compliance and risk-containment reasons — trading systems, payment processing, core banking, and DMZ-facing services frequently live on separate VLANs enforced by network and security teams, sometimes under PCI-DSS or similar regulatory scope.
• In both cases, the network team is often unwilling — or organizationally unable — to open a BGP peering session between the compute layer and the core network, whether for security policy, change-control overhead, or simple separation of duties between NetOps and platform teams.
• Both industries commonly run mixed fleets of physical and virtualized OpenShift workers, with inconsistent NIC naming and bonding configurations across hardware generations.

In these cases, MetalLB’s Layer 2 (L2) mode is the pragmatic choice. The catch: L2 mode only works if your nodes are actually present — at Layer 2 — in the VLAN where the LoadBalancer IP needs to live.

This article walks through the real mechanism behind that requirement, why it breaks by default, and how to fix it cleanly with a concrete multi-VLAN, mixed-hardware example taken from a live OpenShift deployment. The pattern applies directly whether the VLAN in question carries signaling traffic for a telco core, or payment traffic for a financial services platform.

The Core Problem: ARP Doesn’t Cross VLAN Boundaries

When MetalLB runs in L2 mode, it doesn’t do any actual routing. Instead:

1. For each LoadBalancer IP, MetalLB elects one node as the leader (speaker) for that IP.
2. When a router or client on the local segment sends an ARP request — “who has this IP?” — the leader node answers with its own MAC address.
3. All return traffic for that IP is switched to the leader node’s interface, and kube-proxy/OVN handles further load-balancing to the actual pod.

The critical detail is that ARP is a broadcast protocol, and broadcasts never cross VLAN boundaries. If your worker nodes’ primary interface only lives on VLAN 120, and a router or client sits on VLAN 101 asking for an IP that belongs to VLAN 101, that ARP request never reaches your nodes. MetalLB has nothing to answer — not because it’s misconfigured, but because the node is simply deaf to that broadcast domain.

This shows up identically whether VLAN 101 carries a telco signaling network or a bank’s payment-gateway segment. It is the single most common cause of “MetalLB IP is not reachable” tickets in multi-VLAN deployments, in either industry, and it has nothing to do with MetalLB’s own configuration.

The fix: give the node genuine Layer 2 presence in every VLAN that will host a LoadBalancer service — without disturbing the node’s own default routing. That’s exactly what the NMState Operator is for.

What the NMState Operator Actually Does

The NMState Operator manages node-level network state declaratively, through Kubernetes manifests, by driving NetworkManager on RHCOS/RHEL under the hood. Its architecture has two parts:

Component Type Responsibility Handler DaemonSet Runs on every node, applies the desired network state via NetworkManager Infra / Control Plane Deployment Webhook (manifest validation), metrics endpoint, OpenShift console plugin

To let a node hear ARP traffic on VLAN 101 (or any additional VLAN), you create a **NodeNetworkConfigurationPolicy** (NNCP). This tells NMState to:

• Create a VLAN sub-interface on the node (e.g., ens1f0.101) tied to the physical uplink.
• Leave that sub-interface without an IP address — you only need Layer 2 presence to answer ARP, not a routable address on the node itself, which would otherwise interfere with the node’s default route.

One prerequisite that’s easy to overlook: the physical switch port (or virtual switch port, for VMs) must be configured as a trunk so tagged VLAN traffic actually reaches the node’s NIC. NMState configures the OS side; it cannot make an access port carry tagged frames. In regulated environments, this trunk change itself may require a formal network change ticket — plan the dependency into your rollout timeline early, since it’s usually the longest lead-time item in the whole project.

Sizing the NMState Instance for Production

Once the operator is installed, you create a single NMState custom resource to instantiate the handler and infra components:

apiVersion: nmstate.io/v1
kind: NMState
metadata:
  name: nmstate
spec:
  logLevel: info
  metricsConfiguration:
    bindAddress: ':8089'
  probeConfiguration:
    dns:
      host: root-servers.net

For a production cluster — telco or financial — the fields worth tuning are:

• **probeConfiguration** — controls liveness/readiness probe timing for NMState pods. On heavily loaded nodes, tight default timeouts can cause unnecessary pod restarts; increase the timeout/interval here if you see flapping.
• **affinity** — pins the Handler DaemonSet to specific nodes (useful if you deliberately want to exclude certain nodes, e.g., isolated CNF workers or PCI-scoped nodes, from network reconfiguration).
• **tolerations** — required if you have tainted nodes (isolated CNF workers, dedicated trading or payment-processing nodes, NoSchedule master nodes) that still need their network state managed by the Handler.
• **infraAffinity** — pins the control-plane components (webhook, metrics) to infra/master nodes rather than workers, which is standard practice in larger clusters.
• **infraTolerations** — complements infraAffinity; needed if your infra/master nodes carry taints (they usually do).
• **selfSignConfiguration** — controls the self-signed TLS certificate used by the webhook to talk to the Kube API. Defaults are fine for almost all cases, though some financial environments with strict internal PKI policies may require replacing this with an internally-issued certificate — check with your security team before go-live.
• **metricsConfiguration** — Prometheus scrape endpoint settings; OpenShift Monitoring defaults are compatible out of the box.

Verify the deployment before moving on:

$ oc get pods -n openshift-nmstate
NAME                                      READY   STATUS    RESTARTS   AGE
nmstate-console-plugin-7f9986dbf4-dd6mq   1/1     Running   0          53s
nmstate-handler-22lwg                     1/1     Running   0          55s
...
nmstate-operator-d58fb5858-9jdkj          1/1     Running   0          7d22h
nmstate-webhook-cc55b56c5-7w78m           1/1     Running   0          55s

$ oc get nmstate
NAME      STATUS      REASON
nmstate   Available   SuccessfullyDeployed

Step 1 — Mixed Hardware, One Policy Won’t Fit All

This is where real production clusters diverge from tutorials: your workers are not homogeneous. A common pattern — equally common in a telco core and a bank’s private cloud — is a mix of virtualized workers (ens33, ens192, etc.) and bare-metal workers running a NIC bond (bond0). A single NNCP referencing one interface name will silently fail — or worse, only partially apply — across a heterogeneous fleet.

The production-safe approach is to label nodes by network topology first, then write one NNCP per topology, scoped with nodeSelector.

# Virtual workers (e.g., worker-01 through worker-11)
$ oc label node <vm-node-name> network-topology=virtual

# Bare-metal workers (e.g., worker-12 through worker-16)
$ oc label node <bm-node-name> network-topology=physical

Validate the labeling:

$ oc get node worker-11 --show-labels
NAME        STATUS   ROLES    AGE   VERSION   LABELS
worker-11   Ready    worker   19d   v1.35.5   ...,network-topology=virtual,node-role.kubernetes.io/worker=,...

NNCP for virtualized workers

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: vlan-3803-virtual-policy
spec:
  nodeSelector:
    network-topology: virtual
  desiredState:
    interfaces:
    - name: ens33.3803
      type: vlan
      state: up
      vlan:
        base-iface: ens33
        id: 3803
      ipv4:
        enabled: false
      ipv6:
        enabled: false

NNCP for bare-metal workers (bonded NIC)

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: vlan-3803-physical-policy
spec:
  nodeSelector:
    network-topology: physical
  desiredState:
    interfaces:
    - name: bond0.3803
      type: vlan
      state: up
      vlan:
        base-iface: bond0
        id: 3803
      ipv4:
        enabled: false
      ipv6:
        enabled: false

Why ipv4.enabled: false and ipv6.enabled: false?

This VLAN sub-interface exists purely so the node can hear and answer ARP on behalf of MetalLB — it is not meant to become a routable interface for the node's own OS traffic.

Leaving IP assignment disabled avoids polluting the node's routing table or accidentally creating a second default route. In a financial environment this also has an auditing benefit: a network scan of the node will show a VLAN-tagged interface with no assigned address, which is easy to explain to an auditor as "L2 presence only, no routable attack surface."

Validate the rollout

$ oc get NodeNetworkConfigurationPolicy -A
NAME                        STATUS      REASON
vlan-3803-physical-policy   Available   SuccessfullyConfigured
vlan-3803-virtual-policy    Available   SuccessfullyConfigured

$ oc describe NodeNetworkConfigurationPolicy vlan-3803-physical-policy
...
Status:
  Conditions:
    Message:  5/5 nodes successfully configured
    Reason:   SuccessfullyConfigured
    Status:   True

For a node-level sanity check, SSH into a worker and confirm the sub-interface exists in NetworkManager:

$ ssh core@<node-name>
$ sudo nmtui

added vLan on virtualized workers

added vLan on bare-metal workers

Step 2 — Deploy MetalLB and Bind It to the New VLAN Interfaces

Once the VLAN sub-interfaces exist on every worker (across both topologies), instantiate MetalLB:

apiVersion: metallb.io/v1beta1
kind: MetalLB
metadata:
  name: metallb
  namespace: metallb-system
spec: {}

Create the address pool for this VLAN’s IP range:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: ip-addresspool-vlan-3803
  namespace: metallb-system
spec:
  addresses:
    - 10.171.16.200-10.171.16.220
  autoAssign: false
  avoidBuggyIPs: false

Production tip: set autoAssign: false for every pool in a multi-VLAN, multi-tenant cluster. Explicit loadBalancerIP requests per service prevent a service from silently grabbing an IP from the wrong VLAN's pool — a mistake that's merely inconvenient in a telco OAM network, but can be a real compliance issue if a service lands on a PCI-scoped or trading-network VLAN it was never approved for.

Now the piece that actually solves the original problem — the L2Advertisement. This is where you tell MetalLB which interfaces to use for ARP on each node:

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: vlan-3803-adv
  namespace: metallb-system
spec:
  interfaces:
    - ens33.3803
    - bond0.3803
  ipAddressPools:
    - ip-addresspool-vlan-3803

Note that you list both interface names — the virtual-worker one and the bare-metal one — even though no single node has both. MetalLB checks, per node, which of the listed interfaces actually exists and uses that one. This is what lets a single L2Advertisement work cleanly across a heterogeneous fleet without maintaining separate advertisements per hardware type.

End-to-end test

apiVersion: v1
kind: Service
metadata:
  name: loadbalancer-svc
  namespace: default
  annotations:
    metallb.universe.tf/address-pool: ip-addresspool-vlan-3803
spec:
  type: LoadBalancer
  loadBalancerIP: 10.171.16.210
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

$ oc get svc -n default
NAME               TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
loadbalancer-svc   LoadBalancer   192.168.83.22   10.171.16.210   80:30644/TCP   5s

If ARP is answered correctly, 10.171.16.210 is reachable from any host on VLAN 3803 — including routers and clients that never had, and never will have, a route into your cluster's default VLAN.

Making This Actually Production-Ready

The steps above get traffic flowing, but production environments — whether a telco core or a bank’s trading floor — have requirements a lab setup doesn’t. A few things worth locking down before this goes anywhere near a live signaling path, a payment gateway, or a market-data feed:

1. Confirm trunk configuration on both physical and virtual switches. NMState only configures the OS side of the VLAN. If the physical switch port (or the virtual switch’s port group, for VM-based workers) isn’t set to trunk with VLAN 3803 allowed, the sub-interface will come up but never see a single tagged frame. This is worth validating explicitly with the network team before troubleshooting anything at the Kubernetes layer — and in regulated environments, worth getting in writing as part of the change record.

2. Plan for L2 failover timing. MetalLB’s L2 leader election is fast, but not instantaneous — when the leader node fails, there’s a window (typically single-digit seconds, network-dependent) before a new node’s gratuitous ARP propagates and switches/routers update their tables. For latency-sensitive traffic — telco signaling on one hand, order execution or payment authorization on the other — factor this into your SLAs, and validate actual failover time in your specific switch fabric rather than assuming a number. If your financial workloads have hard latency ceilings (e.g., trading systems), this failover window may make BGP with ECMP a better fit for that specific tier, even if L2 mode remains right for everything else.

3. Don’t let one node become the leader for every pool. In L2 mode, MetalLB elects a leader per service, not per node globally, but in small clusters it’s common to see the same node win leadership repeatedly. For services with heavy throughput — high call-volume signaling, or end-of-day batch settlement traffic — monitor leader distribution across nodes so you don’t accidentally concentrate all your VLAN egress traffic on one worker’s NIC.

4. Separate VLANs by function, not convenience. The pattern shown here (VLAN 101/3803 for one address pool, with its own NNCP and L2Advertisement) scales cleanly to additional VLANs for OAM, signaling, and media planes in a telco context, or for DMZ, core banking, and payment-processing planes in a financial context — each with its own IPAddressPool and L2Advertisement, and its own dedicated VLAN sub-interface via a separate NNCP. Keep the naming convention (<iface>.<vlan-id>) consistent; it makes auditing NNCPs and advertisements far easier once you have five or six VLANs in play, and far easier to hand to a compliance auditor as evidence of network segmentation.

5. Keep autoAssign: false everywhere, always. Worth repeating: in multi-VLAN clusters, an address pool with autoAssign: true is a standing risk that a service gets an IP from the wrong VLAN, especially as new pools are added over time by different teams. In a PCI-DSS or similarly regulated scope, this isn't just an operational risk — it can constitute an uncontrolled change to network segmentation.

6. Watch the NMState metrics endpoint. metricsConfiguration.bindAddress exposes Prometheus metrics for the Handler. In production, alert on NNCP reconciliation failures — a policy that silently drops from "SuccessfullyConfigured" after a node reboot or NIC firmware change is a classic source of "it worked yesterday" incidents, and in a regulated environment, an undetected drift like this can also become an audit finding if it isn't caught by monitoring.

7. Taint and toleration hygiene for isolated workloads. If you run isolated worker pools with dedicated taints — CNF nodes in telco, or PCI-scoped/segregated nodes in financial services — remember the Handler DaemonSet needs a matching tolerations entry in the NMState CR, or those nodes will never get their VLAN sub-interface, and will silently fail to answer ARP for any pool that expects them to.

8. Document the “no-IP” design decision for auditors and network teams. Because the VLAN sub-interfaces intentionally carry no IP address, they can look unusual to anyone reviewing node network configuration for the first time — including auditors, security reviewers, or a NetOps engineer troubleshooting an unrelated issue. A short internal note explaining that these interfaces exist solely for MetalLB ARP responsiveness, with no routable address by design, saves a surprising amount of back-and-forth during audits and incident reviews.

Summary

MetalLB’s L2 mode is a legitimate, production-viable alternative to BGP for exposing LoadBalancer services — in telco cores and financial-services platforms alike — but only if your nodes have genuine Layer 2 presence in every VLAN you intend to serve. The Kubernetes NMState Operator is the piece that makes this declarative, auditable, and safe to run at scale: it creates IP-less VLAN sub-interfaces purely for ARP visibility, without disturbing the node’s own routing.

The pattern that scales in mixed-hardware, multi-VLAN environments — whether the VLANs represent signaling planes or payment-processing segments — is:

1. Label nodes by network topology (virtual vs. physical, or whatever axis your hardware varies on).
2. Write one NodeNetworkConfigurationPolicy per topology, each producing a differently-named but functionally identical VLAN sub-interface.
3. Point a single L2Advertisement at all the interface names in play — MetalLB resolves per-node which one actually exists.
4. Keep pools explicit (autoAssign: false) and VLANs scoped by function, with documentation ready for whoever audits your network segmentation next.

No BGP session, no route reflectors, no coordination overhead with the network team beyond “please trunk this VLAN to these switch ports.” Just clean, declarative Layer 2 presence, exactly where MetalLB needs it — whether that’s a telco signaling network or a bank’s payment gateway VLAN.