MetalLB Layer 2 Monitoring Dashboard with Grafana and Prometheus

Modern OpenShift environments rely heavily on LoadBalancer services to expose applications outside the cluster. In cloud platforms, this functionality is usually provided by the cloud provider. In on-premises, edge, or bare-metal environments, MetalLB fills that gap by assigning external IP addresses and advertising them on the network.

This article focuses on MetalLB Layer 2 mode, one of the most widely used deployment models in OpenShift. While MetalLB is relatively easy to deploy, troubleshooting network issues can be difficult without the right visibility. Problems such as IP pool exhaustion, failed ARP responses, uneven VIP ownership, or repeated failovers often remain unnoticed until users begin reporting connectivity issues.

The Grafana dashboard presented in this article was designed to provide operational visibility into MetalLB rather than simply displaying metrics. It combines controller, speaker, and networking metrics into a single view, making it easier to understand how MetalLB is behaving in real time and quickly identify potential issues.

Why Monitor MetalLB Layer 2?

Unlike BGP mode, where multiple nodes can advertise routes simultaneously, Layer 2 mode assigns each Virtual IP (VIP) to a single node at a time. That node is responsible for answering ARP requests (IPv4) or NDP requests (IPv6) from the network.

When the active node becomes unavailable, another MetalLB speaker takes ownership of the VIP and sends Gratuitous ARP or unsolicited Neighbor Advertisements so that switches and clients update their MAC address tables without waiting for cache expiration.

Although this process is automatic, administrators often need answers to questions such as:

• Is every speaker running the latest configuration?
• Which node currently owns a VIP?
• Are ARP requests being answered successfully?
• Is a failover expected or is the cluster unstable?
• Is the IP pool close to running out of addresses?

These operational questions cannot be answered by checking pod status alone.

Understanding the Layer 2 Architecture

MetalLB Layer 2 Architecture

In a Layer 2 deployment, the MetalLB Controller manages IP address allocation and watches Kubernetes resources such as IPAddressPool and Service objects. It decides which external IP should be assigned but does not participate in network advertisement.

The MetalLB Speaker, running on each eligible node, is responsible for advertising VIPs on the network. Only one speaker owns a VIP at any given time and responds to ARP or NDP requests from external clients.

When leadership changes because of maintenance or a node failure, the new speaker immediately sends Gratuitous ARP or Neighbor Advertisements. This updates the network’s MAC address tables and redirects traffic to the new owner with minimal interruption.

Common Operational Challenges

Even in healthy clusters, several conditions can affect application availability if they are not detected early.

IP Pool Exhaustion

Every LoadBalancer service consumes an IP address from a configured pool. Once the pool is exhausted, new services remain in a Pending state until additional addresses are made available.

Monitoring pool utilization helps identify capacity issues before deployments are affected.

Uneven VIP Distribution

Ideally, VIP ownership should be reasonably balanced across available nodes. If one node owns significantly more VIPs than others, it becomes a larger failure domain. A single node outage could trigger multiple simultaneous failovers, increasing recovery time and network activity.

Configuration Synchronization

All MetalLB speakers should operate with the same configuration. If one or more speakers fail to load the latest configuration, VIP ownership and failover behavior may become inconsistent.

Layer 2 Communication

Every external connection begins with an ARP or NDP request. If those requests are not answered, applications may become unreachable even though Kubernetes reports all pods and services as healthy.

Monitoring Layer 2 traffic provides an early indication of these issues.

Failover Stability

Occasional Gratuitous ARP messages are expected during upgrades or planned maintenance. However, frequent or continuous announcements may indicate repeated leader changes, unstable nodes, or network problems that require investigation.

Dashboard Design Philosophy

This dashboard was built around real operational scenarios rather than simply exposing Prometheus metrics.

Instead of asking, “What metric should be displayed?”, the focus was on questions administrators ask during incidents.

• Are all speakers synchronized?
• Is the IP pool healthy?
• Which node currently owns each VIP?
• Is MetalLB responding to every ARP request?
• Are failovers occurring as expected?
• Is one node handling significantly more services than others?

Each panel answers one of these questions, allowing operators to move quickly from detection to diagnosis without switching between multiple tools.

Dashboard Overview

MetalLB Layer 2 Dashboard

The dashboard is organized into logical sections that follow the lifecycle of MetalLB operations:

• Cluster Status validates speaker configuration and overall allocator health.
• Pool Health tracks IP usage, utilization, and remaining capacity.
• Leadership Distribution shows how VIP ownership is spread across nodes.
• VIP Mapping provides a live view of which node and speaker currently advertise each service.
• Layer 2 Networking monitors ARP/NDP requests, responses, and dropped requests.
• Failover Analysis tracks Gratuitous ARP activity and identifies unstable VIPs or excessive leadership changes.

By combining these views into a single dashboard, administrators can monitor capacity, networking behavior, and failover activity without manually correlating multiple metrics.

Dashboard Walkthrough

The dashboard is organized to follow the operational flow of MetalLB. It starts with overall cluster health, moves into IP address management, shows how VIP ownership is distributed, and finally provides visibility into Layer 2 network activity and failover events.

Rather than looking at individual metrics in isolation, the dashboard brings together related information so that potential issues can be identified quickly.

Cluster Status

Cluster Status

The first section provides a quick health check of the MetalLB deployment. It focuses on speaker configuration and IP pool utilization, making it the first place to look during routine health checks or incident investigations.

Speaker Configuration Status

The first two panels monitor whether every MetalLB speaker is running the latest configuration.

Speakers Config Loaded

Metric

sum(metallb_k8s_client_config_loaded_bool)

This panel shows how many speaker instances have successfully loaded the current MetalLB configuration.

A healthy cluster should have every speaker reporting a loaded configuration. If the value is lower than the number of speaker pods, one or more nodes may not be advertising services correctly.

Possible causes include:

• Invalid MetalLB configuration
• Failed configuration reload
• Communication issues between the controller and speaker

Speakers Config Stale

Metric

sum(metallb_k8s_client_config_stale_bool)

This panel highlights speakers still using an outdated configuration.

The expected value is always 0.

A non-zero value indicates that at least one node has not applied the latest configuration changes. Although the cluster may continue serving traffic, future VIP assignments or failovers can behave unexpectedly until the configuration is synchronized.

IP Pool Capacity

The remaining panels provide an overview of IP address allocation and available capacity.

IPs In Use

Metric

sum(metallb_allocator_addresses_in_use_total)

Displays the number of addresses currently assigned to LoadBalancer services.

This metric helps track growth over time and provides a quick indication of how much of the available address space is being consumed.

Total Pool IPs

Metric

sum(metallb_allocator_addresses_total)

Shows the total number of IP addresses available across all configured address pools.

This value changes only when IP pools are expanded or modified.

Pool Utilization

Metric

100 * sum(metallb_allocator_addresses_in_use_total)
/
sum(metallb_allocator_addresses_total)

This panel calculates overall pool utilization as a percentage.

Recommended thresholds:

UtilizationStatusBelow 70%Healthy70–90%Monitor capacityAbove 90%Expand IP pools

High utilization doesn’t immediately impact running services, but it prevents new LoadBalancer services from receiving external IP addresses once the pool is exhausted.

Free IPs Remaining

This panel shows the number of available addresses left in the configured pools.

Monitoring this value is often more useful than watching utilization percentages because it provides a clear indication of remaining capacity.

For example:

• 200 free IPs indicate plenty of capacity.
• 10 free IPs suggest planning for expansion.
• 0 free IPs mean new LoadBalancer services will remain in the Pending state.

Leadership Distribution

VIP Leadership Distribution

One of the most useful sections of the dashboard shows how MetalLB distributes VIP ownership across cluster nodes.

Only one speaker advertises a VIP in Layer 2 mode, making leadership distribution an important operational metric.

Active Services Per Node

Metric

sort_desc(sum by (node) (metallb_speaker_announced))

This bar chart displays the number of services currently announced by each node.

A balanced environment should show a relatively even distribution of VIP ownership.

This indicates that leadership is spread evenly across the cluster.

An imbalanced distribution might look like this:

Node VIPs

worker-1 42

worker-2 6

worker-3 5

Although applications continue to function normally, a failure of worker-1 would require dozens of VIPs to move simultaneously, increasing network convergence time and creating a larger failure domain.

This panel makes those situations immediately visible.

Live Node-to-Service Mapping

Live VIP Ownership

During troubleshooting, one of the most common questions is:

Which node currently owns this LoadBalancer IP?

The Node → Speaker Pod → VIP → Service table answers this instantly.

Each row shows:

• Physical node
• MetalLB speaker pod
• Advertised VIP
• Kubernetes Service
• Advertisement protocol

Instead of searching through Kubernetes resources or MetalLB logs, administrators can quickly identify the current owner of any VIP.

This view is especially useful during:

• Planned maintenance
• Node failures
• Failover verification
• Service migration
• Network troubleshooting

It also helps confirm that VIP ownership matches expectations after upgrades or cluster changes.

Why These Panels Matter Together

Although each panel provides valuable information on its own, their real strength comes from being viewed together.

For example, imagine a new application cannot obtain an external IP address.

The dashboard immediately provides a logical troubleshooting path:

1. Check Pool Utilization and Free IPs Remaining to verify address availability.
2. Confirm all speakers have successfully loaded the latest configuration.
3. Ensure no speakers are reporting stale configurations.
4. Verify VIP ownership using the live mapping table.
5. Confirm leadership is balanced across the cluster.

Within a few minutes, operators can determine whether the issue is related to capacity, configuration, or service advertisement without manually querying multiple Kubernetes resources.

ARP/NDP Request Monitoring

ARP/NDP Requests Received

Whenever a client connects to a LoadBalancer service, it first needs to determine which MAC address owns the Virtual IP. For IPv4, this is done using the Address Resolution Protocol (ARP), while IPv6 uses the Neighbor Discovery Protocol (NDP).

The ARP/NDP Requests Received panel displays the rate at which MetalLB receives these requests for each advertised VIP.

Metric

rate(metallb_layer2_requests_received[$__rate_interval])

How to Interpret the Panel

A low and steady request rate is expected in most environments because clients and switches cache MAC address information. Once cached, they do not need to send repeated ARP or NDP requests.

Higher request rates may occur during:

• Client reconnects
• Switch or router cache expiration
• Application deployments
• Network topology changes

A sudden and sustained increase across multiple VIPs may indicate excessive ARP traffic or instability in the surrounding network.

ARP/NDP Response Monitoring

ARP/NDP Responses Sent

Receiving ARP or NDP requests is only half the picture. MetalLB must also respond to those requests so clients can resolve the VIP and establish connections.

This panel tracks the response rate for every advertised VIP.

Metric

rate(metallb_layer2_responses_sent[$__rate_interval])

What Healthy Looks Like

Under normal conditions, the request and response graphs should closely match.

For every ARP request received, there should be a corresponding response from the active MetalLB speaker.

If response rates consistently fall below request rates, external clients may fail to reach the service even though Kubernetes reports everything as healthy.

Possible causes include:

• Speaker failures
• Network interruptions
• Incorrect Layer 2 advertisements
• Configuration issues

Comparing the request and response panels side by side provides a quick way to verify that MetalLB is actively responding to network discovery traffic.

Dropped Request Detection

Dropped Layer 2 Requests

One of the most valuable panels in the dashboard measures the difference between incoming requests and outgoing responses.

Metric

sum(rate(metallb_layer2_requests_received[$__rate_interval))
-
sum(rate(metallb_layer2_responses_sent[$__rate_interval))

The expected value is always zero.

A value above zero means MetalLB is receiving discovery requests but is not responding to all of them.

Even a small sustained increase deserves investigation because unanswered ARP or NDP requests can prevent external clients from reaching applications.

This panel serves as an excellent candidate for alerting, helping identify network issues before they become widespread.

Gratuitous ARP/NDP Monitoring

Gratuitous ARP Activity

When VIP ownership changes, MetalLB sends Gratuitous ARP (IPv4) or unsolicited Neighbor Advertisements (IPv6). These packets notify switches and connected devices that the VIP is now owned by a different node.

The dashboard tracks these announcements for every VIP.

Metric

rate(metallb_layer2_gratuitous_sent[$__rate_interval])

Expected Behavior

Small background activity is normal and helps refresh network forwarding tables.

Short spikes are also expected during:

• Planned maintenance
• Node reboots
• Cluster upgrades
• VIP migration

These events typically settle within a few seconds.

Continuous or repetitive spikes, however, may indicate repeated leadership changes or an unstable environment that requires further investigation.

Leadership Failover Analysis

Gratuitous ARP by Node

Understanding which node is generating Gratuitous ARP announcements makes it much easier to verify failover behavior.

This panel joins MetalLB metrics with Kubernetes node information to display Gratuitous ARP activity per node.

Metric

sum by (node)(
rate(metallb_layer2_gratuitous_sent[$__rate_interval])
* on(pod) group_left(node)
kube_pod_info{namespace="metallb-system"}
)

Typical Scenarios

Normal operation

All nodes show minimal background activity.

Planned maintenance

The original owner stops advertising the VIP, while another node briefly spikes as it announces the new ownership.

Unexpected failovers

Frequent spikes moving between multiple nodes often indicate repeated leader elections or node instability.

Viewing this panel alongside the VIP ownership table provides a clear picture of how leadership changes are occurring within the cluster.

Measuring Failover Stability

Not every Gratuitous ARP packet indicates a problem. During quiet periods, there may be very little client traffic, making simple percentage calculations misleading.

To avoid false alarms, the dashboard calculates the Gratuitous-to-Request ratio only when there is sufficient network activity.

Metric

(100 *
rate(metallb_layer2_gratuitous_sent[$__rate_interval])
/
rate(metallb_layer2_requests_received[$__rate_interval]))
and
(rate(metallb_layer2_requests_received[$__rate_interval]) > 0.01)

This approach prevents inflated percentages during periods of minimal traffic while still highlighting excessive Gratuitous ARP activity when clients are actively accessing services.

Sustained high ratios under normal traffic conditions may indicate unstable VIP ownership or repeated failovers.

Identifying Unstable VIPs

Failover Severity Ranking

The final panel ranks VIPs based on the cumulative number of Gratuitous ARP announcements.

Metric

sort_desc(metallb_layer2_gratuitous_sent)

This view makes it easy to identify services experiencing significantly more failover activity than others.

Interpreting the data requires considering the observation period.

• A high value accumulated over several weeks is generally expected.
• A rapidly increasing counter over minutes or hours may indicate repeated failovers, unstable nodes, or networking issues.

Combined with the previous panels, this ranking helps narrow investigations to the services experiencing the greatest amount of Layer 2 activity.

Bringing the Metrics Together

Each networking panel provides a different view of MetalLB’s behavior, but together they form a complete operational picture.

A healthy environment typically exhibits the following characteristics:

• ARP and NDP request rates remain stable.
• Responses closely match incoming requests.
• Dropped requests stay at zero.
• Gratuitous ARP activity is minimal during normal operation.
• Brief spikes occur only during planned failovers.
• VIP ownership remains stable across nodes.

When one or more of these patterns changes, the dashboard helps determine whether the issue is related to client discovery, Layer 2 communication, VIP ownership, or failover activity.

By combining these metrics into a single view, administrators can quickly validate network health and troubleshoot MetalLB Layer 2 behavior without relying solely on logs or packet captures.

Get the dashboard manifest from the GitHub : **https://raw.githubusercontent.com/arjun0451/ocp-sysops-kit/refs/heads/main/Monitoring/metallb-monitoring-l2-mode/metallb-l2-dashboard.json**

Conclusion

Monitoring MetalLB extends beyond checking whether pods are running or IP addresses have been allocated. Reliable Layer 2 networking depends on healthy ARP and NDP communication, balanced VIP ownership, synchronized speaker configuration, and predictable failover behavior.

This dashboard combines these operational insights into a single Grafana view, enabling faster troubleshooting, better capacity planning, and improved visibility into how MetalLB behaves in production. Whether investigating connectivity issues, validating failovers during maintenance, or monitoring long-term network stability, these metrics provide the context needed to keep LoadBalancer services reliable and available.

Disclaimer:

The views and insights expressed in this piece are solely those of the author and do not represent the beliefs of their employer. These reflections are firmly based on personal experience. While every attempt has been made to ensure the information is accurate, the author cannot guarantee its correctness; it is ultimately the reader’s responsibility to verify and apply the content wisely. All trademarks mentioned are the property of their respective owners

LinkedIn: https://www.linkedin.com/in/nnreddy51/