Your SIEM Shouldn’t Go Silent When Your Network Does

Photo by l ch on Unsplash

*Jeff Darrington, Technical Marketing Director, Graylog*

A WAN link goes down at a remote site. Firewalls, switches, and routers on the other side of that link keep generating logs. Those logs have nowhere to go. When the link comes back, those logs do not come back with it.

That is not a configuration problem. That is how Syslog was designed. It sends and forgets. Every message is a UDP packet fired at the collection point with no acknowledgment, no sequencing, and no retry. If the path between the device and the SIEM is interrupted, every log generated during that window is gone.

For security and IT teams managing distributed environments, this is not an edge case. It is the default behavior of the most widely deployed logging protocol in the world. And most SIEM evaluations never test it.

SIEM operational resilience is the property that determines whether a security platform continues collecting, processing, and alerting accurately during network outages, cluster stress, and log volume anomalies, or silently fails. Most teams discover the gap at the worst possible moment: during an incident investigation, when the logs that should be there are not.

The Three Ways a SIEM Goes Silent Without Telling You

A SIEM can fail to collect logs in three distinct ways, and none of them generate an alert in most default configurations.

1. Network outages drop Syslog in transit. Syslog is the universal logging protocol for network infrastructure: firewalls, switches, routers, access points, and OT devices. In its most common UDP implementation, each log message is a single packet with no delivery guarantee. Even TCP Syslog, which provides connection-oriented delivery, offers no guaranteed delivery at the application layer. If the connection drops, whether logs are buffered on the source device depends entirely on the vendor and many don’t. There’s no standardized retry or catch-up mechanism you can count on.
2. Log sources silently stop sending. Sources stop sending for reasons that have nothing to do with a network outage: an agent crashes, a configuration change inadvertently removes a forwarding rule, a disk fills up on the source device and logging suspends. In most SIEM configurations, the absence of expected log data generates no alert. The dashboard shows no errors. The coverage gap is invisible until an incident investigation asks why three weeks of logs from a specific device are missing.
3. Cluster stress degrades processing without failing completely. The subtlest failure mode is gradual. A SIEM cluster under load begins buffering messages internally when the processing pipeline cannot keep up with ingest volume. Messages are not lost immediately, but they are delayed. If the buffer fills completely, messages begin to drop. The cluster does not crash. It processes more slowly, then drops events silently, with no obvious indication unless operational metrics are being actively monitored.

The Uptime Institute’s Annual Outage Analysis found that IT and network-related issues now account for 23% of impactful outages, with the trend reflecting growing complexity in distributed environments and increased reliance on third-party services. For organizations running distributed SIEM architectures, each of these failure modes is a realistic scenario, not a theoretical one.

The Forwarder: Moving Collection to Where the Logs Are

The architectural solution to Syslog data loss during network outages is to move the collection point as close to the log sources as possible, with local caching that survives the outage and delivers buffered data when connectivity restores.

A lightweight forwarder service, typically running on a small Linux VM or container with minimal resource requirements, deploys at the edge near the log sources. It receives Syslog streams, Windows log feeds, and other inputs that would otherwise be sent across the WAN directly to the central SIEM cluster. Once it receives those logs, it compresses and encrypts them for transit, and it caches them locally on disk before sending.

The critical property is that the cache is persistent. When the network link to the central cluster fails, the forwarder continues collecting from every source in its proximity. All logs are stored in a journal and buffered. When connectivity restores, the forwarder automatically delivers all cached messages. The central cluster receives a complete, gapless log history for that period. The gap that would otherwise appear in the investigation timeline does not appear.

Three additional properties matter for organizations with distributed environments. First, the forwarder consolidates what would otherwise could be dozens of individual firewall rules, one per device, into a single outbound TLS connection from the forwarder to the central cluster. Second, input configurations are managed centrally through the SIEM UI and pushed automatically to the forwarder. No remote access to the remote site is required when a new log source is added. Third, forwarder deployments are not licensed per instance, meaning organizations with 20 or 50 remote sites can deploy forwarders at every location without changing their license cost.

For lean security teams in particular, this architectural approach means achieving enterprise-level precision without the operational overhead of managing brittle, device-by-device log routing across distributed environments.

The Volume Anomaly That Discovered a Breach

Monitoring log volume per source is not just an operational health check. It is a detection capability.

Standard SIEM alert models are content-based: they look for specific event types, specific field values, and specific patterns in log messages. They work when an attacker generates the kind of log entries that detection rules are written to find. They miss the class of attack where activity manifests not as a specific suspicious event, but as a change in the volume of normal-looking events.

An API being accessed by an attacker at scale produces logs. Those logs look like normal logs. Each individual request may not be suspicious. But a quadrupling of API request volume on a service that normally handles a stable, predictable load is anomalous in exactly the way that a content-based detection rule would never catch.

The same operational health monitoring mechanism that alerts when a log source goes silent can be tuned to alert when volume spikes beyond a threshold. The detection logic is symmetric: too little volume means something stopped working; too much volume means something changed. Volume anomaly detection as a SIEM alert type catches a class of attack that content-based rules miss entirely.

IBM’s Cost of a Data Breach Report found that 42% of breaches were detected by an organization’s own security team or tools, and that internal detection shortened the data breach lifecycle by 61 days while saving organizations nearly $1 million in breach costs compared to incidents disclosed by an attacker. The organizations detecting breaches internally are the ones with monitoring architectures that watch not just what events happen, but whether their monitoring is working at all.

Knowing When Your SIEM Is Not Working

The most dangerous SIEM failure is the one that produces no error message: gradual degradation where processing slows, messages buffer, and eventually drop, without any obvious indication in the standard UI.

Data comes into Graylog and gets written to the journal. Messages are pulled from the journal and go through processing pipelines. Once processing is complete, batches of messages are written to Opensearch, which allows them to be seen in searches and dashboards.

Graylog Data Processing

When the output buffer grows, the data store is receiving messages faster than it can index them. The cause is typically resource constraints on the storage layer or an ingest spike from a specific source. When the processing buffer grows, the processing nodes are receiving messages faster than they can parse and enrich them, usually from CPU saturation or a computationally expensive pipeline rule.

Without metrics connected to a time-series monitoring system, this process is invisible until the buffer fills and drops begin. Connecting Graylog and OpenSearch metrics to Prometheus with a Grafana frontend provides not just a current value for each metric, but historical trending. A buffer that began filling six hours ago and is now at 40% is a different operational situation than a buffer that spiked to 40% in the last five minutes. Both require investigation. Only one is urgent.

Manage and Create Pipelines

High Availability for the Infrastructure That Can’t Go Down

High availability in a SIEM is not a luxury. It is a baseline requirement for any organization where the SIEM is the single point of visibility into what is happening on the network.

The risk is straightforward: a self-hosted SIEM that lives in the same network segment as the devices it monitors faces the same threats those devices face. The event that generates the most logs, a major incident, a power failure, a ransomware attack spreading across the network, is also the event most likely to take down the monitoring infrastructure. The architectural response is separation: collection points live with the devices they monitor, and the central cluster lives somewhere independent.

Each Graylog processing node maintains a local journal. If the connection from a processing node to the data store is interrupted, whether from a brief network hiccup, a scheduled maintenance window, or a node restart, messages are written to the journal on local disk rather than being dropped. When connectivity restores, the journal drains automatically. The gap that would otherwise appear in the log timeline does not appear.

Building Operational Visibility Before You Need It

Operational visibility for a SIEM should be built as part of initial deployment, not in response to a performance problem that has already caused data loss. Five monitoring commitments belong in every deployment checklist.

Export cluster metrics to a time-series monitoring system from day one. The metric that is always empty is still worth watching, because when it starts filling, there is a narrow window to act.

Configure input source silence alerts for every major log source. Five minutes of silence from a high-volume firewall is meaningful. An hour of silence from a domain controller may require immediate investigation.

Configure volume anomaly alerts alongside silence alerts for sources with stable, predictable output. The upper-bound alert and the lower-bound alert are the same mechanism pointed in both directions.

The Logs Your Forwarder Saved Are the Logs the Investigation Needs

The breach discovered through log volume monitoring was not discovered by a sophisticated detection rule. It was discovered because the logs were there to look at, and the right number was being watched.

The organizations that discover incidents through their own visibility infrastructure have done the unglamorous operational work that most SIEM deployments skip: collection infrastructure that does not go silent when the network does, cluster metrics connected to something that records trends over time, and alerts configured not just for what events happen, but for whether the monitoring itself is working.

Your SIEM is only as reliable as its least resilient collection point. The logs your forwarder caches during a WAN outage, the messages your cluster journals through a brief connectivity hiccup, the volume anomaly that tells you something changed: these are not advanced features. They are the baseline architecture that determines whether your security visibility continues when the network does not.

Detection without continuous collection is just an archive of the moments the infrastructure happened to be working. The architecture that closes that gap is the one worth building before the next incident, not after.

Already running Graylog Open? Graylog Enterprise and Security add the Forwarder with persistent disk caching, cluster health metrics, and distributed collection on the same codebase. No migration. Just a license key.

Follow Graylog on LinkedIn for practical guidance on log collection architecture, SIEM operations, and getting more from the security tools your team already runs.