How a Single Regex Took Down a Zabbix Proxy

A small discovery rule mistake caused 100k+ items, massive SYN-SENT traffic, a disabled proxy and monitoring incident

Zabbix is freaking out over Docker because of the regex

Monitoring systems are supposed to help you understand incidents. They are not supposed to become the incident. But one night, that is exactly what happened.

A Zabbix proxy in our environment started restarting over and over again. At first, it looked like a routine issue: maybe the network was unstable, maybe a few agents were flaky, maybe the proxy had a resource problem. Then the proxy itself began to crash. A little later, even the Zabbix UI started returning HTTP 500 errors.

At that point, it was obvious we were no longer dealing with a simple monitoring glitch. This is the story of how we traced that failure back to something deceptively small: a missing exclusion in a network interface discovery rule.

The symptom: a proxy that would not stay up

The first clue came from systemd. The proxy service was not failing at startup. It started normally, ran for a while, and then exited with code 1.

The logs looked roughly like this:

zabbix-proxy.service: Control process exited, code=exited, status=1/FAILURE
zabbix-proxy.service: Failed with result 'exit-code'
Scheduled restart job, restart counter is at ...

That distinction mattered.

If a service fails during startup, I usually look at configuration, permissions, or dependencies. But when it starts fine and dies later, I expect one of three things:

• a runtime dependency problem
• a load problem
• an internal worker/process failure

The proxy log gave us the first real hint:

One child process died (exitcode/signal:1). Exiting ...

So this was not “the proxy randomly stopped.” One of its child processes was dying, and the proxy was shutting itself down afterward.

That raised the real question:

What was killing the proxy’s child processes?

The first theory: “This must be a network issue”

At first, that seemed reasonable.

The Zabbix logs were full of messages like these:

Zabbix agent item "..." failed: first network error, wait for 15 seconds
Zabbix agent item "..." failed: another network error, wait for 15 seconds

And they were not limited to one or two hosts. They showed up in bursts, across many checks.

That looked like network instability.

But once we started eliminating the usual suspects, the picture got stranger:

• no OOM killer
• no segfaults
• no kernel panic
• no obvious file descriptor exhaustion
• no clear UDP receive buffer issue

The “network errors” were real, but they did not behave like a clean network outage.

So instead of treating them as the root cause, we treated them as symptoms.

Then we looked at TCP connection states on the proxy.

That was the first real turning point.

The proxy was full of half-open connections

On the proxy, we checked socket states and found a very large number of outbound connections stuck in SYN-SENT. That is a very specific symptom. It means the proxy is trying to open TCP connections to agents, but the handshake is not completing.

In plain English:

the proxy is actively trying to poll agents, but many of those connections are getting stuck before they even fully connect.

When we grouped those SYN-SENT connections by destination, another important detail appeared: they were not spread evenly across the whole environment. Most of them were concentrated on a handful of agent hosts.

That told us two things immediately:

1. this was not a general proxy networking problem
2. a small set of hosts was driving the proxy into failure

So we moved to one of those hosts.

And that is where the real story began.

The agent was not down. It was overloaded.

On one of the affected hosts, TCP state on port 10050 looked unhealthy.

We saw large numbers of:

• ESTABLISHED
• CLOSE-WAIT
• SYN-RECV
• LAST-ACK

That alone was suspicious, but the Zabbix agent log made it crystal clear:

failed to accept an incoming connection:
reading first byte from connection failed:
[11] Resource temporarily unavailable

That line changed the investigation. The agent was not simply unreachable. It was reachable, but under enough pressure that it could no longer process incoming monitoring connections correctly.

At this point, the problem looked like this:

• the proxy was opening large numbers of passive-check connections
• the agent was accepting more than it could comfortably handle
• old connections were not being cleaned up fast enough
• new ones kept arriving
• the backlog was growing
• the failure was feeding back into the proxy

But there was still one critical question left:

Why were these hosts so much heavier than the rest?

That answer came from item count.

The clue that changed everything: 100k+ items

We used the Zabbix API to pull hosts with the highest item counts.

The top of the list looked something like this:

host-a   107,852 items
host-b   106,298 items
host-c    68,673 items
host-d    58,393 items
host-e    49,576 items

That is not normal.

A typical monitored Linux host might have a few hundred or a few thousand items. Tens of thousands already deserves scrutiny. More than 100,000 means something is fundamentally wrong.

Once we saw those numbers, the proxy behavior made sense. A proxy polling hosts with that kind of cardinality will eventually start to struggle, especially if the checks are passive and connection-heavy.

So the next step was obvious:

What exactly is creating that many items?

We opened one of the worst offenders and looked at the discovered interfaces.

And there it was.

The real root cause: Docker veth interfaces

The affected hosts were container-heavy systems.

Their interface lists were full of names like:

veth0a0eed7
veth0a06040
veth0a065a7
...

Those are Docker-created virtual Ethernet interfaces.

Zabbix low-level discovery was treating them like normal interfaces and generating interface monitoring items for every one of them.

That meant:

• traffic metrics
• errors
• drops
• status checks
• and more

for every ephemeral Docker interface. Once we realized that, the entire incident snapped into focus.

This was not primarily a proxy problem. It was not primarily a network problem. It was not even primarily an agent problem.

It was a monitoring cardinality explosion.

A rough version of the math looked like this:

Math of Failure Chain

And once that happens, the rest is just a chain reaction.

What the failure chain actually looked like

Once we understood the root cause, the incident became easy to explain.

Failure Chain Flow Chart

At that point, even the Zabbix frontend started to suffer and returned HTTP 500s. The monitoring platform was no longer just observing the incident. It had become part of it.

The fix: stop discovering ephemeral interfaces

The good news was that the actual fix was simple once we knew what to change.

The network interface discovery rule was using a global regular expression. Instead of redesigning the whole template, we extended the exclusion logic to ignore ephemeral container-related interfaces.

We added exclusions for:

^veth.*
^docker.*
^br-.*
^cni.*
^flannel.*

That solved the future problem.

But there was still a second issue: tens of thousands of bad items already existed.

So we needed cleanup too.

Discovery fixes do not remove existing damage

This is the part that is easy to miss.

Updating the regex prevents new bad items from being discovered. It does not automatically erase the old ones that are already attached to the host.

We validated the fix with the cleanest reset path available:

1. update the interface discovery regex
2. unlink and clear the template from the affected host
3. link the template back
4. execute network discovery again

That gave us the result we wanted.

At one point, after the template had been cleared, the host was down to a very small item count. After the template was linked again and discovery re-ran, the veth interfaces did not come back.

That was the strongest possible confirmation.

The discovery rule was fixed, and it was fixed in the right place.

Bulk cleanup also mattered

In some cases, template reset was the cleanest path.

In others, we also needed a fast way to remove the already-created veth items in bulk. The Zabbix UI was not a good fit for deleting thousands of items manually, so we used the API.

The approach was straightforward:

• search for items whose names matched veth
• collect the itemids
• delete them in batches

Batch size mattered. We did not want to throw thousands of deletions into one request and hit payload or timeout issues, so we used small batches.

Here is the script we used:

#!/usr/bin/env bash
set -euo pipefail
API_URL="https://zabbix.example.com/api_jsonrpc.php"
API_TOKEN="REDACTED"
HOST_ID="12345"
SEARCH_TERM="veth"
BATCH_SIZE=200
SLEEP_SECONDS=1
ITEM_IDS=$(
  curl -s -X POST "$API_URL" \
    -H 'Content-Type: application/json-rpc' \
    -H "Authorization: Bearer $API_TOKEN" \
    -d "{
      \"jsonrpc\": \"2.0\",
      \"method\": \"item.get\",
      \"params\": {
        \"hostids\": [\"$HOST_ID\"],
        \"search\": {
          \"name\": \"$SEARCH_TERM\"
        },
        \"output\": [\"itemid\"]
      },
      \"id\": 1
    }" | jq -r '.result[].itemid'
)
echo "$ITEM_IDS" | xargs -n "$BATCH_SIZE" | while read -r batch; do
  JSON_ARRAY=$(printf '%s\n' $batch | jq -R . | jq -s .)
  curl -s -X POST "$API_URL" \
    -H 'Content-Type: application/json-rpc' \
    -H "Authorization: Bearer $API_TOKEN" \
    -d "{
      \"jsonrpc\": \"2.0\",
      \"method\": \"item.delete\",
      \"params\": $JSON_ARRAY,
      \"id\": 1
    }" | jq .
  sleep "$SLEEP_SECONDS"
done

That script was important because the fix was not just about stopping new damage. It was also about cleaning up the existing blast radius.

How we knew it was really solved

The best validation came after cleanup.

Once the discovery regex was fixed and the host was reset:

• the total item count dropped dramatically
• veth interfaces did not reappear
• the agent stopped logging incoming connection acceptance failures
• the proxy stopped accumulating huge SYN-SENT bursts
• the issue stopped reproducing

That was the point where this moved from “probable root cause” to “confirmed root cause.”

Lessons learned

There were a few takeaways from this incident that are worth keeping.

1. Monitoring can become the outage

The first visible symptom was a monitoring failure, but the real issue was how monitoring had been configured.

2. Cardinality matters

A host with a few thousand items is normal. A host with tens of thousands is suspicious. A host with more than 100,000 is a serious warning sign.

3. Ephemeral infrastructure needs explicit exclusions

Container environments generate noise. If you do not tell your discovery rules what to ignore, they will turn that noise into permanent load.

4. “Network errors” are not always network problems

In this case, the network symptoms were real — but they were downstream of overloaded agents and aggressive passive polling.

5. Fixing discovery is only half the job

If bad items already exist, you still need to clean them up.

Final thought

• This started as a Zabbix proxy crash.
• Then it looked like a network issue.
• Then it looked like an agent issue.
• In the end, it was something much more subtle: a discovery rule that had been allowed to observe things no one actually needed monitored.
• That is what made the incident interesting.
• And that is why I think incidents like this are worth writing down. Because the hardest failures are often not caused by the thing that first appears broken.
• Sometimes the outage is not in the system you are running.
• Sometimes the outage is in the way you chose to watch it.

As always, if you have any questions you can reach out to me on Linkedin or via email.

Thanks for your time. Have a wonderful day!