Part one of a weekend spent finding Besu’s breaking point at 50 validators, 2,500 messages per round.

What Happens When You Push Hyperledger Besu 3× Past Its Tested Limit? I Spent a Weekend Finding Out

It started, like most of my best rabbit holes do, on a Saturday morning with a cup of chai and a question I couldn’t let go of: How far can you actually push a QBFT validator set?

I’ve spent five years in blockchain, working my way down the stack: from writing Solidity contracts and obsessing over gas optimization, to eventually designing blockchains themselves. Along the way I built a public chain on Avalanche’s Subnet-EVM: custom native token, gasless transactions, everything from the genesis file to the block explorer.

Public blockchains teach you a specific way of thinking: open validator sets, token economics, assuming every participant could be an adversary. So when I recently dove into Hyperledger Besu and permissioned blockchains, it felt like looking at the same world from the opposite side of the mirror. Known validators. No token incentives. Immediate finality through Byzantine Fault Tolerance. No forks, no waiting for “12 confirmations.”

So I did what every curious engineer does when they discover something new and have a free weekend.

I decided to break it.

What the Published Benchmarks Say (and Don’t Say)

When you go looking for concrete data on Besu QBFT scaling, you quickly discover there isn’t much:

• ~14 validators: the most cited independent academic benchmark (built on Hyperledger Caliper) found this is where a QBFT network scales comfortably without noticeable degradation. This number gets repeated everywhere.
• ~30 validators: the Besu maintainers’ own internal testing suggests networks this size still perform reasonably under light workloads. Beyond it: growing transaction pools, slowing block production, rising risk of degradation.
• 4 validators: the number Besu’s official QBFT performance tuning documentation is actually based on.

Beyond 30, there’s almost no published data. That gap was exactly what caught my attention.

I wanted to run 50 validators. simply because I wanted to understand what actually breaks, when, and why.

Why 50 Validators Is Hard

The difficulty isn’t hardware. It isn’t configuration. It’s mathematics.

QBFT, like most classical BFT protocols, requires every validator to communicate with every other validator during each consensus round, an exchange of PRE-PREPARE, PREPARE, and COMMIT messages across the entire set. The communication complexity is O(n²).

Validators    Messages per Round    Relative Load
4             ~16                   1×
14            ~196                  12×
30            ~900                  56×
50            ~2,500                156×

This isn’t a bottleneck you can tweak away in a config file. Every validator you add increases communication load on every existing validator. This quadratic growth is the scaling wall the existing benchmarks quietly point toward.

Understanding the Quorum Math

Before touching infrastructure, I validated the consensus math first, because infrastructure problems are expensive, but consensus math mistakes are fatal.

Besu calculates QBFT quorum using ceiling division:

Quorum = ceil(2n / 3)
       = ceil(2 × 50 / 3) = 34

34 validators must sign every block before it can be finalized.

Fault tolerance follows the standard Byzantine formula:

f = floor((n − 1) / 3) = 16

The network tolerates 16 faulty, malicious, or offline validators. Lose one more, leaving only 33 online, and the network immediately loses quorum. No blocks finalize. Round changes loop indefinitely until validators return.

Here’s a subtle detail: 50 isn’t actually an optimal BFT number. Classical BFT systems are most efficient at n = 3f + 1. A 49-validator network tolerates the same 16 Byzantine faults but only needs 33 signatures instead of 34. The 50th validator adds cost with zero additional fault tolerance. I chose 50 deliberately, to test the protocol beyond its mathematically clean boundary.

The Four Failure Modes I Expected

Before running anything, I dug through Besu’s source code, GitHub issues, and community discussions. That research produced four hypotheses.

1. BFT Event Queue Overflow

Besu processes consensus messages through a single-threaded processor backed by a bounded queue (~1,000 events by default). If PREPARE, COMMIT, and ROUND_CHANGE messages arrive faster than they can be processed, new events get dropped. A few drops are survivable while a healthy supermajority exists, but at scale, they trigger stalled rounds and cascading round changes.

During testing, I saw exactly this:

Queue size exceeded trying to add new bft event

These warnings appeared precisely as validator count increased. Watching the theoretical bottleneck surface in live logs was a satisfying validation of the hypothesis.

2. Vert.x Event Loop Blocking

Besu uses discv4 for peer discovery, which periodically does refresh work on the Vert.x event loop. On large networks, I observed:

Thread vert.x-eventloop-thread has been blocked for 3000–5000 ms

Tracing these warnings led straight to the discovery refresh path. Here’s the thing: in a permissioned network where every validator is already known, discovery adds almost no value. Disabling it and relying entirely on static-nodes.json removes this entire category of overhead.

3. Round Change Cascades

QBFT rotates proposers round-robin. If a proposer is slow or unavailable, validators wait requesttimeoutseconds, then move on. Configure that timeout too aggressively on a large network, and validators start triggering round changes simultaneously, and every round change fires another burst of O(n²) messages into already overloaded queues.

The result is a self-reinforcing feedback loop.

4. Validator Set Transitions

Adding or removing a validator changes the quorum requirement immediately at the transition block. Add a validator that hasn’t fully synced, and effective voting power can temporarily drop below quorum, stalling the chain even though every configuration looks correct.

What I Actually Tested

Rather than chasing TPS numbers, I wanted to understand how the network behaved operationally.

Fault tolerance. I progressively shut down validators while monitoring block production. The network kept producing blocks with 16 validators offline. When I stopped the 17th, block production halted exactly as the Byzantine formulas predicted, and it resumed automatically once enough validators returned, no manual recovery needed. Watching consensus math play out perfectly on live infrastructure was the most satisfying moment of the weekend.

Transaction flow. Continuous transaction submission while watching block production stability, pool growth, validator participation, and finalization. The goal was consensus health under load, not a TPS leaderboard.

Consensus health. Round change frequency, block time variance, missed proposals, CPU, memory, and network traffic, all visualized through Prometheus and Grafana as validator counts climbed.

One Important Caveat

I’m still validating the formal performance benchmarks: standardized throughput, latency, and sustained transaction rates. I’m intentionally not publishing TPS numbers yet. The blockchain ecosystem already has enough loosely sourced benchmark data. I’d rather publish verified results than add another optimistic number without validation.

Why This Matters

If you’re building a consortium or enterprise blockchain, governance may naturally push you toward large validator sets. Maybe every participating organization wants its own validator, or regulation demands distributed control. Either way, you’ll hit the gap between what published benchmarks have tested (~30 validators) and what your architecture actually requires.

A simple “yes, Besu can run 50 validators” doesn’t help engineers make architectural decisions. The questions that actually matter are:

• Which components fail first?
• What warning signs appear before consensus degrades?
• Which fixes are configuration, and which are architectural?

Those are the questions this experiment set out to answer.

What’s Next

This is part one of a series. Next up: how I automated the creation of the 50-validator network, the deployment tooling, the Prometheus/Grafana dashboards, the full benchmarking methodology, and the validated performance results.

If it saves someone else from spending an entire weekend chasing consensus edge cases, it’ll have been worth writing.