The Ceph Problem: 40Gb vs. 100Gb in a World That’s Moving Too Fast

Actually, it always starts the same way: someone with a spreadsheet, a cluster diagram, and this creeping feeling of doubt. Five servers. Twelve NVMe drives each. Dual Xeon Gold CPUs humming under aluminum lids. Terabytes of RAM like candy. And then, somewhere between the PCIe lanes and the power budget, comes the question that derails the whole build: Do we really need 100-gig networking for this?

That was exactly the crossroads one systems architect hit while sketching out a five-node Ceph deployment for Proxmox. His specs were borderline crazy — 6.4TB U.3 NVMes, DDR5 memory, replication across every node. The kind of setup meant to make VMs feel instant. But as he priced out the 16-port switches, a harsh reality emerged: 40GbE gear was way cheaper. 100GbE switches? Triple the cost.

And so the debate began. A debate that’s happening quietly in sysadmin Slack groups, data center basements, and the back corners of homelabs everywhere: When does “future proof” become “financially stupid”?

The Two Speeds of Reality

On paper, 40GbE still sounds fast. It’s four lanes of 10GbE bundled together, capable of moving about 5GB per second. That’s roughly the bandwidth of two consumer NVMe drives running flat out. For traditional workloads — VMs, databases, file shares — it’s decent. Even luxurious.

But Ceph doesn’t behave like traditional storage. It’s not a single disk writing to a filesystem. It’s a distributed mess of object stores — OSDs — replicating, recovering, and balancing data across the cluster constantly. Every write to one node echoes three times across the network. Every rebuild doubles or triples that chatter.

As one engineer in the community put it bluntly:

“Just by looking at your server specs, I’d say go at least 100 Gb/s. If you don’t, you also don’t need U.3 NVMe SSDs and 6542Y CPUs.”

He wasn’t kidding around. A high-end NVMe drive can easily saturate 7 GB/s on its own. Multiply that by twelve, add Ceph’s replication overhead, and 40Gb starts to look less like a backbone and more like a garden hose trying to fill a swimming pool.

The Price of Future-Proofing

Here’s the thing: 40GbE hardware is cheaper because it’s old. Really old, by networking standards. Vendors have been dumping inventory for years as hyperscalers moved to 100, 200, and now 400-gig infrastructures.

For small to midsize operators, that depreciation is a siren song. Why spend $10,000 on shiny new Mellanox switches when you can grab a used 40Gb Arista stack for half the price?

But, as one user pointed out:

“40G is being phased out in favor of 50/100G, so if you’re looking at a new install, go 100G. If it’s a question of budget, 40G is cheap right now compared to 50/100.”

That’s the quiet part most people skip over. Cheap now often means expensive later. The parts are harder to find. The optics are older. The firmware stops getting patched. By the time your cluster needs an expansion or a replacement, the secondary market has dried up, and your entire network fabric looks like a legacy museum.

One commenter nailed the sentiment:

“If you’re already buying Xeons and 12 NVMes per node, saving a few thousand on the switches feels like painting a race car with discount house paint.”

The Real-World Math of 100GbE

Still, the 100-gig argument isn’t just about raw throughput. It’s about breathing room.

Several admins chimed in describing similar setups — five or six nodes, all-NVMe storage, heavy virtual machine workloads — and nearly every one of them ended up with 100GbE links, sometimes even bonded pairs.

“We have 4×100 Gbit per server,” one engineer wrote, “divided in 2×100Gbit LACP for frontend and backend.”

That’s crazy for most clusters. But in practice, the redundancy and flexibility it provides can be the difference between smooth failovers and hours of degraded performance.

And then there’s rebuild time. When an OSD fails, Ceph has to reconstruct the lost data from replicas scattered across the cluster. It’s an intense network operation. A 40GbE link might be fast enough for normal operation, but when you’re pushing terabytes across five nodes to restore redundancy, those minutes or hours matter.

As another sysadmin said with a shrug:

“It doesn’t matter if it’s fully utilized, it’ll save you some hours of sleep when you’ll have to rebuild a failed OSD.”

It’s not just about today’s workloads; it’s about the bad nights when everything breaks.

The VLAN vs. Dual-Network Debate

For years, Ceph best practices suggested separate physical networks for “frontend” (client access) and “backend” (OSD replication). It made sense when bandwidth was limited and failures were localized.

But the times — and the code — have changed. The official docs now warn against physically separating the two, pointing to reliability issues that can cause “flapping OSDs.” Instead, the recommendation is to merge them logically through VLANs or QoS tagging.

One comment summed it up:

“Separating it into a VLAN is OK, but a separate backend network is no longer recommended. My recommendation would be a single dual-port 100GbE card, a pair of switches that support LACP, MLAG, VLAN tags, and per-VLAN QoS.”

The advice reflects a broader trend: simplicity beats segmentation. Modern networks are fast enough to handle both types of traffic without tripping over themselves — as long as they’re properly managed.

And yes, “properly managed” usually means “don’t cheap out on the switches.”

The Budget Builders’ Workaround

Not everyone can just throw money at the problem. One admin shared his balancing act: dual-port ConnectX-4 NICs bought from eBay for $150 each, paired with two 100Gb Mikrotik CRS520 switches at $1,995 apiece. It wasn’t bleeding edge, but it worked.

“I wanted to go with ConnectX-5 or Intel E810,” he admitted, “but they are about twice the price or more. I spent over $125K on servers, so it didn’t leave me much left for network hardware.”

That’s the unsung story of most deployments like this. The hardware is expensive. The drives, the CPUs, the memory — all premium. So when it comes time to order switches, corners start to look tempting.

The community consensus, though, was consistent: if you can stretch for 100Gb, do it. Even used. Even with compromises. A handful of builders recommended buying decommissioned Edgecore or FS.com gear and flashing them with open firmware like SONiC or Cumulus Linux.

“Purchase used off eBay the Edgecore 7712 and run SONiC on it,” one poster said. “$1,000 per switch and rock solid.”

That’s not bad advice. In many ways, 100GbE hardware has matured enough to hit the sweet spot of price and performance — if you know where to look.

40GbE: Still Good Enough?

Despite the enthusiasm for 100GbE, a small but vocal group defended the humble 40-gig setups. Their argument? Most real-world workloads never come close to maxing it out.

One admin wrote:

“Are you expecting to use more than 40GbE/8bits=5GB/s of bandwidth? 5GB/s is still incredibly fast, all things considered.”

And he’s right — if your cluster is primarily running moderate virtual machine loads or mixed RDS traffic, 40Gb might hold up fine. Another user with smaller hosts echoed that sentiment:

“We use redundant 40Gbps and it really seems to be sufficient.”

The key word there is redundant. Most of these 40Gb clusters rely on multiple bonded links — 2× or even 4×40GbE — to spread traffic and add fault tolerance. That setup can outperform a single 100Gb link in certain configurations.

“For scale-out beyond five nodes,” a veteran admin noted, “spreading the TCP out via LACP on multiple 40G links will yield better results than one or two 100G links in the same config.”

In short, 40Gb isn’t dead — it’s just situational. For smaller clusters, or those with lighter workloads, it remains a perfectly valid middle ground.

When Ceph Outruns Everything Else

One of the more sobering comments came from a user who pointed out an uncomfortable truth: Ceph itself can’t always keep up with the hardware beneath it.

“You’re way overprovisioning your nodes for what Ceph can do,” they said. “It can’t take advantage of the full NVMe speeds or that amount of memory.”

That’s an under-discussed reality. Ceph is powerful but complex. Its distributed nature introduces latency and overhead that raw hardware speed can’t fully mask. A single node might benchmark at 60GB/s in isolation, but in cluster operation, those numbers drop dramatically.

The takeaway? Even if you go 100GbE, you might not see the benefit unless your workloads are truly pushing the limits — or unless you’re planning for the long haul.

And that’s where the “future proof” argument circles back. You’re not buying 100GbE for what you need today. You’re buying it for what you’ll need after your first rebuild, your second node failure, or your next big client migration.

So… Which Should You Choose?

If there’s a consensus buried in all the anecdotes, it’s this:

• 40GbE still works for smaller clusters, especially when bonded or used with lower-intensity workloads.
• 100GbE is the safe bet for anything NVMe-heavy, especially with 3× replication and active failover requirements.
• Ceph’s network architecture benefits more from simplicity and redundancy than from chasing theoretical throughput.

The difference isn’t just performance — it’s peace of mind.

As one comment summed it up perfectly:

“It doesn’t matter if you fully use 100GbE. What matters is that you don’t regret not having it when you need it most.”

The Final Word: In a World That’s Moving Too Fast

The real problem isn’t 40 versus 100. It’s present versus future. Storage performance has outpaced networking for years, and the gap keeps widening. CPUs add cores. Drives get faster. RAM gets cheaper. But networks? They’re the last bottleneck standing.

The irony is that 100GbE used to sound like crazy talk. Now it’s just the new normal. The infrastructure world has moved on to 400Gb and even terabit links in hyperscale environments. For smaller operators, that means 100Gb isn’t premium anymore — it’s accessible, reliable, and increasingly necessary.

If Ceph has a weakness, it’s that it punishes hesitation. Its brilliance lies in parallelism — data flowing freely, simultaneously, everywhere. And if your network can’t keep up, it all grinds down to a crawl of replication timeouts and angry dashboards.

So maybe the answer isn’t whether you need 100Gb. Maybe the real question is whether you can afford to keep thinking like 40.