The Sidecar Bottleneck: Rethinking gRPC Connection Pooling

We’ve all been there. You’re monitoring a downstream service during a sudden traffic spike, and you start seeing latency creep up. Requests begin timing out. Naturally, you suspect the database is locking up, the network is saturated, or perhaps the upstream service is buckling under the load. But after digging in, you realize the call isn’t even leaving the pod. The bottleneck is right there in the living room: the connection between the application and its local Envoy proxy sidecar.

This is how we stumbled upon an artificial ceiling in our gRPC client configurations, the assumptions we challenged, and the tradeoffs we navigated to break through it.

The Hidden Ceiling in Our Legacy Setup

To set the scene, our architecture relies heavily on Istio and Envoy sidecars. When a service makes a call, it uses a fully qualified domain name (k8s FQDN) that resolves to a single service IP address. In this setup, there is a strict 1:1 mapping between a gRPC channel and the underlying TCP connection.

Everything was running smoothly until we started hitting unexplained limits during throughput spikes. Digging into the legacy client code, we found the culprit: a fixed connection pool operating with an HTTP/1.1 mindset. When the application spun up, it created a fixed set of TCP connections. Each outgoing request would “borrow” a connection from the pool, blocking it entirely until a response was received. If a sudden burst of requests arrived and the pool was busy, new requests had to wait in line. If they waited too long, they failed.

We were using gRPC, which is built on HTTP/2, but completely ignoring its superpower: multiplexing. By blocking whole channels per request, we were causing self-inflicted latency and throughput starvation.

Evaluating the Fix: Two Divergent Paths

We knew we had to embrace HTTP/2 multiplexing, which allows multiple concurrent requests to flow over a single connection simultaneously. But how exactly should we architect this interaction with the sidecar? We debated two main approaches.

Path 1: The HTTP/2 Purist (Single Connection)

The most theoretically sound approach in HTTP/2 is to use a single, heavily multiplexed TCP connection. Why pool connections when one pipe can handle it all? Before committing, we had to ask ourselves three crucial questions:

• Will a single connection be reliable enough? Yes. Because this TCP connection is entirely local, established on the loopback interface (127.0.0.1) between the application container and the sidecar on the same Kubernetes node. It is incredibly stable and supports massive bandwidth. The dreaded TCP head-of-line blocking isn’t a practical concern over localhost.
• Will a single connection hit any multiplexing limits? Servers enforce a limit on concurrent streams. Envoy defaults this to 1,024, but Istio configurations typically override this to a massive number (roughly 2 billion). We weren’t going to hit a queueing ceiling.
• What about Envoy proxy concurrency? Here is where the theory hit a wall. Envoy’s threading model is highly opinionated: it assigns each downstream connection to exactly one worker thread.

If our client uses only a single TCP connection, the sidecar handles all of that client’s outbound traffic on a single thread. Effectively, our proxy’s outbound processing becomes limited to the clock speed of a single CPU core.

Path 2: The Pragmatic Middle Ground (Non-Blocking Small Pool)

To solve Envoy’s single-thread bottleneck, we looked at a hybrid approach. Instead of one massive connection, we maintain a very small pool (say, 5 to 10 connections). Crucially, unlike our legacy setup, we don’t block them.

We utilise HTTP/2 multiplexing across all of them, distributing the requests using a random load-balancing strategy. By doing this, we trick the Envoy sidecar into distributing the traffic across multiple worker threads, unlocking multi-core concurrency for outbound requests.

Putting it to the Test

We spun up a test environment, pushing the Istio proxy to its absolute limits to see how the single connection (Path 1) fared against the small pool (Path 2) under heavy synthetic load (using an Envoy concurrency setting of 2). The measured latency is internal processing time in Istio proxy.

The data paints a clear picture. Under normal or moderate loads (sub-500 RPS per pod), both approaches perform beautifully. But as we artificially scaled the throughput into the thousands, Path 1’s single-thread constraint caused latency to skyrocket. Because Path 2 could utilize both of the proxy’s worker threads, its average latency was quite literally cut in half.

The Takeaway

In software engineering, everything is a tradeoff. The pure, single-connection approach is incredibly simple and solves the blocking issue for 99% of micro-services, as very few individual pods actually sustain thousands of outgoing RPS without scaling out first.

However, for high-throughput gateway nodes or extremely dense aggregators, that Envoy threading model constraint is a silent killer. In these contexts, Path 2, the pragmatic, non-blocking multiplexed pool is the preferred choice. It balances the architectural simplicity of HTTP/2 multiplexing with the raw reality of how proxy sidecars manage CPU threads under the hood.

References

1. gRPC docs: Performance Best Practices
2. gRFC: MAX_CONCURRENT_STREAMS Connection Scaling
3. Envoy docs: Life of a Request
4. Envoy blogs: Threading model
5. IETF RFC 9113: HTTP/2
6. TCP head of line blocking