A Production Journey Through Istio Gateway Performance: WasmPlugin vs EnvoyFilter

TL;DR: The Goal: The primary objective of our migration was simply to move our ingress layer from NGINX to Istio Gateway. The Problem: Our core challenge emerged from our strict observability requirements: We needed to extract custom metrics across all paths to fulfil our monitoring systems. The Solution: We switched to EnvoyFilter (with Lua), which was much, much more performant.

The Migration and the Cardinality Problem

After the retirement of Nginx, the second recommended choice for routing was Istio, and after some architecture decisions, we chose to go with Istio Gateway with routing HTTPRoute resources.

After the migration, we reached a state where we also needed to migrate our Grafana dashboards to the new setup. The catch is metric cardinality: if the request path is used as a metric label, then /users/123, /users/124, /users/125, and so on each become a separate time series. With unbounded IDs in the path, that count explodes and overwhelms storage and query cost. The fix is to normalize the path before it becomes a label (/users/123 → /users/{id}) so all those requests collapse into a single series. We needed a tool to do that normalization, and we began our journey with WasmPlugin. It was the most well-known option and heavily recommended in the community for exactly this case.

What is the Problem with WasmPlugin?

When we tested the setup in our development environment, everything seemed seamless. The problem began with high load of traffic a.k.a after applying the setup in production.

At first, we distributed the traffic between Nginx and our new Istio gateways: 5% for Istio and 95% for Nginx. This is where the issue started. We found that Istio was consuming a huge amount of CPU compared to the percentage of traffic it received. We did an analysis and found that WasmPlugin was burning the CPU and taking a lot of resources for normalizing the paths, which was done by sequential checks on matched regexes.

Taking it by numbers: for just 5% of our traffic in Istio, it took a total of 10 pods, with each pod burning between 2 to 3 cores. That was a lot compared to how Nginx used to consume.

Second Chance for WasmPlugin: Load Testing and Grouping

The first thing that came to mind was optimizing the plugin. Instead of running every check sequentially, we could group similar paths, check only the group header first, and terminate once matched. So instead of scanning all paths one by one — O(N) — we’d check at most M group headers and then only the paths inside the matched group, roughly O(M + N/M). That’s much better than linear, where N is the number of paths and M is the number of groups. we built a k6 load-testing harness that ramps virtual users (VUs) through stages — a warm-up, a sustained heavy-load phase, a short spike at the top and capture performance flamegraphs.

One thing worth being precise about is what those VU numbers mean. The stages drive concurrent virtual users, not a fixed request rate. We sized the spike at ~16.6k VUs as our “50k RPS” target, but VUs measure concurrency — the actual request rate is bounded by how fast the backend responds, not by what we ask for. So the test isn’t really pinning throughput at 50k RPS; it’s telling us how the filter behaves under heavy concurrency and the CPU pressure that comes with it, which is exactly the dimension WasmPlugin was struggling with.

Here is the full picture of how throughput improved across the iterations:

The pod-level results showed a real improvement. Against the sequential baseline, the spike pinned all 10 gateway pods at roughly 1.5–2 cores each — around 17 cores total just to run the Wasm filter. With grouping, the same sustained load was carried by fewer pods at noticeably lower CPU per pod. But the ceiling didn’t move: at the absolute peak of the spike, even the optimized plugin still saturated all 10 pods. The optimization genuinely helped, yet WasmPlugin remained too heavy for our infrastructure during spikes — so we went looking for another solution.

💡 Note on WasmPlugin: It is important to emphasize that WasmPlugin is not a “bad” solution. In fact, it provides a much cleaner, safer, and more developer-friendly way to extend Envoy compared to native filters. However, for our specific high-traffic production use case, the compute overhead was simply too expensive.

Asking the Community

My manager said, “If you see yourself walking alone and facing issues that no one else is facing, then it’s more probably you are on the wrong road.” So we posted questions in the Istio Slack community, defined our problem, and they suggested we could use either EnvoyFilter or Wasmtime, which are more performant than WasmPlugin.

Starting from the Beginning: EnvoyFilter

The transition from Wasm to EnvoyFilter surprised me so much. The performance was much, much, much better than expected, and it solved a lot of our issues.

We ran the exact same 50k RPS k6 load test against the new EnvoyFilter setup. It was almost hard to believe:

• During the absolute peak spike — the same one that forced WasmPlugin to burn ~17 cores across 10 pods — the EnvoyFilter setup barely reached 6 cores.
• CPU usage hovered between 300m to 600m (0.3 to 0.6 cores) per pod.

Current Design & New Problems

While the performance was great, the setup introduced new headaches. Here is how our design works:

• Istio uses gateways.
• For each gateway, there is 1 EnvoyFilter.
• For each gateway, there is more than 1 service.
• Each service has its own EnvoyFilter.

After compiling and applying this in the cluster, all the EnvoyFilters for each service are combined together to form one huge file of EnvoyFilter that processes the services one by one. If they all have the same priority (the default is 0), they simply get organized alphabetically.

After testing the EnvoyFilter in dev, we applied it in production and hit three distinct problems:

1. Lua Language Limitations

First, the Lua language does not support the heavy regex that we used in production. We had to customize, filter, remove, and reshape the EnvoyFilter Lua script to transform our regex into something that Lua understands and works with.

2. The Alphabetical Regex Collision

Second, because we use Istio as a shared gateway, the regexes for all our services got combined into that one huge Lua file. We observed that some services (let’s call one Service A) were alphabetically ordered before others (Service C). Service A had regex rules that were more generic. By default, the requests meant for Service C got matched with the general, least specific rules of Service A, simply because A came first in the alphabet.

The solution for this was either:

Option A: Fix the general rules of the least specific paths in Service A (which requires more work from different teams). Option B: Raise the priority of the services that have the generic rules. Any number above 0 has less priority according to the documentation, so we could force specific services to be processed first.

⚠️ Filter Ordering ⚠️ When working with EnvoyFilter, execution order is critical and can be notoriously tricky. Because Envoy executes filters in a specific chain, you must carefully ensure the services’ execution order to avoid collisions.

3. The Gateway Design Itself

Third, the problem we faced wasn’t actually related to Wasm or EnvoyFilter; it was because of the gateway design we implemented in the first place. Because the gateway is like one huge building block that serves all our services, if it fails, then all our services fail.

What This Teaches

Because of that final realization, we agreed on shifting to an Istio Service Mesh instead of relying purely on a Gateway.

Here are the action items and lessons we are taking away to solve these issues in the long term:

• Switch to service mesh instead of gateway: Moving away from a monolithic gateway removes the single point of failure and stops all our filters from compiling into one massive file.
• Don’t walk alone: Search and ask what others are doing better.
• Talk about the problem: In my case, this was the first step to solving it. Asking for help is the most valuable thing I learned here.

References

• EnvoyFilter official documentation
• Running distributed tests by K6
• Getting perf to work
• My real-life Istio Performance Tuning