What’s new in STUNner v1.2.0?

STUNner v1.2.0 logo

We are proud to present STUNner v1.2.0, the next release of the STUNner Kubernetes media gateway for WebRTC brought to you by l7mp.io. This release includes many fixes and features to improve the STUNner experience for our production users: security hardening, observability, and improved robustness and stability at scale.

Major changes are upgrading STUNner to the latest Kubernetes, Gateway API, OpenTelemetry, and Pion dependencies and integrating upstream security fixes, hardening STUNner’s Config Discovery Service, optimizing the operator to handle large deployments (+100 Gateways and UDPRoutes), and adding an opt-in pprof endpoint to the operator for live debugging and profiling. This release also includes broad dependency refreshes across STUNner repositories, documentation cleanups and clarifications, test and benchmark improvements, and many small stability fixes throughout the dataplane and the control plane.

Security hardening

The most visible change in this release is the merging of the upstream TURN NONCE size optimization that we implemented in pion/turn to mitigate ongoing TURN amplification attacks. In short, the problem is that malicious users can send unauthenticated TURN requests to STUNner so that the resultant TURN error response is larger than the triggering request, which makes the TURN server a tempting reflector/amplifier in spoofed UDP attacks. Our mitigation reduces the attack surface by making NONCE generation more compact, while preserving the security properties the TURN handshake needs.

If you want the full background, out earlier post is still the best deep dive: Mitigating TURN Amplification Attacks.

Operator profiling

The Gateway Operator now exposes an opt-in pprof endpoint for live debugging and live profiling. This is disabled by default; enable by setting the pprof HTTP endpoint with a operator CLI flag:

./bin/manager - pprof-bind-address=127.0.0.1:6060

Equivalently, use an environment variable:

export STUNNER_GATEWAY_OPERATOR_PPROF_BIND_ADDRESS=127.0.0.1:6060
./bin/manager

Then, you can use kubectl port-forward to connect to the operator’s pprof endpoint:

export STUNNER_OPERATOR_NAMESPACE=$(kubectl get pods -A -l control-plane=stunner-gateway-operator-controller-manager \
    -o jsonpath='{.items[0].metadata.namespace}')
export STUNNER_OPERATOR_NAME=$(kubectl get pods -A -l control-plane=stunner-gateway-operator-controller-manager \
    -o jsonpath='{.items[0].metadata.name}')
kubectl -n $STUNNER_OPERATOR_NAMESPACE port-forward ${STUNNER_OPERATOR_NAME} 6060:6060

Inspect the profile endpoints locally:

go tool pprof -http=:8081 "http://127.0.0.1:6060/debug/pprof/profile?seconds=30"

Do not expose this endpoint publicly. Profiles can contain sensitive runtime details; pprof is a debugging interface, not a service API.

Faster convergence at scale

The most important control-plane change in this release is the operator updater optimization. In previous versions, the updater could end up rewriting objects in the Kubernetes API server even when nothing semantically changed. This churn causes excess Kubernetes API traffic and slows convergence under load, to the point that eventually the Kubernetes API server starts to rate-limit the operator, leading to the control-plane becoming unresponsive.

In v1.2.0 we reworked the updater around resource-specific “lenses” that compare and apply only the parts of an object STUNner actually cares about. This has a practical effect: the operator stops uploading objects to the API server that are already in the desired state. This alone made the operator about 100x faster at scale.

Use the end-to-end operator benchmark to understand the effects:

cd stunner-gateway-operator
make bench

On an average CPU the operator should need about 1 sec to render the dataplane configuration for 500 Gateways, which is about tolerable for even for very large deployments. Let us know if you need still faster convergence, we have plenty of ideas to further optimize the configuration renderer.

Workaround for broken UDP graceful shutdown behavior

STUNner supports graceful shutdown: when a pod receives SIGTERM, it stops advertising readiness but keeps serving existing allocations until after active connections have drained. Unfortunately, some Kubernetes environments have a curious bug in the UDP data path: once the pod turns unready, Kubernetes may stop forwarding UDP packets to the pod too early, even though the process is still alive and intentionally draining existing sessions. For TURN/TCP this is usually survivable. For TURN/UDP it can be disastrous: sessions that should have drained cleanly are blackholed mid-flight.

STUNner v1.2.0 adds a targeted escape path for exactly this situation, which can be enabled with the — force-ready-status CLI flag. When this flag is enabled, STUNner keeps reporting ready status during the graceful termination window instead of deliberately failing readiness. Do not enable this unless you see premature TURN connection terminations during graceful shutdown, because the normal Kubernetes behavior is still the right one. But if your environment drops UDP too early during rollouts or scale-down, this flag gives you a pragmatic workaround.

Add a JSON logger backend

On popular request, STUNner now supports JSON logging in the dataplane. We have always supported JSON logging in the operator, but the dataplane support was missing. Thanks to the recent upstream efforts of the Pion team, we now fully support JSON log format, a critical observability feature for our production users.

Start stunnerd with JSON logging at the second highest log-level (all:DEBUG) and expose a UDP TURN server at 127.0.0.1:5000:

./stunnerd --log=all:DEBUG --log-format=json turn://user1:passwd1@127.0.0.1:5000

For reference, here is the operator command line for enabling JSON logging at the debug level:

bin/manager --zap-log-level=debug --zap-encoder=json

Enjoy STUNner, join us on Discord, and let us know what you like (or dislike?) about STUNner. See here for the commercial version, sporting a security hardened control plane and the world’s fastest Linux/eBPF based UDP TURN server.