Unlock the Full Potential of WebRTC: Essential Features for High-Quality Applications

This article helps you understand the important features offered by WebRTC CPaaS (Communications Platform as a Service) vendors. If you are building large scale audio/video calling applications using any CPASS vendor like Agora, Twilio or LiveKit so on, these features can definitely take your application to the next level.

If you are a beginner to WebRTC you can read about it here. Use of these features are must to build a high quality application.

Here is the list of features:

1. Simulcast
2. Adaptive Bitrate
3. Optimization on platforms
4. Stats
5. AI features

Let’s understand the above terms one by one.

Simulcast

Creating a world-class calling experience user wants the smooth interaction of audio/video call between different peers joined over the internet. Having said that, when peers join the room, each peer has different bandwidth to receive the coming media stream.

Insufficient bandwidth clients can experience high packet loss on the receiving side, causing choppy picture quality or black/blank frames. To give a smooth experience for the end-users, especially those who have the insufficient bandwidth to receive the high quality stream, Simulcast does the magic. Most of the CPASS vendors which are built on SFU(Selective Forwarding Units) model have the Simulcast feature available.

Now lets understand How it works?

Every client publishes the different resolution media stream to the SFU and then based on the implementation, SFU decides which stream to forward to the other client. Commonly, media streams are divided into three modes HD, LD and MD. For example

• high: 1280 x 720 @ 2.5mbps
• medium: 640 x 360 @ 400kbps
• low: 320 x 180 @ 125kbps

By this technique we can provide a far better experience to the users with different resources. If a client is struggling with the slow bandwidth, can request for the lower resolution streams to have a better experience. Each WebRTC SDK provider has a different way of Simulcast implementation, But the end goal is similar. You can explore this feature and check the docs given by the RTC SDK provider you are using.

Adaptive Bitrate

Adaptive Bitrate Streaming (ABR) ensures the best video quality and viewer experience possible, regardless of bandwidth, device, or software. It is a technique that dynamically adjusts the compression level and video quality of a stream to match the available bandwidth.

With this feature, clients can allocate bandwidth between multiple streams based on their requirements. Most CPaaS providers automatically adjust the video quality to ensure optimal performance without compromising reliability.

When the network is constrained, the bitrate and resolution are reduced to maintain the call connection, ensuring enough throughput for audio. When bandwidth is available, the bitrate and resolution are increased to deliver higher-quality video.

Optimization on platforms

Video calling applications build on all platforms like Web, iOS and Android. Mobile applications are highly recommended when creating a live video solution. There are many optimizations that can be made on mobile devices and platform levels. Let’s explore a few examples.

For instance, if a mobile application user is on a screen where the client is both publishing and subscribing to video, and the user then brings the application to the background, there’s no benefit in receiving remote video at that point. As a result, the client would be wasting bandwidth. To avoid this, we can stop the remote video tracks when the app is in the background and re-enable them once the user returns to the app. This simple change can save a lot of bandwidth and other resources.

The active speaker feature is also very useful and is available in all major SDKs. When implementing this feature, you can optimize the published video by adjusting the bitrate and resolution if the current user is not the active speaker.

Another client-side optimization depends on the design and layout of the live session. Based on the video layout, clients can determine the best bitrate for the video stream. For example, if the publisher is not displayed in a large view, you can lower the resolution of the publisher’s video. On the other hand, if the publisher is shown in a larger view, you can send a high-resolution stream. This optimization helps ensure the best experience for users, adjusting to the layout and user changes in real time.

Stats

Debugging RTC (Real-Time Communication) application issues can be complex because we are dealing with real-time media. Therefore, having perfect logging and a robust analytics dashboard is essential. End users may encounter different issues, and identifying the root cause is crucial for maintaining the quality of your product.

To help debug these issues, RTC providers offer Stats APIs and analytics dashboards.

The Stats API provides comprehensive data about the connected session and each media stream within that session. These metrics are invaluable for diagnosing issues that users experience, such as video not being received, audio issues on one end, screen sharing not working, and more. The API offers real-time data about the current session, which can significantly aid in troubleshooting. You can check the stats API given by 100ms and more details about WebRTC stats read here.

Chrome webrtc-internals stats

There are other tools available to debug RTC issues. If the session is running on Chrome browser you can check complete stats on chrome webrtc-internals(chrome://webrtc-internals).

AI Features

AI is all around in today’s world, influencing nearly every aspect of technology. CPaaS (Communications Platform as a Service) vendors are no exception, integrating AI technology with WebRTC. Many SDK providers now offer capabilities that enable live conversations powered by Large Language Models (LLMs) such as ChatGPT.

AI features can be seamlessly integrated into various use cases when building applications. For example, they can be applied to areas like customer support, pre-sales, growth marketing, self-service product assistance, and more. Let’s explore a few vendors and their APIs that make these advancements possible.

WebRTC and AI

LiveKit Agents is an end-to-end framework that enables developers to build intelligent, multimodal voice assistants (AI agents) capable of engaging users through voice, video, and data channels.

Similarly, Integrating Agora’s real-time audio communication capabilities with OpenAI’s language models enables dynamic, conversational AI experiences, you can check here.

Twilio AI Assistants is an opinionated framework to build and host Assistants for customer-facing use cases. At the core of Twilio AI Assistants resides a distinctive customer-aware approach facilitated by Customer Memory.

Soon I will cover a few topics in detail.

Useful Links:

Simulcast and Adaptive Bitrate: https://www.wowza.com/blog/webrtc-simulcast-what-it-is-and-how-it-works

WebRTC Stats: https://developer.mozilla.org/en-US/docs/Web/API/RTCStatsReport