Low Latency Streaming Solutions

Introduction

In 2019, Zenith Media published a seminal report entitled “Online Video Forecasts,” which presented insightful data on the rapidly growing trend of online video streaming. The report found that, on average, individuals spend 100 minutes each day consuming video content online, with certain countries, including China, Sweden, Canada, India, Mexico, the United Kingdom, and the United States, exhibiting even higher usage rates, surpassing the average with daily consumption exceeding 100 minutes by 2021.

The report emphasized the paramount importance of the video experience in determining an individual’s level of engagement and interest in online video content. One key factor that significantly impacts the quality of the video experience is the reduction of latency in the video stream. Thus, minimizing latency is considered a crucial aspect in ensuring a seamless and satisfying video experience for users.

Our data shows that just one buffering event decreases the amount of video watched by 39%. — Mux.com

Allow me to define the term “latency,” which is central to the topic under consideration. Latency is the delay between when an image is captured on your camera and when your viewer experiences it on their screen.

The comprehension of the principles behind low latency streaming holds significant value in terms of improving audience engagement. This is achieved through the utilization of Content Delivery Network (CDN) solutions, which have the capability to significantly reduce latency to mere milliseconds, thereby enhancing the overall viewing experience.

The Live Streaming Process

Streaming, a method of delivering media content in real-time over the internet, involves the sequential transmission of digital video data from a remote storage location to the client device. As opposed to downloading the entire video file beforehand, the streaming process enables the client device to receive and playback the video incrementally, resulting in an efficient utilization of bandwidth and improved user experience.

Typical Video Streaming Pipeline

Video Codecs

A Video Codec is a critical component of digital video technology, serving to both compress and decompress video data. In the late 1980s, various organizations embarked on exploratory efforts to leverage Discrete Cosine Transform-based lossy compression techniques for video encoding, which ultimately gave rise to the H.261 standard. Over the years, subsequent advancements in the field have resulted in the development of new standards, and the current industry standard is the Advanced Video Coding (AVC) standard, also known as H.264 and the High Efficiency Video Coding (HEVC) standard also called as H.265. Another important standard is the AV1 standard which is an open source, royalty free video coding format. With ongoing research and innovation, the Very Versatile Coding (VVC) standard, H.266, has been introduced and purports to offer a theoretical 50% improvement in efficiency compared to its predecessor. This improvement in bitrate efficiency comes at a cost of encoding decoding complexity.

Relative Bitrate Efficiency vs Codec (Lower the Better)

This data was sourced from Jan Ozer. The AV1 efficiency was approximated using the blog from a WinXDVD blog.

Encoding Time vs. Codec (Lower the Better)

This information was procured from a blog hosted by WinXDVD. The VVC encoding time was approximated from another blog by the same company.

Hence, it is evident that the selection of an appropriate video encoder is contingent upon the hardware specifications and whether the video is static or a live stream. In my opinion, optimal efficiency for live streaming video can be attained by utilizing either the AV1 codec or the Advanced Video Coding (AVC) codec, H.264, dependent upon the hardware used for encoding.

Media Streaming Protocols

Media streaming protocols represent a set of standardized procedures and guidelines that facilitate the fragmentation of video files into smaller, manageable segments for efficient delivery to end users. The files must undergo compression, achieved through the use of a codec. Furthermore, the files must be stored in a suitable container format, such as .mp4 or .avi, prior to transmission.

(sourced from a 2021 Video Streaming Latency Report)

Here are a few important protocols:

Real Time Messaging Protocol

The Real-Time Messaging Protocol (RTMP) was originally developed by Adobe. RTMP operates as a TCP-based protocol (with the exception of RTMFP, a variation of RTMP that utilizes UDP) and facilitates persistent connections, enabling low-latency communication and minimizing buffering. It has evolved to serve a modified purpose in modern live streaming infrastructure. A drawback of RTMP is that as it does not use a regular internet protocol (like HTTP), and if you have users working behind a firewall, they probably can’t get past it’s restrictions and they will be blocked and unable to receive data. In contemporary live streaming environments, RTMP primarily serves as a means of delivering content from an encoder to an Online Video Platform, a process commonly referred to as ingestion.

The utilization of RTMP for ingestion in live streaming setups offers a multitude of advantages. Firstly, it facilitates seamless integration with encoders due to its affordability and ease of deployment. Furthermore, the combination of RTMP and HLS results in a highly optimized low latency experience, maximizing the overall quality of the streaming experience.

The RTMP ingest process sits between the Compression and Encoding block and the Online Video Platform block in the typical video streaming pipeline. The implementation of RTMP ingestion necessitates the utilization of an RTMP encoder, which offers cost-effective and readily available options compared to other alternatives based on different streaming protocols.

Secure Reliable Transport

The Secure Reliable Transport (SRT) protocol is a solution designed by Haivision to address the limitations of the RTMP in delivering low-latency streams over challenging network conditions. SRT leverages the speed of the UDP and incorporates error correction mechanisms from the TCP to provide a reliable, low-latency streaming experience. Despite RTMP being widely used in the streaming industry, it is not optimized for interactivity, and SRT outperforms it by offering a significantly lower end-to-end latency of approximately 1 second compared to RTMP’s 5 seconds. Furthermore, SRT is codec-agnostic, which means that it can support a wide range of video codecs without any restrictions. For these reasons, SRT has been replacing the RTMP ingestion protocol in many recent use-cases.

Popular Ingestion Protocols (according to a 2021 Video Streaming Latency Report)

HTTP Live Streaming

HTTP Live Streaming (HLS) is an adaptive bitrate media streaming protocol that is designed to deliver visual and audio media to viewers over the internet. This protocol is developed by Apple and is characterized by its adaptive bitrate streaming nature, which determines the most optimal bitrate for a given client by first determining the bandwidth of the client. HLS was introduced as a successor to the Quicktime Streaming Server (QSS), which was utilized by early smartphones and was instrumental in the early days of digital media delivery. However, QSS relied on non-standard ports for data transfer, which resulted in firewall restrictions and limitations similar to RTMP.

It operates by dividing a video into segments, with each segment being a short clip encoded using the H.264 or H.265 codecs. These segments are stored on an HTTP server and are accompanied by a manifest file with a *.m3u8 extension, which serves as an index for the video chunks.

It boasts a high degree of compatibility with a vast array of devices and firewalls, making it a widely accessible and practical solution for delivering multimedia content. Additionally, the utilization of custom HLS encoders allows for a reduction in latency.

The limitations of the protocol can be attributed to its relatively elevated latency, which can result in delays of up to 30 seconds. Additionally, there is a requirement for a minimum of three video segments to be maintained in the queue prior to permitting video playback, which can be considered as another drawback of HLS.

For this reason, we use the RTMP ingest (mentioned above) with the HSL streaming protocol as it yield the lowest latency possible.

Dynamic Adaptive Streaming over HTTP

Also referred to as MPEG-DASH, DASH is a vendor-independent, adaptive bitrate streaming protocol that provides a viable alternative to the HLS protocol. Similar to HLS, DASH operates by segmenting the video content into small HTTP-based segments and serving them through a dynamic XML-based manifest file that acts as an index to these segments. One of the major advantages of DASH is its codec-agnostic nature, enabling the delivery of video content encoded in any codec format. However, it is worth noting that DASH does not have as widespread support as HLS and may not be supported by all devices.

Web Real Time Communication

WebRTC is a cutting-edge technology designed with low latency as a primary consideration. Developed by Google, it is not a protocol in the traditional sense, but rather an open-source framework that encompasses multiple standard protocols, streaming components, and a web API. Its unique advantage lies in its reliance on the UDP, which allows for a direct, real-time connection between source and endpoint without the need for a prior handshake. This direct connection also eliminates the need for content splicing or splitting, and the technology is equipped with various mechanisms to mitigate the impact of packet loss, garbled order and delayed arrival.

However, there are some trade-offs to consider, such as limitations in terms of scalability and quality. Additionally, the use of WebRTC is restricted to a limited range of video and audio codecs, with VP8/VP9 and H.264 with B frames being the only supported formats video codecs. The standard implementation of WebRTC also incorporates encryption, which increases the handshake times and the transmission bandwidth.

Popular Streaming Protocols (according to a 2021 Video Streaming Latency Report)

Traditional Low Latency Streaming Solution

Latency has become a prominent issue in the realm of live streaming, hindering the delivery of an optimal viewing experience to audiences. The demands of modern streaming audiences have evolved beyond what traditional broadcast methods can provide, necessitating the implementation of ultra-low latency solutions to effectively address this challenge.

Traditional Low Latency Streaming Solutions

The commonly adopted industry standard for live streaming is the HLS protocol, which involves an initial RTMP ingestion, followed by conversion to HLS by the video platform for distribution. While this approach offers a high-quality stream to viewers, it can introduce significant latencies, often exceeding 15 seconds. The use of a standard HLS ingest can further exacerbate this latency.

Low Latency Streaming Solution Proposed by CDNetworks

CDNetworks has innovatively devised a customized version of the WebRTC technology, incorporating enhancements to the conventional CDN network architecture. This custom implementation overcomes the limitations of traditional WebRTC and achieves a remarkable latency of less than 500 milliseconds.

CDNetwork’s Low Latency Streaming Solution

To enhance the delivery of superior streams, CDNetworks has made several optimizations, including:

• Support for the advanced H.265 and AAC video and audio transcoding formats.
• Incorporation of B frame support for a seamless playback experience.
• The elimination of DTLS encryption from the standard WebRTC and the implementation of proprietary security strategies that have a lower transmission latency.

In addition to these optimizations, CDNetworks also offers traditional livestream features such as watermarking, transcoding, and recording, making it a highly viable platform for live streaming.