Understanding Codec Challenges: CPU, GPU, and VPUs in the Era of Perceived Video Quality

Video technology is evolving at an unprecedented pace, bringing with it a host of new challenges for encoding workflows. Accurately measuring and retaining perceived quality is a constantly evolving battlefront. Consistently retaining perceptual quality gains and measuring it remains to be a multi-disciplinary approach beyond causal benchmarking; it’s a cornerstone for fine-tuning workflows and unlocking the full potential of advanced technologies like high resolution & HDR content distribution in live streaming & VOD scenarios while retaining consumer engagement.

C21 Live Encoder has existed in three eras: CPU in the beginning, GPU several years later and VPUs since 2024, with each phase offering significant lessons and opportunities for optimization.

The Codec Landscape: Complexity Meets Opportunity

Video encoding is at the heart of media delivery, and its significance has only grown with the proliferation of streaming platforms and their evolving demands for high-quality delivery. Today, encoding workflows rely on three primary processing architectures:

1. CPU (Central Processing Unit): Traditionally the workhorse for encoding, CPU-based encoder implementations offer excellent flexibility and software compatibility. However, their performance is often limited by processor constraints and the inevitable trade-offs between encoding quality and speed.
2. GPU (Graphics Processing Unit): With their massively parallel processing capabilities and dedicated on-die Silicon to offload video encoding tasks, GPUs have become a favorite for high-performance video encoding & post-processing, particularly for live streaming and low latency applications where acceptable compromises on rate control and tunability are expected. GPUs, however, bring challenges in power efficiency, cost and runtime considerations.
3. VPU (Video Processing Unit): Emerging as a specialized alternative, VPUs are purpose-built appliances for video encoding tasks, with optional dedicated co-processing units for AI inference, scaling, etc. However, they differ from GPUs by being explicitly dedicated for pre-programmed functions and cannot be re-purposed for GPU-based computing. Modern FPGA-based designs do allow for in-field upgrades to functions with firmware updates and extended recovery options without depending on complex host-based device drivers, making them ideal for environments where mimical downtimes are demanded. While they offer hardware-optimized encoding and decoding capabilities and at extremely low power draw, their limited flexibility can be a drawback in some scenarios.

The Challenge of Perceived Quality

Unlike bitrate or latency, perceived quality introduces a subjective element into the evaluation of codecs. Standardized metrics like PSNR (Peak Signal-to-Noise Ratio) or SSIM (Structural Similarity Index) are useful but can fail to fully align with human perception, especially when comparing results across different architectures like CPU, GPU, and VPUs, even across identical codec implementations.

Key challenges include:

• Encoding Efficiency: Balancing compression efficiency and visual quality varies between architectures. For instance, GPUs excel in throughput with compromises on rate control options that can produce artifacting on lower bitrates. VPUs, on the other hand, have shown great promise in this front due to their extremely specialized focus on rate control tunability in constrained bitrate flows.
• Scalability: While CPUs manage varied workloads effectively, VPUs and GPUs can struggle with non-standard formats or niche requirements. A common challenge with both GPU and VPU-accelerated flows is the omission of interlaced encoding which legacy broadcast flows still mandate.
• Power Consumption: High-quality encoding often comes at the cost of power, particularly evident in GPU and CPU -heavy workflows.

As we tune for visual quality, and that’s why we stick to Netflix’s VMAF.

Different Architectures, Different Results

Testing the same content across CPU, GPU, and VPU encoders often reveals surprising differences.

Equivalent settings/options were also applied to the reference comparison encoders used in this exercise.

Encoding notes

This section is necessary to understand the VMAF scoring we used, and it’s applicability for our playouts.

• For all codecs, identical rate control (CBR), GOP presets (for low latency) and key frame intervals (2 seconds for 60 fps content) are set.
• All codec implementations here have coding unit-level rate control enabled in these test sessions.
• Coding init level rate control strength and tolerance(s) can be further refined in the encoding session.
• The ASICs allow for resilient mode(s) to be enabled for all codecs.
• This implies signalling HRDs, repeated headers and Access Unit Delimiters for H.264/AVC and HEVC.
• For AV1, separate flags are available for AV1 error resilient encoding mode, and it has no performance impact.
• Long Term Reference Frames (LTRs) are enabled throughout all tests. This is also related to the resilience modes above.
• Enhanced compatibility on restricted encoding profiles and legacy devices can also be enforced by limiting the length of long term reference frame count to 1.

Visualization ofVMAF Scores on C21 Live Encoder for CPU-GPU-VPU on H264-HEVC-AV1

Visualization of VMAF Scores on C21 Live Encoder for H264-HEVC-AV1 on CPU-GPU-VPU

Summary of VMAF Scores for the NETINT Codensity G5 VPUs in production

The following summarizes the VMAF scores across different codecs and resolutions, highlighting their quality performance:

1. HEVC

• Best performance at 720p (81.09), followed closely by 1080p (79.58).

• As expected, the quality degradation even at extremely low resolutions is acceptable and comparable to software encoding. The 360p variant scored the lowest (55.15).

2. H.264/AVC

• Similar performance to HEVC, peaking at 720p (80.95) and 1080p (79.26).

• Minimal quality degradation even at extremely low resolutions compared to H.265/HEVC, and is comparable to software encoding. The 360p variant scored the lowest (54.75), slightly below the equivalent HEVC encoder.

3. AV1

• Consistently delivers the best scores across resolutions.

• Tops at 720p (81.13) and 1080p (79.65).

• Notably stronger performance at 576p (70.55) and 360p (55.26) compared to other codecs.

Key Insights

• CPU-based Encoding: Typically excels in high-bitrate scenarios but can falter in real-time applications due to processing limitations.
• GPU-based Encoding: Fast and scalable but may compromise quality in low-bitrate applications.
• VPU-based Encoding: Optimized for standard workflows with unmatched efficiency but struggles with adaptability to specific legacy requirements such as interlaced encoding, among others.
• AV1 outperforms HEVC and H.264/AVC at mid-to-lower resolutions, showing its efficiency for modern encoding demands.
• HEVC and H.264 show near-parity at higher resolutions but decline similarly at lower levels.
• However, their scores are nearly identical to their equivalent software-based encoders.

Of note is that the VMAF scores acquired with libvmaf_cuda, weighted against NVIDIA’s scaling algorithm(s) are virtually identical to those measured against libvmaf across multiple runs.

This implies that the scalers in use on the NETINT VPUs are going toe to toe with a GPU-based scaler for a fraction of the power. The VMAF point difference(s) at 1080p between GPU & VPU ASICs are due to the low-power mode(s) that the VPU scaler runs on by default, and the results are visually indistinguishable.

Why It Matters for Your Workflow

Understanding these differences is critical for media professionals tasked with delivering the best viewer experience. Choosing the right architecture isn’t just about technical specs; it’s about aligning your workflow to your content goals.

At Cires21, we’re committed to providing tools like the C21 Live Encoder that simplify these complexities. Our solutions ensure optimal performance across architectures, helping you measure, compare, and optimize perceived quality like never before.