On-Device Video Quality Enhancement

A blurry video stream is not fixed by simply adding a sharper filter.

Real on-device video quality enhancement has a harder constraint: every decoded frame must be enhanced before the next frame arrives. At 60 fps, the system has about 16 ms per frame for decoding, enhancement, rendering, UI work, and synchronization. If the app copies each frame into CPU memory, converts formats, runs a model, then copies it back to the GPU, the feature may look good in a demo and still fail in production.

The important idea is this: on-device video enhancement is a frame pipeline problem first, and an AI model problem second.

The key idea

Client-side video enhancement usually combines several techniques: super-resolution, deblocking, denoising, tone mapping, and region-based enhancement for faces or subtitles. The business reason is clear: transmit a lower-bitrate stream, then recover perceived quality locally on the device. The source material correctly emphasizes this trade-off and the need for a zero-copy-oriented pipeline.

But the engineering boundary is often misunderstood.

The goal is not to “make every frame beautiful.” The goal is to improve the frame without breaking playback.

That means the frame should stay as close as possible to the hardware decode and render path. On Apple platforms, Core ML can use CPU, GPU, and Neural Engine depending on configuration and model support, and MLComputeUnits lets developers restrict which compute units are allowed. For Metal rendering, CVMetalTextureCache is designed for sharing Core Video image buffers with Metal kernels.

On Android, AHardwareBuffer represents memory that can be shared across hardware components, which makes it relevant to decoder, GPU, camera, and rendering pipelines.

The practical rule is simple: avoid turning video frames into ordinary images unless you are willing to pay the performance cost.

Core example: a small enhancement switch with real frame boundaries

Problem:

The user enables AI enhancement during video playback.
The app should enhance frames only when the device can afford it.
The UI should not know whether the engine uses Core ML, Metal, Vulkan, GPU delegates, or a fallback shader.

Bad direction:

Decoded frame
  -> convert to UIImage or Bitmap
  -> run enhancement
  -> convert back to texture
  -> render

This design is easy to write but expensive to run. UIImage and Bitmap are convenient UI abstractions, not ideal real-time video frame abstractions.

Better direction:

Decoded frame
  -> hardware-backed frame buffer
  -> enhancement engine
  -> renderable output frame
  -> display

The code should reflect that boundary.

iOS:

import SwiftUI
import Observation
import CoreML
import CoreVideo

enum EnhancementMode {
    case off
    case regionOnly
    case fullFrame
}

@Observable
final class VideoEnhancementState {
    private let engine: VideoEnhancementEngine
    private let policy: EnhancementPolicy

    var enabled = false
    var mode: EnhancementMode = .off
    var message: String?

    init(engine: VideoEnhancementEngine, policy: EnhancementPolicy) {
        self.engine = engine
        self.policy = policy
    }

    func setEnabled(_ value: Bool) {
        guard value else {
            enabled = false
            mode = .off
            engine.stop()
            return
        }

        let selectedMode = policy.bestMode()

        guard selectedMode != .off else {
            enabled = false
            mode = .off
            message = "Enhancement is unavailable under current device conditions."
            return
        }

        enabled = true
        mode = selectedMode
        message = nil
        engine.start(mode: selectedMode)
    }

    func process(_ frame: CVPixelBuffer) -> CVPixelBuffer {
        guard enabled else { return frame }
        return engine.enhance(frame)
    }
}

final class VideoEnhancementEngine {
    private lazy var modelConfiguration: MLModelConfiguration = {
        let configuration = MLModelConfiguration()
        configuration.computeUnits = .cpuAndNeuralEngine
        return configuration
    }()

    func start(mode: EnhancementMode) {
        // Warm up model, allocate reusable buffers, prepare Metal/Core ML path.
    }

    func stop() {
        // Release temporary resources or downgrade to normal rendering.
    }

    func enhance(_ frame: CVPixelBuffer) -> CVPixelBuffer {
        // Key point:
        // Keep the API at CVPixelBuffer level.
        // Avoid converting every frame into UIImage or Data.
        return frame
    }
}

struct EnhancementPanel: View {
    @State private var state: VideoEnhancementState

    init(state: VideoEnhancementState) {
        _state = State(initialValue: state)
    }

    var body: some View {
        VStack(alignment: .leading) {
            Toggle("AI Enhance", isOn: $state.enabled)
                .onChange(of: state.enabled) { _, value in
                    state.setEnabled(value)
                }

            Text("Mode: \(String(describing: state.mode))")

            if let message = state.message {
                Text(message).font(.footnote)
            }
        }
        .padding()
    }
}

The important line is not the placeholder model call. It is this boundary:

func process(_ frame: CVPixelBuffer) -> CVPixelBuffer

That keeps the enhancement path compatible with decoder output, Core Video, Metal, and Core ML-oriented workflows.

Android: Kotlin + Compose

import androidx.lifecycle.ViewModel
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.update

enum class EnhancementMode {
    OFF,
    REGION_ONLY,
    FULL_FRAME
}

data class EnhancementState(
    val enabled: Boolean = false,
    val mode: EnhancementMode = EnhancementMode.OFF,
    val message: String? = null
)

class VideoEnhancementController(
    private val engine: VideoEnhancementEngine,
    private val policy: EnhancementPolicy
) : ViewModel() {

    private val _state = MutableStateFlow(EnhancementState())
    val state: StateFlow<EnhancementState> = _state

    fun setEnabled(enabled: Boolean) {
        if (!enabled) {
            engine.stop()
            _state.value = EnhancementState()
            return
        }

        val selectedMode = policy.bestMode()

        if (selectedMode == EnhancementMode.OFF) {
            _state.update {
                it.copy(
                    enabled = false,
                    mode = EnhancementMode.OFF,
                    message = "Enhancement is unavailable under current device conditions."
                )
            }
            return
        }

        engine.start(selectedMode)

        _state.update {
            it.copy(
                enabled = true,
                mode = selectedMode,
                message = null
            )
        }
    }

    fun process(frame: HardwareVideoFrame): HardwareVideoFrame {
        return if (state.value.enabled) engine.enhance(frame) else frame
    }
}

interface VideoEnhancementEngine {
    fun start(mode: EnhancementMode)
    fun stop()
    fun enhance(frame: HardwareVideoFrame): HardwareVideoFrame
}

data class HardwareVideoFrame(
    val timestampNs: Long
    // Production code may wrap a decoder surface, hardware buffer,
    // external texture, or Vulkan-compatible image handle.
)

import androidx.compose.material3.Switch
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue

@Composable
fun EnhancementPanel(controller: VideoEnhancementController) {
    val state by controller.state.collectAsState()

    Switch(
        checked = state.enabled,
        onCheckedChange = controller::setEnabled
    )

    Text(text = "Mode: ${state.mode}")

    state.message?.let {
        Text(text = it)
    }
}

Again, the key detail is the frame type:

fun process(frame: HardwareVideoFrame): HardwareVideoFrame

The UI never receives a Bitmap. The enhancement engine can later choose a GPU path, Vulkan path, hardware buffer path, or lightweight shader fallback without changing the UI.

Why this works

This design works because it protects the hot path.

Video playback is a continuous pipeline. Every avoidable copy increases latency, memory bandwidth, and thermal pressure. Once the phone heats up, the system may throttle CPU or GPU frequency, which makes the next frames even harder to process.

That is why production systems usually need multiple modes:

High-end device:
full-frame super-resolution + denoise + tone mapping

Mid-range device:
region-based enhancement for face or subtitle areas

Low-end or hot device:
disable AI enhancement and keep stable rendering

The best user experience is not always the strongest model. Sometimes the best choice is a weaker enhancement that never drops frames.

One practical limitation

Do not describe the pipeline as perfectly zero-copy unless you have measured it.

Real devices may still require format conversion, texture synchronization, intermediate buffers, or runtime-specific layout changes. The better wording is zero-copy-oriented: design the system to avoid CPU round trips on the critical path, then verify with profiling tools.

This is especially important on Android now that NNAPI is deprecated in Android 15. New implementations should hide the inference backend behind an engine interface so the app can migrate between runtime choices without rewriting playback logic.

Practical rule

Design the frame abstraction before choosing the enhancement model.

If the frame type is UIImage or Bitmap, you are probably building an image feature.

If the frame type is CVPixelBuffer, hardware buffer, decoder surface, external texture, or a renderable frame wrapper, you are building a real-time video feature.

Conclusion

On-device video quality enhancement is valuable because it lets products trade network bandwidth for local computation.

The common mistake is treating it as a model showcase. In production, the pipeline decides whether the model can run every frame without damaging playback.

The core implementation idea is to keep frames in hardware-friendly representations and downgrade gracefully when the device cannot afford full enhancement.

Interview-friendly sentence: real-time video enhancement is not just about generating sharper pixels; it is about doing it inside the frame budget.