Measure, Don’t Guess: Building viser, a Content-Adaptive Video Encoding Optimizer in Rust

Most video on the internet is encoded with a one-size-fits-all bitrate ladder that wastes bandwidth on simple content and starves complex content. viser is a bet that we can do better — by measuring every video instead of guessing.

There is a small act of quiet waste happening millions of times a day. Somewhere a streaming service takes a video — a news anchor reading the headlines, a fast-cut action sequence, a grainy noir thriller — and runs all of them through the same table of bitrates. The anchor gets far more bits than her flat, static frame could ever use. The thriller gets nowhere near enough, and its shadows dissolve into blocky mud. The table doesn’t care. It was written once, in 2010, and it has been copied ever since.

viser is built on a simple conviction: every video deserves to be looked at. Not guessed at from a lookup table, not predicted from a thumbnail — actually encoded, actually measured, and given the bitrate ladder that wrings the most perceptual quality out of every single bit. The name blends vision and optimizer (and nods to French viser, “to aim”). It is fifteen Rust crates, roughly 8,000 lines, and a belief that the right answer is worth the work of finding it.

This is the story of how it works, why the hard parts are hard, and what it took to make a tool whose every numerical claim you can reproduce yourself.

The Problem: One Ladder Does Not Fit All

Every streaming service faces the same decision, over and over: at what resolutions and bitrates do I store this video so a player can adaptively switch between rungs as the network changes?

The easy answer is a fixed bitrate ladder — one table of (resolution, bitrate) rungs applied to everything. Apple’s original 2010 HLS recommendation defined ten rungs from 64 kbps up to 8564 kbps at 1080p, and most of the industry copied it. It is simple, and it is wasteful, because content does not have uniform complexity:

ContentAt 3 Mbps, 1080pTalking head (news anchor)Excellent — bits are being wastedAnimation (Pixar-style)Very good — some bits wastedSports (football game)Acceptable — could use more bitsFilm grain (dark thriller)Poor — severely underbitrated

A fixed ladder simultaneously overspends on the anchor and starves the thriller. The talking head reaches transparent quality at 1.5 Mbps and 720p; the action sequence needs 8 Mbps and 1080p to reach the same perceived quality. Netflix proved the fix in 2015 — per-title encoding — and demonstrated 20–30% bitrate savings. The catch is that doing it correctly is a measurement problem, and measurement problems are where good intentions quietly go wrong.

viser exists to do that measurement correctly, at four levels of granularity, for three codecs, with results you can reproduce.

Four Granularities, One Framework

“Content-adaptive encoding” is not one algorithm. It is a family, distinguished by how finely you adapt:

MethodGranularityThe hard partPer-TitleWhole videoBuild a per-codec convex hull, pick N Pareto-optimal rungsPer-ShotScene (2–30s)Detect shot boundaries, then allocate a global bit budget across shotsSegment-Level CRF1-second segmentsTune CRF per segment with closed-loop VMAF verificationContext-AwarePer device classGenerate mobile/desktop/TV ladders with different caps and codecs

All four sit on the same foundation, and all four had to be right not just in shape but in number. A convex hull that is “roughly right” produces a ladder that is roughly right, which means bandwidth you can measure on a bill. The whole craft of the project is in turning “roughly” into “exactly.”

When There’s No Single Right Answer to Diff Against

Here is what makes this kind of tool intimidating to build: the output of a per-title analysis is a judgment — a set of rungs — and reasonable engineers disagree about the last rung. There is no canonical perfect-ladder binary you can run alongside yours and demand identical output. So what do you validate against?

The breakthrough is to realize that the oracle is the published science, decomposed into pieces that each have a knowable right answer:

1. The convex hull has a correct answer. Given a set of (bitrate, VMAF) points, the upper Pareto frontier is a deterministic geometric object. Andrew’s monotone chain produces it in O(n log n), exactly. You can pin it against hand-computed hulls: empty input, a single point, all-collinear points, interior points that must be removed, unsorted input.
2. BD-Rate — the Bjøntegaard delta that says “codec A needs X% less bitrate than codec B for equal quality” — is a published numerical method with reference implementations and known outputs. Feed in the canonical test vectors; demand the same percentage back.
3. The reference ladders are public. Netflix and Apple’s fixed ladders are documented rung-for-rung. viser carries them as comparison baselines, and a test asserts they are reproduced exactly — because a tool that can’t even echo a fixed ladder has no business computing an adaptive one.
4. The encode and measurement bottom out at ffmpeg and libvmaf. viser orchestrates them rather than reimplementing them, so the oracle for “did this encode hit the target bitrate” is ffmpeg’s own output, probed back and checked.

This decomposition is the heart of the whole thing. You cannot diff a judgment — but you can diff every input to the judgment. Get the hull exactly right, get BD-Rate exactly right, get the bitrate measurement exactly right, and the judgment that rides on top of them is correct by construction. Every module in viser had to answer one question: what could I be objectively wrong about? — and then a test was written to pin it. That discipline is what turns an intimidating problem into a tractable one.

Architecture: Fifteen Crates in Four Layers

viser is fifteen crates arranged in four strict layers, each crate with a single responsibility:

Foundation   viser-ffmpeg     FFmpeg/FFprobe wrapper — encode, probe, cache
             viser-quality    VMAF / PSNR / SSIM / SSIMULACRA2 / Butteraugli
             viser-encoding   Shared config, preset mapping, temp cleanup
             viser-checkpoint SHA-256 resumable state

Core         viser-hull       Convex hull (Pareto frontier) + BD-Rate
             viser-ladder     Rung selection with crossover enforcement
             viser-shot       Shot/scene detection (FFmpeg scdet)
             viser-complexity Spatial / temporal / DCT analysis

Pipelines    viser-pertitle   Whole-video ladder
             viser-pershot    Per-shot + Trellis bit allocation
             viser-persegment Segment-level CRF adaptation
             viser-contextaware Device-specific ladders

Application  viser-cli        The clap binary
             viser-compare    Browser comparison player
             viser-chart      R-D curve charts (plotters)

The dependency graph only ever points down. A pipeline crate composes foundation and core crates; it never reaches sideways into another pipeline. viser-pershot is allowed to depend on viser-pertitle (a shot is just a short title), but viser-pertitle knows nothing about shots.

This is where Rust’s crate model earns its keep. Every cross-module call is an explicit, typed, public API — there is no accidental reach-around between optimization logic and ffmpeg-shelling logic, because the crate boundary makes it impossible to write one by accident. Fifteen shallow crates, not a deep inheritance tree. Wide and flat, where each piece is small enough to hold in your head and test in isolation.

Race the Encodes, Walk the Budget

A full per-title search is 5 resolutions × 9 CRF values × 3 codecs = 135 trial encodes. Serially, that's an afternoon. All at once, it's a fork bomb that thrashes the machine into swap and produces slower wall-clock time than a careful subset. The art is in racing the encodes while walking the resource budget — and viser models that bound explicitly:

// Bound concurrent encodes to num_cpus/2 — encoders are already
// internally threaded, so oversubscribing thrashes the cache.
let permits = Arc::new(Semaphore::new(num_cpus::get() / 2));

let trials = trial_matrix.into_iter().map(|trial| {
    let permit = permits.clone();
    async move {
        let _slot = permit.acquire().await?;   // backpressure, not a fork bomb
        let encoded = encode(&trial).await?;
        let vmaf = measure_vmaf(&source, &encoded).await?;
        Ok(RdPoint::new(trial, encoded.bitrate, vmaf))
    }
});

The semaphore is the entire concurrency story, and it is visible. There is no hidden task that outlives its parent, no channel that deadlocks if a consumer panics. Tokio’s structured concurrency means a failed trial propagates and cancels the rest rather than orphaning them.

And because every long analysis is checkpointed — each completed trial hashed by its (resolution, codec, CRF, preset) config under a SHA-256 key — a crash three hours into a 135-trial sweep resumes from trial 130, not trial zero. The same hash is the cache invalidation: change the preset, the hash changes, stale results are silently ignored. Long, expensive work that survives a crash is not a luxury; it's what makes the whole "measure everything" philosophy practical instead of fragile.

What Makes This Genuinely Hard

It would be dishonest to pretend the language makes this correct. Rust makes it safe; the domain makes it hard, and no language helps with the domain. The genuinely difficult parts are worth naming, because they are also the parts that make the project interesting:

Resolution crossovers are content-dependent. Each resolution traces an R-D curve that flattens at a quality ceiling — 480p tops out around VMAF 98 because downsampling destroys high-frequency detail that no number of bits can recover; 1080p reaches 99.7. The bitrate at which 1080p overtakes 720p on the hull is not a constant. For a talking head it happens low; for film grain it happens high, because the lower resolution hasn’t hit its ceiling yet. The hull finds the crossover automatically — but only if you compute it across all resolutions jointly, a subtlety that quietly produces a plausible-but-wrong ladder if you get it backwards.

VMAF is expensive, and the cost compounds. Every one of those 135 trials needs a VMAF measurement, and VMAF runs on the CPU through libvmaf. The honest accelerations — n_subsample=5 to measure every fifth frame, fast presets for the search and slow presets only for final delivery, eliminating dominated resolutions after a single probe CRF — are all approximations, each trading accuracy for time. So viser exposes them as flags rather than baking in a guess. The trade is the user's to make, with eyes open.

Per-shot allocation is a global optimization, not a local one. You cannot optimize each shot independently — that spends your whole budget on the first complex shot. The correct method is Trellis/Lagrangian allocation: find the single quality-per-bit slope λ at which the sum of per-shot bitrates hits your global target, then assign every shot the operating point at that slope. It’s a one-dimensional search over λ with a convex-hull lookup per shot inside the loop, and every failure mode (empty hull, single shot, identical shots, duration weighting) earned its own test.

HDR breaks the metric. libvmaf is SDR-centric. An HDR source measured with an SDR model produces a number that looks fine and means nothing. viser detects HDR from probe metadata and refuses it by default, gated behind an explicit --allow-hdr for people who know they're doing something best-effort. Honesty as a feature, refusal as a kindness.

None of these are language problems. They are video problems that any serious implementation has to get right — which is exactly why the 170+ tests exist: the compiler cannot catch a wrong crossover bitrate, so a human writes the assertion that does.

The Heart: Convex Hull and BD-Rate

The core of the whole project is about 540 lines in viser-hull, and it is the part I trust most, because it is the part with the least ambiguity. The upper hull is textbook Andrew's monotone chain, specialized to (bitrate, quality) space:

/// Upper convex hull of R-D points: the Pareto frontier where no other
/// configuration achieves higher quality at the same or lower bitrate.
fn upper_hull(mut points: Vec<RdPoint>) -> Vec<RdPoint> {
    points.sort_by(|a, b| a.bitrate.total_cmp(&b.bitrate));
    let mut hull: Vec<RdPoint> = Vec::new();
    for p in points {
        // Pop while the last turn is clockwise or collinear — those
        // middle points are dominated and must leave the frontier.
        while hull.len() >= 2 && cross(hull[hull.len() - 2], hull[hull.len() - 1], p) >= 0.0 {
            hull.pop();
        }
        hull.push(p);
    }
    hull
}

total_cmp instead of partial_cmp is a small thing that matters: a stray NaN from a failed measurement must not silently corrupt the sort, and Rust forces you to confront float ordering rather than let < quietly do the wrong thing. The crate is #![forbid(unsafe_code)]; nothing in hull geometry needs raw pointers.

BD-Rate sits beside it — the cubic fit over log(bitrate) versus quality, integrated over the overlapping quality range, returned as a percentage. It has the most published reference outputs, and therefore the most assertive tests: minimum-point handling, negative efficiency (when codec A is worse), non-overlapping ranges, and the near-singular matrices the cubic fit produces on pathological inputs.

Validation: Reproduce the Known Before Trusting the Novel

The test suite is 170+ tests across the workspace, and its philosophy is “pin everything that has a knowable answer”:

cargo test -p viser-hull        # convex hull, BD-rate — 24 tests
cargo test -p viser-ladder      # rung selection, crossover, savings — 19 tests
cargo test -p viser-pershot     # Trellis Lagrangian allocation — 12 tests
cargo test -p viser-complexity  # complexity + screen-content detection — 21 tests
cargo test -p viser-ffmpeg      # probe, encode args, metadata adapter — 26 tests

The coverage maps directly onto “what could each module be wrong about”:

• Hull: empty, single point, interior-point removal, unsorted input, per-codec separation.
• BD-Rate: minimum points, negative efficiency, non-overlap, singular matrices, cubic fit.
• Trellis: empty, single shot, duration weighting, identical shots, empty-hull fallback, λ-search bounds.
• Ladder: empty, zero rungs, bitrate/VMAF filters, max-VMAF cap, sorted output, the Netflix and Apple reference ladders reproduced exactly, savings computation.
• Screen content: slides classified 90%, natural video 0%, code-capture 70%, empty input.

The whole workspace runs in about half a second, which means the tests run on every save, which means the math stays pinned. Fast tests are honest tests — the ones too slow to run get run never, and the modules they protect drift. The single most valuable test in the suite is the cheapest one: the assertion that viser reproduces the Netflix and Apple fixed ladders rung-for-rung. Reproduce the known before you trust the novel.

Safety and Honesty

The pure-math and orchestration crates are unsafe-free, the workspace denies the usual footgun lints, and the whole thing is edition = "2024" on a pinned MSRV of 1.88. There is no unsafe in hull geometry, ladder selection, or Trellis allocation, because none of those domains need it — they are arithmetic and Vecs.

What Rust really buys here isn’t raw speed — the bottleneck is ffmpeg, not the optimizer. It’s the elimination of a class of silent wrongness. A missing bitrate can’t quietly default to 0 and distort a frontier; Option, exhaustive match, and the absence of an implicit zero-value force every "what if this is missing" to become a decision the compiler makes you write down. In a tool whose entire value is the trustworthiness of a number, that guarantee is the foundation everything else stands on.

The Frontier: Beyond the Obvious

The four methods are the spine, but the most exciting part of viser is where it goes past the textbook:

• Two extra perceptual metrics — SSIMULACRA2 and Butteraugli run in viser-quality alongside VMAF, PSNR, and SSIM. VMAF is excellent, but it is one model; cross-checking against a second perceptual metric catches the cases where VMAF is over-confident.
• Screen-content detection. Slides, code screencasts, and UI captures behave nothing like camera footage — static frames, razor-sharp edges, DCT energy concentrated completely differently. viser-complexity classifies content from spatial/temporal/DCT heuristics so the pipeline can choose a strategy instead of treating a slide deck like a football game.
• A pure-Rust probe engine — revelo. By default viser reads metadata with ffprobe, but an optional engine called revelo (--features revelo) extracts resolution, frame rate, codec, color space, and dynamic range with no external binary on the path at all. More on what that unlocks below.
• Audio-aware ladder budgets. per-title analyze detects the source audio bitrate and reserves it, so a "3 Mbps" rung is 3 Mbps of total stream, not 3 Mbps of video plus whatever audio happens to cost.
• A browser comparison player that shows reference and encoded side-by-side with a per-frame VMAF timeline — because at some point a human has to look, and a number on a chart is not the same as seeing the artifact disappear.

What I’d Want You to Take From This

If there’s a lesson worth carrying out of this project, it’s smaller and more hopeful than “rewrite it in Rust”:

1. When you can’t diff the answer, diff the inputs to the answer. A judgment isn’t testable; the geometry, the numerical method, and the measurement underneath it all are. Decompose until every piece has a knowable right answer, and the judgment becomes correct by construction. This turns intimidating, “expert-only” problems into ones an ordinary careful engineer can ship.
2. Reproduce the known before trusting the novel. The cheapest test can be the most valuable one. If your tool can’t echo a published reference, it has no authority to compute something new.
3. Make your approximations into flags, not assumptions. Every speed/accuracy trade you hide is a guess you’re making on someone else’s behalf. Expose it, and the trade becomes their informed choice.
4. Measure, don’t guess. The whole project is one sentence long: look at the actual video, run the actual encodes, trust the number you can reproduce. It is more work than a lookup table. It is also just better, and the difference shows up on a bandwidth bill and in a viewer’s eyes.

The Result

viser is three things at once, depending on how you reach for it:

# A CLI: compute an optimal ladder across codecs and resolutions
viser per-title analyze -i video.mp4 \
  --codecs libx264,libsvtav1 \
  --resolutions 480p,720p,1080p \
  --preset veryfast --parallel 4 -o analysis.json

# Then deliver the selected rungs as final encodes
viser per-title deliver --analysis analysis.json \
  --output-dir delivery --mode capped-crf --manifest delivery/manifest.json

# A probe — with no ffprobe required
viser inspect probe video.mp4 --probe-engine revelo

# And a set of Rust crates you can compose yourself

Four optimization methods, three codecs, five quality metrics, fifteen crates, about 8,000 lines of unsafe-free Rust, under a permissive BSD-2-Clause license — and every numerical claim pinned to a test you can run in half a second.

cargo install viser-cli --features revelo

Most video on the internet is still encoded by a table written before the iPhone had a Retina display. It doesn’t have to be. Every video deserves to be looked at — and now there’s a tool that does exactly that.

viser is open source at github.com/vbasky/viser.

One More Thing: How revelo Changes the Equation

Everything above quietly leans on a dependency I’ve barely mentioned: to set up its trial matrix, viser has to read the video first — resolution, frame rate, codec, color space, dynamic range. The default way to do that is to shell out to ffprobe, which means a native binary has to be installed, found on PATH, version-matched, and trusted to parse a container the same way every time. For a tool whose whole promise is reproducibility, leaning on an external parser is the one soft spot in the foundation.

revelo is the answer to that — a pure-Rust probe engine built directly into viser. Enable it with --features revelo, ask for it with --probe-engine revelo, and viser reads container and stream metadata itself, in process, with nothing external on the path:

cargo install viser-cli --features revelo
viser inspect probe video.mp4 --probe-engine revelo   # no ffprobe anywhere

That single change rewrites the equation in a few ways:

• One fewer native dependency. The metadata path becomes pure Rust. Distribution gets dramatically simpler — closer to “download one binary and run it” than “install this toolchain first.”
• Reproducibility all the way down. The probe is no longer a black box whose behavior depends on which ffprobe build happens to be installed. The same input yields the same parse, pinned by the same fast tests as the rest of the workspace (codec mapping, color transfer, pixel format, frame-rate formatting all have their own assertions).
• Memory safety on untrusted input. Parsing arbitrary media containers is exactly the kind of byte-wrangling that has produced a decade of CVEs in C parsers. Doing it in unsafe-free Rust turns a whole class of buffer bugs into compile-time impossibilities — which matters the moment you point the tool at a file you didn't create.

ffprobe stays the friendly default, because it is battle-tested and ubiquitous. But revelo is the direction of travel: a video optimizer that, end to end, depends on nothing but itself — and answers you can reproduce on any machine, with no toolchain to install first. The measurement philosophy, finally, reaches all the way down to the first byte.

Additional Reading

If this piece sparked your curiosity, the science docs go a layer deeper:

• Per-Title Encoding — convex hulls, rate-distortion optimization, and how a ladder gets selected
• Per-Shot Encoding — shot detection and Trellis constant-slope bit allocation
• Segment-Level CRF — closed-loop CRF tuning with complexity analysis
• Quality Metrics — VMAF, PSNR, SSIM, and BD-Rate, demystified
• Netflix: Per-Title Encode Optimization — the 2015 post that started it all