Variable frame rate is why your ABR ladder drifts out of sync

The bug report said the audio was out of sync. It was, but only after about four minutes. The first minute played fine. That detail is the whole story, and it took me longer than I want to admit to hear it properly.

A video that is out of sync from frame one has a fixed offset. You add a delay, you ship the fix, you move on. A video that starts in sync and comes apart as it plays does not have an offset. It has a clock problem. Somewhere in the pipeline, audio time and video time are advancing at different rates, and the gap between them opens a little wider with every passing second. In a user-upload product, the overwhelmingly common cause of that is variable frame rate footage that nobody normalized on the way in.

What VFR actually is, and why your users keep sending it

Constant frame rate means every frame occupies the same slice of time. Thirty frames per second, each one 1/30th of a second wide, forever. It is the assumption baked into nearly every tool that touches video downstream: your segmenter, your thumbnail extractor, your player’s seek logic, your editor’s timeline.

Variable frame rate breaks that assumption on purpose. The gap between frames is not fixed. Some frames sit 1/30th of a second apart, some 1/60th, some considerably longer, because the encoder only produced a frame when there was something new to encode. It is an efficiency win in the context that invented it. A screen recording of someone reading a static slide for twelve seconds does not need 360 frames of the same slide.

Which is exactly why you keep receiving it. Screen capture is the biggest VFR factory in existence. OBS, QuickTime’s screen recorder, Loom, phone screen recorders, and most game capture tools all emit VFR by default, because the underlying system generates a new frame only when the screen content actually changes. If your product accepts uploads from anyone doing a tutorial, a demo, a bug report, or a gameplay clip, a meaningful slice of your ingest is variable frame rate and you probably are not measuring how much.

Then there is the harder truth: your users have no idea. Nobody opens QuickTime and thinks about frame timing. They record, they upload, and they file a ticket four days later saying the audio drifts. The problem entered your system as a perfectly valid MP4 that plays fine in the app that made it.

Why the drift happens on your side, not theirs

Audio does not do variable. An audio stream advances at a fixed sample rate, relentlessly, with no notion of “nothing interesting happened so I’ll skip ahead.” When you feed a VFR video and a CFR audio track into a transcoder that has been told to produce a fixed output frame rate, something has to give.

If the transcoder assumes the input is constant and simply stamps out frames at the target rate, it is effectively re-timing the video against a clock the source never agreed to. Frames that were meant to be held for a long beat get replayed too quickly, or dropped. The video’s total duration ends up subtly different from the audio’s. Because the error accumulates rather than resetting, the first few seconds look perfect and the divergence becomes obvious somewhere in the middle. The viewer who watches thirty seconds of your video never complains. The one who watches ten minutes always does.

Drift is the loudest symptom but not the only one. Segment durations in an HLS or DASH ladder get lumpy, because the segmenter is trying to cut on a keyframe cadence expressed in frames while the frames themselves are irregular in time. Thumbnail extraction at a fixed timestamp lands somewhere other than where you asked. And if you are encoding the same source into multiple renditions, each rendition can make slightly different decisions about which frames to keep, which is a fast way to end up with a ladder whose renditions do not line up cleanly for adaptive switching.

A file that plays fine in QuickTime can still be structurally hostile to a streaming pipeline. Playing correctly once is not the same as being safe to transcode.

Detecting it is trickier than the popular answer suggests

The advice you will find in most places is to run ffprobe and compare r_frame_rate against avg_frame_rate. If they disagree, you have VFR. This is useful and it is also not quite true, and the gap between "useful" and "true" is where you will waste a day.

avg_frame_rate is what it says: total frames divided by duration. r_frame_rate is not the real frame rate at all. It is FFmpeg's guess at a base rate, computed as something close to a lowest common multiple of the timings it observed, so that every frame's timestamp can be expressed as a whole number of ticks. A file can report 30000/1001 for one and 30/1 for the other and be perfectly constant. Another file can report identical values and still contain irregular gaps.

So treat the mismatch as a cheap first-pass filter, not a verdict. It costs almost nothing and it correctly flags most of what you care about. When you need certainty, go one level down and look at the actual per-frame timing. Ask ffprobe to dump packet or frame timestamps and check whether the deltas between them are uniform. Uniform deltas mean the file is effectively constant regardless of what the header fields claim. Varying deltas mean it is genuinely variable, and now you know rather than suspect.

The practical shape of this in an ingest pipeline is two-tier. Run the cheap header comparison on every upload. For anything that trips it, run the more expensive timestamp inspection before deciding what to do. You do not want to pay for a full packet dump on every file, and you do not want to normalize files that did not need it.

[IMAGE: side-by-side timeline diagram showing evenly spaced CFR frames vs irregularly spaced VFR frames, with an audio track running at a constant rate underneath both]

Normalize once, at the front door

The fix is not clever and that is a feature. Convert to constant frame rate as the first transcoding step, before anything else in your pipeline gets an opinion about the file. Everything downstream then operates on the assumption it was already making, and you stop debugging timing problems in three different services.

If you are writing new FFmpeg commands in 2026, reach for -fps_mode cfr rather than the -vsync cfr you will find in older tutorials. The -vsync family is deprecated and lives on borrowed time; -fps_mode is the current spelling and it is clearer about what it does. Pin an explicit output rate rather than letting the tool infer one, because inference is what got you here.

The rate you pick is a product decision more than a technical one. Thirty is the safe default for talking-head, screen-capture, and general user-generated content. Sixty is worth it when motion detail is the point, which mostly means gameplay and sports. Going higher than the source’s own peak rate buys you nothing except bitrate. Going far below it will visibly stutter footage that had genuinely smooth motion.

There is a real cost to normalizing, and you should know it going in. A twelve-second static slide that occupied a handful of frames as VFR becomes 360 frames as 30fps CFR. Your encoder does more work and, depending on how good your rate control is, your output may be larger. For screen-recording-heavy libraries this is not a rounding error. The answer is not to skip normalization; the answer is to let a modern encoder’s rate control handle the redundancy, since near-identical consecutive frames compress extremely well, and to size your transcode fleet knowing that screen capture is your worst case rather than your best.

The part worth internalizing

The reason this bug is expensive is not that it is hard to fix. It is one flag. It is expensive because it presents as three unrelated tickets in three different parts of your product: an audio sync complaint from support, a jittery-segment-duration observation from whoever owns the packager, and an off-by-a-second thumbnail from the design team. Nobody connects them, because nothing about a lip-sync report suggests a frame timing problem.

That pattern generalizes past this one bug. In video pipelines, an enormous share of your weird, intermittent, hard-to-reproduce failures are not failures of your code. They are your code correctly processing input whose properties it silently assumed and never verified. Frame rate is one such property. Color space is another. Rotation metadata is a third, and it will ruin your week in a completely different way.

So the useful habit is not “add -fps_mode cfr to the command." It is: pick one assumption your pipeline makes about incoming files, write the ffprobe check that tests it, and run that check against a week of real uploads. Whatever comes back will tell you more about your product than any amount of reading will.