Captions in HLS are not a manifest entry. They are a pipeline.

A founder I know shipped a video product where every upload got automatic English subtitles. The captions looked great in the editor preview. They synced perfectly when played back on the staging environment. The team announced the feature on a Thursday, and by Friday afternoon support was forwarding a steady drizzle of “the captions are wrong” complaints, none of which made any sense from the dashboard.

After half a day of staring at network tabs, the engineer on call figured it out. The captions were exactly right. They were also delivered as a single, gigantic WebVTT file that started its clock at zero, while the HLS video manifest started its media clock somewhere around two seconds in. iOS happily reconciled the difference. Chrome on Android did not. The seek bar made it worse: every seek snapped the WebVTT clock back to zero, while the video kept its own time, so the further into the video you went, the worse the drift got.

This is the kind of bug that PSNR-style optimism hides until it hits a production audience. Captions in HLS look simple from outside. They are a manifest entry, a file, an attribute. They are not simple. They are a pipeline, and the parts of that pipeline that decide whether your captions actually display correctly are the parts almost nobody documents end-to-end.

What “captions in HLS” actually means

There are at least three different things people call “HLS captions”, and conflating them is where most teams’ bugs come from.

The first is CEA-608 / CEA-708 captions, which ride inside the video bitstream itself. They are a relic of broadcast television, encoded into the H.264 or HEVC NAL units, and players surface them through a flag on the track. If your source already has them embedded (anything that came through a broadcast workflow probably does), you do not need a separate subtitle track for that language. You also cannot edit them without re-encoding the video.

The second is WebVTT delivered as a sidecar, a separate .vtt file referenced through EXT-X-MEDIA in the master manifest. This is what most web tutorials show first, and it is also the implementation that ships the bug I started with. A single sidecar file works for short content, breaks at scale, and falls apart on low-latency live.

The third, and the one a serious player team should be running today, is segmented WebVTT or fMP4-wrapped WebVTT, packaged into per-chunk segments that line up with the media segments, each carrying an X-TIMESTAMP-MAP cue so the player can reconcile the WebVTT clock with the MPEG-2 PTS clock the video segments are on.

If you do not know which of those three your pipeline produces, that is the first bug.

The dance with X-TIMESTAMP-MAP

The single most useful line in RFC 8216 (the HLS spec) for subtitle engineering is the description of X-TIMESTAMP-MAP. It is a tiny declaration at the top of a WebVTT segment that looks like this:

WEBVTT
X-TIMESTAMP-MAP=MPEGTS:900000,LOCAL:00:00:00.000

00:00:00.000 --> 00:00:03.500
Hello, and welcome to the show.

What that line is saying: the moment my WebVTT clock reads 00:00:00.000 corresponds to MPEG-2 PTS value 900000 on the media track. 900000 in a 90kHz PTS clock is ten seconds. If the player is currently at PTS 945000, my cue is rendered fifty milliseconds after the start.

Without that line, a player has to guess. Some guess by aligning the first cue to the first media sample. Some assume zero. Some align to whatever the previous segment did. The result is the cross-browser inconsistency that hurts so much, because the bug only shows up under specific seek and reload patterns.

Adding the cue is not optional for serious HLS subtitles. Adding the cue correctly takes more thought than people give it. The MPEG-2 PTS value comes from the encoder, not from the subtitle file, so the packaging step has to read it back. Most teams I have worked with were doing the read-back wrong on the first attempt.

Sidecar vs fMP4: pick the one your players actually run

For a long time, the safe answer for HLS subtitles was “sidecar WebVTT”. Apple invented HLS, Apple shipped WebVTT support natively, every other player followed. The downside is that the player makes its own decisions about how to fetch and parse a giant .vtt, and a one-megabyte subtitle file for a feature-length movie is a network and parse cost the player wears at startup.

The modern answer for production-grade pipelines is fMP4-wrapped subtitles, where the WebVTT cues are carried in their own ISO BMFF segments that look structurally the same as the video and audio segments. They get the same fetch behavior, the same byte-range optimizations, the same CMAF-driven low-latency story. Shaka Packager and modern mp4box releases both produce them. Safari, Shaka Player, ExoPlayer, and HLS.js all handle them, with HLS.js fixing several edge cases in its 1.6.x line (the latest as of April 2026 is 1.6.16, which closed off a few WebVTT progressive-loading and LL-HLS-part-loading bugs I had personally hit on a side project).

If you are running plain WebVTT sidecars and the team has not had bugs yet, the questions to ask are: do users seek often, do users on Android Chrome complain about drift, do you ship low-latency live. The bigger any of those signals get, the more “sidecar is fine” stops being fine.

The OTT-grade reality check: IMSC sits above WebVTT

If your product touches premium OTT, broadcaster delivery, or anything with regulatory captioning requirements, you will run into IMSC 1.1 the moment a partner sends you a mezzanine file. IMSC, the constrained TTML profile that the industry settled on as the canonical interchange format for OTT subtitles, is what most premium pipelines author in. WebVTT is what gets generated at packaging time, on the way out the door to HLS. The 2026 architectural reassessments coming out of the streaming press are pretty consistent about this: IMSC as source, WebVTT for HLS, segmented WebVTT or fMP4 for serious players.

The reason this matters even if your product is consumer-facing is that the moment you accept third-party captions (a creator uploads a .dfxp, a customer ingests a TTML file from a captioning vendor) you are implicitly committing to a conversion step. The conversion is mostly fine; the corners that hurt are positioning, styling, and forced narratives, which IMSC supports more richly than the WebVTT spec. If you flatten everything to plain text in the converter, you will lose a class of captions that an accessibility reviewer cares about, and you will not find out until you fail an audit.

The pieces I would not skip

A handful of things are worth getting right the first time, because the cost of redoing them later compounds:

Drop X-TIMESTAMP-MAP into every WebVTT segment, not just the first. Some players cache the first segment’s timing and reuse it; others fetch out of order. Putting the cue in every segment is the only safe choice.

Match subtitle segment duration to video segment duration. If your video is on six-second segments, do not ship subtitles on thirty-second chunks. The player has to over-fetch, and seek behavior becomes spongy.

Test with the player your largest user segment runs. AVPlayer on iOS, ExoPlayer on Android, HLS.js on Chromium, and Shaka Player on a TV box all have slightly different ideas about how to render the same WebVTT. The unit tests that pass in your laptop’s Safari are not the ones that decide whether your support inbox stays quiet.

Carry positioning data through the pipeline, even if you do not render it. The moment you onboard a customer that cares about accessibility, you will be glad you did not lose align, line, and position in the converter.

What this changes about how teams ship subtitles

The shortcut version of subtitles, single sidecar .vtt, no timestamp map, no segmenting, is fine for a hackathon. It is not fine for a product that has a support inbox and a churn metric. The good news is that the tooling has caught up: FFmpeg 8.1.1 has a real webvtt muxer, shaka-packager and mp4box both produce segmented and fMP4-wrapped output that the modern players actually read correctly, and the player libraries themselves have spent the last year closing the long tail of edge-case bugs.

If you are building a video product that any non-trivial number of people will watch with captions on, the work that matters is in the packager and the converter, not in the manifest. The next thing worth looking at is probably how your worst-network user sees your worst-rendered language. That, like most of the things that decide whether viewers stay or leave, is hiding two layers below the dashboard.