Frontend System Design: Video Streaming (Youtube & Netflix)

A high-level design walkthrough of how the browser actually plays a two-hour movie it can never fully download.

When an interviewer says “design YouTube” or “design Netflix,” most people jump straight to the recommendation feed. The interesting frontend problem is somewhere else entirely: a 4K movie is several gigabytes, and the browser cannot download it before you press play. Everything in this design exists to solve that one constraint.

This is the high-level design — requirements, architecture, the tech choices and component breakdown, the data model, the video player, and the performance work that keeps it all smooth. Let’s build it up layer by layer.

1. Requirements

Functional

• Browse videos — categorised lists the user can scroll through.
• Autoplaying hero banner — the large featured video at the top of the page.
• Video player with: Video controls — play, pause, seek, change playback rate. Audio controls — adjust volume, switch audio language. Subtitles — language-specific tracks, toggle on/off.

Non-Functional

• Video performance — multiple resolutions and quality levels, fast startup.
• Page performance — fast initial load.
• Device support — works across phones, TVs, browsers.
• Auth — gated content, sign-in.
• SEO — discoverable metadata.
• Two-way pagination — vertical (rows of categories) and horizontal (videos within a row). Netflix supports horizontal pagination natively.

These non-functional points drive almost every decision below. “Fast startup” is why we stream instead of download. “Multiple quality levels” is why we need adaptive bitrate. “SEO” is why parts of the page are server-rendered.

2. Streaming Terminology

Before the architecture, a shared vocabulary. Skip this if it’s familiar, but most of the design only makes sense once these terms click.

Streaming vs. downloading: With an image, the browser downloads the whole file, then renders it. With a Netflix movie that approach is hopeless — the file is too large to download up front. So we stream: break the video into small chunks (segments, usually 2–10 seconds each) and pull them down progressively as the user watches.

Buffering: Downloading chunks ahead of the playhead so playback stays smooth. That grey portion of the YouTube timeline is content already buffered but not yet played.

Bitrate: Amount of data transferred per second (kbps/Mbps). Higher bitrate means better quality but more bandwidth.

Frame rate: Frames rendered per second — 24 (cinematic), 30, 60. Higher is generally smoother.

Resolution: Pixel dimensions, from 144p up to 4K (2160p).

Codec: The algorithm that encodes and decodes the audio/video. The common ones:

• H.264 (AVC) — universally supported, the safe default.
• H.265 (HEVC) — better compression than H.264, but carries licensing costs.
• VP9 — Google’s royalty-free codec, used heavily on YouTube.
• AV1 — the newest, best compression, royalty-free, but heavier to decode.

Container vs. codec (worth separating): .mp4 and .webm are containers — boxes that hold the streams. H.264, VP9, AV1 are codecs — what's inside the box. WebM typically holds VP9/AV1; MP4 typically holds H.264.

Bandwidth: How much data the connection can pull at a time. The codec compresses; the bandwidth delivers.

Poster: The thumbnail/placeholder shown before playback begins.

Captions: Closed captions can be toggled on/off; open captions are burned into the video.

Manifest file: The index of the stream — it lists every available rendition (resolutions, bitrates) and the URLs of their segments. HLS uses .m3u8; DASH uses .mpd. This file is the brain of adaptive streaming; the player reads it to decide what to fetch next.

Subtitle formats: SRT, WebVTT (the web-native one), TTML (Timed Text Markup Language), SCC.

Playback controls: Play, pause, volume, autoplay.

Seeking vs. scrubbing: Seeking is jumping to a point on the timeline. Scrubbing is dragging along it — often with a thumbnail preview riding under your cursor.

3. High-Level Architecture

A layered, MVC-style breakdown keeps the responsibilities clean.

Views

• Hero section — the autoplaying featured player.
• Video list — rows of categorised videos.
• Video player — the full-screen playback surface.

Controllers

• Video controller — manages playback: streaming, buffering, and the requests to the CDN for the next segments.
• Recommendation controller — pulls and orchestrates feed data, paginating sections as the user scrolls.

Services

• Video delivery service (CDN) — serves the actual video segments from the edge, close to the user.
• Recommendation service — the backend that returns categories and metadata.

Storage & Cache

This layer is what makes the experience feel instant.

• Client-side data cache. Netflix uses a GraphQL client cache (think Apollo cache) so categories and metadata don’t refetch on every interaction. Recommendation payloads are usually pre-computed and stored in something like Redis on the server, not calculated per request.
• In-browser video memory. This is the clever bit. The video element can’t consume raw binary, and we can’t hand it a multi-gigabyte MP4. So:

1. The browser fetches segments as binary.
2. The player wraps them in a Blob and creates a temporary object URL.
3. That URL gets attached to the <video> element.
4. Memory is finite, so once a segment is done, the object URL is revoked to free it.

A minimal sketch of that flow with the MediaSource API:

const video = document.querySelector('video');
const mediaSource = new MediaSource();

// createObjectURL hands the <video> tag a URL it understands
video.src = URL.createObjectURL(mediaSource);
mediaSource.addEventListener('sourceopen', () => {
  const sourceBuffer = mediaSource.addSourceBuffer(
    'video/mp4; codecs="avc1.42E01E"'
  );
  // fetch a segment as an ArrayBuffer, then append it
  fetch('/segments/seg-1.m4s')
    .then(r => r.arrayBuffer())
    .then(chunk => sourceBuffer.appendBuffer(chunk));
});
// when the segment is no longer needed:
// URL.revokeObjectURL(objectUrl);

You’re essentially minting and tearing down disposable URLs inside the browser to keep memory under control.

4. Tech Choices

Every requirement above quietly points at a technology decision. These are the ones that matter — with the why, not just the what. (None of these are the only valid answer; the reasoning is the point.)

Framework — React, with a meta-framework like Next.js: The UI is highly stateful and interactive — player, carousels, menus — which is exactly what a component model handles well. A meta-framework layers server rendering on top. Netflix runs React in production.

Rendering — hybrid SSR + CSR: Server-render the shell and metadata so the page is discoverable and paints fast; hydrate into a client app for the interactive browse-and-play experience. The landing route leans SSR, the watch route leans CSR.

Data fetching & cache — a GraphQL client (e.g., Apollo): The browse page is a patchwork of heterogeneous rows; one typed GraphQL query fetches exactly what each section needs, sidestepping the over- and under-fetching you’d get from REST. The normalised client cache then makes back-and-forth navigation feel instant. Netflix’s path here went Falcor → GraphQL.

Player engine — a streaming library over raw MediaSource: Don’t hand-roll MSE. Shaka Player, hls.js, or dash.js give you adaptive bitrate, manifest parsing, gap-jumping, retries, and DRM out of the box. Pair the engine with a separate controls layer (Media Chrome, or a video.js skin).

Protected content — Encrypted Media Extensions (EME): Premium catalogues need DRM (Widevine, PlayReady, FairPlay). User-generated platforms lean on it far less.

Adaptive protocol — HLS and/or MPEG-DASH: DASH is open and codec-agnostic; HLS is required on Apple devices. Large platforms ship both and choose per device.

Codecs — AV1/VP9 where the device handles them, H.264 as the universal fallback — decided at runtime via the Media Capabilities API.

State management — a light global store (Zustand, Redux Toolkit) for cross-cutting state like session, user, and volume/quality preferences, with local state for everything else. Crucially, keep playback state out of the global tree — more on that in the next section.

Styling — component-scoped CSS (CSS Modules or CSS-in-JS) so styles don’t leak across a large tree.

Images — AVIF/WebP with fallbacks, plus sprite sheets for seek-preview thumbnails.

5. Component Architecture

Zooming in from the layered view, here is how the frontend decomposes into components.

The component tree

<App>
├─ <AppShell>                  // nav, layout, auth gate
│   └─ <BrowsePage>
│        ├─ <HeroBanner>            // autoplaying featured preview
│        └─ <VideoRow> × N          // one per category (virtualised)
│             └─ <VideoCard> × M    // poster, hover preview
│
└─ <WatchPage>
     └─ <VideoPlayer>
          ├─ <VideoSurface>         // <video> + MediaSource engine
          ├─ <PlayerControls>
          │    ├─ <PlayPauseButton>
          │    ├─ <Timeline>        // seek + scrub thumbnails
          │    ├─ <VolumeControl>
          │    ├─ <QualityMenu>
          │    └─ <SubtitleMenu>
          └─ <SubtitleRenderer>

Smart vs. presentational

Split components into containers that own data and side effects (the feed container, the player container) and presentational components that are pure and reusable (a <VideoCard>, a <PlayPauseButton>). The presentational layer has no idea where data comes from, which keeps it testable and portable.

Where state lives

This is the decision that makes or breaks player performance.

• Browse/feed state lives in the GraphQL cache, read through a feed container.
• Session & preferences (user, volume, default quality) live in the global store.
• Playback state (current time, buffered ranges, active quality) is high-frequency — timeupdate fires several times a second. If that lived in global state, the whole app would re-render constantly. So it stays in a dedicated player context/store scoped to the player subtree, and the streaming engine itself lives in a ref, deliberately outside React's render cycle.

Data flow

Data flows top-down through props. The deeply nested control components read shared playback state through the player context instead of prop-drilling five levels down, and interactions bubble back up via callbacks (onSeek, onQualityChange).

Designed for performance

• Virtualise the rows — render only the cards in or near the viewport; a catalogue has thousands.
• Lazy-mount the player — don’t load the heavy player bundle until the user opens a title.
• Memoise cards so a feed update doesn’t re-render every tile.
• Isolate the engine — keep the MediaSource engine in a ref so segment appends never trigger React renders.

A note on scale

At a large org, these areas often harden into separately deployable micro-frontends — Browse, Watch, and Account as independent apps sharing a common design system and component library. It’s overkill for a single team, but it’s how the boundaries above tend to solidify once many teams own different surfaces.

6. Data Model

The model maps directly onto the two-way pagination requirement.

Recommendation — the top-level feed.

Recommendation {
  videoSections: VideoSection[]
  pagination: { pageNo, limit }   // vertical scroll
}

VideoSection — one horizontal row.

VideoSection {
  category: string
  videos: VideoMeta[]
  pagination: { pageNo, limit }   // horizontal scroll
}

VideoMeta — a single tile.

VideoMeta {
  id: string
  title: string
  poster_image_url: string
  video_preview_url: string
}

Vertical pagination loads more sections as you scroll down the page. Horizontal pagination loads more videos within a row as you scroll sideways. Netflix optimises this by returning the first slice of every horizontal row in a single shot, so the visible grid fills immediately and only deeper scrolls trigger follow-up requests.

7. The Video Player

The player is where you decide how much control you want — and pay for it in complexity.

Three layers of control

• The <video> tag. Dead simple, but you barely control it. Good for a plain MP4, not for adaptive streaming.
• MediaSource API. Lets you create object URLs and feed segments into the player yourself. This is the control point for buffering and quality switching.
• Libraries on top. You rarely write MediaSource by hand. Shaka Player (Google, open source, supports DASH and HLS), video.js, hls.js, dash.js, and Media Chrome (web components for the control UI) handle the messy parts.

A short history

Video on the web went Flash → <video> tag → MediaSource API. Each step traded simplicity for control. Flash was a plugin nightmare; the <video> tag was clean but limited; MediaSource gave engineers the fine-grained control streaming actually needs.

Containers, briefly

• WebM — modern, excellent compression (VP9/AV1), popular for online streaming, but not supported on every device.
• MP4 — the workhorse for storage, editing, broadcasting, and broad-compatibility streaming.

Why <video> + MediaSource (with a library) wins

The combination unlocks features the bare tag can’t touch:

• Automatically jumping gaps in the content buffer.
• Offline storage and DRM-protected content.
• Authentication on the stream.
• Preloading and cast support.
• HLS / MPEG-DASH / Microsoft Smooth Streaming support.
• Thumbnail previews while seeking.
• PWAs and service workers, network filters, automatic retries.

Drawbacks of the bare <video> tag

• Limited adaptive streaming support.
• Limited customisation.
• You often need two libraries kept in sync (playback engine + control UI).
• No support for advanced features like live streaming or scrubbing previews.

A note on protocols: HLS (Apple) and MPEG-DASH are the two dominant adaptive streaming standards today, with Microsoft Smooth Streaming as the older sibling. The manifest formats differ (.m3u8 vs .mpd), but the idea is identical — describe every rendition and let the player choose.

8. Performance

This is where a streaming frontend is actually won or lost.

Adaptive Bitrate Streaming (ABR)

The headline technique. The server stores the same video at several quality levels — the “bitrate ladder.” The player monitors the network and the buffer, and switches renditions on the fly: drop to 480p when the connection chokes, climb back to 1080p when it recovers. The user just sees uninterrupted playback. MediaSource is what makes mid-stream switching possible.

Media Capabilities API

navigator.mediaCapabilities.decodingInfo() tells you whether the device can not only play a given codec but decode it smoothly and power-efficiently. Use it to pick AV1 where it's well-supported and fall back to H.264 where it isn't, instead of guessing.

CSR vs. SSR

Netflix runs a hybrid: the metadata and initial content are server-rendered (fast first paint, SEO), while the interactive application renders on the client. You get a discoverable, quick-loading shell that then hydrates into a rich SPA.

SPA vs. MPA

Most large streaming sites end up as a combination of SPA and MPA — single-page fluidity for the browsing experience, multi-page boundaries where they help with load and isolation.

The optimisation checklist

• Lazy loading — only render rows near the viewport.
• Buffer only active video — don’t buffer players the user can’t see.
• CDN everywhere — serve segments from the edge as much as possible.
• Preload media — the preload attribute on the video tag for the hero.
• Resource hints — preconnect, dns-prefetch, and link-level preload to warm up the CDN connection early.
• Separate audio and video tracks — switch language or quality independently.
• Image & thumbnail optimisation — YouTube packs seek-preview thumbnails into sprite sheets so one request covers a whole timeline of previews.
• Avoid GIFs — use a short muted video instead; far smaller, far smoother.
• Player responsiveness — adapt the UI to screen size.

Bandwidth optimisation

• Selective autoplay — don’t autoplay everything on the page.
• Pause on non-visible tabs — stop pulling data when the tab is backgrounded.
• Buffer sufficiently — enough headroom to ride out a dip, not so much you waste data.

Memory optimisation

• Cap the buffer size and keep an eye on memory pressure.
• Revoke object URLs once segments are consumed.
• Prioritise and evict — finite memory means old segments have to go.

9. YouTube vs Netflix

This article has treated “YouTube and Netflix” as a single problem, but the two make very different frontend choices — because they’re solving for different content, business models, and audiences. Where their designs diverge is where the interesting decisions live.

Closing

The whole design comes back to that one constraint at the top: you can’t download the movie. Everything — chunked segments, manifests, the MediaSource API, object URLs, adaptive bitrate, the buffer-and-revoke memory dance — exists to stream a file too large to hold, while making it feel like it was already on the device.

Get the streaming layer right and the rest of the platform (the feed, the rows, the recommendations) is a comparatively ordinary frontend problem. Get it wrong and no amount of UI polish saves the experience.