SSAI Reimagined on Google Cloud: Designing a per-user streaming system at scale

Server-Side Ad Insertion (SSAI) has evolved significantly over the past decade. Traditional SSAI systems focused on stitching ads into a shared stream, often sacrificing personalization and scalability. Modern requirements demand:

• Per-user ad experiences
• Low latency live streaming
• High concurrency (millions of viewers)
• Fault-tolerant ad delivery

In this article, we explore a reimagined SSAI architecture built entirely on Google Cloud, designed for per-user playlists, scalable ad insertion, and graceful degradation under failure.

Core Design Principles

This system is built on a few key principles:

1. Per-user playlist generation
2. Stateless compute, stateful edge (Redis)
3. Asynchronous ad fetching and transcoding
4. Fallback-first architecture
5. Sliding window live playback

High-Level Architecture

End-to-end SSAI pipeline with per-user playlist generation, async ad decisioning, and Redis-backed low-latency ad serving

This system separates the playback path from ad decisioning to ensure low latency and high reliability. The manifest manipulator generates per-user playlists with embedded redirections. Ad fetching and transcoding happen asynchronously, while the ad server uses a low-latency cache (Redis) to serve replacement segments in real time. If ads are not ready, the system seamlessly falls back to the original content, ensuring uninterrupted playback.

The system is composed of the following core components:

• Manifest Manipulator (/getMaster)
• Manifest Manipulator (/getVariant)
• Ad Fetcher (/Fetch)
• Ad Decision Engine (/getMyAd)
• Ad Server (/Serve)
• Transcoder (/Transcode)
• Quartile Tracker (/QFire)

All services are deployed using:

• Cloud Functions
• Cloud Tasks
• Cloud Run Jobs
• Firestore (Document DB)
• MemoryStore (Redis)

Data Model-Persistence vs. Real-Time Performance

To achieve sub-10ms manifest manipulation, we split our data between Firestore (for long-term state and metadata) and Memorystore (for the high-speed playback edge)

Document DB: The Persistence Layer (Firestore)

Firestore acts as our “Source of Truth” for stream configurations and ad assets.

1. Live Stream Metadata StreamID: Unique identifier for the ingest source. MasterURL: The original source manifest. Variants: Array of available bitrates (e.g., 360p, 720p, 1080p) and their source paths.
2. User Session State UID / UserIP: Identity and location for ad targeting. PlaybackContext: Last served segment and timestamp (used for session recovery).
3. Asset Transcode Registry (Keyed by MD5_Hash(MediaURL)) Transcode Status: [Pending | Started | Failed | Succeeded] Metadata: File duration, segment duration, and source MP4 path. SegmentMap: A nested list mapping Variant_ID to the actual storage path (GCS/S3) for each .ts file.

Memorystore: The High-Concurrency Edge (Redis)

We use Redis to eliminate the “Firestore Latency Tax” during the critical path of segment requests.

1. Playback State Key Pattern: play:{StreamID}:{UID} Value Structure: {last_seq, last_request_time} Purpose: Tracks the user’s sliding window position to ensure we serve N segments behind the live edge.
2. Ad Mapping Key Pattern: ad:{StreamID}:{Variant}:{UID}:{Seq} Value Structure: {transcoded_path, quartile_url} Purpose: The “Stitcher” uses this to instantly swap a content segment for a personalized ad segment.

Playback Flow

Actors and role play, what does each service does and how !

Manifest manipulator — Serving Master (/getMaster)

Serves modified master and variant playlist(s). SSAI usually decorates these playlists embed User, Variant and Fallback stream information. This information is subsequently used to fetch & replace ads, serve fallback in case of down-time. Query Params : Ingest-ID

1. Create User ID, SSAI will create/track new user when Master playlist is requested
2. Serves master playlists with different variant playlists as in input by adding User ID, Ingest ID , Variant bitrate as query parameters
3. Every variant will also have input variant as query parameter fallback, this way manipulator doesn’t need to remember which variant it is serving , also helps to serve fallback via redirection in case manifest manipulator is down.

#EXTM3U
#EXT-X-VERSION:6

# Variant 1 – 360p
#EXT-X-STREAM-INF:BANDWIDTH=800000,AVERAGE-BANDWIDTH=600000,RESOLUTION=640x360,FRAME-RATE=25.000,CODECS="avc1.42e01e,mp4a.40.2"
https://ssai.example.com/getvariant?ingestId=stream123&variant=360p&uid=s101&fallback=https://ingest.example.com/live/360p/index.m3u8

# Variant 2 – 480p
#EXT-X-STREAM-INF:BANDWIDTH=1400000,AVERAGE-BANDWIDTH=1100000,RESOLUTION=854x480,FRAME-RATE=25.000,CODECS="avc1.4d401f,mp4a.40.2"
https://ssai.example.com/getvariant?ingestId=stream123&variant=480p&uid=s101&fallback=https://ingest.example.com/live/480p/index.m3u8

# Variant 3 – 720p
#EXT-X-STREAM-INF:BANDWIDTH=2500000,AVERAGE-BANDWIDTH=2000000,RESOLUTION=1280x720,FRAME-RATE=25.000,CODECS="avc1.64001f,mp4a.40.2"
https://ssai.example.com/getvariant?ingestId=stream123&variant=720p&uid=s101&fallback=https://ingest.example.com/live/720p/index.m3u8

# Variant 4 – 1080p
#EXT-X-STREAM-INF:BANDWIDTH=5000000,AVERAGE-BANDWIDTH=4200000,RESOLUTION=1920x1080,FRAME-RATE=25.000,CODECS="avc1.640028,mp4a.40.2"
https://ssai.example.com/getvariant?ingestId=stream123&variant=1080p&uid=s101&fallback=https://ingest.example.com/live/1080p/index.m3u8

Manifest manipulator — Serving Variant (/getVariant)

Variant serving flow serves delayed playlist(s), delay helps fetch and replace ads for impending ad-breaks. Variant playlist shall included redirections to ad-fetcher and ad-replacer in sliding window being served. Query params: Ingest ID, User ID, Variant ID

1. Serves variant playlists with redirections to fetcher and to replacer /serve (during ad-break). Serves 2 segments behind Live (if Live is at 102, serves 100 to requesting player)
2. Manipulator shall serve variant playlist 2 segments behind live. Delta = floor ((Now — Last_request_time) / segment_duration) candidate_seq = last_served_seq + delta candidate_seq = candidate_seq — 2 // 2 segments behind live next_seq = min(candidate_seq, effective_live_edge)
3. Fetcher(/fetch) redirections fetches ads for { User ID , User IP } for a given Ingest ID from ad network
4. Redirection Server (/serve) serves replacement ad segment fetched for user, if not found redirects to input ad segments
5. Manipulator updates last requested time and last requested segment per user in document DB or in memory store (millions of concurrent users will require that).

#EXTM3U
#EXT-X-VERSION:6
#EXT-X-TARGETDURATION:5
#EXT-X-MEDIA-SEQUENCE:200

# --- MAIN CONTENT ---
#EXTINF:5.0,
[https://cdn.example.com/live/main200.ts?uid=s101](https://cdn.example.com/live/main200.ts?uid=s101)
#EXTINF:5.0,
[https://cdn.example.com/live/main201.ts?uid=s101](https://cdn.example.com/live/main201.ts?uid=s101)

# --- AD PREP (Pre-fetch / Buffer phase) ---
#EXTINF:5.0,
[https://ads.example.com/fetch?dur=10&uid=s101&adSeq=204&fallback=https://cdn.example.com/live/main202.ts?uid=s101](https://ads.example.com/fetch?dur=10&uid=s101&adSeq=204&fallback=https://cdn.example.com/live/main202.ts?uid=s101)
#EXTINF:5.0,
[https://cdn.example.com/live/main203.ts?uid=s101](https://cdn.example.com/live/main203.ts?uid=s101)

# --- AD BREAK START ---
#EXT-X-DISCONTINUITY
#EXT-X-CUE-OUT:DURATION=10

#EXTINF:5.0,
[https://ads.example.com/serve?variant=360p&uid=s101&seq=204&fallback=https://cdn.example.com/live/main204.ts?uid=s101](https://ads.example.com/serve?variant=360p&uid=s101&seq=204&fallback=https://cdn.example.com/live/main204.ts?uid=s101)
#EXTINF:5.0,
[https://ads.example.com/serve?variant=360p&uid=s101&seq=205&fallback=https://cdn.example.com/live/main205.ts?uid=s101](https://ads.example.com/serve?variant=360p&uid=s101&seq=205&fallback=https://cdn.example.com/live/main205.ts?uid=s101)

#EXT-X-CUE-IN
#EXT-X-DISCONTINUITY

# --- RESUME MAIN CONTENT ---
#EXTINF:5.0,
[https://cdn.example.com/live/main206.ts?uid=s101](https://cdn.example.com/live/main206.ts?uid=s101)
#EXTINF:5.0,
[https://cdn.example.com/live/main207.ts?uid=s101](https://cdn.example.com/live/main207.ts?uid=s101)

Ad Fetcher- Trigger Ad fetch (/fetch)

This service shall queue cloud task to fetch ads and redirect back to input content segment Query Params: Adv dur, fallback URL, Start-Seq, UID, Ingest-ID, User-IP

1. Queue a Cloud task (onTaskDispatched , no delay) , send advert request for ad duration , UID, User-IP parameters. Meat of work is called as /getMyAd below
2. 307 redirect to input fallback URL

Ad Fetcher- Trigger Ad fetch (/getMyAd)

This service shall send VAST request to Ad-Server or SSP follows waterfall or simply sends a single request to a Header Bidding Multiplexer. Query params :Stream ID, UID, U-IP, ad Start seq , ad duration

1. Engage with Ad-Server/SSP and Get and parse VAST response, Vast response has mediaURL for advertisement video.
2. Transcoder Cache Hit : Gets transcoded segments from Transcoder DB for mediaURL, if found add memory store for replacement segment(s)

• Key = {StreamID_Variant_UID_AdSeg_Seq}, Value = {Transcoded segment path, Quartile URL to be fired}
• Above key values shall be added for each variants, every segment
• Return Success

3. Transcoder Cache Miss: Advertisement is not pre-transcoded, queue transcoding.

• *Transcoding job shall be queued with unique ID same as MediaURL hash. This way same job can’t be de-duplicated, even if millions of users get same ad response
• Return Success*

Replaced Advertisement Server (/serve)

This service reads memory store for replacement advert segment for a given user, variant and segment seqeuence. Query Params : Stream ID , UID, Seg-Sequence, Variant, Fallback URL

1. Fetch replacement segment from Memorystore for StreamID_Variant_UID_AdSeg_Seq combination , if not found redirect to input fallback URL
2. When fetch is successful, queue cloud task for firing quartile URL
3. Redirect to fetched replacement ad segment.

Quartile Tracker (/QFire)

Fires quartiles , this is done as soon as player downloads ad-segment. Query Params: URL to be fired

1. SSAI lived long with limitation where in quartiles were fired from server side, this doesn’t mean advertisement is watched, it only meant advertisement segment is downloaded.
2. Nowadays SGAI is used to fire quartiles from client player. ( May be another time)

Transcoder (/Transcode)

Transcodes advertisment in set of resolutions(360p,480p, 720p, 1080p) Query Params : mediaURL, streamID,

1. Retrieve the variants expected for streamID from Document DB
2. Queue transcoding job for mediaURL
3. Once transcoding is complete successfully add record for transcoded file segments, record structure covered in Data model
4. Key for record added shall be Haskhey for mediaURL

Transcoder JOB

1. This is a Google Cloud Run Job
2. Cloud run job runs a dockerized ffmpeg engine that use MP4 URL , downloads transcodes and uploads to Google store or S3 bucket
3. Bucket may be behind a CDN for caching

Key Innovations

1. Per user SSAI: Each user gets unique playlist, personalized ads and independent playback state
2. Sliding Window + Delay Buffer: Serving 2 segments behind live allows ad-decision time and absorbs latency for building replacement ad segments
3. Redis store for playback state: Redis read latency of order of few m seconds ensures millions of concurrent viewers
4. Async everything: Ad fetching , transcoding, quartile firing all async. This ensures no blocking in playback path and gives predictable latency.
5. Fallback- First Design: Ad not ready → play in-stream slates, Transcoder slow → play content, Ad network fails → play content. Thus, playback never fails

Scaling Considerations

1. Horizontal Scale**

• **Cloud Functions auto-scale
• Redis handles high QPS reads

2. Memory Strategy

• TTL-based eviction for ad mappings
• Short-lived keys reduce footprint

3. Transcoding Deduplication

• Prevents duplicate jobs under load
• Critical during high concurrency events

Handling Failures

System is intended to handle failure, show must go on. Live stream can’t go down in any situation

Fallback pattern

Future Enhancements

1. SSHB (Server-Side Header Bidding) instead of waterfall
2. Client-side quartile firing (SGAI) via EXT-X-DATERANGE
3. CDN-aware caching strategies
4. Dynamic ad pod optimization
5. Real-time analytics pipeline

Conclusion

This architecture demonstrates how SSAI can be reimagined for modern scale

1. Fully cloud-native
2. Designed for per-user personalization
3. Built for millions of concurrent streams
4. Resilient by design

By combining Redis for speed, Firestore for durability, and Cloud Tasks for orchestration, we achieve a system that is both scalable and fault-tolerant.