Design YouTube: Resumable Uploads, a Transcoding DAG, Adaptive Streaming and View Counts

Listen

Requirements. Creators upload videos, possibly many hours long and many gigabytes. We process each into formats every device can play. Viewers watch on any screen and any network, and can jump to any point. The watch page shows title, channel, view count and likes. Recommendations, live streaming and comments are out of scope today; mention that the recommendation design looks like the TikTok funnel.

Qualities. Playback should start within a second or two and rarely stall. Uploads must survive flaky connections; nobody wants to restart a ten-gigabyte upload at ninety percent. A new video should be playable within minutes, at least in common resolutions. Originals are never lost. View counts must be approximately right and resistant to inflation. And cost matters, because bandwidth will be the biggest bill by far.

Ask scale

Uploads: YouTube has publicly said more than five hundred hours of video arrive every minute. That's about seven hundred twenty thousand hours a day. At around two gigabytes per hour for an HD original, that's more than a petabyte of originals daily, and the renditions roughly double it. That's on the order of an exabyte a year, so storage must be tiered: hot for popular videos, cold for the huge tail nobody watches.

Viewing: eight hundred million daily viewers watching forty minutes each at an average of two and a half megabits a second. Work it through and that's about six hundred petabytes a day, roughly fifty-five terabits per second on average. No data center serves that. The design has to make the CDN do almost all the work, with the origin seeing only a small percentage of requests.

upload rate   ~500 hours/min
  x 1,440 min = 720K hours/day
original      ~2 GB per hour
  = ~1.4 PB/day of originals
renditions    ~1x original more
  = ~3 PB/day total
  x 365       = ~1 EB/year

-> tiered storage: hot, warm,
   cold for old, unwatched

DAU                 800M
watch time/user     40 min/day
  = 1.9 trillion s of video
avg bitrate         2.5 Mbps
  = 4.8e18 bits/day
  = ~600 PB/day
  / 86,400 s = ~55 Tbps avg

video starts: 4B/day = 46K/s
-> CDN; origin sees a few %

Define API

The API. An upload starts by creating a session, which returns a session URL. The client sends the file in pieces with byte ranges. If the connection drops, it asks the session for the last byte received and continues from there. The watch page returns metadata and a manifest URL. The player fetches the manifest and segments from the CDN. And the player sends playback events, which drive view counts and quality metrics.

POST /v1/uploads
  { size, title } -> { sessionUrl }
PUT  sessionUrl
  Content-Range: bytes 0-8M/9.8G
GET  sessionUrl -> next offset
GET  /v1/videos/{id}
  -> { title, views, manifest }
GET  {cdn}/v/{id}/master.m3u8
POST /v1/playback/events
  { videoId, pos, state }

Data model

Data model. A video row has its channel, title, duration, visibility and a processing status. Each rendition row describes a codec, resolution and bitrate and points at its manifest. Segments are files in object storage with predictable keys. Counts live separately because they change constantly. Metadata is small enough for sharded SQL, and it's read far more than written, so a cache in front absorbs most reads.

video   (videoId, channelId,
         title, duration, status,
         visibility, createdAt)
rendition (videoId, codec, height,
         bitrate, manifestKey)
segment  key: v/{id}/{rend}/{n}.m4s
channel (channelId, ownerId)
counts  videoId -> views, likes

metadata: sharded SQL or
  wide-column, heavily cached

High-level design

The upload path. The creator's file streams into an upload session and lands as the original in raw storage. When the upload completes, a pipeline planner inspects the file and builds a graph of jobs: split, encode each segment in each rendition, make thumbnails, extract audio for captions. Workers run those jobs in parallel and write the results to video storage, and the planner updates the video's status as stages finish.

The watch path. The viewer's app fetches the video page from the watch API, which reads cached metadata. Then the player pulls the manifest and segments from a nearby CDN edge. On an edge miss, the request goes to a regional origin shield, and only if that misses too does it reach video storage. Meanwhile the player sends playback events into a stream, which feeds the view counter and quality dashboards.

Deep dive: processing

Deep dive one: processing. Encoding a two-hour video in one piece would take hours. Instead, split it at keyframes into short chunks, and encode each chunk for each rendition as an independent job. Hundreds of workers finish in minutes. The planner treats this as a DAG: split before encode, encode before stitching the manifests. Publish as soon as a couple of common resolutions are ready, and let higher resolutions and more efficient codecs arrive later.

Make the pipeline reliable. Jobs sit on a durable queue. Each worker takes a lease on a job; if it crashes, the lease expires and another worker retries. Every job writes to a key determined by video, rendition and segment, so a retry simply overwrites the same output. A corrupt file fails after a few attempts and is marked failed rather than retried forever. Because jobs are small and retryable, you can run them on cheap preemptible machines.

Deep dive: streaming

Deep dive two: streaming. The master manifest lists every rendition. Each one is a sequence of segments a few seconds long. The player measures how fast segments arrive and how full its buffer is, and picks the bitrate for the next segment. When your train enters a tunnel, it drops to four eighty p instead of stalling. Seeking just means requesting segment number N. Start at a lower rung for a quick first frame and climb as the buffer fills.

The CDN has to handle a long tail. A small fraction of videos gets most views, and those sit in every edge cache. But billions of rarely watched videos can't be everywhere, so a regional origin shield catches misses from many edges, and only its misses reach storage. Big platforms also place cache servers inside internet providers' networks to cut transit cost. Before a huge premiere, prewarm the edges.

Deep dive: views

Deep dive three: view counts. A view isn't a page load; it's a playback event that passes rules, such as watching past a threshold. Events flow through a stream partitioned by video. A filter drops bots and repeats from the same viewer within a window. An aggregator sums per minute and writes one update per video, instead of millions of increments to one hot row. The displayed count is cached and may lag, and because the raw log is kept, counts can be audited and recomputed.

Trade-offs

Failures and monitoring. If encoding backs up, publish low resolutions first and autoscale workers on queue depth. If a CDN edge fails, traffic shifts to the next one through DNS or anycast. If a shield gets hammered by misses for one popular segment, coalesce requests so only one goes to storage. The key metrics are viewer-side: time to first frame and rebuffering ratio, plus encode lag and the bandwidth bill.

Trade-offs. Every extra rendition and codec costs storage and compute, but saves bandwidth and improves playback. Efficient codecs like AV1 are expensive to encode, so spend that only on videos that will be watched a lot. Approximate, delayed view counts trade freshness for scale and fraud resistance. At ten times the scale, push more caches into provider networks and move old, unwatched videos to cold storage, keeping only a few renditions.

Say it like this

The summary. Uploads are resumable sessions into object storage. A pipeline builds a DAG of small idempotent jobs, splitting at keyframes and encoding segments in parallel, so common resolutions publish in minutes. Playback uses HLS or DASH, with the player adapting bitrate per segment, served from CDN edges backed by regional shields. Views are playback events that are filtered and aggregated per minute. Metadata is small, relational and cached.