Start practising

SYSTEM DESIGN · PATTERNS · 6 OF 7

Large Blobs: Uploading, Storing and Serving Big Files at Scale

How to handle photos, videos, documents and backups without choking your servers: split metadata from bytes, upload straight to object storage with presigned URLs, chunk and resume big uploads, confirm completion with storage events, serve through a CDN, process asynchronously, deduplicate by content hash, and control cost with storage tiers.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
1 / 15

Press play to hear the narration, or step through at your own pace.

Notes

The signals

The signal is any object measured in megabytes rather than bytes. Photo and video uploads, chat attachments, a file sync product, backups, datasets. Sizes range from a megabyte to terabytes, uploads come from flaky phone networks, and the same file is usually downloaded far more often than it was uploaded. Each of those facts shapes the design.

Signals in the question
  • upload photos, videos, docs
  • attachments in chat or email
  • file sync and sharing
  • backups, logs, ML datasets
  • object size: 1 MB to many TB
  • uploads from flaky mobile
  • networks
  • many more downloads than
  • uploads

The naive design

The naive design streams the upload through your app servers and stores it in a database column. Each upload ties up a server for minutes. You pay to move every byte twice. If the train enters a tunnel at ninety percent, the user starts over. Gigabyte rows bloat backups and replication. And every download comes back through the same servers. All of it is avoidable.

Why it breaks
  • app servers hold a thread and
  • memory per upload for minutes
  • bandwidth bill doubles: in to
  • you, then out to storage
  • connection drops at 90%?
  • start again from zero
  • DB rows of 2 GB: backups,
  • replication and caches suffer
  • every download hits origin

Move 1: split the two

Move one: separate the two kinds of data. The bytes go to object storage like S3, which is designed for eleven nines of durability and grows without you planning capacity. The database holds a small row per file: owner, name, size, type, a content hash, a status and the object key. Queries, permissions and listings use the row; nothing ever reads the bytes just to show a folder.

Metadata vs bytes
files(
  id, owner_id, name,
  size, content_type,
  sha256, status,
  object_key, created_at)

object storage: the bytes
  key: u/42/f/9a1c.mp4
  ~11 nines durability
  scales without capacity
  planning

Move 2: presigned URLs

Move two: presigned URLs. The client tells the API it wants to upload a two gigabyte file. The API checks permissions and quota, inserts a pending row, and signs a URL that allows a single upload to one specific key for fifteen minutes, with a size limit. The client sends the bytes directly to object storage. Your servers never touch a single byte of the file.

Presigned upload
  • 1 client asks the API to
  • upload (name, size, type)
  • 2 API checks auth and quota,
  • inserts row status=pending
  • 3 API signs a URL for one key,
  • valid 15 min, size capped
  • 4 client PUTs bytes directly
  • app servers touch zero bytes

Move 3: chunked upload

Move three: chunk it. Split the file into parts of, say, ten megabytes, two hundred parts for two gigabytes. Upload four to eight in parallel, each with its own signed URL and checksum. When the connection drops at part one hundred fifty, resume at one fifty-one, not at the start. A final complete call stitches the parts together. On S3, parts range from five megabytes to five gigabytes, up to ten thousand of them.

Multipart and resumable
2 GB / 10 MB parts = 200 parts
upload 4 to 8 in parallel
each part: own signed URL
  and checksum
drop at part 150? resume
  at 151, not 1
complete(list of part ETags)

S3 limits: part 5 MB to 5 GB,
  10,000 parts, object 50 TB

Completing the upload

How does the database learn the upload finished? Do not just trust the client to say so. Object storage emits an object created event onto a queue; a worker checks the size and hash and flips the row from pending to uploaded. The client can call complete too, but the event is the source of truth. A lifecycle rule cleans up abandoned multipart uploads, and a sweeper expires pending rows older than a day.

Closing the loop
  • storage emits "object created"
  • to a queue
  • worker verifies size + hash,
  • sets status = uploaded
  • client may also call /complete,
  • but the event is the truth
  • pending rows older than 24 h
  • → abort the upload, delete
  • orphaned parts (lifecycle)

Move 4: serve via CDN

Move four: serve downloads from a CDN. The API checks that this user may see the file and hands back a signed CDN URL or cookie that expires in an hour. The edge serves the bytes and goes to object storage only on a miss. Range requests let video players seek and broken downloads resume. Keep the bucket private, so the signed link is the only way in. Concepts episode twelve covers CDNs.

Downloads
  • API checks permission, returns
  • a signed CDN URL or cookie
  • CDN serves from the edge,
  • fetches origin once on a miss
  • Range requests: seek in video,
  • resume a broken download
  • public, immutable files:
  • versioned keys, long cache
  • bucket itself stays private

Move 5: process async

Move five: process asynchronously. The upload event lands on a queue, and workers do the slow jobs: scanning for malware, making thumbnails, transcoding video into several bitrates cut into short segments, extracting text for search. Outputs are written back as new objects and the row moves to ready. The user sees processing, then the file appears. Nothing slow happens inside the upload request.

Processing pipeline
  • upload event → queue → workers
  • virus and content scan
  • thumbnails for images
  • transcode video to several
  • bitrates, 4 to 10 s chunks
  • extract text for search
  • outputs written back as new
  • objects, rows updated
  • status: uploaded → ready

Dedupe and sync

For file sync products there is a further trick. Split files into blocks of a few megabytes and hash each one. The client sends the list of hashes first, and the server replies with the ones it already has. Only new blocks are uploaded. A file becomes an ordered list of block hashes in the metadata. Edit one page of a gigabyte file and you upload about four megabytes instead of the whole thing.

Dedupe and delta sync
split file into 4 MB blocks
hash each block (SHA-256)
client sends the hash list
server: "I already have 248
  of 250 blocks"
→ upload only 2 blocks

file = ordered list of block
  hashes in metadata
edit 1 page of a 1 GB file:
  upload ~4 MB, not 1 GB

Cost

Cost matters at petabyte scale. Standard object storage is roughly two cents per gigabyte per month; infrequent access and archive tiers cost a fraction of that but are slower or charge to read. Add lifecycle rules, like move to infrequent access after thirty days and archive after a year. Data transfer out often costs more than storage, which is one more reason to cache at the CDN.

Cost and lifecycle
  • standard tier: roughly 2 cents
  • per GB-month
  • infrequent and archive tiers:
  • a fraction of that, slower
  • or paid retrieval
  • rule: after 30 days → IA,
  • after 1 year → archive
  • egress often costs more than
  • storage: CDN caching helps
  • delete derived files you can
  • regenerate

Trade-offs

The trade-offs. Presigned URLs remove load from your servers but move retry and chunk logic into the client. Chunking makes uploads resumable at the price of more requests and bookkeeping. Async processing makes uploads fast but the file is not ready instantly. And cross-user dedupe saves storage but can reveal that a file already exists, so many products dedupe per user only.

Trade-offs
  • presigned URLs: less load,
  • but upload logic moves to
  • the client
  • chunking: resumable, more
  • requests and bookkeeping
  • async processing: fast upload,
  • file not ready instantly
  • dedupe: saves storage, leaks
  • "this file exists" unless
  • scoped per user

Where it shows up

You will use this pattern in Google Drive, with chunks, dedupe and sync; in YouTube and Netflix, with transcoding, segments and a CDN; in Instagram and TikTok for media upload; in WhatsApp and Slack for attachments; and even in Pastebin, where large pastes live in object storage while the database keeps just the metadata.

Problems that use it
  • Google Drive: chunks, dedupe,
  • sync
  • YouTube, Netflix: transcode,
  • CDN, adaptive bitrate
  • Instagram, TikTok: media
  • upload + CDN
  • WhatsApp, Slack: attachments
  • Pastebin: large pastes in
  • object storage

Say it like this

The phrasing. Bytes go to object storage, metadata to Postgres. The API hands out presigned URLs for a multipart upload in ten megabyte parts, so uploads are resumable and never pass through our servers. The storage event marks the row uploaded, workers generate thumbnails or transcodes, and downloads go through a CDN with signed, expiring URLs.

In the interview
  • "Bytes go to S3, metadata to
  • Postgres. The API returns
  • presigned URLs for a 10 MB
  • multipart upload, so it is
  • resumable and never passes
  • our servers. The storage
  • event marks the row
  • uploaded, workers make
  • thumbnails, and a CDN
  • serves signed downloads."

Interview questions

Why not store files in the database?

Large binary rows bloat backups, replication and buffer caches, and serving them ties up database connections. Object storage is cheaper, more durable and built for large sequential reads, so the database should hold only the metadata and the object key.

What is a presigned URL?

A URL signed by your service that grants temporary permission for one operation, such as PUT or GET on one object key, with an expiry and optional size or type constraints, so clients can talk to object storage directly without your credentials.

How do you make large uploads resumable?

Use multipart upload: split the file into parts, upload them independently with checksums, record which parts succeeded, and on reconnect upload only the missing parts before calling complete with the part list.

How does the backend know an upload has finished?

Subscribe to the object storage creation event through a queue and have a worker verify size and hash before marking the metadata row uploaded. A client complete call can speed this up but should not be the only signal.

How would you serve private files through a CDN?

Keep the bucket private, check permissions in the API, and return a short-lived signed CDN URL or signed cookie. The CDN validates the signature at the edge and caches the object for later authorised requests.

How do file sync services avoid re-uploading unchanged data?

They split files into blocks, hash each block, and send the hash list first. The server replies with the blocks it lacks, so only changed or new blocks are uploaded, and a file is stored as an ordered list of block hashes.

How do you keep storage costs under control?

Apply lifecycle rules that move old objects to infrequent-access and archive tiers, delete abandoned multipart uploads and regenerable derivatives, deduplicate where privacy allows, and cache downloads at a CDN to reduce egress.