Notes
The problem
The problem. Big files in a database row bloat every backup, every replica and every cache. Files on an app server's local disk vanish when that server dies and can't be shared. And streaming gigabytes through app servers ties up threads for minutes. You want storage that's cheap, effectively unlimited, extremely durable and served over plain HTTP. That's an object store.
- a 2 GB row bloats backups,
- replication and caches
- local disks: lost when the
- server dies, not shared
- app servers streaming GBs:
- tied-up threads, timeouts
- want: cheap, near-infinite,
- durable, served over HTTP
The model
The model is simple. A bucket holds objects, each addressed by a key like videos slash user forty-two slash clip ninety-one. An object is bytes plus metadata: size, content type, a checksum called the ETag, your own tags. You can put, get, delete, and list by prefix. Gets support byte ranges, which is how video seeking works. You can't edit part of an object; you replace it. And folders are an illusion; the slash is just a character.
bucket / key → object videos/u42/2026/clip-91.mp4 object = bytes + metadata (size, type, ETag, tags) PUT, GET (with Range), DELETE, LIST by prefix no in-place edit or append: rewrite the whole object flat namespace; "/" is just text
Inside
Inside, there are two very different halves. The metadata service maps bucket and key to the object's version, size, checksum and the list of chunks that make it up; it's a sharded, replicated key-value store. The data nodes are simple machines with lots of big disks that store chunks by id. Stateless front ends check authentication and stream bytes between the two. Each half scales on its own.
- metadata service:
- bucket + key → version,
- size, checksum, chunk list
- sharded, replicated KV store
- data nodes:
- dumb, huge disks, store
- chunks by chunk id
- front ends: auth, signatures,
- stream bytes, stateless
Write path
A put, step by step. The front end checks the signature, splits the stream into chunks, and a placement service picks data nodes in three different zones for each chunk. Each chunk is written and checksummed. Only when every chunk is durable is the metadata record written, pointing at those chunks, and the client gets 200 with the ETag. Committing metadata last means a reader never sees half an object.
1 auth + signature check 2 split stream into chunks (e.g. 64 MB) 3 placement picks nodes in 3 zones per chunk 4 write + checksum each chunk 5 all durable → write metadata 6 return 200 + ETag metadata last = no half object
Durability
How is each chunk protected? Simple replication keeps three full copies: three times the storage. Erasure coding is what large stores use for most data. Split a chunk into ten pieces and compute four parity pieces, spread all fourteen across machines and zones. Any ten of the fourteen rebuild the chunk. That survives four failures with only one point four times the storage, at the cost of more CPU and slower rebuilds.
- 3 full copies:
- 3.0x storage, fast rebuild,
- survives 2 lost copies
- erasure code, e.g. 10 + 4:
- split into 10 data pieces,
- compute 4 parity pieces
- any 10 of 14 rebuild it
- 1.4x storage, survives 4
- more CPU, slower rebuild
Presigned uploads
Here's the move interviewers want to hear. The client asks your app for permission to upload. The app checks it, picks a key, and returns a presigned URL: a signed link that allows exactly one put to that key, for fifteen minutes, up to a size limit. The client uploads directly to the object store, and an event notification tells your app it's done. Your servers never touch the bytes.
- 1 client asks app for upload
- 2 app checks permissions,
- returns presigned URL
- (key, expiry, size limit)
- 3 client PUTs to storage
- 4 storage event or client
- callback → app saves key
- app never touches the bytes
Multipart
For big files, use multipart upload. Start an upload and get an id, then upload parts in parallel, each from five megabytes to five gigabytes, and retry only the part that failed on a flaky connection. Completing the upload stitches the parts into one object. That's how a single object can reach terabytes. Set a lifecycle rule to abort abandoned uploads, because their parts are billed until you do.
init → upload_id PUT part 1..N (5 MB - 5 GB, max 10,000 parts) in parallel retry only the failed part complete(upload_id, ETags) → one object objects up to terabytes abort stale uploads with a lifecycle rule (they cost money)
Failures
With millions of disks, several fail every day, so repair is a constant background job. When a disk dies, the system finds every chunk that had a piece there and rebuilds it elsewhere from the surviving pieces. Scrubbers re-read data and verify checksums to catch silent corruption. If a whole zone goes dark, reads rebuild from the other zones. The faster repair runs, the smaller the window where a second failure could hurt.
- disk dies: its chunks are
- re-created elsewhere from
- the surviving pieces
- silent corruption: scrubbers
- re-read and checksum data
- in the background
- zone down: reads rebuild
- from the other zones
- faster repair = fewer losses
Durability math
S3-class stores advertise eleven nines of durability. Put concretely: store ten million objects and you'd expect to lose one about every ten thousand years. That comes from spreading pieces across three or more zones, repairing quickly, and checksumming everything. What it doesn't cover is you deleting the object or a bug overwriting it. For that, turn on versioning, object lock for compliance, and cross-region replication for disasters.
- "11 nines" = 99.999999999%
- → store 10M objects, expect
- to lose 1 every ~10,000 yrs
- it comes from: pieces in
- 3+ zones, fast repair,
- checksums everywhere
- it does NOT cover: you
- deleting it → versioning,
- object lock, cross-region copy
Serving and tiers
For reads, put a CDN in front, so popular objects are served from the edge and the store only sees misses; private files use signed URLs. Then think about cost over time. The standard tier costs roughly twenty dollars per terabyte per month. Infrequent-access tiers are about half, with a fee per read. Archive tiers are a few dollars, but restores take minutes to hours. Lifecycle rules move objects down as they age.
- reads: put a CDN in front;
- private files via signed URLs
- Range GETs for video seeking
- hot (standard): ~$20/TB/mo
- infrequent: ~half, fee on read
- archive: ~$1-4/TB/mo,
- minutes-hours to restore
- lifecycle rules move by age
Gotchas
A few gotchas. S3 now gives strong read-after-write consistency, but don't assume every S3-compatible store does. Request rates are limited per key prefix, a few thousand writes a second each, so spread very hot paths across many prefixes. And listing millions of keys is slow and paged, so never treat it as a query. Keep your own table of objects in a database: key, owner, size, status. The store holds bytes; your database holds meaning.
- S3 today: strong read-after-
- write for PUT and LIST
- per-prefix limits (S3: ~3,500
- writes, 5,500 reads /s); spread
- hot paths over many prefixes
- LIST is slow for millions:
- keep your own index of keys
- in a database, with owner,
- size, status
Sizing
Sizing. A million uploads a day at five megabytes each is five terabytes a day, about one point eight petabytes a year. At about twenty dollars a terabyte, that's roughly thirty-six thousand dollars a month by the end of the first year, which is why lifecycle rules matter. Upload bandwidth is five terabytes over a day, about fifty-eight megabytes a second, and three times that at peak, around one point four gigabits.
1M uploads/day × 5 MB = 5 TB/day → ~1.8 PB/year cost (standard ~$20/TB/mo): year 1 end ≈ $36k/month move >90 days to archive upload bandwidth: 5 TB / 86,400 s ≈ 58 MB/s peak 3x ≈ 175 MB/s ≈ 1.4 Gbps
Say it like this
In the interview, say one sentence first: blobs go in object storage, metadata goes in our database. Then the upload flow: presigned URL, multipart for big files, an event marks the object ready. Downloads through a CDN with signed URLs. Durability from erasure coding across zones, versioning against mistakes, and lifecycle rules for cost. If asked how it works inside, describe the metadata service and data nodes.
- "Blobs in object storage,
- metadata in our database"
- upload: presigned URL,
- multipart for big files,
- event → mark ready
- download: CDN + signed URLs
- durability: EC across zones,
- versioning for mistakes
- cost: lifecycle to archive