Start practising

SYSTEM DESIGN · TECHNOLOGIES · 10 OF 12

Object Storage: How S3-Style Blob Stores Hold Media, Backups and Data Lakes

How S3-style object storage works: buckets, keys and immutable objects, metadata and erasure-coded data behind the API, strong read-after-write consistency, presigned URLs for direct uploads, multipart uploads, per-prefix request rates, storage classes and lifecycle rules, events and CDNs, where it hurts, the numbers worth knowing and how to use it in an interview.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
1 / 16

Press play to hear the narration, or step through at your own pace.

Notes

What it is

Object storage keeps any blob of bytes under a key, reached over HTTPS. You never provision disks or think about capacity; it just grows. Amazon S3 is the reference, and most others offer an S3-compatible API, so tools work across them. You pay for gigabytes stored per month, for each request, and for data transferred out to the internet.

In one line
  • store any blob by key over
  • HTTPS; no servers, no disks,
  • no size planning
  • S3, Google Cloud Storage,
  • Azure Blob, Cloudflare R2,
  • MinIO / Ceph (self-hosted)
  • pay per GB-month, per
  • request and per GB out

Data model

The model is tiny. A bucket holds objects. Each object has a key, its bytes, up to fifty terabytes on S3, and metadata like content type. There are no real folders: the slashes are just characters in the key, and listing by prefix makes them look like directories. Objects are immutable: you cannot edit the middle of a file or append to it; you write a new version of the whole object.

Buckets, keys, objects
bucket: media-prod
key:    videos/u42/2026/10/abc.mp4
object: bytes (up to 50 TB)
        + metadata (Content-Type,
          custom x-amz-meta-*)
        + version id (if enabled)

no real folders: "/" is just
part of the key; LIST by prefix
no partial update: rewrite whole

Behind the API

Behind the API there are roughly three layers. Front ends handle authentication and routing. A metadata index, partitioned by key, maps each key to its version, size and where its chunks live. Storage nodes hold the data, split into data chunks plus parity chunks using erasure coding and spread across at least three availability zones. Losing several chunks, or a whole zone, still leaves enough pieces to rebuild the object. That is where the eleven nines come from.

How it is stored
  • front end: auth, routing
  • metadata index: key →
  • version, size, chunk
  • locations (partitioned
  • by key prefix)
  • storage nodes: data split
  • into k data + m parity
  • chunks (erasure coding)
  • across 3+ zones
  • lose m chunks: still readable

Consistency

Consistency used to be an interview gotcha, but since late twenty twenty S3 has been strongly consistent. Once a write succeeds, the next read and the next listing see it, including overwrites and deletes. Google and Azure behave the same way. What you still do not get is locking: two writers to the same key means the last one wins, unless you use conditional writes that succeed only if the object is absent or unchanged.

Consistency today
  • S3 since Dec 2020: strong
  • read-after-write for PUT,
  • overwrite, DELETE and LIST
  • same for GCS and Azure Blob
  • still no locking: two
  • writers to one key →
  • last write wins
  • use conditional writes
  • (If-None-Match / If-Match)

Presigned uploads

The most important pattern: presigned URLs. Do not stream gigabytes through your own servers. The client asks your API to upload; the API checks permissions, chooses the key and returns a URL signed for, say, fifteen minutes. The client sends the bytes directly to object storage. Your servers handle a tiny request, and storage absorbs the heavy traffic. Downloads of private files work the same way.

Presigned URLs
  • 1. client: "I want to upload"
  • 2. API checks auth, picks a
  • key, signs a URL valid
  • for e.g. 15 minutes
  • 3. client PUTs bytes straight
  • to the object store
  • 4. store emits an event
  • your servers never touch
  • the gigabytes

Multipart uploads

For large files, use multipart upload. Start an upload and get an id, then send parts of five megabytes to five gigabytes each, in parallel, and finally complete it with the list of parts. A failed part is retried alone, which matters on phone networks. A single put is capped at five gigabytes, and multipart is recommended above about a hundred megabytes. Add a lifecycle rule to clean up uploads that were never completed, or you pay for orphaned parts.

Multipart upload
1. CreateMultipartUpload → id
2. UploadPart 1..N in parallel
   part size: 5 MB - 5 GB
   max parts: 10,000
3. CompleteMultipartUpload
   (list of part ETags)

single PUT max: 5 GB
use multipart above ~100 MB
lifecycle: abort stale uploads

Events + CDN

Two things usually sit around the bucket. Event notifications: when an object is created, a message goes to a queue or a function, which generates thumbnails, transcodes video into several resolutions or scans for viruses. And a CDN in front for reads, so popular files are served from the edge, with fewer storage requests and lower egress bills. For private content, the CDN checks signed URLs or cookies. See the CDN building block.

Around the bucket
  • events: ObjectCreated →
  • queue or function →
  • thumbnails, transcoding,
  • virus scan, indexing
  • CDN in front for reads:
  • edge hit = no storage
  • request, lower egress cost
  • private files: signed
  • CDN URLs or cookies

Request rates

Throughput scales by key prefix. S3 supports at least three thousand five hundred writes and five thousand five hundred reads per second per prefix, and there is no limit on the number of prefixes, so spreading keys across ten prefixes gives ten times the rate. Latency to the first byte is tens of milliseconds, far slower than a database. For big files, fetch byte ranges in parallel to get very high throughput.

Request limits (S3)
  • per prefix, per second:
  • 3,500 PUT/COPY/POST/DELETE
  • 5,500 GET/HEAD
  • no limit on number of
  • prefixes: 10 prefixes =
  • 10x the rate
  • first byte: ~10-100 ms
  • big files: parallel ranged
  • GETs for throughput

Classes + lifecycle

Storage classes trade storage price against access cost. Standard is for hot data, about two cents per gigabyte a month in the main US region. Infrequent access is cheaper to store but charges per gigabyte read. Glacier tiers are archives, and the deepest costs about a tenth of a cent per gigabyte but takes hours to restore. Lifecycle rules move objects automatically, for example to infrequent access after thirty days and to an archive after six months.

Storage classes (S3)
Standard          hot, ~$0.023/GB-mo
Standard-IA       infrequent, + fee
                  per GB read
Intelligent-Tier  moves it for you
Glacier IR        archive, ms reads
Glacier Deep      ~$0.001/GB-mo,
                  hours to restore

lifecycle: IA after 30 days,
  Glacier after 180, delete at 7 yr

Great at

Object storage is great for anything large and written once: user photos and videos, attachments, static website assets behind a CDN, database backups and log archives, and data lakes, where analytics engines like Spark, Athena or Trino query Parquet files in place. S3 is designed for eleven nines of durability, which in practice means you will not lose an object to hardware failure.

Where it shines
  • • user media: photos, video,
  • attachments, documents
  • • static sites and assets
  • (behind a CDN)
  • • backups, DB snapshots, logs
  • • data lakes: Parquet files
  • queried by Spark, Athena,
  • Trino, ClickHouse
  • • designed for 11 nines
  • durability

Where it hurts

Where it hurts. You cannot append to or edit part of an object. There is no query language: listing by prefix returns a thousand keys per page and gets slow on huge buckets, so keep your own index of objects in a database. Every request takes tens of milliseconds, which is bad for millions of tiny, hot objects. Request fees add up with small files. And transferring data out to the internet is a large part of many bills.

Pain points
  • • no append or partial edit
  • • no queries: LIST by prefix
  • only, and it is slow
  • (1,000 keys per page)
  • • tens of ms per request:
  • bad for tiny, hot objects
  • • per-request cost for
  • millions of small files
  • • egress to the internet
  • is expensive

Use it when

The rule in interviews: blobs go to object storage, metadata goes to a database. Store the file under a key in the bucket, and store the key, owner, size and status in Postgres or DynamoDB, where you can query them. Avoid object storage for small records that change often, for anything that needs queries, and for low-latency disk that a running process writes to; that is block storage or a network file system.

Use it when / avoid it when
  • USE when
  • files, media, backups, logs
  • anything over ~1 MB
  • data lakes and archives
  • AVOID when
  • small records updated often
  • you need queries (DB)
  • low-latency shared disk
  • (block storage / EFS)

Numbers

Numbers to have ready. Objects up to fifty terabytes, a single put up to five gigabytes, multipart parts from five megabytes to five gigabytes, up to ten thousand parts. Three thousand five hundred writes and five thousand five hundred reads per second per prefix. Eleven nines of designed durability. Tens of milliseconds to the first byte. About two cents per gigabyte a month for standard storage, and a thousand keys per listing page.

Numbers worth knowing
max object              50 TB
max single PUT          5 GB
multipart part          5 MB - 5 GB
max parts               10,000
per prefix              3,500 w / 5,500 r
durability (designed)   99.999999999%
first-byte latency      ~10-100 ms
Standard price          ~$0.023/GB-mo
LIST page               1,000 keys

Say it like this

How to say it. Clients ask for a presigned URL and upload directly to object storage, using multipart for large videos. The object created event goes onto a queue, and workers transcode the video into several resolutions. The video's metadata and processing status live in Postgres. Playback goes through a CDN. A lifecycle rule moves originals to an archive tier after ninety days. That is the standard media pipeline.

In the interview
  • "Clients get a presigned URL
  • and upload straight to S3,
  • multipart for big videos.
  • ObjectCreated triggers a
  • transcode job via a queue.
  • Metadata and status live in
  • Postgres. Reads go through
  • a CDN. Old originals move to
  • Glacier after 90 days."

Interview questions

How would you let users upload large files?

The API authorises the user and returns a short-lived presigned URL, and the client uploads directly to object storage, using multipart upload for large files so parts go in parallel and failed parts are retried alone. Your servers never carry the bytes.

Why store files in object storage and metadata in a database?

Object storage is cheap and durable for large immutable blobs but has no query language and slow listing, while a database can index and query owner, size, status and timestamps. The database row holds the object key as a pointer.

Is S3 strongly consistent?

Yes, since December 2020 S3 provides strong read-after-write consistency for PUTs, overwrites, deletes and LIST operations. It still has no locks, so concurrent writers to the same key need conditional writes or last write wins.

How does object storage achieve such high durability?

Each object is split into data and parity chunks with erasure coding and spread across devices in several availability zones, so it can be rebuilt even after losing multiple chunks, with continuous background checks repairing damaged pieces.

What are the request rate limits and how do you go beyond them?

S3 supports at least 3,500 write and 5,500 read requests per second per prefix. Spread keys across many prefixes to multiply throughput, and put a CDN in front for hot reads.

What is a multipart upload?

An upload split into parts of 5 MB to 5 GB, up to 10,000 parts, which are sent in parallel and then combined with a complete call. It speeds up large uploads, allows resuming, and is required above the 5 GB single PUT limit.

How do you reduce object storage costs?

Use lifecycle rules to move cold objects to infrequent-access or archive classes and to delete old versions and incomplete uploads, and serve popular content through a CDN to cut request and egress charges.

When is object storage the wrong choice?

For small records that are updated often, for data that needs queries, and for low-latency file systems that running processes write to; use a database, block storage or a network file system there.