Design Pastebin: Storing and Serving Text Snippets with Expiry

Listen

Listen first. Users create a paste of up to ten megabytes of text and get a link. Anyone with the link can read it. Pastes can expire after an hour, a day, or never, can be public or unlisted, and can be set to burn after the first read. Non-functionally: reads should be fast, pastes must not be lost before they expire, and unlisted links must not be guessable.

Scope out what you will not build: editing, since a change can simply be a new paste; accounts and history; full-text search across all pastes. Syntax highlighting happens in the browser, so the server just stores text. Then ask two questions that matter: the maximum size, ten megabytes, and the read to write ratio, about ten to one. Pastes are read less than short links, but they are much bigger.

Ask scale

Ask about scale. Assume ten million new pastes a day. A day is about a hundred thousand seconds, so that is a hundred writes a second, three hundred at peak. At ten reads per paste, a thousand reads a second, three thousand at peak. With an average paste of ten kilobytes, outbound bandwidth is about ten megabytes a second. Traffic is modest. The interesting number is storage.

Storage is where the design is decided. Ten million pastes at ten kilobytes is a hundred gigabytes a day, about thirty-six terabytes a year, around nine after compression. The metadata, about two hundred bytes per paste, is under a terabyte a year. So split them: text goes to object storage, which is cheap and practically infinite, and the small metadata goes to a database.

10M new pastes / day
  10M / 10^5 s ≈ 100 writes/s
  peak ×3 ≈ 300 writes/s

reads 10:1
  ≈ 1,000 reads/s
  peak ≈ 3,000 reads/s

bandwidth: 1,000 × 10 KB
  ≈ 10 MB/s out, fine

avg paste 10 KB (long tail
  up to 10 MB)
10M × 10 KB = 100 GB/day
  × 365 ≈ 36 TB/year
  compressed ~4x ≈ 9 TB/year

metadata: ~200 B per paste
  10M × 200 B = 2 GB/day
  ≈ 0.7 TB/year

→ text in object storage

Define API

Define the API. POST a paste with its content and optional expiry, visibility and burn flag, and get back a key and a URL. GET by key returns the raw text, four hundred four if it never existed, or four ten Gone if it expired. DELETE needs a secret owner token returned at creation, since we have no accounts. Rate limit creation per IP.

POST /v1/pastes
  { content, expires_in?,
    visibility?, burn? }
→ 201 { key: "q7Lm2xRt", url }

GET /v1/pastes/{key}
→ 200 text/plain
→ 404 / 410 Gone if expired

DELETE /v1/pastes/{key}
  (owner token required)

Data model

The data model: a metadata row keyed by the paste key, holding the object key, size, a content hash, visibility, the burn flag, a hash of the owner token, and created and expiry times, with an index on expiry for cleanup. The text itself is an object named after the key. Lookups are always by key, so a key-value store fits, but Postgres at under a terabyte a year is perfectly fine too.

pastes (metadata DB)
  key          PK  char(8)
  object_key       text
  size_bytes, sha256
  visibility       enum
  burn_after_read  bool
  owner_token_hash
  created_at, expires_at
  INDEX (expires_at)

object: pastes/q7Lm2xRt.gz

High-level design

Draw the high-level design and trace a create. The request goes through the load balancer to a stateless paste service. It generates a key, compresses the text, writes the object to storage, and then inserts the metadata row. Order matters: object first, row second. If we crash in between, we leave an orphan object that a cleanup job removes, never a row pointing at nothing.

Now a read. Public pastes are served from the CDN when possible. On an edge miss, the service checks Redis for the paste. On a cache miss it reads the metadata row, checks expiry and visibility, fetches the object, decompresses it and returns it, filling the cache for small pastes. Most reads of a popular paste never get past the CDN or Redis.

Deep dive: keys

Deep dive one: keys. Unlike the URL shortener, unlisted pastes rely on the key being secret, so a sequential counter is out: anyone could walk the IDs. Use eight random base sixty-two characters from a cryptographic random generator. That is about two hundred trillion possibilities, so after billions of pastes the space is still almost empty. Insert, and on the rare primary-key conflict, just pick again.

unlisted = security by key, so
  no sequential counters

8 random base62 chars
  62^8 ≈ 2.2 × 10^14
  after 4 billion pastes the
  space is 0.002% used

INSERT ... key; on PK conflict
  pick again (very rare)
random from a crypto RNG

Deep dive: storage

Deep dive two: why not keep the text in the database? Thirty-six terabytes a year of large text columns would make backups, replication and failover slow and expensive. Object storage is far cheaper per gigabyte and extremely durable. A nice refinement: most pastes are tiny, so store pastes under about four kilobytes inline in the metadata row, saving the extra round trip for the common case.

Deep dive: big pastes

Deep dive three: the long tail of big pastes. Most pastes are a few kilobytes and travel in the POST body. But someone will paste a nine megabyte log file. For anything over about a megabyte, the client asks for a presigned upload URL, capped at ten megabytes, and sends the bytes straight to object storage. The row starts as pending and flips to ready when the storage event arrives. Your app servers never buffer megabytes of text. See the Large blobs pattern.

Deep dive: storage

Text compresses very well, three to five times with gzip or zstd, so store it compressed and serve it with a gzip content encoding, letting the browser decompress. Identical pastes, like the same error log pasted a thousand times, can share one object found by content hash. If you dedupe, keep a reference count, so deleting one paste does not delete everyone else's copy.

Deep dive: caching

Deep dive four: caching. Because a paste never changes, there is nothing to invalidate. Set the cache lifetime to the time left until expiry. Public pastes go on the CDN and in Redis. Unlisted ones stay out of shared edge caches, or are marked private. Burn after reading must never be cached anywhere. And keep multi-megabyte pastes out of Redis memory; let the CDN carry those.

Deep dive: expiry

Deep dive five: expiry. Correctness comes from the read path: if the expiry time has passed, return Gone, even if cleanup has not run yet. A background sweeper uses the expiry index to find expired rows in batches, deletes the objects, then the rows. Burn after reading needs an atomic step: delete the row and return it in one statement, and only the request whose delete succeeded gets the content. Two simultaneous readers cannot both see it.

read: if expires_at < now
  → 410, even before cleanup
sweeper every minute:
  SELECT key ... WHERE
  expires_at < now LIMIT 5000
  delete objects, then rows

burn after read:
  DELETE ... RETURNING object_key
  only the reader who deletes
  gets the content

Trade-offs

Trade-offs. The real growth problem is storage, not requests, so compression, expiry and cheaper storage tiers matter most. If object storage is slow, the cache protects hot reads. If the database is down, creates fail but cached reads keep working. Pastebins attract abuse, so cap size, rate limit per IP and scan for leaked keys and malware. Monitor read latency, error rates and the sweeper backlog. At ten times, shard metadata by key.

Interview summary

The summary. A hundred writes and a thousand reads a second, thirty-six terabytes of text a year. Metadata in a database, compressed text in object storage, with tiny pastes inline. Keys are eight random base sixty-two characters so unlisted links cannot be guessed. Reads flow CDN, Redis, storage, and because pastes are immutable they cache until expiry. Expiry is checked on read, a sweeper deletes, and burn after reading is an atomic delete.