Design Google Drive: Chunked Uploads, Deduplication, Sync and Conflicts

Listen

Requirements. Users upload and download files, up to around ten gigabytes. They organise them in folders: rename, move, delete. Any change syncs automatically to all their devices. They can share files and folders with view or edit permission, and restore older versions for thirty days. Real-time co-editing of documents is a separate problem, see the Google Docs design, and full-text search is out of scope.

Qualities, in priority order. Durability above everything: losing a user's file is the worst possible bug, so aim for the eleven nines that object stores offer. Transfers must be efficient, because people edit big files on slow connections and you should only send what changed. Metadata must be strongly consistent, so a move or rename is all or nothing. Sync should take seconds.

Ask scale

Assume five hundred million registered users, a hundred million active daily, each with about two devices. Each active user writes two files a day: two hundred million writes, about two thousand a second, six thousand at peak. Metadata traffic is much higher: listing folders and checking for changes adds up to around fifty thousand requests a second. And we have well over a hundred million devices waiting to hear about changes.

Storage. At half a megabyte of new data per write, that's a hundred terabytes a day, over thirty petabytes a year. Deduplication claws back a good chunk, because people store the same files, and versions add some back. Object storage with erasure coding keeps durability at around one and a half times the raw size, much cheaper than three full copies. Metadata is about a kilobyte per file: a hundred billion files is a hundred terabytes, which a sharded relational database can hold.

registered users     500M
DAU                  100M
devices per user     ~2

file writes/user/day 2
  200M / 100,000 s  = 2K wps
  peak x3           = 6K wps

metadata reads, list + sync
  100M x 50 = 5B/day = 50K rps
open sync connections ~150M

avg new data/write    0.5 MB
200M x 0.5 MB      = 100 TB/day
  x 365            = ~36 PB/yr

dedupe saves       ~30-50%
versions add       ~20%
object store keeps ~1.5x
  overhead (erasure coding)

metadata 1 KB per file
  x 100B files     = 100 TB

Define API

The API separates bytes from metadata. First the client sends the hashes of a file's blocks and asks which ones the server is missing. It uploads only those, each addressed by its hash. Then it commits a new version: the list of block hashes plus the version it started from. To sync, a device asks for changes since its cursor. Downloads fetch blocks by hash. Sharing is its own call.

POST /v1/blocks/check
  { hashes[] } -> { missing[] }
PUT  /v1/blocks/{sha256}  (bytes)
POST /v1/files/commit
  { path|fileId, blocks[],
    baseVersion } -> { version }
GET  /v1/changes?cursor=..
  -> { changes[], cursor }
GET  /v1/blocks/{sha256}
POST /v1/files/{id}/share

Data model

Data model. A file row has an owner, a parent folder and a pointer to its latest version. A version row is an ordered list of block hashes. A block row is keyed by its SHA-256 hash with a reference count, so identical content is stored once. A per-user journal records every change with an increasing sequence number. Use a relational database for metadata, because moving a folder or renaming a file must be a single transaction.

file    (fileId, ownerId, parentId,
         name, latestVersion, deleted)
version (fileId, version, size,
         blockHashes[], ts, deviceId)
block   (sha256 PK, size, refCount)
journal (userId, seq, fileId, op)
acl     (fileId, principal, role)

SQL for metadata: moves and
renames need transactions

High-level design

The upload path. The client splits the file into blocks and hashes each one. It asks which of the forty blocks the server lacks; for an edited file, maybe only three. It uploads those three to the block service, which writes them to object storage keyed by hash. Then it commits version eight to the metadata service, which checks every block exists and writes the new version in one transaction.

The sync path. In the same transaction, the commit appends an entry to the user's change journal. The notification service tells the user's other devices that something changed, without saying what. Each device then asks for changes since its last cursor, sees that this file is now version eight, compares block lists, and downloads only the three blocks it doesn't have.

Deep dive: chunking

Deep dive one: chunking. Fixed four-megabyte blocks are simple, but insert a byte near the start of a file and every block boundary shifts, so everything looks new. Content-defined chunking fixes that: a rolling hash slides over the bytes, and you cut a chunk wherever the hash matches a pattern. Boundaries follow the content, so an insert only changes the chunk around it. Each chunk's key is its SHA-256, which makes deduplication automatic.

Content addressing has more benefits. Uploads are naturally resumable: a dropped connection loses one block, and on retry the client asks again which blocks are missing. Re-uploading a block is harmless because the key is its hash. The server recomputes the hash to verify integrity. One caution: deduplicating across users lets an attacker test whether someone else stored a given file, so many designs dedupe only within a user or account.

Deep dive: sync

Deep dive two: sync. The backbone is the journal: every commit appends an entry with a sequence number that only grows. Each device remembers the last sequence it applied, its cursor. To sync, it asks for everything after the cursor and moves the cursor forward. Notifications are just a nudge to check now, sent over a long poll or mobile push. If a nudge is lost, a periodic poll catches up, because the journal, not the notification, is the source of truth.

The desktop client does a lot of work. A file watcher notices edits. A small local database maps every path to the last synced version and its block hashes, so the client can tell exactly which blocks changed. It waits a couple of seconds after the last write, so saving a big file doesn't trigger ten uploads. Remote changes are written to a temporary file and swapped in atomically. Offline edits are queued and replayed later.

Deep dive: conflicts

Deep dive three: conflicts. Two devices edit the same file offline, both starting from version seven. Each commit says which version it was based on. The first commit wins and becomes version eight. The second is rejected because its base is no longer the latest. Rather than guess at merging arbitrary files, the losing device keeps its work as a clearly named conflicted copy. Versions are cheap because they share unchanged blocks, and blocks unused by any version are deleted later by a reference-counting job.

Trade-offs

Failures. If blocks upload but the commit fails, those blocks are orphans with no references, and garbage collection removes them later. The metadata database is the source of truth, so shard it by owner, replicate synchronously and fail over fast. If a notification node dies, devices reconnect and read the journal, losing nothing. A folder shared with a whole company is a hot spot: cache its listing. Monitor sync lag, commit error rate and blocks awaiting garbage collection.

Trade-offs to name. Relational metadata gives you transactional moves and renames at the cost of harder sharding; shared folders that cross owners are the awkward case. Conflicted copies are simple and never lose data, but push merging onto the user. Deduplicating per user favours privacy over maximum savings. Erasure coding is far cheaper than three copies but slower to rebuild after a disk loss.

Say it like this

The summary. Files are split into content-defined chunks keyed by their hash, so uploads send only missing blocks, are resumable, and dedupe for free. A file version is an ordered list of block hashes, committed transactionally in a relational metadata store with an optimistic base-version check. Every commit appends to a per-user journal; devices get a nudge and pull changes since their cursor. Conflicts produce a conflicted copy.