Design a CI/CD Pipeline: From Git Push to Safe Production Deploys

Listen

Listen. A push, pull request, tag, schedule or button press triggers a pipeline. The pipeline is defined in a YAML file inside the repository, as jobs with dependencies. Jobs run in isolation, logs stream live, and the pass or fail status goes back to the pull request. Builds produce artifacts, and deploys support approvals, canaries and rollback.

Qualities. Feedback must be fast: a job should start within ten seconds, and a typical pull request should finish in about ten minutes. Jobs run other people's code, so isolation is a security requirement, not a nice-to-have. Secrets must never appear in logs or reach pull requests from forks. A lost runner should cause a retry, never a pipeline stuck forever.

Ask scale

Scale, for a hosted service. A million pipeline runs a day with five jobs each is five million jobs, about sixty starts a second. With an average job of eight minutes, Little's law says around twenty-nine thousand jobs run at any moment, near ninety thousand at the weekday peak. Logs add five terabytes a day and artifacts fifty, so deduplication matters.

1M pipeline runs/day × 5 jobs
  = 5M jobs/day ≈ 60 starts/s

average job 8 min = 480 s
  concurrent ≈ 60 × 480 ≈ 29,000
  weekday peak ×3 ≈ 90,000 runners

logs: 1 MB/job → 5 TB/day
artifacts: 50 MB/run → 50 TB/day
  keep 30 days ≈ 1.5 PB (dedupe!)

Define API

The API. Signed webhooks from the git host start runs, and users can also start one by ref and commit. Clients read run status, follow live logs over server-sent events, cancel runs and approve deploys. Runners use their own protocol: they long poll to claim a job, then report state and extend their lease.

POST /v1/hooks/git  (signed)
POST /v1/repos/{r}/runs { ref, sha }
GET  /v1/runs/{id}
GET  /v1/jobs/{id}/logs?follow=1
     (server-sent events)
POST /v1/runs/{id}:cancel
POST /v1/deploys/{id}:approve

runner protocol:
POST  /v1/runner/claim (long poll)
PATCH /v1/jobs/{id} {state, lease}

Data model

Data model. Runs and jobs are relational, with clear states and relationships, so they live in SQL. Each job lists the jobs it needs, the runner labels it requires, and its lease. Logs are written as chunks to object storage. Artifacts are stored by the hash of their content, so identical outputs are stored only once.

runs (SQL)
  run_id, repo, sha, ref,
  trigger, status, created_at
jobs (SQL)
  job_id, run_id, name, needs[]
  labels, state, runner_id,
  lease_until, attempt

logs: chunks in object storage
artifacts: blobs by SHA-256
  (same content stored once)

High-level design

The high-level design. A webhook arrives; the pipeline service fetches the YAML at that commit, turns it into a graph of jobs and saves them. Jobs with no unfinished dependencies go onto the queue. A runner claims one, checks out the code, runs the steps, streams logs, uploads artifacts and reports the result.

Deep dive: the DAG

Deep dive one: the graph. Lint, unit and end-to-end tests run in parallel; build needs all three; deploy needs build. When a job finishes, the service checks each child: if all its needs passed, enqueue it; if any failed, skip it. A matrix expands one job into several, one per runtime version. And a new push to the same branch cancels the older run, which saves a surprising amount of compute.

lint ─┐
unit ─┼─→ build ─→ deploy-stg
e2e  ─┘          └→ deploy-prod
                     (approval)

job done → for each child:
  all needs passed? → enqueue
  any failed? → skip child
matrix: unit × {node18, node20}
new push on branch → cancel
  the older run

Deep dive: runners

Runners pull work instead of being pushed to. Each one long polls for jobs matching its labels, such as Linux with eight CPUs. Pulling means a company can run its own runners behind its firewall and still connect. While running, the runner holds a lease and heartbeats; if it disappears, the job is requeued. Per-organisation limits keep one tenant from taking every runner.

Deep dive: isolation

Deep dive two: isolation. Every job is untrusted code. Run each one in a fresh micro virtual machine, such as Firecracker, that boots in a fraction of a second and is destroyed when the job ends, so nothing leaks to the next tenant. Plain containers share the host kernel, which is a weaker boundary. Add CPU, memory, time and network limits.

Secrets. Store them encrypted, inject them only when the job starts, and mask their values in the log stream. Pull requests from forks get no secrets at all, or anyone could print them. Better still, let jobs reach the cloud through OIDC: the job receives a token valid for minutes, scoped to one repository and branch, so there is no long-lived key to steal.

Deep dive: logs

Logs. The runner sends numbered chunks every second or every sixty-four kilobytes. The log service pushes them to anyone watching, over server-sent events, and appends them to object storage. Because chunks are numbered, a viewer who reconnects asks for everything after chunk forty-one and sees no gaps. Old logs expire after ninety days.

Deep dive: speed

Deep dive three: speed. Most build time repeats work. Cache dependencies keyed by the hash of the lockfile, and container layers by digest. Keep a warm pool of booted machines so jobs start in seconds. Split a forty-minute test suite across eight runners for five minutes. In a monorepo, build and test only what the change actually affects.

Deep dive: deploys

Deploys. Build once and promote the same image digest through staging, a canary and production, so what you tested is exactly what you ship. The canary takes five percent of traffic for about fifteen minutes while the system compares error rate and p ninety-nine against the current version, rolling back on its own if they are worse. Lock each environment to one deploy at a time. See Concepts episode 39.

Trade-offs

Failure modes. Webhooks get lost, so also poll the git host for new commits now and then. A dead runner means a lease expiry and a retry. Flaky tests are the biggest real pain: retry once, but track each test's flake rate and quarantine the worst. If the cache is down, builds are slower but still correct. A bad deploy is caught by the canary.

Monitor queue wait at p ninety-five for each runner label, run duration, success and flake rates, cache hit rate, runner utilisation and cost, and how often deploys roll back. At ten times the scale, shard the pipeline database by organisation, add regional runner pools near the code, and autoscale runners on queue depth.

Say it like this

In the interview: a webhook turns the repository's YAML into a graph of jobs, and ready jobs go to a queue by label. Runners pull work, hold leases and heartbeat, inside a fresh micro-VM per job. Secrets are masked, short-lived and never given to forks. Logs stream as numbered chunks. Caching, warm pools and test sharding keep it fast, and one immutable artifact is promoted through a canary.