Start practising

SYSTEM DESIGN · CONCEPTS · RUNNING IT IN PRODUCTION · EPISODE 39

Deploying and Scaling: Containers, Autoscaling, Canary

How code gets to production safely and how a fleet grows with traffic: containers and images, Kubernetes basics, CI/CD pipelines, rolling, blue-green and canary deployments, feature flags, database migrations without downtime, horizontal autoscaling and its limits, and how to describe it in an interview.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
1 / 10

Press play to hear the narration, or step through at your own pace. Every picture here is the same one used in the YouTube video.

Notes

Containers

A container image packages your app with exactly the runtime and libraries it needs. You build it once, and the same image runs in testing and production, which ends 'works on my machine'. Containers start in seconds. Kubernetes runs them as pods across many machines, restarts crashed ones and scales their number.

Containers
FROM eclipse-temurin:21-jre
COPY app.jar /app.jar
CMD ["java","-jar","/app.jar"]

same image in test and prod
"works on my machine" solved
starts in seconds

Kubernetes runs them as pods,
restarts failures, scales them

CI/CD

A CI/CD pipeline turns every commit into a release candidate: build, run tests, create an image tagged with the commit, deploy to staging, run checks, then promote to production. Small, frequent releases are safer than big ones, because when something breaks you know exactly which change did it.

CI/CD pipeline
  • push → build → unit tests
  • → build image, tag with commit
  • → integration tests
  • → deploy to staging
  • → automated checks
  • → deploy to production
  • (gradually, see next)
  • small, frequent releases
  • are safer than big ones

Rolling

A rolling deploy replaces instances a few at a time, waiting for each new one to pass health checks. No extra capacity needed, but old and new versions run side by side for a while. Blue-green builds a complete new fleet next to the old one and switches all traffic at once, so rollback is just switching back.

Rolling and blue-green
  • rolling: replace pods a few
  • at a time; health checks gate
  • each step
  • + no extra capacity
  • − v1 and v2 run together
  • blue-green: full v2 fleet beside
  • v1, switch traffic at once
  • + instant rollback (switch back)
  • − double capacity briefly

Canary

A canary release sends a small slice of traffic, say five percent, to the new version, and compares its error rate, latency and business metrics with the old one. If it looks good, increase to twenty-five, then a hundred. If it looks worse, roll back automatically. Only a few users ever saw the bad version.

Canary release
  • v2 gets 1% → 5% → 25% → 100%
  • compare v2 vs v1 at each step:
  • error rate, p99, business KPIs
  • worse? roll back automatically
  • only 5% of users ever saw it
  • the safest default for
  • large user bases

Feature flags

Feature flags go further: the new code ships switched off, and you turn it on for staff, then ten percent of users, then everyone, without a deploy. If it misbehaves, flip it off in seconds. Deploying and releasing become separate decisions. Just remember to remove old flags.

Feature flags
if flags.on("new_checkout", user):
    new_checkout()
else:
    old_checkout()

deploy ≠ release
turn on for staff → 10% → all
kill switch in seconds,
no redeploy
clean up old flags!

Migrations

Database changes are the risky part, because old and new code run at the same time during a deploy. Use expand, migrate, contract: add the new column, deploy code that writes to both, backfill old rows in small batches, switch reads over, and only much later drop the old column. Every step is safe with both versions running.

Migrations without downtime
  • expand → migrate → contract
  • 1. add new column (nullable)
  • 2. deploy code that writes both
  • 3. backfill old rows in batches
  • 4. switch reads to new column
  • 5. later: drop the old column
  • every step works with both
  • old and new code running

Autoscaling

Autoscaling watches a metric like CPU, requests per second or queue depth, and adds instances when it rises, removing them later to save money. Target around sixty to seventy percent, not a hundred, because new instances take time to start, so pre-scale before known peaks. And remember: stateless servers scale this way; your database doesn't.

Horizontal autoscaling
  • scale on CPU, requests/s,
  • or queue depth
  • target: 60–70% CPU, not 100%
  • limits:
  • new pods take time to start
  • → pre-scale for known peaks
  • the DB does not autoscale
  • like stateless app servers
  • set min and max replicas

Say it like this

In an interview: stateless services packaged as containers on Kubernetes behind a load balancer, a CI/CD pipeline with automated tests, canary releases with automatic rollback, feature flags for risky changes, expand-and-contract schema migrations, and autoscaling on CPU, request rate or queue depth with a minimum spread across zones.

In the interview
  • stateless services in containers
  • on Kubernetes, behind an LB
  • CI/CD with automated tests
  • canary releases + auto rollback
  • feature flags for risky features
  • expand/contract migrations
  • HPA on CPU / rps / queue depth,
  • min replicas across zones

Interview questions

What are the benefits of containers?

A container image bundles the application with its runtime and dependencies, so the same artefact runs identically in every environment, starts quickly and can be scheduled densely by an orchestrator like Kubernetes.

Compare rolling, blue-green and canary deployments.

Rolling replaces instances gradually with no extra capacity. Blue-green runs a full new environment and switches traffic at once, enabling instant rollback at the cost of double capacity. Canary sends a small percentage of traffic to the new version, compares metrics and expands or rolls back.

What are feature flags used for?

Turning features on or off at runtime for chosen users or percentages without redeploying, enabling gradual rollouts, experiments and instant kill switches; they separate deployment from release.

How do you change a database schema without downtime?

Use expand-migrate-contract: add new structures in a backward-compatible way, deploy code that writes both, backfill in batches, switch reads, and remove old structures only after all code stops using them.

What metrics should drive autoscaling?

CPU utilisation for compute-bound services, requests per second or latency for web tiers, and queue depth or lag for workers. Keep a target below saturation to allow for start-up time.

What are the limits of autoscaling?

Instances take time to start, so sudden spikes outpace it; stateful systems such as databases do not scale by adding replicas easily; and downstream dependencies or quotas may become the bottleneck.

Why are small, frequent releases safer?

Each change is small, so problems are easier to detect and attribute, and rollbacks are cheap; large infrequent releases bundle many risks together.