Design an Online Code Judge: Sandboxed Runs, Fair Verdicts and Contest Surges

Listen

Listen. Users read a problem with its limits and sample tests, then submit code in one of about twenty languages. We compile it and run it against hidden tests, and return a verdict: accepted, wrong answer, time limit exceeded, memory limit exceeded, runtime error or compile error. Contest mode adds a timer, live standings and penalties. Plagiarism detection is out of scope.

Order the qualities. First, security: user code must never reach the network, the host, other submissions or the hidden tests. Second, fairness: the same code should always get the same verdict and a similar run time. Third, speed: most verdicts within ten seconds. And contests must survive a ten times burst without losing a single submission.

Ask scale

Scale. Practice traffic is about two million submissions a day, only twenty-five a second. Contests are the real load: fifty thousand people, with about a hundred and fifty submissions a second in the first fifteen minutes. Each one costs around seven CPU seconds, two to compile and a hundred tests at fifty milliseconds. That is about a thousand busy cores, so roughly a hundred and thirty eight-core judges.

practice: 2M submissions/day
  ≈ 25/s average

contest: 50,000 people, 2 hours
  first 15 min ≈ 150 submits/s

cost per submission:
  compile 2 s + 100 tests × 50 ms
  ≈ 7 CPU-seconds
150/s × 7 s ≈ 1,000 busy cores
  → ~130 eight-core judges

Define API

The API. Submitting returns two oh two with a submission id right away, because judging takes seconds. The client then gets progress, like running test thirty-seven, over a WebSocket, or polls the submission. A separate run endpoint executes code on the user's own input. Contests have a paged standings endpoint.

GET  /v1/problems/{slug}
POST /v1/submissions
  { problem_id, language,
    source, contest_id? }
  → 202 { submission_id }
GET  /v1/submissions/{id}
  → { status, verdict, time_ms,
      memory_kb, failed_test }
WS   /v1/submissions/{id}/events
POST /v1/run  (custom input)
GET  /v1/contests/{id}/standings

Data model

Data model. Problems and submissions are relational and modest in size, so SQL works well. A problem records its time and memory limits, its checker, and which version of the tests to use. Test files live in object storage under a versioned path. Each submission stores its status, verdict, time, memory, and the first failing test, which is shown only for samples.

problems (SQL)
  id, slug, time_ms, memory_mb,
  checker, tests_version
test sets (object storage)
  tests/{problem}/{version}/
  001.in 001.out … (hash-named)
submissions (SQL)
  id, user_id, problem_id, lang,
  source_ref, status, verdict,
  time_ms, memory_kb, failed_test
  contest_id, created_at

High-level design

The high-level design. The API checks the request, saves the submission as queued and puts a job on the judge queue. A judge worker claims it, fetches the test set, which it caches locally, compiles the code inside a sandbox, runs each test inside a sandbox, compares output with the expected answer, and writes the verdict back.

While judging, the worker publishes progress events: compiling, running test thirty-seven, then the verdict. A small push service holds the user's WebSocket and forwards them. If the socket drops, the client simply reads the submission record. Progress is a nicety; the database row is the truth.

Deep dive: sandbox

Deep dive one: the sandbox. Assume every submission is an attack. Code will try to open sockets, read the test files or other users' code, fork until the machine dies, allocate sixty-four gigabytes, print ten gigabytes of output, loop forever, or exploit the compiler itself with a tiny file that expands into gigabytes. Each needs its own defence.

Defend in layers. The outer layer is a throwaway micro VM or a user-space kernel like gVisor, one per submission. Inside, there is no network at all, a read-only file system except a small capped work directory, and cgroups limiting memory, CPU and process count, so a fork bomb stops at sixty-four processes. A seccomp allow-list blocks unusual system calls. Compilation runs in the same sandbox.

Deep dive: fair verdicts

Deep dive two: fairness. Limit CPU time, and add a wall clock cap of about double plus one second to catch code that just sleeps. Pin each run to its own core so neighbours cannot slow it down, and use one hardware class and the same compiler flags everywhere. A run that times out just past the limit can be retried once, so one noisy moment does not decide a contest.

Correct verdicts need more than string comparison. Ignore trailing whitespace, and when many answers are valid, like any shortest path or a floating point result, run a checker program that validates the output. Stop at the first failed test unless the problem gives partial credit. If a test turns out to be wrong, bump the test version and rejudge the affected submissions at low priority.

compare: exact, or ignore
  trailing whitespace
checker program when many
  answers are valid, or floats:
  |a − b| ≤ 1e-6
stop at first failed test
  (or run all for partial score)

tests fixed? bump tests_version
  → rejudge affected submissions
    at low priority

Deep dive: contests

Deep dive three: contests. Fifty thousand people submit in the first minutes. Unlike most spikes, this one is on the calendar, so scale judges up half an hour before the start, pre-warm sandboxes for each language and pre-load the test sets. Contest jobs go before practice jobs. Limit each user to one submission per problem every ten seconds. The queue is durable, so a slow verdict is acceptable and a lost one never is.

Live standings. Rank by problems solved, then by penalty: minutes taken plus twenty for each wrong attempt before solving. Pack both into one score, solved times a million minus penalty, and keep it in a Redis sorted set, exactly as in the leaderboard problem. Many contests freeze the public board in the last hour and reveal the final ranks at the end.

rank by solved desc, then
  penalty asc
penalty = minutes to solve
  + 20 per wrong attempt

ZADD standings:{contest}
  score = solved × 10⁶ − penalty
ZREVRANGE for a page,
  ZREVRANK for "my rank"
freeze public board in the
  last hour; reveal at the end

Trade-offs

Failure modes. If a judge dies mid-run, its lease on the job expires and another judge starts over, which is safe because judging the same code twice gives the same verdict. If a sandbox is ever escaped, the attacker lands in a throwaway VM on a judge that holds no secrets and can only write its own verdict. Standings can always be rebuilt from verdicts.

Monitor queue wait by priority, judge time per language, how often a rejudge flips a verdict, which means unfair timing, and sandbox violations, alerting on any new kind. At ten times the scale, add regional judge pools, cache test sets on local disks everywhere, and shard the submissions table by user.

Say it like this

In the interview: submissions are accepted asynchronously into a durable queue. Judges lease jobs and cache test sets. Every compile and run happens in a throwaway micro VM with no network, a read-only file system, cgroup limits and a syscall filter. Timing is fair through CPU limits and pinned cores. Checkers and versioned tests keep verdicts right, and contests get pre-scaling, priority and sorted-set standings.