Start practising

AI ENGINEERING · AGENTS · PRODUCTION · 5 OF 5

Guardrails and Evals: Keeping AI Agents Safe and Measurably Good

How production AI systems stay safe and get measurably better: layered guardrails on input, tools and output, prompt injection and least privilege, human approval, then evaluation: building a task set from real traces, code graders, model judges and people, judging whole agent runs, gating releases on regressions, and monitoring quality, cost and drift in production.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
1 / 18

Press play to hear the narration, or step through at your own pace.

Notes

Why it is hard

Why is this harder than normal software? The same input can produce different outputs on different runs. The inputs are unbounded, and they arrive not only from users but from every web page, email and document the agent reads. Wrong answers sound just as confident as right ones. And agents take real actions, like refunds and emails. Ordinary unit tests cannot cover that space, so you need both guardrails and evals.

New kinds of failure
  • • same input, different output
  • • any text can arrive, from
  • users AND from tool results
  • • confident wrong answers
  • • real actions: refunds,
  • emails, deleted files
  • unit tests alone cannot
  • cover this

Layers

Think in layers, like any security design. Check what comes in. Limit what the agent may do through its tools. Check what goes out. Ask a person before the actions that really matter. And log everything. Each layer misses some cases. A model based filter can be fooled, a rule can be too narrow. Stacked together, they catch most problems, and you design each layer assuming the others might fail.

Defence in depth
  • input: what comes in
  • tools: what it may do
  • output: what goes out
  • people: what needs a yes
  • logs: what happened
  • each layer misses some;
  • together they catch most
  • assume each one can fail

Input checks

On the way in, start with cheap rules. Limit the size of a message and how often a user can send them. Mask secrets and personal data, like card numbers and identity numbers, before they ever reach the model or the logs. Then use a small, fast classifier model to check the topic is in scope and to flag obvious attempts to override the instructions. Rules are fast and predictable. The model catches what rules miss.

Checks on the way in
  • • size limits, rate limits
  • • mask secrets and personal
  • data (cards, ids, keys)
  • • topic scope: is this
  • something we handle?
  • • jailbreak / injection
  • classifier
  • rules first (fast, cheap),
  • small model second

Prompt injection

Prompt injection is the signature risk. Direct injection is a user typing ignore your instructions. Indirect injection is worse: the instructions hide inside a web page, an email or a file that the agent reads while doing its job. No filter catches all of it. The danger peaks when one agent can read untrusted content, can see private data, and can send data out. Remove any one of those three and the worst outcomes become impossible.

Prompt injection
  • direct: the user types
  • "ignore your instructions"
  • indirect: a web page, email
  • or file contains them
  • danger is highest when one
  • agent has all three:
  • • reads untrusted content
  • • sees private data
  • • can send data out

Tool policy

The strongest guardrail is not a filter, it is less power. Give each task only the tools it needs. Run tools with the user's own credentials, never a shared admin key. Cap amounts, counts, recipients and file paths. Run any generated code in a sandbox with no network and a throwaway disk. Allow outbound network calls only to known hosts. Then even a fully hijacked agent can do little damage.

Least privilege
  • • allowlist tools per task
  • • scope credentials to the
  • user, never an admin key
  • • caps: amounts, counts,
  • recipients, file paths
  • • sandbox code execution:
  • no network, temp disk
  • • network egress allowlist

Output checks

On the way out, check before anything reaches the user or another system. Does structured output match its schema, with every value in range? Does the reply leak a secret, another customer's data, or the system prompt? For RAG answers, is every claim supported by the sources? Is the tone acceptable? When a check fails, retry with the error, repair the output, or fall back to a safe refusal.

Checks on the way out
  • • schema: valid JSON, fields
  • present, values in range
  • • leaks: secrets, other
  • users' data, system prompt
  • • grounding: claims backed
  • by retrieved sources
  • • tone and policy
  • fail → retry, fix or refuse

People

Some actions need a person to say yes: anything irreversible like deleting data or paying money, anything over a cost threshold, and anything that speaks to customers on your behalf. Show the reviewer the exact action and the reason, not a vague summary, or approvals become rubber stamps. And always give users an easy way to reach a real person when the agent is out of its depth.

Human approval
  • approve when the action is:
  • • irreversible (delete, pay)
  • • costly (over a threshold)
  • • external (emails customers)
  • show the exact action and
  • why, not a vague summary
  • also: an easy "talk to a
  • person" handoff

Why evals

Now evals. Without them, the usual workflow is change the prompt, try three examples, it looks better, ship it. And quietly break ten other cases. An eval is a fixed set of tasks you run against every change, scoring each case, so you compare numbers instead of impressions. Changing the model, the prompt, a tool description or the retrieval settings should all go through the same suite before release.

Without evals
  • change the prompt, try three
  • examples, looks better, ship
  • → quietly broke ten others
  • with evals:
  • same fixed task set, every
  • change, a score per case,
  • compare to the last release

Building the set

Build the eval set from real usage, not imagined examples. Start with twenty to fifty real cases, and every time something fails in production, add it as a new case. Cover the common requests, the awkward edge cases, hostile inputs like injection attempts, and things the agent should politely decline. Write down what good looks like for each case. And keep a held out slice you never tune against, so you notice overfitting.

Building the task set
  • • start with 20-50 real cases
  • • add every bug from
  • production as a new case
  • • cover: common, edge, hostile
  • (injection), out-of-scope
  • • write what "good" means
  • for each case
  • • keep a held-out slice you
  • never tune against

Graders

Who grades the outputs? Use code wherever you can: exact matches, valid JSON, tests passing, the right tool called with the right arguments. It is fast, cheap and never moody. For qualities code cannot judge, like helpfulness, tone or faithfulness to sources, use a model as a judge with a clear rubric. And use people to create gold labels and spot check, because they are slow and costly but they are the ground truth.

Three kinds of grader
  • code: exact match, regex,
  • JSON valid, tests pass,
  • right tool called
  • → fast, cheap, reliable
  • model judge: rubric scores
  • for tone, helpfulness,
  • faithfulness → scalable
  • people: gold labels, spot
  • checks → slow, trusted

Model as judge

A model judge works when it is narrow. Ask one clear question, such as is every claim in this answer supported by these sources, and ask for a pass or fail with the reasons, rather than a vague score out of ten. Use a separate judge for each quality. Then check the judge itself: grade a hundred cases by hand and measure how often the judge agrees. If agreement is low, fix the rubric before trusting any number it produces.

Judge prompt
Given SOURCES and ANSWER:
Is every claim in ANSWER
supported by SOURCES?
Reply: {"supported": true|false,
        "unsupported_claims": [...]}

one question per judge
pass/fail beats 1-10 scales
measure agreement with humans

Judging agents

Agents need extra care. Grade the end state, not the agent's claim. Was the refund actually issued in the test environment, do the tests actually pass? Then look at the path: did it use the right tools, avoid forbidden calls, and stay within its step and cost budgets? And because results vary, run each task several times. An agent that succeeds one time in three is not ready, however good that one run looked.

Agent evals
  • end state: was the refund
  • issued? do tests pass?
  • (check the sandbox, not
  • the agent's claim)
  • trajectory: right tools,
  • no forbidden calls, steps
  • and cost within budget
  • consistency: run each task
  • several times, pass rate

Release gates

Wire the suite into your release process. Every change to the prompt, the model, a tool or the retrieval settings runs the evals and compares against the current baseline. If a key metric drops more than a set margin, block the release. And read the individual cases that changed, not just the averages, because an average can hide one serious new failure behind many small improvements.

In CI
  • on every prompt, model,
  • tool or retrieval change:
  • • run the suite
  • • compare to the baseline
  • • block if a key metric
  • drops more than X points
  • • read the diffs, not just
  • the averages

In production

Evals before release are not the end. In production, run cheap graders on a sample of real traffic. Collect user signals like thumbs up or down, edits to drafts, and handoffs to a person. Track cost and latency per task, and how often each guardrail fires. Alert on sudden shifts, which often mean a provider model changed. And send every bad case back into the eval set. That loop is how quality actually improves.

Watch it live
  • • run cheap graders on a
  • sample of real traffic
  • • thumbs up/down, edits,
  • handoffs to a person
  • • cost and latency per task
  • • guardrail trigger rates
  • • alert on sudden shifts
  • • bad cases → eval set

Numbers

Some numbers to have ready. Start with twenty to fifty eval cases and grow to hundreds or a couple of thousand. Run each agent task three to five times. Aim for a model judge that agrees with people around eighty five to ninety percent of the time. Input guardrails should add tens of milliseconds, not seconds. Grade one to ten percent of live traffic, and block releases that drop a key metric by a few points.

Numbers worth knowing
eval set to start       20 - 50 cases
eval set mature         200 - 2,000
runs per agent task     3 - 5
judge vs human target   ~85-90% agree
input guard latency     10 - 100 ms
live sample graded      1 - 10%
regression gate         ~2-5 points

Say it like this

How to say it. Inputs are masked for personal data and checked by a scope classifier. Tools are least privilege with the user's own credentials, and refunds need approval. Outputs are checked for schema and leaks. We keep a hundred and fifty eval cases from real tickets, graded by code where possible and a faithfulness judge elsewhere, run three times each, and block any release that drops a key metric by three points.

In the interview
  • "Input masking and a scope
  • classifier, least-privilege
  • tools, refunds need approval,
  • output schema and leak checks.
  • 150 eval cases from real
  • tickets, code graders plus a
  • faithfulness judge; releases
  • blocked on a 3-point drop."

Interview questions

What are guardrails in an LLM application?

Checks around the model that limit what goes in, what it can do and what comes out: input validation and masking, scope and injection classifiers, least-privilege tool policies, output schema and leak checks, and human approval for risky actions.

What is prompt injection and how do you defend against it?

It is when instructions hidden in user input or in content the agent reads take over its behaviour. Defend in layers: least-privilege tools, scoped credentials, approval on writes, output checks, and never giving one agent untrusted input, private data and outbound reach together.

Why are evals necessary for LLM systems?

Outputs are non-deterministic and a change that fixes one case can break others. A fixed eval set scored on every change replaces impressions with comparable numbers and catches regressions before release.

How would you build an initial eval set?

Start with 20 to 50 real cases from logs, covering common requests, edge cases, hostile inputs and out-of-scope requests, with a written definition of a good answer for each. Add every production failure as a new case and keep a held-out slice.

When should you use a model as a judge?

For qualities code cannot check, such as faithfulness, helpfulness or tone. Keep each judge narrow with a clear rubric and pass/fail output, and measure its agreement with human labels before trusting its scores.

How do you evaluate an agent rather than a single answer?

Check the actual end state in a test environment, inspect the trajectory for correct tool use, forbidden calls and budget, and run each task several times to measure a pass rate rather than trusting one run.

Which agent actions should require human approval?

Actions that are irreversible, expensive or external, such as deleting data, moving money or messaging customers. Show the reviewer the exact action and reason so approvals stay meaningful.

How do you monitor an LLM feature in production?

Run cheap graders on a sample of live traffic, collect user feedback and handoffs, track cost, latency and guardrail trigger rates, alert on sudden shifts, and feed bad cases back into the eval set.