Notes
A plain LLM
Start with a plain language model. You send it text, it sends text back. It knows a great deal from training, but nothing about your systems. Ask it where order eleven eighty two is and the honest answer is that it has no idea. It cannot look the order up, cannot read a file, cannot send an email. On its own, a model can only talk. Everything that makes an agent useful is built around that limit.
- input: text (and images)
- output: text
- knows: what it was trained on
- does not know: your data,
- today's date, live state
- cannot: call an API, read a
- file, send an email
The definition
Here is a working definition. An agent is a program where a language model decides which action to take next, sees the result of that action, and repeats until the goal is met or it gives up. The key word is decides. In normal software, the developer writes the order of steps. In an agent, the model chooses the steps at run time. Four parts make that possible: the model, the tools, the memory and the loop.
- a program where an LLM
- decides which action to take
- next, sees the result, and
- repeats until the goal is met
- four parts:
- model → chooses
- tools → act
- memory → remembers
- loop → keeps going
Workflow or agent
It helps to think of a spectrum. At one end is a single model call, like summarising an email. Next is a workflow, where you write the steps in code and the model fills in each one. Then a router, where the model picks one of a few paths. At the far end is an agent, where the model picks every step and decides how many there are. More autonomy means more flexibility, but also more cost and more ways to go wrong.
- single call
- summarise this email
- workflow (you fix the steps)
- classify → route → reply
- router
- model picks one of N paths
- agent
- model picks every step,
- as many as it needs
The parts
Here is the whole picture. The user states a goal. The agent runtime, which is ordinary code you write, sits in the middle. It builds the prompt, calls the model, runs whatever tool the model asks for, stores results in memory, and checks every step against guardrails. The model never touches the outside world directly. It only asks. Your code decides whether to do what it asks. Keep that split in mind, because it is where safety lives.
- runtime: your code. Builds
- the prompt, calls the model,
- runs tools, enforces limits
- model: reads the context,
- returns text or a tool call
- tools: functions with a
- name, description, schema
The model
The model is the decision maker, and it only knows what is in its context window. That window holds the system prompt with its role and rules, the list of available tools, the conversation so far, every tool result, and any documents you retrieved. If a fact is not in that window, the model cannot use it. And every token in the window costs time and money on every single step. Managing that window is half of agent engineering.
- system prompt: role, rules
- tool list: names + schemas
- conversation so far
- tool results so far
- retrieved documents
- all of it must fit in the
- context window, and every
- token costs time and money
Tools
Tools are how an agent acts. A tool is a plain function with a name, a description in plain language and a schema for its inputs. Get order, search the docs, run this query, send this message. The model reads the descriptions and, when it wants something, writes a structured request to call one. Your runtime runs the real function and hands back the result. The next episode covers tool calling in detail.
name: get_order
description: Look up one order
by id. Returns status + dates.
input: { order_id: string }
the model writes the call,
your code runs it Memory
Memory comes in two kinds. Short-term memory is simply the transcript of this run, the messages and tool results, kept in the context window. When it grows too long you trim it or summarise it. Long-term memory survives between runs. Facts like the user prefers email, or this project uses Postgres, are saved to a store and fetched when they look relevant. Retrieval, which episode four covers, is the usual way to fetch them.
- short-term
- the messages and tool results
- of this run, kept in context
- trimmed or summarised when
- it gets long
- long-term
- facts saved across runs
- (user prefers email, uses
- Postgres), fetched on demand
The loop
The loop ties it together. Send the context to the model. If it returns a final answer, you are done. If it asks for a tool, run the tool, append the result to the context, and go around again. Always cap the number of steps, so a confused model cannot spin forever and run up a bill. That loop, a few lines of code, is the core of every agent framework. Episode three takes it apart properly.
while steps < max_steps:
reply = model(context)
if reply is a final answer:
return reply
result = run_tool(reply)
context.append(result)
return "stopped: step limit" A worked example
Here is one run. A customer writes that order eleven eighty two is late and asks for a refund. The model first calls get order and learns it is two days late. It then calls get policy and reads that late orders over one day qualify. It asks to issue the refund. Because refunds move money, the runtime pauses for a human to approve. Then the model writes the final answer. Three tool calls, one answer, nothing hard coded.
user: my order 1182 is late,
can I get a refund?
1 get_order(1182) → late 2 days
2 get_policy("late") → refund
allowed if over 1 day late
3 issue_refund(1182) → needs
approval → human approves
answer: refund issued, 3-5 days Autonomy levels
Decide up front how much the agent may do on its own. Read only, where it can look things up but never change anything. Suggest, where it drafts an action and a person approves it. Act with limits, where it may act inside caps, such as refunds under fifty dollars or edits to one repository. And full autonomy, which is rare and only sensible where mistakes are cheap and easy to undo. Most production agents live in the middle two.
- read-only: may look things
- up, never change anything
- suggest: drafts an action,
- a person clicks approve
- act with limits: may act,
- inside caps (under 50 USD,
- only this repo)
- full: acts alone; rare, and
- only where mistakes are cheap
When to use one
When is an agent the right call? When the steps cannot be known in advance, because the path depends on what you discover along the way. Debugging a failing test, researching a question across several sources, or triaging a support ticket that could need any of twenty tools. It also helps a lot when success can be checked automatically, like tests passing, because the agent can then correct itself before it finishes.
- • the path depends on what
- you find along the way
- • many possible tools, few
- used per task
- • tasks a person would do
- with a browser and APIs
- • results can be checked
- (tests pass, ticket closed)
When not to
And when is it the wrong call? If you can draw the flowchart, write the workflow in code instead. Extract fields from an invoice, validate them, save them. That is three fixed steps, and an agent would only add cost and randomness. The same goes for anything that must answer in under a second, or that regulators expect to run in a fixed audited order. Start with the simplest design that works and add autonomy only when it pays for itself.
- • the steps are always the
- same: extract → validate →
- save
- • latency must be under a
- second
- • every step must be audited
- in a fixed order
- • one model call already works
Where it fails
Here is the trap. Each step has a small chance of going wrong: a bad tool choice, a wrong argument, a misread result. Those chances multiply. At ninety five percent per step, ten steps succeed only about sixty percent of the time, and twenty steps about a third. So good agents keep runs short, give the model clear and few tools, check results after each step, and let the model retry when a check fails.
success per step steps whole run
99% 10 ~90%
95% 10 ~60%
95% 20 ~36%
fixes: fewer steps, better tools,
checks after each step, retries Cost and latency
Agents are slow and expensive compared with a single call. Each step is a full model call, usually half a second to a few seconds. A task takes anywhere from three to thirty steps, so seconds to many minutes. And because the context grows each turn, later steps cost more than early ones. Prompt caching helps a lot, because the system prompt and tool list stay the same on every step and can be reused cheaply.
model call 0.5 - 5 s steps per task 3 - 30 task wall time 5 s - 10 min context per step grows each turn cost per task cents to dollars prompt caching cuts repeated prefix cost and time a lot
Say it like this
How to say it. An agent is a language model in a loop that picks tools until the goal is met. In this design it has four read tools and one write tool that needs approval. It stops after fifteen steps and hands off to a person. Every step is logged, and we measure it against a fixed set of tasks before each release. Naming the stopping rule and the approval point is what makes the answer sound like production experience.
- "An agent is an LLM in a loop
- that picks tools until the
- goal is met. Here it has
- four read tools and one
- write tool behind approval.
- Max 15 steps, then it hands
- off to a person. We log
- every step and evaluate on
- a fixed task set."