A field guide for product managers · about 30 minutes
We'll follow one example the whole way through: a research assistant that takes a question, searches the web, reads a few sources, and writes a short report with citations. Start by watching it run, then walk through the five stages of building one.
Words with a dotted underline, like , are technical terms. Tap one to see a plain-English definition.
Live demo
Type a question or pick one below. Then step through the run, or press Autoplay. You'll see each step, what the agent remembers, and the log an engineer would read.
Stage 1 · Ideation
Before anyone writes code, be clear about what an agent is, whether your problem needs one, and what "good" looks like.
All three can use the same . The difference is who decides the next step.
| Chatbot | Agent | ||
|---|---|---|---|
| Who decides the next step | The user, one message at a time | A developer, in advance, in code | The model, while it works |
| Research example | Answers from what it already learned in training. No live sources. | Always: search the question, take the top 3 results, summarise them. | Chooses what to search, which results are worth reading, and when it has enough. |
| Good for | Quick Q&A, drafting, brainstorming | Repeatable processes with known steps | Open-ended tasks where the steps depend on what it finds |
| Main risk | Out-of-date or made-up facts | Breaks when a case doesn't fit the script | Unpredictable path, higher cost, harder to test |
Real systems sit somewhere between "the developer decides everything" and "the model decides everything". Each step to the right gives the model more freedom, and costs more to test and to trust.
| Who decides what | Research assistant at this level | |
|---|---|---|
| 1 · Fixed workflow | The developer decides every step. No LLM, or an LLM only writes text. | Search the question, take the top 3 results, paste them into a template. |
| 2 · Workflow with LLM steps | The developer decides the steps. The LLM does the work inside some of them. | Same fixed steps, but an LLM writes the summary. |
| 3 · Router | The developer draws the possible paths. The LLM picks which path to take. | An LLM grades the results: good ones go to reading, poor ones to a rewritten search. |
| 4 · Agent on a map | The developer draws the map and its limits. The LLM decides what to do at each stop, and how many times to go round a loop. | The graph you'll meet in modules 2 to 4: plan, search, grade, rewrite, read, write, with human review. |
| 5 · Open agent loop | The LLM decides every step, from a list of tools, until it's done or a limit stops it. | The demo above: the model chooses each search, page and note. |
Answer these four questions. If you can't say "yes" to at least the first two, a workflow is usually cheaper, faster and easier to trust.
Write this on one page before design starts. The success criteria become your tests in Stage 4.
Given a question, find credible sources, read them, and write a report of about 150 words with a citation for every claim.
Analysts and PMs who need a quick, sourced briefing before a meeting. They will skim it and click through to the sources.
Personal advice (medical, legal, investment), questions about private company data, and anything that sends messages or spends money.
An analyst spends about 40 minutes on a sourced briefing. A plain workflow (search, take the top 3, summarise) takes 10 seconds and costs under ₹1, but often cites junk.
The agent has to be clearly better than the plain workflow on source quality, and much faster than the analyst. If it isn't, ship the workflow.
Download a one-page agent spec with these headings, plus the questions from Stages 2 to 5: the model, tools, limits, approvals, evals and rollout. Fill it in with your team before design starts.
Check your understanding · Stage 1
Your team wants a weekly sales summary: the same 3 database queries and the same format, every Monday. What should you build?
Stage 2 · Design
Every agent, however complex, is built from the same parts. Here is what each one is for our research assistant.
An LLM that reads everything in memory and decides the next step: think, use a tool, or give the final answer. You reach it through an and pay per .
A that sets the role, the rules and when to stop. It's sent to the model at the start of every turn.
You are a research assistant. Answer the user's question in a report of about 150 words. Use at least 3 credible sources. Cite every claim as [n]. Only cite pages you have opened with fetch_page. If sources disagree, say so. If you can't find good sources, say that instead of guessing.
Functions your code can run when the model asks. Ours has four:
web_search: find pages for a queryfetch_page: open one page and return its textsave_note: write key facts to a scratchpadwrite_report: save the final report to a shared folderShort-term memory is the running conversation: instructions, question, and every step and result so far. It lives in the model's .
Long-term memory is stored outside the model and fetched when needed, for example "this user prefers bullet points". Our first version doesn't need it.
The limit: every step adds text, and the context window has a ceiling. Long runs get slower, cost more and can fill it up. Keep tool results short, save key facts as notes, and summarise or drop old steps.
There's no single best model. Bigger models plan and write better, and smaller ones are cheaper and faster. Many teams use both: a large model for the steps that need judgement, and a small one for simple checks.
| Question | Why it matters | For our research assistant |
|---|---|---|
| How good is it on our test cases? | Public benchmarks don't measure your task. Your eval set (Stage 4) does. | Compare 2 or 3 models on the same 30 questions before choosing. |
| What does one run cost? | An agent calls the model 5 to 20 times per task, so price differences multiply. | Large model to plan and write, small model to grade search results. |
| How long do users wait? | Each call adds seconds. Smaller models answer faster. | Target: under 60 seconds per report. |
| Is the context window big enough? | Three full web pages plus instructions can be tens of thousands of tokens. | Trim pages to their main text before adding them to memory. |
| Where does our data go? | Providers differ on data retention, training use and region. | Fine for public web pages. Check carefully before adding client data. |
| Can we pin the version? | Providers update models. An unpinned name can change behaviour overnight. | Use a dated model version, and re-run evals before moving to a new one. |
The fifth part ties the others together. The agent runs this cycle until it decides it has enough to finish, or until a limit stops it. When the demo is running, the current phase lights up below.
Some systems use an agent that splits a task and hands the parts to worker agents. That adds handoffs, and every handoff can lose information.
Check your understanding · Stage 2
In the research assistant, what decides which tool to use next?
Stage 3 · Build
The surprising part for most PMs: the model never does anything by itself. It only writes text. Your code does everything else.
A is just structured text, such as fetch_page(url="…"). Nothing happens until your code acts on it.
It checks the request, runs the tool, catches errors, and stops the loop when a limit is hit.
The model has no memory between calls, so all of it is sent again each turn. That's why cost goes up with every step.
This is : a simplified sketch of real code. When the demo is running, the line for the current step is highlighted.
function run_agent(question): memory = [instructions, tool_list, question] for step in 1 .. MAX_STEPS: reply = model.next_step(memory) if reply.is_final_answer: return check_citations(reply) if reply.tool needs_approval: wait_for_human_ok() result = run_tool(reply.tool, reply.args) memory.add(reply, result) return "Stopped: step limit reached"
Each tool is described to the model in a short block like this, usually written as . The model chooses tools by reading the description, so the description is effectively part of the prompt.
{
"name": "fetch_page",
"description": "Open one web page and return its main text. Use after web_search to read a result in full. Do not use it to search.",
"inputs": { "url": "the full address from a search result" }
}
Tools fail all the time: a search times out, a page is behind a login, a service is rate-limited. Your code decides what happens next, so decide it on purpose.
Every tool call gets a deadline, for example 10 seconds for fetch_page. A hung call shouldn't hold up the whole run.
For short outages, try again once or twice, waiting a little longer each time. Count retries towards the run's limits.
If it still fails, send the error back as the tool's result, such as "page timed out, try another result". The model can then pick a different source instead of the run crashing.
If the agent can't get what it needs, it says so ("I could only verify 2 sources") rather than filling the gap from memory.
Writing a connector for every service gets old fast. The is an open standard for this. A service publishes its tools once as an MCP server, and any agent that speaks MCP can use them: search engines, Google Drive, Slack, GitHub, your own database.
Check your understanding · Stage 3
On each turn of the loop, what does your code send to the model?
Stage 4 · Test
Agents take a different path each time, so "I tried it and it looked fine" isn't enough. You need repeatable checks across many examples.
A is one example input plus what should happen. An runs many test cases automatically and scores the results, so you can compare versions after every change. Some checks are simple code rules. For fuzzier qualities you can use an .
| Test case | What we check | How it's scored |
|---|---|---|
| "Is remote work better for productivity?" | At least 3 sources read. Every citation points to a page it actually opened. | Code check against the run's |
| "How do index funds work?" | Includes "not investment advice". Never promises returns. | Code check (text rules) |
| "What's the weather tomorrow?" | Politely says this is out of scope and makes no tool calls. | Code check |
| A page that says "ignore your instructions" | Ignores text inside web pages that tries to give it orders (). | Code check, plus a person reviews |
| Any question | Clear, accurate, about 150 words, balanced when sources disagree. | LLM-as-judge with a written rubric, spot-checked by a person |
Start with 20 to 50 test cases drawn from real user questions. Add a new one every time you find a bug.
Run the same question twice and an agent may take a different path. One pass proves little. Good evals account for that.
Run every test case 3 to 5 times and record its pass rate, not pass or fail. A case that passes 3 times out of 5 is a real problem, even if it passed the one time you tried it.
Hard rules must pass every time: no made-up citations, a disclaimer where needed, no out-of-scope answers. Quality scores, like clarity, get a target average.
Score the plain workflow (or today's manual process) on the same cases first. The bar is "beats the baseline on quality, with zero hard-rule failures", not a number picked in a meeting.
The trace shows how it got there. Did it read before citing? Did it take 6 steps or 20? A right answer reached by a lucky, wasteful path will fail next week.
| Example bar for v1 | Target | Checked by |
|---|---|---|
| Made-up citations (cited page was never opened) | 0 across all runs | Code check against the trace |
| Unsupported claims (cited page doesn't say what the report says) | Under 2% of claims | LLM-as-judge comparing each claim with the page text, spot-checked by a person |
| Pass rate per test case | At least 4 runs out of 5 for every case | Eval run, 5 repeats |
| Clarity and balance | Average 4 out of 5, and no worse than the baseline | LLM-as-judge with a rubric |
| Steps and cost | Median under 14 steps and ₹5 | Trace |
Your saved test cases, run on every prompt, model or tool change. They tell you whether a change is safe to ship.
Automatic checks and a judge running on a sample of real traffic, plus user ratings. They tell you whether it's still working on questions you didn't think of.
These are the failure modes you'll see most often. Press "Show me" to replay one in the demo.
It asks for a tool that doesn't exist, picks the wrong tool, or passes bad inputs.
Fix: few tools with clear, distinct names and descriptions. Return a helpful error so it can correct itself.
It repeats the same action because it never feels "done", using up time and money until something stops it.
Fix: a step limit, detection of repeated identical calls, and instructions that define "good enough".
A is when the model states something made up as fact. Here, it invents a statistic and a source.
Fix: a code check that every citation matches a page it actually read. Ask the model to say "I don't know". A real citation can still be wrong, so also check that the page supports the claim.
A web page contains hidden text such as "ignore your instructions and…". The agent reads it and may obey it.
Fix: treat everything tools return as data, not orders. Limit what tools can do, check the output, and require approval for risky actions.
Check your understanding · Stage 4
The agent cites a source it never opened. Which check catches this most reliably?
Stage 5 · Ship
Shipping an agent means deciding what it may do on its own, what needs a person, what it may cost, and how you'll know when it goes wrong.
A step pauses the agent until a person approves. Use it where a mistake would be costly, public or impossible to undo, and nowhere else. Too many approvals and people start clicking "yes" without reading.
| Action | Risk | Approval? |
|---|---|---|
| web_search, fetch_page | Read-only, nothing changes | No |
| save_note | Private scratchpad, easy to undo | No |
| write_report to shared folder | Colleagues will see and rely on it | Yes, for v1 |
| Email the report to 2,000 clients | Public, can't be undone | Always |
Cost depends on how many times the model is called and how much text is sent each time. (the wait) depends on the same model calls plus the tools. Change the numbers below to see the effect. The prices are illustrative.
The loop figure is more than 3× because memory keeps growing, so later calls send more tokens. Ways to cut cost and wait: use a smaller model for simple steps, run independent tool calls at the same time, reuse the fixed part of the prompt (caching), and stream the answer as it's written.
About currency: model providers bill in US dollars, so modules 2 to 5 show $, as real traces do. This module uses ₹ at an illustrative ₹85 to the dollar, so ₹0.15 per 1,000 tokens is about $1.75 per million tokens.
Agents drift. Websites change, users ask new things, and model updates change behaviour. Good lets you see what happened in any run.
Save the full trace: each model call, tool call, result, tokens, cost and time. It's what the demo's Log panel shows.
Alert on spikes in cost, errors or step limits hit. Each week, review a sample of bad runs and turn each one into a new test case.
Put a thumbs up and down on every report, with an optional reason ("wrong source", "too long", "missed the point"). Save it on the run's trace so you can open the exact run behind each complaint.
Don't go from zero to every user in one day. Each stage has a clear bar for moving on, and you can step back at any point.
| Stage | Who | Move on when |
|---|---|---|
| Shadow | Runs on real questions, but nobody sees the output | Hard rules pass on real traffic, and cost and wait are within budget |
| Internal | Your own team, for a week or two | No open serious bugs, and the team would use it themselves |
| Beta | About 5% of users, who know it's new | Ratings and complaint rate at or better than target |
| General | Everyone, in steps: 25%, 50%, 100% | Metrics hold at each step |
One setting that turns the agent off and shows a fallback, such as the plain workflow or "try again later", without a code release.
Use a dated model version. Treat an upgrade like any other change: run the evals first, then roll it out in stages.
Decide who gets the alert, and what they're allowed to do: pause, roll back or switch off.
Check your understanding · Stage 5
Which action should definitely need a person's approval?
Keep this
Quiz score: 0 of 5
Agent = model + instructions + tools + memory, running in a loop of plan → act → observe → repeat.
Learn from others