Module 7 · Agent Field Guides · about 25 minutes
The research assistant from modules 1 to 5 passes its evals. Now it has to reach real users without surprises, feel trustworthy when they use it, and pay for itself. This module covers the product work around launch: staged rollout, designing the experience, unit economics, build or buy, and what the first six weeks really looked like.
Words with a dotted underline, like , are technical terms. Tap one for a plain-English definition.
Interactive demo
Each click is one week. Look at the numbers, then decide: move to the next stage, hold, roll back, or hit the kill switch. Pick a scenario to see what can go wrong between stages.
1 · Rollout
Agents fail in ways your test cases didn't imagine. A staged rollout limits how many people see each new failure, and gives you a moment to fix it.
| Stage | Who sees it | Typical length | Move on when |
|---|---|---|---|
| Shadow | Nobody. The agent runs on real questions, and its output is logged but not shown. | 1 to 2 weeks | Hard rules pass on real traffic, and cost and wait are within budget |
| Internal | Your own team | 1 to 2 weeks | No open serious bugs, and the team uses it without being asked |
| Beta | About 5% of users, who know it's new and how to give feedback | 2 weeks | Task success, ratings and complaints at or better than target |
| General | Everyone, in steps: 25%, 50%, 100% | About a week per step | The numbers hold at each step, and on-call is quiet |
Write the bar down before the stage starts. Deciding afterwards whether 82% is "good enough" is how teams talk themselves into bad launches.
A decides who gets the agent: a percentage, a team, a customer tier. Moving between stages becomes a setting, not a release.
One switch turns the agent off for everyone. Decide in advance what users see instead: the plain workflow, a "temporarily unavailable" note, or a human channel.
Use a dated model version, not "latest". A provider's update can change behaviour overnight. Treat a model upgrade like a release: evals, then shadow, then stages.
Name who may move a stage forward, who may roll back, and who may hit the kill switch. Rolling back should need one person, not a meeting.
Check your understanding · Rollout
At 25%, made-up citations jump from 0 to 3.5 per 1,000 overnight. Nothing in your code changed. What do you do first?
2 · Experience
An agent is slower and less predictable than a normal feature. Good design makes that visible and manageable, instead of hiding it.
Say what it does and doesn't do, before the first question: "Researches public web sources and writes a short cited summary. Doesn't use company data or give investment advice."
A 40-second wait feels broken behind a spinner. Stream steps as they happen: "Searching… Reading 3 sources… Writing…"
Put citations next to each claim, with links. People trust what they can check, and checking catches mistakes.
"I could only verify 2 sources" or "Sources disagree on this" beats a confident paragraph that hides the gap.
When it can't finish, say why and offer the next step: try a narrower question, see the sources it did find, or ask a person.
Edit the draft, remove a source, or ask for a shorter version. Corrections are also your best feedback data.
Thumbs up or down on each report, with quick reasons: "wrong source", "too long", "missed the point". Save it on the run's trace.
Show the reviewer exactly what will happen and what changed, with approve, edit and reject. Ten approvals a day get read; two hundred get rubber-stamped.
For focused, individual tasks, remote work can raise output[1]. Hybrid work looked like a safe middle ground in a two-year trial[2]. The main cost is cross-team collaboration[3].
Sources mostly agree. One study covered a single company, so treat it with care.
Check your understanding · Experience
Users say reports “feel unreliable”, though evals score them well. Which change is most likely to help?
3 · Economics
Model tokens are the cost everyone tracks, and often not the biggest one. Add the platform, the human review, and the work redone when the agent fails.
| Lever | What it changes | Watch out for |
|---|---|---|
| Raise the success rate | Every failure costs an analyst's time. Going from 88% to 94% often saves more than halving token costs. | Measure success on real tasks, not just eval cases |
| Route simple steps to a small model | Grading and note-taking rarely need the large model. Often cuts token cost by a third or more. | Check the small model on your eval set first |
| Cache the fixed part of the prompt | Instructions and tool descriptions repeat every turn. Providers charge less for cached input. | Only helps when the start of the prompt stays the same |
| Trim what goes into memory | Main text only, not whole web pages. Fewer tokens on every later turn. | Don't cut the evidence the report needs |
| Review smarter | Review only flagged or low-score reports, plus a small random sample. | Keep the random sample, or you can't measure what the flags miss |
| Model | How it works | Good for | Risk |
|---|---|---|---|
| Included in the plan | No extra charge | Driving adoption of a core feature | Heavy users eat your margin. Add a fair-use limit. |
| Usage-based | Per report, or credits | Costs that vary a lot between users | Users ration it and never build the habit |
| Outcome-based | Per successful task | Clear, checkable outcomes | You must agree on what "success" means, and you carry the cost of failures |
Check your understanding · Economics
Token cost is ₹4 per report. Success is 88%, and an analyst redoes each failure at ₹1,000. Where's the bigger saving?
4 · Build or buy
Before the team spends a quarter building an agent, check whether someone sells one that's good enough.
| Build on frameworks | Build on a provider's agent platform | Buy a product | |
|---|---|---|---|
| Example | LangGraph, LangChain and Langfuse, as in modules 2 to 5 | A model provider's agent SDK or hosted agent service | An off-the-shelf research or support assistant |
| Best when | The agent is core to your product, or needs your data and rules | You want speed and are happy with one provider | The job is common, and not where you compete |
| Control | Full | High, within the platform's limits | Low: settings and prompts at most |
| Time to first version | Weeks to months | Days to weeks | Days |
| Ongoing work | Your team owns everything: evals, upgrades, on-call | Shared with the provider | Mostly the vendor's |
| Ask about | Who maintains it in a year? | How hard is it to switch provider? | Where does our data go? Can we see traces and run our own evals? |
5 · Case study
An illustrative story, built from problems teams commonly hit. Each week shows what happened, how it was spotted, and what changed.
One in five "read" pages was a login wall. The agent cited them anyway, from the snippet. Spotted by the citation check: the cited text wasn't in the page. Change: fetch_page now reports "page needs a login", and the agent picks another source.
The team skimmed the first line and stopped. Spotted by feedback reasons: "too long" on a third of reports. Change: a one-line answer first, then the detail. Thumbs-down fell by half.
Users asked about their own company's numbers, which it can't see. It searched the web and guessed. Spotted by reviewing low-score traces. Change: an input check that says what it can't do, and 12 new test cases.
The provider updated the model behind an unpinned name. Made-up citations reappeared. Spotted by the citation-accuracy alert within an hour. Change: rolled back to 5%, pinned a dated version, and added "upgrade the model" to the release checklist.
The legal team started researching 200-page PDFs. Cost per report tripled for them. Spotted by cost per user in the dashboard. Change: pages trimmed to the relevant sections, notes moved to a small model, and a fair-use limit.
89% task success, 0 made-up citations in automated checks, p95 wait of 44 seconds and ₹3.90 per report. Weekly review of 50 bad traces keeps feeding the eval set, now at 140 cases.
Evals get you to launch. Monitoring, feedback and a staged rollout find the problems your evals didn't imagine, while only a few users can see them.
Keep this
Ship in stages, design for trust, price on all-in cost, and keep a kill switch within reach.
Learn from others