Module 6 · Agent Field Guides · about 35 minutes
Metrics for a compliant wealth AI bot
What to measure when an AI assistant talks to investors. Modules 1 to 5 followed a research assistant. This module takes the same building blocks into a regulated setting: a wealth assistant for a mutual fund distributor. It answers fund and portfolio questions, helps start a SIP, and hands off to a relationship manager. Think of the research assistant's index-fund report, turned into a live conversation with investors. Seven categories of metrics, each split into P0, P1 and P2 priorities.
Tap any word with a dotted underline, like , for a plain-English definition. Use the P0 / P1 / P2 buttons at the top to filter the metrics table and cards.
Measured, alerting and owned before go-live. Most are breach metrics: zero-tolerance, where a single breach is an incident. A few are guard metrics that must be live at launch and alert on a spike, such as negative feedback or spend.
no launch without theseMeasured from launch and reviewed weekly. They drive fixes, and release go/no-go decisions.
weekly reviewAdded once the bot is stable. Reviewed monthly or quarterly. They shape the roadmap and long-term trust.
monthly / quarterlyHands-on demo
Spot the breach
Eight replies the wealth assistant drafted. For each, decide whether it's safe to send, or which P0 rule it breaks. Then see which automated check would catch it.
At a glance
The seven categories and their top metrics
Each row is a category with its owning team. Click a category name to jump to the detail: what each metric means, how it's measured, the target and how often it's reviewed.
| Category | P0 · launch blockers | P1 · core health | P2 · optimisation |
|---|
Measuring the P0s
How to prove a zero
Many P0 targets are "0". But finding no breaches in a sample doesn't mean there are none. Two questions decide whether a "0" on the dashboard means anything: how much did you check, and how good are the checks?
Zero in a sample isn't zero overall
If people review n replies and find no breaches, the most you can say with 95% confidence is that the true rate is below about 3 ÷ n. Statisticians call this the rule of three.
The checks need checking
Automated checks run on 100% of replies, so they see far more than any reviewer. But a check that misses 1 breach in 5 will still report "0" on a bad day. That's why detector recall is now a P0 in the compliance category: measure how many known breaches the checks catch, before launch and on every release.
- Seed a test set with real breaches from testing and red-teaming, labelled by Compliance.
- Run the checks on it with every prompt, model or check change. Recall below the bar blocks the release.
- Feed it with every breach that human review finds and the checks missed.
Don't judge every reply in the reply path
Running an LLM judge on every reply before it's sent roughly doubles model calls and adds seconds of waiting. Layer the checks instead, from cheap and instant to slow and thorough.
| Layer | Runs on | When | Cost and delay | Good at |
|---|---|---|---|---|
| Rules and small classifiers | 100% of replies | Before sending: can block | Milliseconds, almost free | "Assured returns", missing disclosures, numbers vs the NAV feed, PII patterns |
| LLM-as-judge | 100% of replies, or a large sample | Just after sending: flags within minutes | One extra model call per reply judged, using a small model where it's accurate enough | Advice-boundary wording, subtle nudges, made-up facts |
| Human review | Every flag, plus a random sample | Daily and weekly | Reviewer time | Confirming breaches, measuring how much the other layers miss |
Check your understanding · Proving a zero
Compliance reviewed 300 replies last month and found no advice-boundary breaches. What can you honestly report?
Instrumenting
Wiring it up in Langfuse
Each P0 card above now shows how it's calculated and where its data comes from. Most of it maps onto the Langfuse features from module 5. The same ideas apply in other tracing tools.
| What the metrics need | Langfuse feature | Example in the wealth assistant |
|---|---|---|
| A result from every automated check | Scores sent from your code | advice_boundary, disclosure_present, numbers_match, pii_leak |
| Judgement calls on live replies | LLM-as-judge evaluators | advice_judge, fact_check, run on every reply or a sample |
| Breakdowns by intent, language and customer tier | Tags and metadata | intent:sip_setup, lang:hi, tier:retail, risk_profile |
| Conversation-level outcomes | Sessions, plus a join with your CRM | Compliant resolution rate, repeat contact within 7 days |
| Human confirmation and weekly sampling | Annotation queues | Every flagged reply, plus a random 200 a week for Compliance |
| Detector recall and release sign-off | Datasets and experiments | The seeded compliance set, run on every prompt or model change |
| Real-time P0 alerts | Alerts and webhooks | Any confirmed prohibited_claim pages Compliance and the on-call engineer |
| An audit trail you can replay | Traces with model and prompt version, plus your retention settings | Every conversation reproducible for the required retention period |
Check your understanding · Wiring it up
Where should the “no assured returns” check run so that investors never see such a claim?
Responding
When a P0 fires: severity and runbook
An alert is only useful if everyone knows what happens next. Agree severities and a runbook with Compliance, Engineering and Service before launch.
| Severity | Looks like | Respond within | First actions |
|---|---|---|---|
| Sev 1 | A P0 breach that is ongoing or reached many investors: repeated assured-return claims, another investor's data shown, the approval step bypassed | 15 minutes, any time | Switch off the bot or the affected intent, page Compliance and the data protection officer, preserve traces |
| Sev 2 | A single confirmed P0 breach, contained: one advice-boundary reply, one stale NAV | Same business day | Fix or roll back the prompt, contact the investor if needed, search for similar conversations |
| Sev 3 | P1 health slipping: rising escalations, latency or cost over target | Next weekly review | Find the cause in traces, add test cases, plan the fix |
- Detect. The alert fires and pages the named owner, not a shared inbox.
- Contain. Use the kill switch, turn off the affected intent, or move the prompt label back. Speed beats diagnosis here.
- Preserve. Export the affected traces and note the live model and prompt versions before anything changes.
- Assess. Filter traces to find every affected conversation and investor. Count them.
- Notify. Compliance and the data protection officer decide who must be told, and when, under your licences and laws such as the DPDP Act.
- Fix and test. Add every bad conversation to the eval set and the seeded compliance set. Re-run before releasing the fix.
- Review. Hold a blameless review within a week. Update checks, thresholds and this runbook.
Check your understanding · Incidents
An alert shows the bot quoted “assured returns” to about 40 investors this morning. What comes first?
Keep this
P0 launch checklist and review rhythm
Every P0 metric in one list. Before go-live, each one should be measured, alerting, and owned by a named person. Tick them off as you confirm them with your teams. Ticks are saved in this browser only.
P0 checklist
Review rhythm
Questions to ask before launch
- Which P0 metrics are automated today, and which depend on manual sampling?
- Who gets paged for each P0 alert, and what's the response time?
- Can we reproduce any conversation from six months ago, exactly?
- Who signs off a prompt or model change, and where is that recorded?
- How do we know our automated compliance checks aren't missing things? What's their recall on the seeded set?
- How many replies does Compliance review each week, and what does a clean sample actually prove?
- Who can switch the bot off, and has anyone practised it?