Module 5 · Langfuse in depth · about 25 minutes
Module 2 introduced Langfuse as the "camera and receipts" for an AI app. This module opens it up. We'll keep the same example, a research assistant that takes a question, searches the web, reads the results and writes a short cited report, and use Langfuse to debug a real-feeling customer complaint.
Words with a dotted underline, like , are technical terms. Tap one for a plain-English definition.
Interactive demo
A customer says a report was wrong and slow. Use a Langfuse-style screen to find the run, work out what went wrong, then test and ship a fix. Hints guide you, and every step has a "Show me" button.
"Tuesday's index-fund report cited a dodgy source and took ages."
| Question | Tags |
|---|
Score = citation_accuracy from an automatic judge, 0 to 1.
research-reportThe write step fetches this prompt from Langfuse by its production label. Version 4 is a fix for Priya's problem, waiting on the staging label.
Moving a label changes which version this line returns. No code release is needed. Apps pick up the change once their cached copy refreshes.
research-questions (10 items)Each item is a saved question with notes on what a good answer cites. Item 1 was added from Priya's ticket. Both versions answer every item, then evaluators score the answers: a code check for citation accuracy, an LLM judge for clarity, and the cost. Ten items is enough to confirm this fix, not to prove v4 is better overall (see Section 7).
| # | Question | Citation v3 | Citation v4 | Clarity v3 | Clarity v4 | Cost v3 | Cost v4 |
|---|
Run the experiment first. Many teams require an eval before a prompt reaches production.
1 · What Langfuse is
Langfuse records what your app does, stores and versions its prompts, and scores the quality of its answers. Engineers, PMs and reviewers all work from the same screens.
The code is public. Its GitHub repository says it is , except for folders marked as enterprise. A few add-on features need a paid licence key.
Use Langfuse Cloud, which offers EU and US , or it on your company's servers. Section 9 covers what that means for data.
Langfuse is not made by LangChain Inc, the company behind LangChain and LangGraph. The Langfuse GitHub README says the team has been part of ClickHouse since January 2026.
It has a ready-made connector for LangChain and LangGraph, and works just as well without them: with plain provider , other frameworks, or anything that speaks .
is LangChain's own product for the same job. Both cover tracing, prompts and evals. The differences are mostly about who makes it, where it runs and how open it is.
| Langfuse | LangSmith | |
|---|---|---|
| Made by | The Langfuse team, separate from LangChain | LangChain Inc |
| Source code | Open source (MIT core, some paid add-ons) | Commercial product; its SDKs are open source, the platform is not |
| Hosting | Langfuse Cloud, or self-host for free | LangChain's cloud; self-hosting is offered on enterprise plans |
| Frameworks | Framework-neutral, with many integrations including LangChain and LangGraph | Deepest fit with LangChain and LangGraph; also works without them |
| Typical reason to pick it | You want open source, self-hosting, or you mix frameworks | Your team is all-in on LangChain and LangGraph and wants one vendor |
Both products change fast. Check current features, plans and pricing on each vendor's site before deciding.
LangSmith isn't the only alternative. These cover the same ground with different trade-offs.
| Tool | In short | Often picked when… |
|---|---|---|
| Arize Phoenix | Open-source tracing and evals, built on OpenTelemetry. Arize also sells a hosted platform. | The team wants open source with a strong evaluation focus |
| Braintrust | A commercial platform centred on evals, datasets and prompt experiments | Evaluation is the main need, and a hosted product is fine |
| Helicone | Open source. Sits between your app and the model provider as a proxy, logging every call. | You want logging and cost tracking with almost no code changes |
| W&B Weave | Tracing and evals from Weights & Biases | The company already uses W&B for machine-learning work |
| Datadog LLM Observability, and similar | LLM tracing inside a general monitoring platform | The company wants one place for all monitoring, and needs less prompt management |
Because Langfuse and most of these accept OpenTelemetry, switching later is easier than it used to be. Try two on a week of your own traces before choosing.
Langfuse is the open, framework-neutral choice for tracing, prompts and evals. LangSmith is the in-house option if you're fully on LangChain.
2 · The data model
Everything Langfuse shows is built from a few kinds of record. Knowing them helps you ask engineers for the right things.
A is the full record of one run: one question in, one report out, with total time and cost.
An is one piece of work inside a trace. Observations nest, so a step can contain the LLM call and tool calls it made.
A groups traces that belong together, such as every turn of one chat. Replay a whole conversation in order.
A user ID ties traces to a customer. That's how you go from "Priya complained" to Priya's runs in seconds.
are short labels for filtering, such as finance or report. is extra detail, such as the app version.
A is a quality grade on a trace or observation: a number, a category or yes/no, with an optional comment.
The first three are the classics. Recent SDKs add more specific types so the screen can show what each step is. The docs list these for Python SDK 3.3.1 and later and JS/TS SDK 4 and later.
| Type | What it records | In the research assistant |
|---|---|---|
| SPAN | A generic timed piece of work, as a | Each step: plan, search, read, write |
| GENERATION | One LLM call, as a : prompt, reply, model, , cost | The model call inside plan, grade and write |
| EVENT | A moment with no duration, as an | "Report sent to user" |
| TOOL | One action, such as a function or API call | web_search, fetch_page |
| AGENT | A step where an LLM decides what to do next and may use tools | The whole assistant, if it chose its own steps |
| CHAIN | A fixed sequence linking steps | prompt → model → clean up output |
| RETRIEVER | A read-only lookup in a database or document store | Searching the company's own research notes |
| EVALUATOR | A step that grades another step's output | The grade step |
| EMBEDDING | A call that turns text into numbers for search, with tokens and cost | Indexing fetched pages |
| GUARDRAIL | A safety check that blocks harmful or off-limits content | Blocking a report that promises returns |
Check your understanding · Data model
Priya complains about Tuesday's report. Tracing is on, but nobody can find her run among 40,000 traces. What was missing?
Ask for user IDs, session IDs and tags from day one. Without them you can't find a customer's run when they complain.
3 · Getting data in
Engineers add a little code to the app, and the app sends records to Langfuse in the background. Which way they choose depends on how the app is built.
A is a one-line label above a function. @observe records that function as an observation, with its input, output and timing. Functions called inside it nest automatically.
If the app uses LangChain or LangGraph, a handler records every step, LLM call and tool call without decorating each function.
Langfuse accepts traces in the OpenTelemetry standard. The current SDKs are built on it too. Any tool that already emits OpenTelemetry data can send to Langfuse.
Ready-made cover many providers and frameworks, such as a drop-in wrapper for the OpenAI SDK, LlamaIndex, LiteLLM and the Vercel AI SDK. The docs keep the full list.
# Simplified. Names from the v3+ Python SDK docs. from langfuse import observe, get_client, propagate_attributes @observe() # this run becomes the trace def research(question, user_id, session_id): with propagate_attributes(user_id=user_id, session_id=session_id): notes = read(search(question)) return write_report(question, notes) @observe(as_type="tool") def fetch_page(url): ... @observe(as_type="generation") def write_report(question, notes): ...
# Simplified. from langfuse.langchain import CallbackHandler handler = CallbackHandler() app.invoke({"question": q}, config={ "callbacks": [handler], "metadata": { "langfuse_user_id": "priya@acme.example", "langfuse_session_id": "chat-118", "langfuse_tags": ["finance", "report"]}}) get_client().flush() # short scripts: send before exit
The SDK sends records in batches in the background, so tracing adds little delay for users. Exact names vary by SDK version: the current docs describe a newer major version than v3, and older code may use update_current_trace(...) instead of propagate_attributes. Ask engineers which version you're on.
Adding tracing is usually a small code change, not a project. If engineers say it's hard, ask which of these four routes they looked at.
4 · Cost and token tracking
Model providers charge per token. Langfuse turns the tokens in each LLM call into a dollar figure, then adds it up per trace, user, tag or model.
Token counts usually come straight from the provider's reply. Langfuse multiplies them by the model's price from a . If the app sends a cost itself, that wins.
Langfuse ships prices for popular OpenAI, Anthropic and Google models. Teams add their own for fine-tuned, self-hosted or discounted models. Price changes apply to new calls only.
Cost sits on generation and embedding observations. A slow web fetch costs time, not tokens, unless its text is later sent to a model.
Because every trace carries a user ID and tags, a can show cost per customer, per feature tag or per model. The Metrics API exports the same numbers to other tools.
| Step | Model | Tokens in → out | Price per 1M tokens (in / out) | Cost |
|---|---|---|---|---|
| plan | chat-large | 610 → 170 | $3 / $15 | $0.0044 |
| grade | chat-small | 1,150 → 40 | $0.25 / $1.25 | $0.0003 |
| read (notes) | chat-small | 2,600 → 210 | $0.25 / $1.25 | $0.0009 |
| write | chat-large | 1,900 → 300 | $3 / $15 | $0.0102 |
| Total | 6,980 | $0.0158 |
Illustrative prices. The write step is about two thirds of the bill because it uses the large model, which costs 12 times more per token than the small one, on a long input. Its 1,900 input tokens ($0.0057) cost more than its 300 output tokens ($0.0045). At 10,000 reports a month that's about $158, or roughly ₹13,400 at ₹85 to the dollar.
Ask for cost per run and per feature, not just the monthly bill. Tags on every trace are what make that breakdown possible.
5 · Latency analysis
is how long the user waits. Langfuse shows it two ways: inside one trace as a timeline, and across many traces as percentiles.
The timeline draws each observation as a bar. Long bars are slow steps. Bars that overlap ran at the same time. Bars in a staircase ran one after another and might be sped up.
Averages hide bad experiences. is the time that 95% of runs beat. If p95 is 22 s, one user in twenty waits longer than that.
Track p95, not the average. It's the wait your unluckiest users feel, and the timeline shows which step to fix.
6 · Prompt management
Instead of hard-coding prompts, the app fetches them from Langfuse by name. Every edit becomes a new numbered version, and labels decide which version is live.
Every save creates a new version: v1, v2, v3. Old versions are kept, so you can always see what changed and when.
A is a movable pointer to one version. production is what the app fetches by default. Teams add their own, such as staging.
To , move the production label to the previous version. No deploy needed.
The lets you try a prompt against a model in the browser, and compare versions side by side, before saving.
When the app records which prompt it used, each trace shows the version, for example research-report v3. You can compare cost, latency and scores by version.
can notify Slack or another system whenever a prompt version is created or changed.
# Simplified. langfuse = get_client() prompt = langfuse.get_prompt("research-report", label="production") text = prompt.compile(question=q, notes=notes) # fills {{question}} and {{notes}} # Pass the prompt along when recording the LLM call, so the trace shows its version.
The SDK keeps a of prompts in memory, so fetching adds almost no delay, and the app keeps working with the cached copy if Langfuse is briefly unreachable.
Moving a label changes what every user sees, without a code release or a code review. Treat it like a release.
Use project roles so most people can draft and test prompts, but only a few can move production. Langfuse can protect chosen labels so only certain roles may change them; check which plan includes this.
No label move without an experiment on the dataset, with the scores written down. In regulated products, add a recorded sign-off (module 6 calls this release sign-off coverage).
Send a webhook to the team channel on every change, and watch live scores for a day after. Know who rolls back, and how fast.
Check your understanding · Prompts
A colleague improved the report prompt and tried it on two questions. They want to move the production label now. What do you ask for?
Prompt management lets you ship and undo wording changes in minutes. Pair it with an eval so "ship" isn't a guess, and limit who can move the production label.
7 · Evaluation
An measures answer quality. Langfuse supports several sources of scores, and a way to test changes before they ship.
Code checks (does every [n] match a real source?), user feedback (thumbs up or down), human reviewers, and LLM judges all write scores in the same format.
An evaluator is set up in Langfuse with a judging prompt. Langfuse runs it, not your app, on a sample of live observations or experiment items, and stores a score with the judge's reasoning.
An is a to-do list of traces for people to review and score, such as every report with a low judge score.
A is a saved set of test questions. An runs a version of your app on every item and scores the results, so you can compare versions side by side.
# Simplified. Names from the current Python SDK docs. dataset = langfuse.get_dataset("research-questions") result = dataset.run_experiment( name="research-report v4", task=run_assistant_with_v4, # answers one item evaluators=[citation_accuracy, clarity_judge])
Experiments can also be run from the Langfuse screen, and a GitHub Action can fail a code change if scores drop below a threshold.
Small datasets are good for catching a specific bug, and bad at proving one version is better overall. A rough rule: on a dataset of n items, a pass rate can be off by about 1 ÷ √n just by chance.
| Dataset size | Pass rate could be off by about | Good enough to… |
|---|---|---|
| 10 items | ± 30 points | Confirm a specific fix, such as Priya's case, and catch obvious breakage |
| 100 items | ± 10 points | Spot a big improvement or a big drop |
| 400 items | ± 5 points | Compare two good versions with some confidence |
Two things help a lot. Compare versions on the same items, item by item, and count where they differ. And run each item more than once, because the model's answer varies between runs. The demo's 10-item experiment shows the workflow; a real decision needs a bigger dataset.
Every complaint should end up as a dataset item. Then each fix is tested against every past failure, not just the latest one.
8 · Dashboards and alerts
Traces answer "what happened in this run?". Dashboards answer "how are we doing this week?". Alerts make sure someone notices without checking.
Ready-made views of cost, usage, latency and scores over time, broken down by model, user or tag.
Build your own charts with filters, such as "cost of finance reports per day" or "average citation score by prompt version".
An watches a metric over a time window, with a warning level and an alert level. It can post to Slack, call a webhook, or start a GitHub Action.
| Metric | Warn at | Alert at | Window | Why |
|---|---|---|---|---|
| p95 latency | > 15 s | > 20 s | 1 hour | Catches slow tools before complaints arrive |
| Average citation_accuracy | < 0.8 | < 0.7 | 1 day | Quality drop, such as after a prompt change |
| Total cost | > $40 | > $60 | 1 day | A runaway loop or a traffic spike |
| Error count | > 10 | > 50 | 1 hour | Search provider outage |
Alert limits depend on the Langfuse Cloud plan. Thresholds here are examples to tune.
Check your understanding · Latency
The average report takes 11 seconds, but customers still complain it's slow. Which number should you look at?
Pick three or four numbers you care about and put alerts on them. A dashboard nobody opens doesn't protect users.
9 · Privacy and data
A trace stores prompts and replies. If a user types a name, email or account number, it ends up in Langfuse unless you stop it. Decide this before launch.
runs inside your app and replaces before anything is sent. Engineers can also switch off recording a function's input or output entirely.
On Langfuse Cloud, data stays in the region you pick, EU or US. Self-hosted, everything stays on your servers: a database, an analytics store, a cache and file storage, all run by your team.
Tag traces with an , such as production or development, so test data doesn't mix with real customer data.
Projects and roles control who sees traces. Keep access to people who need it, and agree how long traces are kept.
# Simplified. Runs in your app, before data leaves it. EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+") def hide_emails(data, **kwargs): if isinstance(data, str): return EMAIL.sub("[email]", data) return data langfuse = Langfuse(mask=hide_emails) @observe(capture_input=False) # don't record this input at all def lookup_account(account_id): ...
Newer SDK versions also offer a masking hook that runs at export time. The parameter names depend on the version.
Tracing records what users type. Agree on masking, region and access with your security team before the first real user.
Keep this
@observe(): record a functionCallbackHandler(): record LangChain or LangGraphget_prompt(name, label=…): fetch a promptrun_experiment(…): test a version on a datasetmask=…: hide data before it's sentLearn from others