How Do Evals for High-Stakes AI Agents Work?

Evals for high-stakes AI agents turn production traces into a gated test suite. The Langfuse workshop by Doneyli De Jesus and Annabell Schäfer traces a banking assistant, masks PII, builds golden datasets from real traffic and expert annotations, and blocks a prompt release when scores miss their thresholds.

Evals for high-stakes AI agents: the loop in one pass

Evals for high-stakes AI agents are a repeatable loop, not a single test suite: trace every step, monitor for failures, curate datasets, run experiments, and score each run with judges or code checks. In a 2026 AI Engineer workshop, Doneyli De Jesus and Annabell Schäfer of Langfuse applied that loop to a fictional banking assistant.

The loop has five stages, and each one feeds the next. Production traffic reveals failures, failures become dataset items, dataset items become scored experiments, and experiments become gates that stop a bad prompt from reaching users. Run in order, they look like this:

1. Trace every step — capture each model call, tool call and decision in the agent run.
2. Monitor for failures — surface broken traces and regressions from live traffic.
3. Curate datasets — turn real failures into reproducible eval items.
4. Run experiments — replay prompts and prompt versions against those datasets.
5. Score and gate — apply judges or code checks, then block a bad prompt from reaching users.

Langfuse describes itself as an open-source platform for tracing, monitoring and improving AI applications. The workshop used a demo banking assistant built for financial institutions, with PII masking, prompt versions and a GitHub CI job that runs the eval gates.

The stage order matters because each stage consumes the output of the one before it:

Stage	Input	Output	Blocks release?
Trace	Live agent runs	Step-level records	No
Monitor	Traces	Flagged failures	No
Curate	Failures	Dataset items	No
Experiment	Dataset items, prompt versions	Scored runs	No
Gate	Scores	Pass or fail on CI	Yes

The presenters framed the loop as the practice that separates teams shipping reliable agents from teams shipping demos. Every stage depends on the one before it, which is why skipping tracing to jump straight to judges tends to produce scores nobody trusts.

Why a banking assistant with green metrics still lies

A traditional observability dashboard can stay fully green while an AI agent invents an outcome. In the support example Schäfer opened with, a user asks for a refund, the agent confirms it, and the user thanks it, but the refund tool was never called and nothing happened until the customer returned weeks later.

That gap comes from nondeterminism. The agent can describe an action it never performed, and no latency alert or error rate catches the mismatch because nothing failed at the infrastructure layer.

For customer-facing financial agents, the consequences go beyond annoyance. De Jesus listed financial liability, legal liability and reputational risk as the reasons banks want tool calls, guardrails and quality scores recorded alongside every response.

The fix is to mine for the signals where behavior diverges from intent. Tracing gives you the raw material, and the rest of the loop turns that material into tests that would have failed the refund case before a customer found it.

What Langfuse records in a trace, and what it masks

A trace records the full tree of an agent run: the root span with aggregate latency and cost, each LLM generation, each tool call with its inputs and outputs, and the metadata around them. Langfuse renders that tree so an engineer can open one run and see exactly which tools fired.

De Jesus demonstrated the masking layer with a deliberately unsafe input. A user typed a credit card number in clear text, and the stored trace showed the sensitive portion redacted because a masking function matched the configured PII patterns before persistence.

That boundary matters for regulated buyers. Observability data sits in a second system, so banks want account numbers and card numbers kept in their systems of record rather than duplicated into a tracing tool. Masking at ingestion addresses that concern without disabling tracing.

Traces also carry the scores attached to a run. In the demo, compliance checks, guardrail hits and intent categories appeared next to the trace, so an engineer could filter straight to the conversations flagged for investment advice.

What survives masking, and what does not

The masking boundary splits a trace into two layers with different retention rules. The comparison below maps what the demo showed for each layer.

Trace element	Captured in the trace	Masked at ingestion
Root span latency and cost	Yes	No
LLM generation prompt and completion	Yes	PII patterns only
Tool call inputs and outputs	Yes	PII patterns only
Card and account numbers	Yes, redacted	Yes
Compliance check scores	Yes	No
Guardrail hit flags	Yes	No
Intent category labels	Yes	No

Verifying the masking layer before a regulated rollout

The demo showed masking in action on a single input, but a production rollout needs a repeatable check. The steps below follow the same pattern De Jesus used: send something unsafe on purpose and inspect what persisted.

1. Configure the PII patterns in the masking function so card and account numbers match before any test traffic is sent.
2. Send a probe run with a known-format credit card number in clear text, mirroring the demo input.
3. Open the persisted trace and confirm the sensitive portion is redacted while the surrounding tool calls and generations remain readable.
4. Check that scores for compliance checks, guardrail hits and intent categories still attach to the same trace after masking.
5. Re-run the probe whenever the masking patterns change, so the ingestion boundary stays verified across releases.

Masking protects the observability store, not the agent's own inputs: the model still sees the clear-text value at inference time. The redaction applies at the point of persistence, which is why the trace can be inspected for debugging without duplicating card numbers into a second system.

Datasets, expert annotation and the failure-mode catalog

A dataset is a set of inputs with expected outputs or expected behavior, and it functions as the test suite for an AI application. De Jesus compared it directly to the QA cases a software team would run before merging to production.

Datasets in the demo came from two directions. Teams with an existing spreadsheet imported it as CSV, while teams without one promoted good production traces into a dataset, and both routes are also available through the API and CLI.

Human annotation is the step that gives a dataset its authority. A domain expert works through a sampled queue, assigns a simple score and leaves a short comment, and the annotation stays attached to the trace as evidence.

An LLM then reads the comments and groups them into failure-mode categories, because pattern matching across free text is a task LLMs handle well. The demo dashboard ranked out-of-scope requests, informal register and unresolved disputes as the top failure modes over seven days.

LLM judges, code evaluators and why you mix them

An evaluator turns an agent run into a score. Langfuse supports LLM-as-a-judge evaluators, where a second model grades the output, and code evaluators, where a deterministic function checks something like an exact match or a required citation.

De Jesus recommended blending both kinds. A judge can assess tone, source correctness or formal register, but it carries a nonzero chance of a wrong verdict on any given run, while a code check returns the same result for the same input every time.

The failure-mode list drives which evaluator you pick. A category such as unsolicited upsell or language match becomes a named score with a threshold, and that score then appears in every experiment run against the dataset.

Judge quality needs its own check. Schäfer described labeling a representative sample by hand, comparing the judge against those human labels, and retuning until the judge agrees closely enough, with a preference for starting stricter and loosening later.

Experiments and CI gates that block a prompt change

An experiment runs a dataset against one agent configuration, so you can vary the prompt, the model or the tool set and compare scores. De Jesus showed prompt version four being promoted to production, which fired a configured automation that launched a GitHub Actions job.

The job ran the experiment, scored the new version against every evaluator, and returned a pass or fail against minimum thresholds the team had set. The demo run passed on most-include, source recall, language match, correctness and no unsolicited upsell.

The gate checks the new version against the same evaluators each run, so a regression on any single metric fails the build:

Evaluator	What it guards	Demo run result
Most-include	Coverage of the must-mention facts	Pass
Source recall	Grounding in retrieved sources	Pass
Language match	Response in the expected language	Pass
Correctness	Factual accuracy against the dataset	Pass
No unsolicited upsell	Refusal to push unrequested offers	Pass

Isolating one variable per experiment keeps the result readable. Changing the prompt and the model at once leaves you unable to tell which change moved the score, which is why Schäfer advised changing one thing at a time.

To keep an experiment readable, run it in this order:

1. Change exactly one variable — the prompt, the model or the tool set.
2. Run the dataset against the new configuration and score every evaluator.
3. Compare each score against the team's minimum thresholds.
4. Promote only if every evaluator passes; otherwise fix and rerun.

The gate produces evidence rather than an automatic merge. De Jesus was explicit that he was not letting the pipeline push to main on its own, though the same plumbing could OpenAI pull request and wait for review.

How mature teams differ from first-time teams

Entry points into the loop depend on stakes. Schäfer described low-stakes internal chat agents shipping with tracing and a few monitors first, adding datasets and evals later, while regulated customer-facing agents need ground truth data and hard limits defined before launch.

Regulated teams converge on three ways to build that ground truth. Expert curation brings domain reviewers into the dataset, shadow runs compare agent output against a human process running in parallel, and full human-in-the-loop operations gradually automate the slices where confidence is high.

The presenters also drew a line around full automation of the improvement loop itself. Asking a coding agent to fix everything without human taste in the loop, they argued, tends to produce lower-quality systems even when nobody has to review anything.

Their recommended starting point for a team beginning tomorrow was the Langfuse agent skill, a set of markdown files that teaches a coding agent the instrumentation, dataset and evaluator patterns, paired with manual review of a curated sample.

Frequently asked questions about evals for high-stakes AI agents

• What is an eval for a high-stakes AI agent? An eval is a scored check on one agent run, such as an LLM judge grading source correctness or a code function testing an exact match. A set of evals run against a dataset turns a single test into a repeatable quality gate for releases.

• Do I need LLM-as-a-judge or code-based evaluators? Most teams need both. Judges handle open-ended qualities like tone and source accuracy, while code evaluators return identical results for identical inputs and anchor the blended score when a judge drifts.

• How do I stop a prompt change from reaching production? Version the prompt, trigger a CI job on each change, run the dataset experiment and compare every score against a minimum threshold. A run that misses a bar fails the gate and the change waits for review.

• Can I trace a banking assistant without storing customer PII? Yes, if masking happens before persistence. Langfuse applies configured PII patterns at ingestion, so redacted values appear in the stored trace while the raw data stays in the institution's systems of record.

• How do domain experts fit into an eval pipeline? Experts annotate a sampled set of traces with a simple score and a short comment, and an LLM groups those comments into failure-mode categories. That keeps expert time small while still grounding the metrics in human judgment.

Building the written record of your own eval practice

The most transferable lesson from the Langfuse workshop is that the loop only works when someone writes down what failure looks like. Traces, dataset items, annotation comments and thresholds are all written artifacts, and the teams that ship reliable agents are the ones that keep producing them.

That habit of turning lived practice into durable text is what the wider developer community around Gustavo Dev Doido, the Crazystack Typescript program and the Bootcamp do Dev Doido keeps emphasizing: document the reasoning, not just the result. Skala Blog follows the same principle by converting recorded explanations into articles that can be searched, cited and corrected later.

If your team records walkthroughs, conference talks or internal explanations of how you evaluate agents, that material already contains the structure of a solid article. Skala Blog turns a YouTube URL into a transcription and then into a publishable draft you can review before it goes live.

Source video