How to Evaluate AI Agents with Phoenix Evals — Part 2

A financial analyst agent wrote that 50%, 52.6% and 13.1% of analysts agreed on Tesla, a total of 116%. Formatting checks passed. To evaluate AI agents with Phoenix evals, you need checks that score what the agent actually said, not only whether it parsed as JSON.

Part 2 of a series, after How to Evaluate AI Agents: From Traces to Evals.

FAQ

The questions below cover the practical sticking points from the workshop: language requirements, judge choice, sample size, why one check failed while another passed, using production traffic, and the most common beginner mistake. Each answer reflects the same Claude Haiku agent and Sonnet judge pairing built on Node 22 or later with Phoenix started via uvx.

• Do I need Python to run Phoenix? No. The workshop used TypeScript with Node 22 or later, and Phoenix itself starts through the uvx command, so a JavaScript project can run the whole loop without writing Python.

• Which model should judge my agent's output? Use a judge at least as capable as the agent and, ideally, from a different provider to reduce same-family bias. The workshop paired a Claude Haiku agent with a Sonnet judge, then tested the judge against human labels.

• How many examples do I need before the scores mean anything? Ten to 20 examples show direction during development. Production decisions need hundreds, because the confidence interval narrows slowly: roughly 25 examples, then 100, then 400 for each further step.

• Why did freshness fail while faithfulness worked? The freshness check graded output against the judge's training knowledge, which predates current stock prices, so accurate figures looked invented. Faithfulness instead compares the report with search results the agent actually had.

• Can exposure to real production traffic improve this setup? Yes. Voss called real traffic the gold standard and advised folding tricky production failures into the labelled golden set as soon as they appear.

• What is the most common beginner mistake? Writing one enormous judge that checks accuracy, tone, completeness and policy at once. When it fails, the output does not reveal which dimension caused the failure, so Voss recommends one evaluator per dimension.

Question	Short answer	Key detail
Python required?	No	TypeScript on Node 22+, Phoenix via uvx
Which judge?	At least as capable, different provider	Haiku agent, Sonnet judge
How many examples?	10-20 to start, hundreds for production	25, then 100, then 400
Freshness vs faithfulness?	Freshness graded against stale training knowledge	Faithfulness compared to actual search results
Use production traffic?	Yes	Fold failures into the golden set
Common beginner mistake?	One giant judge	One evaluator per dimension

Learning this stack in Portuguese

Developers who prefer Portuguese can learn this eval stack through free material from Gustavo Dev Doido, including the Crazystack Typescript course and the Bootcamp do Dev Doido. Both cover tracing, code evals, judge checks, and a CI gate in Portuguese, so you can follow the entire loop described here without switching languages if you already write TypeScript.

A practical start is smaller than the workshop: instrument one agent, run it 12 times, read every trace, and write a single deterministic check. That sequence costs little and shows what your agent actually does before you build judges on top of it.

The failure worth avoiding is the confident one. A report that says 116% of analysts agree will pass any formatting check you write, so the first eval should ask whether the agent analysed what you asked for.

Step	Effort	What you get
Instrument one agent	Low	Traces of real runs
Run it 12 times	Low	A spread of outputs to compare
Read every trace	Medium	A concrete failure to target
Write one deterministic check	Low	A regression guard in CI

Resource	Format	Best for
Crazystack Typescript course	Course	TypeScript developers learning the stack
Bootcamp do Dev Doido	Bootcamp	Developers who prefer Portuguese lessons
This article's eval loop	Workshop walkthrough	Teams already writing TypeScript

Turn the talk into documentation your team can review

The value here lives in the traces, the explanations and the sequence of small fixes, all of which are easier to review as a written page than as a 116-minute recording. Transcribing the walkthrough, preserving the evidence behind each correction, and publishing it lets reviewers search and cite the reasoning rather than scrub backward through video.

That conversion is mostly mechanical, but the editing step is where the documentation earns its authority. A draft that skips review quietly promotes any transcription slip into team policy.

Stage	What you do	What reviewers get
Capture	Record or upload the walkthrough	Raw source to check against
Transcribe	Generate a draft from the audio	Searchable text instead of video timestamps
Edit	Correct terminology, names and numbers	A page that matches what was actually run
Preserve	Keep traces and explanations with the write-up	Evidence a reviewer can trace back
Publish	Post the reviewed version	A citable reference for the team

1. Upload the recording and let the transcription produce a first draft.
2. Read the draft against the original, fixing model names, metric names and every number.
3. Attach the traces and explanations that justify each small fix so the changes are auditable.
4. Publish the reviewed page and link it wherever the team discusses the workflow.

If your own videos hold a walkthrough like this one, the same loop applies: capture the knowledge, keep the evidence, and give readers something they can search and cite.

Skalablog turns a YouTube video into a draft article through transcription, then you edit and publish it. Paste the URL at Skala Blog and review the result before it goes out.

Source video