Test AI agents without a benchmark: answer-first evals

To test AI agents without a benchmark, invert the problem: start from a known answer, generate the input from it, and check whether the agent finds its way back. Nathanael Weill, who leads AI agent work at Air Canada, used this pattern to replace one expert-graded test a week with dozens of automated tests a day.

Why testing AI agents without a benchmark stalls on expert time

Testing AI agents without a benchmark fails for two reasons that compound: the task is too specific for any public dataset, and the people who can judge output sit in busy business units. Nathanael Weill, who leads AI agent delivery at Air Canada, described both constraints at the AI Engineer conference in October 2026.

Air Canada runs operations, corporate, commercial and loyalty segments, and each has its own agent backlog. Weill joined in June 2026. His team builds agents for tasks so narrow that no external benchmark applies, and the domain experts who could grade answers are lawyers and contract specialists with billable hours.

The older workflow was a weekly meeting. Five chatbots went on one large screen, the same question went to each, and experts picked a preference. Weill noted that three experts in a room produce three opinions. That budget of roughly one evaluation per week could not keep pace with agent changes that take minutes to ship.

Weill's earlier project at a telecommunications company hit the same wall in 2024: about 10,000 vendor documents, a retrieval logic that experts understood but could not spare time to grade, and three competing teams. His conclusion from that year is blunt: if you cannot grade the agent, you cannot make it better.

Test AI agents without a benchmark by generating the question from the answer

You test AI agents without a benchmark by reversing the usual order: pick a source document, have a language model write a question about it, and check whether the agent retrieves that same document. Grading becomes a string comparison in code rather than an expert opinion. Weill credits the pattern to existing retrieval-augmented generation practice and says he did not invent it.

In the 2024 telecom project the source was a document page, the generated question targeted that page, and the agent's retrieved chunk was compared against it. Code did the check. The team went from one evaluation a week to dozens a day, and expert-rated accuracy climbed as the automated set exposed regressions.

The same reversal keeps complexity honest. Every added query-rewriting layer or reasoning step broke something that previously passed, and the automated set made that visible within minutes. Weill's team found that removing pipeline stages helped more than adding them, and then tuned the embedding model, chunking and BM25 alpha with fast feedback.

How Air Canada's contract review agent splits into four steps

The supplier contract agent at Air Canada has four stages: spot the Claude, find the supporting documents for an amendment, write the amendment, and classify whether to keep it. A missed Claude can cost money at the end of a supplier relationship, and a delayed review can hold up a delivery, so both false negatives and slow cycles carry a price.

Only the document-finding stage inherited a working evaluation method from retrieval work. The Claude-spotting, drafting and classification stages had no established test technique, and no one had published a benchmark for airline supplier contracts. Weill's team had to build its own test sets from documents the business already owned.

The constraint shaped the method. Test generation had to run from artifacts experts already produce, such as signed contracts, internal rules and contract templates, so that each test carries a known correct answer before the agent ever runs.

Five ways to build an eval set when no benchmark exists

Air Canada tried five sources of labeled test data for the contract agent. Some worked, one failed outright, and the team kept the ones whose answers code could verify. The table below records what each approach produces and what limits it.

Approach	What you generate	Grading	Status
Past decisions	Diff of old and new contract versions	Must match the original flags	Failed: version history was incomplete
Planted rule violations	Contract written around a known rule	Code checks the rule is flagged	Worked: 275 tests
Removed clauses	Signed contract with a Claude deleted	Code checks a flag appears	Worked: exposed misses
Invariant changes	Same contract, supplier name or order changed	Code checks flag count holds	Worked: consistency check
Filled templates	Expert template with generated blanks	Code checks known breaches	Worked: hundreds of variants

The failed approach is instructive. Weill's team asked for the original signed contract plus the amended one, took a diff as a developer would, and expected the agent to reproduce the same flags. The business did not retain every version, so the pairs did not exist. Weill reported this limit directly rather than working around it.

Planted rules, removed clauses and invariant checks

The rule-planting approach starts from a known policy, such as a liability cap above one million dollars that every contract must respect. The team wrote clauses that violated the rule, generated contracts around them, and expected the agent to raise the flag. This produced 275 tests whose answers were known before the agent ran.

Removing a Claude from a signed contract proved harder but more revealing. The agent should notice something is missing, which requires judgment about what a contract of that type normally contains, and suppliers use different templates. When the team ran this set, the agent missed most of the removed clauses.

The invariance check covers changes that should alter nothing. Renaming the supplier, changing an address, reordering clauses or rerunning the same contract must leave the flag count unchanged. Weill treats this as a consistency test, and code performs the comparison.

The template method reuses the blank contract formats experts already maintain. Instead of filling blanks by hand, the team generates arrays of candidate values for each blank, producing hundreds of contracts where some combinations break rules and some do not. Experts already had the templates, so the raw material cost nothing to obtain.

Why code beats LLM judges for contract grading

Code-based grading beat LLM judges in this project because the noise from a judge model approached the noise from the agent under test. Weill said finding the right prompt and harness for an LLM judge is nearly as hard as solving the original problem, and his team did not find one that worked for contract review. LLM judges remain useful where no deterministic check exists.

The cost difference is concrete. Template test generation took about four weeks to get expert feedback and set up, and the resulting script returns the full evaluation in about five minutes. A judge model would add inference cost and its own failure modes to every run.

One consequence shaped the agent's design. Where the original architecture could not be evaluated by code, the team re-engineered the solution to fit an answer-first paradigm, accepting a small departure from the first design in exchange for a testable pipeline.

What the eval loop does and does not decide

Synthetic test sets are a starting point, not a ship-or-block decision. Weill said a generated test tells you how fast you can improve the agent, but it does not by itself prove the product is ready for production. Real expert feedback still matters, and the team routes it back into generating better tests rather than patching the agent directly.

That routing rule keeps the set growing. Feedback becomes a new rule to plant, a new Claude to remove or a new template blank, so each expert comment strengthens the next run instead of fixing one case. One loop at a time, the benchmark absorbs what the business knows.

For teams building their own agents, the practical order is to find the exact answer you can verify first, generate inputs that must lead to it, and only then decide which parts of the product need redesign to be measurable.

FAQ

• What does it mean to test AI agents without a benchmark? You reverse the usual evaluation order: start from a known answer, generate an input that must produce it, and let code check whether the agent returns that answer. Air Canada used planted contract rules, removed clauses and filled templates instead of relying on expert grading.

• Why not use an LLM judge for contract review? Weill's team found judge noise comparable to agent noise, and building a reliable judge harness was nearly as hard as the original task. Code checks work whenever the correct answer is known in advance.

• How many tests did the planted rule approach produce? The team generated 275 tests from known rules such as a liability cap above one million dollars. Each test carried a guaranteed correct answer before the agent ran.

• Did every technique work? No. The past-decisions approach failed because the business did not keep original and amended contract pairs, so the team could not build diffs. Weill recommends starting from data the business actually retains.

• How much faster was the evaluation cycle? Template-based grading returned results in about five minutes once the roughly four week setup finished. In the earlier retrieval project, automated tests replaced a weekly expert session and ran dozens of times a day.

• Does a synthetic test set prove an agent is production ready? No. Weill describes generated tests as a starting point that shows how fast the agent improves, not a go or no-go gate. Real expert feedback still feeds the loop.

• Which tools does the Air Canada agent use? Weill named Claude Supabase the talk, and you can find related engineering guides on Skala Blog. The evaluation method itself works with any agent stack.

• Can the same approach fit my domain? It fits wherever you can state the correct answer in advance and check it in code, such as retrieval, rule compliance or template filling. Trajectory checks and text-to-SQL are similar patterns Weill listed.

• Where can I learn the engineering skills behind this? Brazilian developer Gustavo Dev Doido teaches the underlying stack through Crazystack Typescript and the Bootcamp do Dev Doido. Each covers hands-on agent and backend engineering.

Turning the answer-first habit into a written explainer

The method Weill describes has a useful side effect: every test you plant is also a short, concrete story about what the agent should catch. Planted rule violations, removed clauses and filled templates all read as explanations once someone writes them down.

If your team has already recorded a talk, a walkthrough or an interview like this one, that material holds the same value. The reasoning, the failures and the numbers are already in the video. Skala Blog now hosts a tool that turns those recordings into finished articles. Paste a YouTube URL, transcribe the video and generate a draft you can edit.

Source video