Agent verification loops are the mechanism that lets a coding agent prove a task is finished before you review it. They give the agent a deterministic command it can run again and again, so a failure returns a concrete error rather than a confident claim. Nick Nisi and Zack Proser of WorkOS built a working example in their Beyond Done workshop at the AI Engineer conference in New York in October 2026.
How do agent verification loops actually work?
Agent verification loops work by giving a coding agent a deterministic command that exercises the application and reports pass or fail with evidence. The loop runs until the probe passes, so a failure drives the next fix instead of reaching a human reviewer. Nick Nisi (an engineering manager at WorkOS) and Zack Proser (an engineer on WorkOS's Applied AI team) presented the pattern in 2026.
The core claim comes from a short post by Lauren Tan, a software engineer, which the speakers quoted: "If an agent can verify its own work, nothing else matters." The useful reading is narrower than the slogan. Verification does not make an agent correct. It makes incorrectness detectable without a human reading every line.
Proser described a failure mode that motivates the whole approach. If a developer is juggling a dozen or more terminal sessions, each lane of work gets a slice of attention. Proser said his normal day involves 10 to 15 agents running at once, and he described an average of 18 or 30 terminal tabs. At that volume, an agent that reports success, and a build that passes, can still ship a feature that does nothing. Three things have to line up for the agent to catch that itself:
- A goal stated as an outcome, not a procedure.
- Tools that let the agent observe the running application.
- A repeatable check whose output names what failed.
The second item is where most teams are missing a step. An agent in a terminal cannot see a rendered page, a failed network request, or an empty database row unless something exposes them as data the agent can read.
Why does a prompt-based check drift across agent runs?
Prompt-based checks drift because large language models are non-deterministic, so thirty agents given the same verification instructions can take thirty different paths. A single command backed by real browser and shell calls behaves the same way every time. Proser made this point directly: the same prompt across many sessions produces variance, while a scripted probe does not.
Written instructions like "always verify in the browser" leave room for interpretation. "Reload the page and confirm the email persists" is better. A list of five such reminders is better still, and that is roughly what a saved skill or a project instruction file contains. Nisi argued the ceiling is low: anything omitted from the instructions becomes an assumption the agent invents. If you do not fill in an outcome, the model infers one, and its assumption may not match yours.
The speakers' prescription is to move the check into code and hand the agent one command to run. The command can then be re-run as many times as needed, and its output feeds back as a concrete error message rather than a sentence the agent must interpret. Good error messages matter more here than in hand-written software, because the agent is the reader that has to act on them.
This is also where the idea connects to ordinary test practice. The loop is red, green, refactor, with the agent running all three steps. What changes is who reads the failure.
What did the Midnight Snack Club demo show about false success?
The Midnight Snack Club demo showed that a UI can confirm a success message while silently discarding the data. Nisi typed an email into the waitlist form, saw a confirmation, and found the address on the page. After a reload, the list was empty, because nothing had been persisted. In the live run, snack join [email] succeeded while snack storage reported empty storage. Proser framed that as the moment the system pays for itself: the agent learns it is not done and keeps working.
That gap is the whole case for verification. A test that checked only the confirmation element would pass forever. The agent's report would be accurate as far as it went and wrong about the outcome a user cares about.
The speakers built a small CLI called snack, which they present as a template rather than a reusable product. It bundles the checks an agent would otherwise have to imagine:
| Command | What it does | Why it matters |
|---|---|---|
snack up / snack down | Starts or stops the app | Gives the agent a known state |
snack doctor | Checks Node, Chromium, session, build | Catches environment failures early |
snack join | Submits the waitlist form | Drives the real UI |
snack storage | Reads local storage | Proves persistence |
snack snapshot | Returns page state as JSON | Gives the agent readable evidence |
snack screenshot | Captures the page | Attaches visual proof to a PR |
snack prove | Runs a deterministic assertion | Passes or fails with detail |
snack judge | Asks a model to assess output | Handles cases a strict assertion cannot |
In the workshop, snack doctor checked that Node and Chromium were installed, that a session existed, and that the app built. That health check is the part teams skip, and it is the cheapest one to add.
What is the difference between an agent verification loop and an LLM judge?
A deterministic check compares exact state and a judge asks a model to assess whether the output matches intent, and the two catch different failures. The snack prove command uses Playwright, Microsoft's browser automation library, to fill the email field, click the join button, and assert that the status text contains the submitted address.
Judges handle the cases where exact text matching becomes brittle. In the workshop, a confirmation that reads "your seat is confirmed" instead of "waitlisted" broke the strict assertion even though the feature worked. Proser and Nisi describe these smaller models as system-one thinking, a reference to the fast, automatic mode in the psychology literature, as opposed to slower deliberative reasoning. The analogy they used: you know a light is yellow without thinking, but explaining why it looks yellow takes deliberation.
The distinction matters because each carries a different failure mode. A deterministic probe can pass while the application is broken in a way the probe does not cover. A judge can approve output that looks right and behaves wrong. Nisi's warning was blunt: a screenshot is not proof on its own, because the agent may have reached that state by a path a real user could never take.
The practical arrangement is to use the deterministic check wherever the expected state can be written down, and reserve the judge for fuzzy output where a strict assertion would fail on harmless wording changes.
How do receipts and evals support agent verification at scale?
Receipts are the stored outputs of each verification run, and they turn a one-off check into a dataset an agent can learn from later. Nisi's framing was that status comes from records rather than from what a model claims. A probe that fails produces a receipt naming what broke, which the agent can use to fix the code before the next attempt.
Proser built a tool he calls CTX (short for context) as an experiment in centralizing that authority. It started as a cross-agent memory layer so ChatGPT, Claude Code, and other harnesses could read and write the same memories. It runs as a small web app on Cloudflare. Agents connect to it over MCP and ask whether a task is complete; a verifier checks the artifact directly, for example querying the GitHub API to confirm a pull request exists and its merge checks pass. Only then does the workflow node count as satisfied, and otherwise CTX sends back the list of failing checks.
The value compounds. Every run, pass or fail, is data about how the application behaves, and that data can become the baseline an eval suite measures future changes against. Proser argued the valuable artifact may end up being the verification harness rather than the application source, because a project with good verifiers can be rewritten and reproven. He cited the Bun team's move from TypeScript to Rust as an example of a rewrite that LLM help made cheaper.
Nisi described a second tool, Riker, aimed at the last stretch of a pull request: watching reviews, fixing what they raise, rerunning the checks, and keeping the evidence. The goal is to stop a developer from spending the final 10 percent of a task nudging an agent turn by turn.
Neither tool is presented as finished. Both speakers describe their work as experimentation, which is the honest status for a practice this young.
What the failure modes look like
The workshop's sabotage exercise made the limits concrete. A hidden bug was planted with --bug random, and the agent was asked to find and prove it. The outcomes they listed included naming the bug with evidence, claiming success when the feature was broken, blaming a bug that did not exist, and presenting no evidence at all.
An agent that says nothing is wrong when nothing is wrong is a correct answer too, which is easy to forget when you are hunting for a result. Nisi also noted a stale-proof problem: a receipt from an earlier run can describe code that no longer exists, so the harness has to be flexible enough to invalidate old evidence. In the live run, the doctor command showed a previous proof marked fail, but the source had changed, so that receipt no longer applied.
Which tools should you add to an agent verification loop first?
The highest-value starting point for a web app is a small CLI that wraps the checks a developer would perform by hand, plus a health command that confirms the environment is sound. Proser and Nisi both name this as the change that unlocked throughput on their own teams.
Everything after that is specific to the app. A sign-on flow needs cookie and expiration checks. An e-commerce flow needs the full journey from login to cart, including whether the catalog took 7 seconds to appear. The speakers were explicit that no generic tool can ship this for you, because what counts as correct differs by product and by business.
Chrome DevTools access lets an agent inspect rendering, network traffic, and performance instead of guessing at a bottleneck. Proser and Nisi also pointed at loop-driven development, which structures a task so the agent can observe state and correct itself rather than working blind until the end. For native clients on iOS or OSX, the same idea applies with different tooling.
For teams on Claude Code, Anthropic terminal-based coding agent, Nisi suggested putting the instruction to run verification in a shared project instruction file so every session inherits it. Hooks and CI can enforce the same step for local runs and pull requests. Nisi's own preference was an agents.md file in the repo, worded as "before you OpenAI PR, run this", because every agent reads it without being asked.
What does agent verification not prove?
Agent verification proves the specific things the probe covers and no more, so a passing loop is evidence about a check rather than a guarantee about the product. That limit is why the speakers keep the human review step for anything that matters.
A judge can be convinced by output that satisfies the surface of a request. Proser described reviewing an agent change of 10,000 lines that did nothing, passed review, and broke nothing. A clean verification report would not have flagged it as wasted work.
Stored receipts raise their own questions about what gets retained, where it lives, and who can read it. The workshop does not address access control or data handling for those artifacts, so treat that as decisions your team makes rather than properties the pattern supplies.
The recursive self-improvement framing that Proser raised, in which codebases evolve and hill-climb against their own evals, is a direction rather than a current standard. Build the harness you can measure today and treat the rest as a bet.
How does the WorkOS team structure agent-driven work?
WorkOS organizes agent work around cleanly decomposed tasks, each with a prompt that states the outcome and a way to verify it, rather than one long session asked to fix everything. Proser said that pattern is what lets agents finish work without constant intervention. He cited colleagues shipping around 900 pull requests a month with no incidents.
The alternative he described, a single multi-day session with a running list of fixes, produces a large pull request and a slow review. Decomposition keeps each lane small enough that a probe can cover it and a reviewer can read the evidence quickly.
For context, WorkOS makes enterprise features such as single sign-on and directory sync available to SaaS products through an API. That background is where the speakers' internal tooling experience comes from, not a claim that the product itself verifies agent output.
Frequently asked questions about agent verification loops
What are agent verification loops?
Agent verification loops are a repeating cycle in which a coding agent runs a deterministic check on its own work, reads the failure, fixes the problem, and reruns the check until it passes. The output serves as evidence for a human reviewer.
Do I need an LLM judge to verify agent work?
Not always. Start with deterministic assertions, such as a Playwright test that checks the expected state directly. Add a judge only where exact text matching would fail on harmless wording changes, and keep in mind that a judge's approval is weaker evidence than an exact assertion.
Why is a passing screenshot not proof?
A screenshot shows one moment of state, and an agent can reach that state through a path a real user could never take. The workshop recommends driving the actual flow start to finish and asserting on the results.
Does verification replace code review?
The speakers treat it as a filter that reduces review load, not a replacement. Their own claim about a team member shipping around 900 pull requests a month is anecdotal and comes from someone who still reviews important changes.
What should I build first?
A small CLI that starts the app, checks that it is healthy, exercises the main user flow, and reports its state in a form an agent can read. Add a judge later if fuzzy output needs one.
How do I get my whole team to run the checks?
Put the instruction in a repository-level agents.md file so every agent session inherits it, and back it up with hooks and CI. Nisi said the file approach works because agents read it without being prompted, while a skill depends on the model choosing to use it.
Can I run verification in CI instead of locally?
CI catches regressions, but the local loop is where the speed comes from. Proser runs the checks locally so the agent can self-correct before he returns; CI would add a slower feedback cycle.
What is CTX?
CTX is Proser's experiment in centralizing verification authority in a web service. Agents connect over MCP, ask whether a task is complete, and CTX checks the artifact (such as a PR via the GitHub API) before marking the node satisfied.
What is Riker?
Riker is Nisi's tool for babysitting pull requests: it watches reviews, fixes what they raise, reruns checks, and keeps the evidence. Its purpose is to remove the final 10 percent of turn-by-turn nudging.
How do receipts help future evals?
Every stored run, pass or fail, becomes a record of how the application behaved. That record can seed an eval suite so future changes can be measured against a known baseline, and agents can hill-climb against it rather than start from nothing.
Where does Skala Blog fit?
The workshop's argument is that evidence beats assertion. A recorded talk holds the explanation, the caveats, and the reasoning that a plain summary tends to drop, which is why so much of it never reaches a written form.
If you have that kind of material sitting in a YouTube video, you can paste the URL into Skala Blog, let it transcribe the talk, and generate a structured article you review and edit before publishing.
For developers in Brazil working through the same ideas, the resources at Crazystack and the Crazystack Typescript track cover the TypeScript side of the same stack, and the Bootcamp do Dev Doido is another route into the material. Gustavo Dev Doido writes about developer education in Portuguese, and his work is one example of what a well-structured transcript can become.
The habit worth copying from the workshop is proof. Write the piece, then check it against the source before you publish. If you have a talk that deserves to be read, try turning a video into an article with Skala Blog.
Fork this article
Start a new branch from the same video, shaped your way. You keep the credit; the original keeps the attribution.
A fork in another language is filed as a translation of this article, so the two pages point at each other. You can unlink it later from the editor.
0/240
You are creating
- Format
- For
- Language
- Source
- Your angle
You will be asked to sign in before it is generated.
Buy credits