Unsupervised AI agents: when the check is the trust

Three stages, one check: that is the shortest description of Evan Kormos's argument for unsupervised AI agents. Kormos, a player-coach at Coinbase with 20 years of software delivery, describes moving from fully supervised agents to sampled review and then to human exception handling once measured confidence clears a policy threshold.

What makes unsupervised AI agents safe in a policy-driven workflow?

Unsupervised AI agents are safe to ship when three conditions hold: the work runs inside a policy boundary, each node has a check that defines acceptable output, and measured confidence clears a threshold the organization has written down. Everything else is supervision wearing a different name.

Evan Kormos set that out in an October 2026 talk at the AI Engineer conference, published as "How to Know When an AI Agent Can Work Unsupervised." He describes himself as a player-coach at Coinbase, a publicly traded US cryptocurrency exchange, with about 20 years of software delivery and 15 in enterprise financial services.

His starting move is to separate policy from the running system. Policy is the institution's original intent, written with words like must, may, and should. The system is the product-driven code that changes constantly. The build state where policy is rewritten and the run state where agents execute are different moments in time, and treating them as one is the first way trust gets muddled.

Kormos also separates the workflow shape from the work inside it. The outer process is a directed acyclic graph (DAG), a set of steps and dependencies that move forward without looping back. One widely used open-source implementation of that idea is Apache Airflow, which expresses workflows and their dependencies as DAGs in Python. The details of one node are a loop.

That split gives the article its central rule: keep the outer flow deterministic and make the inner work adaptive. An agent can iterate freely inside a node because the node's exit condition is fixed by policy.

DAG outside, loop inside: how agents fit into a workflow

A DAG fixes which steps run, in what order, and what must finish before what; the loop inside a node repeats work until the output is good enough to pass the gate. Agents belong inside the loop, not in charge of the dependency graph.

Kormos walks through commercial lending: identity checks and financial review can run in parallel, then a credit memo, risk review, documentation, and decision follow in sequence. Listing a token in a specific region is another flow he names. Software delivery is the example he develops.

Adding agents changes volume across a DAG, not just speed inside a node. If code review finishes faster, the security scan and the deployment path have to keep up. Kormos calls the result a set of new bottlenecks, and he argues operational visibility into the whole process matters more than the tooling used to author it.

Exceptions need somewhere to go as well. A canceled shipment or a change that fails a security standard should surface as a monitored path rather than a stuck ticket. Kormos warns against designing for every edge case up front; instrument the flow instead.

Parallelism is where agents pay off most in this model. Work that once cost too much to coordinate can run as independent reviews feeding a single join point, as long as the required checks stay in place.

Why the check decides whether autonomy is earned

The loop is only as trustworthy as the check, because more iterations do not produce better work unless something reliable decides what counts as good enough. Kormos puts the check at the center of the whole argument, ahead of model choice or context design.

A check has two parts: a written definition of acceptable output, agreed with domain experts, and a way to evaluate an output against it. The first part is the hard one. Outputs from a probabilistic system vary, and more than one answer may be acceptable, so exact string comparison rarely works.

His comparison is traditional software. Unit tests and delivery gates give teams confidence that a change did not break something else. Agents need an equivalent mechanism, because the same request can produce different answers on different runs.

That mechanism is evals plus traces. Evals run a representative dataset repeatedly, in series, so quality shows up statistically rather than in one pass. Traces record what actually happened at runtime. Together they let a team argue about agent quality with evidence instead of a feeling about a model.

More passes without a better check just produce more confident output of unknown quality. The check is where autonomy is earned or refused.

Which nodes should an agent drive and which stay human?

A node is a candidate for automation when you can check its output against policy easily, bound its loop, trace what happened, and fall back to a person. The decision node in a lending flow usually fails the first test, so it stays human.

Kormos suggests asking, per node, how much value an agent adds versus a human owner. The screening questions are practical rather than philosophical: can we check it, can it run in a bounded loop, can we trace it, can we hand it back to a human, and how much would full autonomy be worth here?

The more useful question is not whether AI is involved in a process but whether AI is expected to take over a given node. Assistance and autonomy are different commitments, and they need different evidence before you flip that switch.

Injecting the DAG state into the agent helps as well. If the agent knows where its node sits in the larger flow and what long-horizon goal the node serves, its output is easier to check and the loop easier to bound.

High-quality input material is the upstream condition. Kormos argues better context yields better output at lower compute cost, then sets that topic aside to focus on the check. Treat that as his assertion about input quality, separate from the measured claims about evals.

The three stages from supervised to sampled to exceptions

An agent earns autonomy in three stages: full supervision on day one, sampling once measured confidence clears a policy threshold, and human handling of exceptions alone once that confidence holds. Each stage changes what humans look at, not whether anyone looks.

In the supervised stage, people verify every output while the team builds representative datasets, defines what good looks like with subject-matter experts, and confirms the confidence mechanism works. Traces are being collected, but humans are the check.

Sampling is the riskiest transition. Humans review a slice of output, and systematic evaluation through traces has to cover what nobody read. The question becomes whether the sampled quality matches the full-quality bar, and that is a statistical claim requiring enough volume to support it.

In the final stage, humans handle exceptions and keep refining the datasets. Kormos describes this as people remaining participants in quality, not leaving the process.

One caveat belongs here: the threshold that marks each transition is a policy decision, not a technical constant. A regulated firm sets it from its own risk appetite, and Kormos does not supply a number that would transfer to another organization.

Graphs, tooling and the regulation question

Graph tooling is secondary to the check, and Kormos names it that way: workflow engines help represent and run policy-driven processes, while evaluation supplies the evidence those processes are reliable.

Component	What it represents	Named example	What it does not prove
DAG	Outer process, fixed dependencies	Apache Airflow	That any single step is correct
Cyclic loop	Iteration inside one node	LangGraph	That the loop terminates on quality
Evals	Repeated scoring on a dataset	Teams' own harnesses	That production traffic matches the dataset
Traces	Runtime record of a run	OpenTelemetry is one option	That a human reviewed the output

Kormos mentions LangGraph as a way to manage state across the cyclic inner loop; the transcript renders it as a repeat of the LangChain name, and the tool he means is the one linked above. He calls both graph forms a simplification, noting real flows can be far more complex.

Regulated deployment needs a boundary here. Local execution, allowlists, exception routing, and human fallback support a controlled architecture. Auditability of the eval process supports it. None of that by itself establishes compliance, data residency, or suitability for a specific regulator, and a team should not read Kormos's framework as a compliance certificate.

Tooling also competes for attention with measurement. Kormos is explicit that operational visibility of the process earns more of a team's effort than authoring the workflow graph.

What an agent-autonomy rollout looks like week by week

A rollout moves one node at a time from supervised output to sampled review, and only then to exception-only handling, with the check written before any stage starts.

1. Map the flow as a DAG and mark every step that must be gated by policy.
2. Pick one node with a clear definition of acceptable output and a human fallback.
3. Write the check with subject-matter experts and build a representative dataset.
4. Run evals in repeated series and collect traces from real runs alongside them.
5. Set the confidence threshold in policy, and record who approved it.
6. Move to sampling, then verify that the unsampled population still meets the bar through traces.
7. Shift humans to exceptions and keep them in dataset refinement.

The sequence matters more than the calendar. Sampling before datasets are representative produces a false sense of coverage, and exception-only handling before traces are trustworthy removes the last check without replacing it.

Volume changes deserve a review at each step. Speeding up one node shifts load downstream, and a bottleneck that appears after autonomy is granted can look like an agent quality problem when it is really a capacity problem.

Dataset maintenance is ongoing work, not a one-time build. Kormos keeps humans in dataset refinement at every stage, including the last one.

What could go wrong with autonomy, and how do cost and quality interact?

Autonomy can fail in four ways that this model anticipates: an undefined check, a stale dataset, untraced runs, and a removed human fallback. Each is a process gap rather than a model gap.

An undefined check lets the loop run on an implicit standard that nobody wrote down, so nobody can audit it. A stale dataset keeps evals green while production drifts away from the cases the dataset covers. Untraced runs leave no evidence to review when sampled output looks wrong. A removed fallback turns an exception into an outage.

Cost sits inside the same loop. Kormos argues stronger input context lowers compute cost, which makes effort spent on context pay back across many iterations. He offers that as reasoning, not as a benchmark, and no measured cost figure appears in the talk.

Quality is where the strongest claim lands. Kormos frames evals plus traces as a complementary confidence mechanism that sits beside unit tests, letting a team present research and statistics about agent quality instead of an impression of the model.

The blunt summary he gives is worth keeping: a loop is only as trustworthy as its check. Everything else in the framework follows from how well that check is built.

How does this apply to developers learning to build AI workflows?

The same model applies to small teams: build the check first, keep the outer flow deterministic, and let agents iterate only where a gate can stop them.

Brazilian developer communities have been teaching this pattern in Portuguese-language courses. Gustavo Dev Doido, the instructor behind the Crazystack Typescript program and the Bootcamp do Dev Doido, covers TypeScript full-stack work and AI-assisted delivery for developers entering the field. The names appear here to orient readers who arrived from those courses; the technical argument is Kormos's.

Anyone building along those lines can compare notes with the community at crazystack.com.br, which publishes course material for the same stack.

The lesson transfers directly. A student project with one agent and one gate is the miniature version of a lending pipeline: define acceptable output, score it repeatedly, keep a human able to take over.

Frequently asked questions about unsupervised AI agents

• How do you know when an AI agent can work unsupervised? The agent can work unsupervised when its output passes a written check against policy, evals on a representative dataset clear a threshold the organization approved, and traces confirm the unsampled work also meets the bar. Until then, humans remain the check, either on every output or on a sample.

• What are evals and traces in an agent workflow? Evals are repeated scores of agent output on a representative dataset, used to judge quality statistically instead of by a single comparison. Traces are runtime records of what the agent actually did on real requests, used to confirm that the cases humans did not review still meet the quality standard.

• Does a faster agent make the whole process faster? Not necessarily. Kormos notes that accelerating code review pushes volume onto security scans and deployment, creating new bottlenecks. Process-wide visibility is needed to see where work accumulates after a node is automated.

• Can this framework make an agent suitable for regulated work? The framework supports an audit-friendly design with policy boundaries, evals, traces, and human fallback. Suitability for a specific regulated environment depends on controls the organization supplies, and a talk about workflow design does not establish compliance with any regulation.

• Which team owns the check? Kormos places definition of quality with subject-matter experts from the domain and the eval pipeline with AI engineers. Quality is a team sport in his phrasing, with product signals feeding an MLOps pipeline that delivers and monitors the applications.

• What is a DAG in an agent workflow? A DAG is a directed acyclic graph, a representation of a process where each step depends on earlier steps and no path loops back. It describes the outer process. The repeated work inside one node is a loop, and that loop needs its own check.

• Why not compare agent output to an expected string? Probabilistic models can return different, equally acceptable answers for the same request. Exact comparison rejects valid variation, so teams build representative datasets and evaluate in repeated series instead.

• What happens to humans after autonomy is granted? They stay in the process. Kormos keeps humans handling exceptions and refining datasets at every stage, including the final one, so quality work continues after routine review ends.

• Is the three-stage model a standard? It is Kormos's proposal from an October 2026 talk, not a published standard. It describes a progression from supervision to sampling to exception handling, and an organization should set its own thresholds and evidence requirements.

• Where can I see the full talk? The session was published by the AI Engineer channel as "How to Know When an AI Agent Can Work Unsupervised" in October 2026. The framework above follows its structure: policy, workflow graphs, the check, node selection, and the three stages.

From a recorded talk to a publishable article

Kormos's argument rests on a single idea worth carrying into your own material: trust follows the check, and the check is something a team builds, measures, and reviews.

If you have explained a framework like that on video, the reasoning already exists as speech, with chapters, examples, and the caveats that make it credible. Skalablog turns that recording into a written article you can edit and publish. Paste a YouTube URL, let the transcript come through, and review the draft before it goes out.

The talk version of this piece

• Ready to turn a video into an article? Skala Blog

Source video