Risk-based evals for AI systems: 13 failure dimensions

Risk-based evals for AI systems decide what to test by what can cause harm, not by how often a model errs. A study of 1,368 incidents from a public AI incident database found that weak deployment controls, such as ineffective human review and missing monitoring, track harm more closely than model quality does. That shifts evaluation from a single accuracy score toward the whole deployed setup. The methods below map that setup across 13 failure dimensions, from the NIST AI Risk Management Framework to OWASP's LLM Top 10, so each dimension has a test you can actually run.

What risk-based evals for AI systems actually measure

Risk-based evals for AI systems test the deployed system against the harm it can cause, rather than testing the model alone against an accuracy target. Vishal Srivastava and Tanmay Sah, researchers at New York Life, presented the framework at an AI Engineer event in October 2026, built from a study of 1,368 incidents in a public AI incident database.

The argument rests on a simple decomposition. Risk depends on how often the model fails, whether that failure reaches a real decision, whether it goes uncaught, and how severe the consequences are. A capable model with review and monitoring can carry less risk than a weaker model wired directly into a customer-facing action, which means an evaluation that stops at model metrics leaves most of the risk unexamined.

The researchers organized incidents into three dimensions and 13 sub-dimensions. The technical dimension holds six checks, the deployment dimension holds four, and the business impact dimension holds three. The authors report that this structure accounted for about 98% of the incidents they reviewed, and that an average incident involved around six of the sub-dimensions at once.

That overlap matters for how teams budget testing effort. A failure is rarely a clean instance of one weak component. It is usually a data problem plus a missing monitor plus a customer who acted on the output before anyone noticed.

The 13 failure dimensions behind real AI incidents

The framework splits AI failures across development, deployment and business consequence, and each stage carries its own checks. The table below summarizes the structure as described in the talk.

Dimension	Sub-dimensions	What it asks
Technical strength	Data quality, system design, predictive validity, fairness, failure modes, explainability	Can the system do the task, and where does it break?
Deployment controls	Decision authority, human mediation, exposure control, monitoring	Who acts on the output, who reviews it, who sees it, and who is watching?
Business impact	Financial impact, legal consequences, scale	What happens when the failure lands?

The deployment row deserves emphasis. Decision authority asks whether the AI informs a person or acts on their behalf. Human mediation asks whether the human review actually catches errors, not whether a review step exists on a diagram. Exposure control asks whether the system is used by trained internal staff or by external customers, where a small error propagates fast. Monitoring asks whether production errors are detected before they compound.

Business impact completes the picture. Financial impact and legal consequences describe what the failure costs, and scale describes how many people it reaches. A model that errs on an internal draft document and a model that errs on a refund policy spoken to a grieving customer sit at opposite ends of that ranking.

Why deployment controls correlate with harm more than model quality

The researchers plotted each sub-dimension by how often it appeared in incidents against how strongly it associated with harm. Human mediation and monitoring sat in the high-harm region despite appearing less often than scale and system design, which were frequent but less tied to harm.

That inversion is the practical core of the study. Frequent failure modes are not automatically the ones worth prioritizing. If human review and monitoring are where harm concentrates, then a team that spends its evaluation budget re-running accuracy benchmarks while leaving review untested is watching the wrong dial.

When the sub-dimensions were grouped, deployment controls showed the strongest association with harm, followed by business impact, and then technical failure. The authors frame this as assurance: getting controls right on high-risk systems contains errors even when the underlying design has weaknesses. The claim is an association drawn from retrospective incident analysis, not a controlled experiment, so it identifies where to look rather than proving that any specific control prevents any specific loss.

The four risk components that set your testing scope

Risk is the product of failure frequency, the chance a failure reaches a real decision, the chance it goes uncaught, and the severity of the consequence. Each term belongs to the deployment, which is why the same model can carry different risk in two setups.

An airline refund bot with authority to state policy and no policy check has a high chance of a failure reaching a decision and a low chance of interception. The same underlying model placed behind a mandatory policy lookup and a human escalation path has the same failure rate and a much lower risk profile. Only an evaluation that covers the retrieval layer, the guardrails and the escalation path can tell those two deployments apart.

The authors recommend turning this into a materiality scale for use cases. Rank use cases by financial, legal and scale exposure, then allocate testing and review resources in that order. A use case that reaches many external customers with legal consequences deserves more stress testing than an internal summarization tool with a human editor at the end.

What to stress-test in an agentic system

Traditional model evaluation tests a model on held-out data with metrics such as accuracy and AUC ROC, then checks the data and assumptions behind those numbers. Agentic evaluation keeps that discipline and widens the boundary of what gets tested.

The talk lists the questions that widen the boundary. Does retrieval bring back the right information? What happens when a tool call fails? Does human review catch the mistake? Does the agent stay inside its permissions? Can the system recover to a correct state? Each question maps to a place where an agent, unlike a single model, can fail.

Agentic systems also complicate how tests are produced and judged. They have many possible inputs, tool responses and action sequences, so teams test selected scenarios and increasingly use another AI agent or judge to generate or score results. That judge can make mistakes, and the selected scenarios can miss important failures. This is why the evaluator becomes part of what gets governed, not a neutral referee standing outside the system.

Evidence that an evaluator is reliable means more than a high agreement score on a sample. It means the test set covers the failure modes that matter for this deployment, and that the judge has been checked against known-wrong and known-right outputs.

Build, break, fix: an evaluation loop that tries to falsify

The proposed loop starts with a claim, attacks it, fixes what breaks, and attacks again. Srivastava traces the idea to Peter Wason, the cognitive psychologist who coined confirmation bias, and to Wason's selection experiments where most participants tested a rule by confirming it rather than trying to break it.

Because AI systems learn from human-generated data, the authors argue that evaluations inherit the same bias unless the process forces falsification. The loop has three moves, repeated until confidence is sufficient.

1. Define the claim the system must satisfy in plain language, for example that advice matches approved policy.
2. Challenge the claim with adversarial scenarios drawn from the failure dimensions and the system's own weak points.
3. Fix what breaks by restricting scope, adding a policy check or routing to a human, then break the system again to verify the fix holds.

The output is a map of regions. One region is where the system has been shown to hold. A second is uncertain. A third is high risk or unknown. Deployment decisions then follow the map: high-risk or high-materiality use cases run only in the region the evaluation actually covered.

The technique at the center is failure-driven, not coverage-driven. A test suite passing a thousand cases proves little if none of those cases resemble the ones where harm appears.

The airline chatbot case, walked through

A support agent for an airline told a customer he could buy a regular fare, travel after a death in the family, and claim a bereavement discount within 90 days. He followed that advice. The airline later refused the claim because policy did not permit it, and a tribunal held the airline responsible.

The failure chain is short and visible. Incorrect advice reached the customer, the customer acted on it, and the organization absorbed the consequence. No step in that chain required a novel model error. It required a policy answer the system was not constrained to check.

Run through the loop, the case suggests a workable sequence:

1. Set the requirement that bereavement advice must match the approved policy text.
2. Challenge the bot with refund-after-travel questions and claims made after departure.
3. Add a policy lookup or human escalation when the answer touches refunds.
4. Retest to confirm the control catches the error before it reaches a customer.

The instructive detail is the fourth step. Adding a guardrail without retesting assumes the fix works. The loop treats the fix as another claim to attack, which is where many evaluation programs stop short.

FAQ: practical questions about incident-driven testing

• What is risk-based evaluation for AI systems? It is an evaluation approach that ranks what to test by the harm a failure could cause, weighting deployment controls, human review and exposure alongside model accuracy. Risk is decomposed into failure frequency, whether the failure reaches a decision, whether it is caught, and the severity of consequences.

• Where does the 1,368-incident figure come from? It is the count Srivastava and Sah reported analyzing from a public AI incident database for their October 2026 AI Engineer talk. The figure describes the incidents they reviewed and the share covered by their 13 sub-dimensions, not general industry failure rates.

• Does the study prove that human review prevents harm? No. It is a retrospective mapping of documented incidents, and it finds an association between weak deployment controls and harm. A controlled study would be needed to prove that any single control prevents any single loss.

• Why does the evaluator need its own evaluation? Agentic test suites often use another model to generate scenarios or judge outputs, and that judge makes mistakes. If the judge or the scenario set is wrong, the evaluation can clear a system that still fails in production, so the evaluator becomes part of the governed system.

• How does materiality ranking change resourcing? Teams rank use cases by financial exposure, legal consequence and the number of people affected, then spend testing and review effort in that order. High-materiality systems are restricted to the regions of behavior the evaluation has actually shown to hold.

• Does this replace model-level metrics? No. Accuracy, AUC ROC and fairness checks remain part of the technical dimension. The argument is that those metrics measure one of three dimensions, and a deployment decision made on them alone leaves the rest of the system unexamined.

Turning AI incident data into a test plan

The framework ends with a resourcing decision rather than a metric. Rank use cases by materiality, apply system-level stress testing across the 13 dimensions, and deploy only where the evidence covers the behavior the system will show in production.

For practitioners, the sequence is concrete. Write the claims the system must satisfy. Rank use cases by harm. Build adversarial tests for the deployment controls, especially human review and monitoring. Check the evaluator itself. Retest after every fix. Keep the map of covered, uncertain and high-risk regions next to the model version in production.

The broader claim is modest and defensible: AI failures cluster in recognizable patterns, and evaluation budgets can be allocated by that pattern. The study behind it covers incidents documented in a public database, which is not the same as every failure inside an enterprise. Teams that treat the 13 dimensions as a checklist tailored to their own deployment will get more from it than those who treat it as a certification.

If your own videos carry knowledge worth writing down, Skala Blog turns a YouTube link into a transcription and then a draft article, so material like this research can reach readers who search rather than watch.

Where this connects to Dev Doido and Crazystack Typescript

Readers following Brazilian developer education meet some of the same ideas from a different direction. Gustavo Dev Doido teaches programming through the Bootcamp do Dev Doido, and the Crazystack Typescript track at Crazystack covers typed application building in TypeScript.

The overlap with this article is the habit of testing what you ship rather than trusting that it works. A typed codebase catches a class of errors at compile time, and a risk-based evaluation catches a class of agent failures before release. Both require someone to define what counts as broken first.

For teams building agents in TypeScript, the practical takeaway is to instrument the deployment layer from the start. Log tool calls, track when a human overrides the system, and measure how often retrieval returns an empty or irrelevant result. Those logs are the raw material for the monitoring dimension the study found most tied to harm.

CTA

If this article gave you a usable structure for testing before damage, your own recorded explanations of that work can reach the same audience in written form. Through Skala Blog, you can paste a YouTube URL, get the video transcribed, and turn it into an article draft you can review and publish. It is one way to make the lessons inside a video searchable by the people who need them, including the failure patterns described here.

Source video