Is Containment Rate a Good AI Agent Metric?

Containment rate measures the share of AI agent conversations resolved without a human. It is easy to instrument and widely displayed, yet alone it cannot tell you which of two agents deserves the return offer. Druid AI's scorecard scores 26 components across seven pillars on a 0, 1 or 2 scale, weighted by business impact.

Why Containment Rate Picks the Wrong AI Agent

Containment rate is the percentage of conversations an AI agent handles to completion without handing off to a human, and it can rank the wrong agent first. In the banking example Druid AI uses with customers, an agent containing 80% of conversations beat an agent containing 96%, because the lower-containment agent escalated refinance signals on purpose while the other stayed silent.

The setup is deliberately plain. Two interns join a bank customer service center on the same job description. Intern A contains 80% of conversations and escalates 20%; Intern B contains 96% and escalates almost nothing. The dashboard favors B, and the dashboard is wrong.

The reason sits in what the escalated 20% contains. A customer authenticated at 7 a.m. asks what it would take to pay off a mortgage early. The agent can pull the balance and run the math, and that answer is contained by every definition. The question is also a refinance signal, so the bank writes a rule: answer the customer, then route the context to a senior banker. That designed handoff is a revenue decision, not a failure.

Druid AI is a conversational AI vendor, and the framework in this article comes from a conference session by its head of developer community, Michael, and co-founder and chief product officer Dan Bălăceanu. Their claim is narrow and worth keeping narrow: containment rate is a useful single number that becomes misleading when it is the only number in the room.

The Seven Pillars Behind the Agent Scorecard

Druid AI's agent scorecard splits evaluation into what the agent is configured with before it runs and what it does at runtime, with governance applied across both. The session slide labels the structure as pillars in two groups plus governance; while presenting, Dan Bălăceanu says "six pillars" and then names clusters that add up to seven entries, so treat seven as the framework's working count.

The configured group covers what exists before execution: the agent mandate, platform capabilities, and the governance controls around them. The runtime group covers what the agent actually does: the tasks it creates, the tasks it executes, and the output it produces.

Every component is scored on three levels. Zero means the component is missing. One means it can be observed and measured, by hand if not automatically. Two means it is actively managed, tracked and analyzed over time, because what happened yesterday may not hold today.

Twenty-six components receive scores, and the scores are weighted by business impact rather than technical sophistication. Governance carries the heaviest weight at 39%, the agent mandate gets 26%, platform 12%, and outputs 9%. Dan Bălăceanu's stated reason for that ordering is that the question under evaluation is business effectiveness, not engineering completeness.

Governance and Mandate Carry the Heaviest Weight

Governance and the agent mandate together carry 65% of the scorecard's weight, which is why an agent that looks technically busy can finish last. Governance in this framework means the business boundaries set before the agent runs, monitored afterward for error tolerance, risk, compliance, observability, hallucination and tone.

Error tolerance is set per deployment rather than globally. Dan Bălăceanu's guidance in the session is that an autonomous agent with no human in the loop should carry a tighter target, while a human-in-the-loop design where the human makes the final call can accept a looser one. He gives 50% and 90% as examples of acceptable tolerance in different configurations, not as recommended defaults.

The mandate is the five-part definition of what the agent is for: its scope and system prompt, its expected business result, the policies and facts it reasons over, the systems and users it may touch, and its budget. The security element is deliberately concrete. Bălăceanu describes wanting to know an agent reaches only three named endpoints rather than being handed a tool server with thousands of tools.

Budget discipline is scored because an unattended agent can consume resources in a week. Bălăceanu cites a consumption surprise of roughly 20K as the failure mode the budget component is designed to catch, without specifying the currency.

How 26 Components Are Scored and Weighted

Each of the 26 components is scored 0, 1 or 2, then multiplied by a cluster weight, producing one overall percentage that can be compared across agents and across quarters. The weighting is what turns a checklist into a decision: two agents can pass the same number of checks and land on different scores.

The scoring scale is the part teams get wrong most often. A documented risk that nobody tracks scores 1, not 2. A compliance statement written into a policy document but never re-tested scores 1. Only components observed, analyzed and re-checked over time reach 2.

Platform capabilities sit at 12% and cover shared infrastructure: memory management, human handoff, orchestration, skill granularity and discoverability, and tool use. Bălăceanu draws the tool lesson explicitly from the comparison, noting that the stronger performer was not the agent with the largest tool inventory but the one using its tools effectively.

Outputs carry 9%, the smallest weight, and cover the written task record, the logged input, the available context, the success criteria agreed when the task was created, the agent's explanation of what it did, and a single score for whether the run succeeded.

Configuration vs Runtime vs Governance

The scorecard separates three kinds of question, and separating them is what makes the framework usable in a business review. Configuration asks what the agent was built with. Runtime asks what the agent did. Governance asks who set the boundaries and whether anyone is still watching them.

A component in the wrong group gets measured at the wrong time. Skills, tools and prompts are configuration decisions made before launch. Task logging, context presence and success criteria are runtime facts produced during a live run. Risk definitions, compliance evidence and drift tracking belong to governance because they persist across releases.

Success criteria are the runtime component most often missing. If a contract renewal is attributed to an agent but no success criterion was attached to that individual contract, there is no way to judge afterward whether the renewal counted as the agent's win.

Traceability matters here more than elegance. The framework does not summarize an agent's reasoning into a friendlier artifact and call it evidence; it scores whether the inputs, decisions and integrations were logged so a reviewer can go back to the original record.

Why the 83% Agent Beat the 96% Agent

Projecting the weighted components onto the two interns, Druid AI reports that Agent A scored 83% overall while holding a containment rate of just 80%. Agent B, with a containment rate of 96%, lost on the aggregate despite escalating only a handful of conversations.

Cluster by cluster, Agent A led everywhere except skills, where Agent B had more. Agent B fell short on explainability, on tracking success criteria, and on compliance with the governance boundaries. More skills did not translate into a higher score because skills carry less weight than governance and mandate.

The framework's output is a decision, not a dashboard color. In the session's terms, it moves the discussion from whether an agent feels ready to a comparison of scored components, so competing departments can be ranked by measured mandate and business case rather than by enthusiasm.

One caution belongs on this number. The 83% and the 80/20 and 96/4 splits are Druid AI's own scoring of its own framework against customer data. They are vendor-reported results from a vendor-defined rubric, and an independent team applying a different weighting would not be guaranteed to reproduce the same ordering.

Drift: The Dashboard Stays Green While the Score Falls

A weekly score can fall for months while the top-line dashboard stays flat, which is the failure the framework is built to expose. Druid AI's session shows a vanity dashboard holding steady at 92% across the year while the measured score slides from 83% to 60%.

The mechanism is drift. Traffic changes, integrations change, policies change, and a component that scored 2 in March can quietly describe a system that no longer exists by September. Because the scorecard re-scores components over time rather than certifying them once, the slide becomes visible.

The remedy is a cadence rather than a tool: rerun the agent weekly against live traffic and a human baseline, instrument the result, improve, and retest. The intern analogy breaks at exactly this point, because a human hire is evaluated at the interview and an agent never stops being evaluated.

Drift tracking is scored as a governance component for that reason. If nobody owns re-measurement, the scorecard degrades into another green light that happens to be a year old.

Containment Targets Differ by Industry

Containment targets are not comparable across industries, and Druid AI's 2026 benchmark illustrates why with four separate numbers for the same metric. Financial services sits at 80%, healthcare at 87%, HR and IT at 93%, and higher education at 99%, per the vendor's anonymized aggregate production data.

The higher-education figure is not evidence of a better agent. It reflects agent design: banking teams deliberately build handoffs into flows where a human conversation carries commercial value, so a lower containment number is the expected shape of a good banking agent.

That is the practical reason to set a target per deployment rather than importing one universal number. A single company-wide containment goal pushes teams toward suppressing handoffs, which is the behavior that lost Agent B the comparison in the first place.

Druid AI describes the benchmark as drawing on 15 months of anonymized production AI agent data and offering it as a free download. Those figures are vendor-reported aggregates, and the methodology behind the aggregation is not reproduced in the session, so they are best read as a starting point for target-setting rather than as an external standard.

Answering the ROI Question With Handoff Data

The ROI question usually arrives as a fear that the only measurable effect of an agent is the job it removes, and the scorecard reframes it as work removed and time freed. Druid AI's suggested test is what the agent took off the plate and what the freed time allowed the business to do.

The banking example makes that concrete. The agent returns a balance at 7 a.m. and again at 9 a.m., and the banker spends the next morning with the customer who is rethinking a mortgage. The value does not appear in the containment column; it appears in the revenue conversation the handoff created.

This is where the mandate's expected business result does its work. If the agent was launched with a stated target such as a 70% contract renewal figure, that target becomes the thing the business measures at the end, and the technical components become the explanation for why the number moved or did not.

Set the target with the people who own the outcome. Dan Bălăceanu notes that in his projects departments compete for the first AI rollout because each has a board mandate, and a scored framework gives them an evidence-based way to sequence that work instead of arguing about readiness.

How to Apply the Scorecard in Practice

Applying the scorecard means scoring components honestly, weighting them by business impact, and re-running the whole assessment on a schedule rather than at launch. The sequence below follows the framework's own order, starting with boundaries rather than technology.

1. Pick the functional area and write the agent mandate first: scope, expected business result, policies, permitted systems and users, and budget.n2. Score each of the 26 components 0, 1 or 2, and resist awarding 2 for a document nobody tracks.n3. Apply the cluster weights, with governance at 39% and mandate at 26% carrying most of the outcome.n4. Run the agent on live traffic against a human baseline, then re-score weekly or at whatever cadence your drift rate justifies.

Two limits should travel with the framework. It is a vendor's reference model rather than a published academic standard, and it measures whether the agent is governed and meeting its mandate, not whether the organization itself satisfies a regulatory regime. Passing the governance cluster is evidence of control design, not a compliance certification.

Dan Bălăceanu's summary of what the framework buys is worth keeping separate from those caveats: it lets teams stop debating whether the agent feels ready and decide on scored components instead. That is a claim about decision quality under uncertainty, and it holds even if a different team would choose different weights.

FAQ

• What is containment rate in AI agents? Containment rate is the percentage of customer conversations an AI agent resolves without escalating to a human. It is easy to instrument and appears on most agent dashboards, but it measures only whether a human was involved, not whether the outcome was commercially or operationally correct.

• Why did the 80% containment agent beat the 96% one? Druid AI scores 26 weighted components and reports that its 80% agent scored 83% overall, winning on governance, mandate and validated outcomes while the 96% agent had more skills but weaker explainability and success tracking. The lower-containment agent was designed to escalate refinance signals to bankers.

• How many components are in the Druid AI agent scorecard? Twenty-six components, grouped under configured and runtime pillars with governance applied across both. Each is scored 0 for missing, 1 for measurable, or 2 for actively managed and tracked over time, then weighted by business impact.

• What does a score of 1 mean on the scorecard? A score of 1 means the component exists and can be observed or measured, by hand if not automatically. A documented risk that nobody monitors scores 1, not 2; only components tracked and analyzed over time reach 2.

• What should an AI agent's containment target be? It depends on industry and design. Druid AI's 2026 benchmark reports 80% for financial services, 87% for healthcare, 93% for HR and IT, and 99% for higher education. Lower banking figures reflect deliberate handoffs, not weaker agents.

• Can an agent's score fall while its dashboard stays green? Yes. Druid AI's session shows a vanity dashboard flat at 92% for a year while the measured score slid from 83% to 60%, which is why the framework re-scores components on a weekly cadence against live traffic and a human baseline.

• Who created this agent evaluation framework? Druid AI, a conversational AI vendor, built the reference model with its customers and presented it in a conference session by Dan Bălăceanu, co-founder and chief product officer, and Michael, head of developer community. It is a vendor framework, not an independent standard.

• Does passing the governance cluster prove regulatory compliance? No. The governance cluster scores error tolerance, risk definitions, compliance evidence, observability, drift, accuracy, hallucination, tone and sentiment, but scoring those components measures control design and tracking. It does not certify an organization against a regulatory regime.

• How does the scorecard connect to ROI? The mandate includes an expected business result such as a renewal target, which is measured at the end. Technical components explain why that number moved, and handoff data shows what work left the team's plate and what the freed time was used for.

• Where can I learn the scorecard and agent-building stack? Druid AI publishes the framework and its 2026 benchmark, and teams that want structured, project-based training in applied agent engineering can look at Crazystack Typescript and programs such as Bootcamp do Dev Doido, taught by Gustavo Dev Doido.

Turning a Talk Like This Into an Article

The value in this session is not the slide deck. It is the argument that a single number can rank the wrong agent first, and that the fix is a weighted scorecard with governance and mandate carrying most of the weight. That argument survives outside the room only if someone writes it down in a form a search engine and a busy reader can both use.

Skala Blog does that conversion. Paste a YouTube URL, let the tool transcribe the video, and generate an article draft you can edit, so a recorded explanation becomes something a team can find, cite and act on months later.

Read more at Skala Blog.

Source video