How to analyze your AI support agent to improve resolution and reduce escalations

You improve an AI support agent by analyzing it in five ways: how many conversations end in real resolution, where its knowledge fails, which ticket topics repeat, how it behaves after each change, and what humans do after an escalation. Feeding it more data without that analysis mostly adds noise.

The five analyses of an AI support agent

1. Resolution: conversations handled vs conversations resolved with confidence.
2. Knowledge: real gaps vs answers the AI had and didn't find.
3. Topics: product area crossed with what triggered the escalation.
4. Behavior: rule breaks before and after each change.
5. Human follow-up: what the person did after the AI passed the case on.

An AI support agent is software that answers customers and can act in other systems. Most teams improve one by training it: more documents, more past conversations, more instructions. When resolution goes up, nobody knows why. When it doesn't, nobody knows that either.

The pattern shows up well beyond support. MIT's NANDA initiative (an AI research project) covered it in its 2025 report The GenAI Divide. Just 5% of integrated AI pilots extract millions in value, and most show no measurable impact on profit and loss (P&L), as Virtualization Review reported. Pilots stall for many reasons. Not knowing what works is one of them.

Without analysis, every improvement is a guess.

How much of what your AI handles is really resolved?

Handled and resolved are two different numbers. At Metricool, over more than five months in production, 86% of support conversations never reach a human. Only 72% end resolved with confidence: the customer confirmed it, or the conversation shows the problem was solved.

The full method is in how we measure AI support beyond "not escalated".

Fourteen points sit between handled and solved.

Two situations show up again and again in that gap. The customer stops replying after the AI asks a clarifying question, such as which account or which plan. Or the AI runs an action and the customer never confirms it solved the problem.

Both count as success on a containment dashboard (one that counts conversations that never reached a person). Neither tells you the customer got what they came for.

Split escalations by type of case

The first useful cut is by type of case. On one recent day, one deployment handled 406 conversations and 50 reached a human, 12.3% (internal Zentesimal data).

Of those 50, 33 were policy handoffs: 18 customers asked for a person and 15 cases needed manual intervention.

Nine hit a capability gap: seven carried an attachment the AI can't read and two an image that failed to process. The other eight were spread across smaller reasons.

A policy handoff waits for a business decision. An unsupported attachment waits for a new capability. Same escalation rate, two different jobs.

One average hides several different problems.

Why does the AI miss answers it already has?

Most escalations that look like missing knowledge are retrieval failures: the AI searches, and the right article never comes back.

Knowledge review is the most common analysis, and the first to run out of road. Teams are investing in it: in a Gartner survey of 321 customer service leaders (October 2025), 58% planned to upskill agents as knowledge management specialists. As knowledge bases get tidier, with clear sections and topics, the obvious gaps close early.

Adding more sources doesn't repair retrieval. Which sources to connect, and in what order, is covered in which data sources raise an AI assistant's answer rate.

Between 6 and 13 October 2026, a Zentesimal voice of customer (VoC) agent read 5,657 Metricool support conversations. Of those, 1,790 reached a human, and 906 were fully classified (internal Zentesimal data, one week).

Only 32 handoffs came from a real knowledge gap. In 155, the human answered with an article the knowledge base already had.

The content existed. The search, the way the answer was tied to the article, or a playbook (a set of rules that detects one situation and triggers one action) left it out.

Picture a library with every book and a broken index.

Which ticket topics point to a product or documentation problem?

Classifying every ticket by topic, reason for opening and product area shows where to fix the product, the documentation, or both. Crossing product area with escalation trigger in that same week gave this:

Escalation triggers by product area (one week, 906 classified escalations)

In billing, most handoffs wait on a decision about what the AI may refund on its own. In scheduling, one publishing flow topped the documentation gaps. In account settings, the AI lacks a tool to act.

Two fixes the topic analysis found

Topic analysis decides who owns the fix: product, documentation, or both.

In one anonymized customer case, renewal escalations clustered on the same error. The cause was a product incident: a bug that stopped customers from renewing. The analysis sent it to the product team, where more documentation would not have helped.

In another anonymized case, repeated plan questions traced back to two help articles that contradicted each other. One of them was among the most visited in the help center. Rewriting it removed the contradiction the AI was drawing from.

While a bug waits for its fix, a known-issue note lets the AI tell the customer what is happening instead of passing the chat to a person.

How do you know if a training change helped?

Comparing violations before and after each change shows its effect, especially at high volume. These rules are limits the AI must never cross, such as promising a refund or claiming an outage without a source.

Risk frameworks treat that monitoring as part of running AI. The NIST AI Risk Management Framework, from the US standards agency, lists it under Manage 4.1:

Post-deployment AI system monitoring plans are implemented, including mechanisms for capturing and evaluating input from users and other relevant AI actors, appeal and override, decommissioning, incident response, recovery, and change management.

On one day with 406 conversations, 41 crossed at least one limit, a rule integrity of 89.9% (internal Zentesimal data, one customer deployment).

The most frequent breaks were poor cancellation handling (16), tone or language that didn't match the customer (11) and an outage claimed without a source (8).

When a fix breaks something else

A limit that fixes one behavior can silence another. In one customer case, the AI kept offering demos the business doesn't run. A new rule against demo offers cut those breaks from 11 to 1 between two consecutive days.

Then the AI stopped offering discount coupons too. The playbook for that type of case told it to offer a coupon, and the limit blocked the message every time. Playbook and limit contradicted each other, and the coupon never reached the customer.

Across several adjustments on those same two days, conversations with at least one violation fell from 63 to 41.

"We think it improved" becomes "it improved here and got worse there."

What do humans do after the AI escalates?

What a person does after a handoff is the cheapest map of what the AI is missing. In the 906 classified cases from that Metricool week, the human:

• acted in the back office (the internal admin tools) in 161;
• answered from an existing article in 155;
• asked the customer for more information in 103;
• looked up account data in 55.

From there, every case passed to a person falls into one of three groups.

Three groups of escalations by what the AI needed

A further 6% stayed unclear. The four most repeated missing tools (subscription refund, change of email, password reset and account reactivation) account for 136 conversations in one week.

A person who refunds a subscription opens the back office, checks the payment and sends the money back. A tool with the right permission follows the same three steps.

Every escalation is an exam a human already graded.

Method note: only 906 of 1,790 escalations had the new classification fields filled in, so the shares may move once the rest is backfilled. A rerun on the full set gives the final split.

What the analysis showed

• Handled and resolved sit 14 points apart in production: 86% against 72%.
• Retrieval failures explained 155 of 906 classified handoffs. Real knowledge gaps explained 32.
• 77% of classified escalations were avoidable, fully or with one small change.
• A limit that stops one behavior can silence another. Comparing violations day by day catches it.
• Product area crossed with escalation trigger separates bugs from documentation gaps.

How Zentesimal does this analysis

Zentesimal Workers are AI agents that run inside a team's existing tools, such as the helpdesk, the CRM (customer relationship management system) and the product itself. Each Worker covers one stage of the customer lifecycle.

In support, a Worker reviews every conversation that reaches a human: why it escalated, what the person did and whether the AI could have handled it.

Those findings tell the team whether to improve knowledge, add a tool, define a policy or fix the product. Each adjustment is then measured against the Worker's rules, on real volume.

The analysis also runs from Claude or ChatGPT. With the Zentesimal connector, a support lead asks in the chat the way they would ask a teammate: "Why did billing handoffs go up this week?" or "Which answers did a human find in the knowledge base that the AI missed?"

The answer comes from the Worker's real conversations and results. A rule or a playbook can be adjusted in the same conversation, then checked the next day against the new numbers.

Today, 17% of classified escalations need a person, mostly deep technical cases that call for product knowledge. The other 77% works as a list of jobs: tools to connect, policies to define, articles to fix.

Escalations stop being a cost and become a list of improvements.

Book a demo