Your AI Shouldn't Escalate When It's Unsure. It Should Escalate When Being Wrong Is Expensive.

Every guide to this question gives you the same list. Escalate when the user asks for a human. Escalate when sentiment turns negative. Escalate after two failed attempts. Escalate when the model's confidence drops below a threshold.

Four of those five are defensible. The last one is the one almost every vendor leads with, and it rests on an assumption the research does not support: that a language model knows when it is wrong.

It largely does not. And once you accept that, the escalation question changes shape. It stops being "how sure is the model?" and becomes "what does being wrong cost here, and can we undo it?" That is a question about your business, not about the model, which is why no vendor's default threshold can answer it for you.

This is the design we use. It is more work than a confidence slider, and it is the part of an automation project that most determines whether people trust the thing six months in.

The trigger everyone recommends is the weakest one available

Ask a model how confident it is and it will tell you a number. The number sounds like information. Mostly it is a stylistic artefact.

Xiong et al. tested this directly in "Can LLMs Express Their Uncertainty?" (ICLR 2024). Their section 5.1 is titled, without hedging, "LLMs tend to be overconfident when verbalizing their confidence", and they suggest models "might be imitating human expressions when verbalizing confidence" — the number is a learned speech habit as much as a measurement.

Two further findings from that paper matter here, and they point in different directions.

Sampling beats asking. Their section 5.3 reports that five-sample answer consistency "consistently outperform[s] verbalized confidence in calibration and failure prediction", with the largest gains on arithmetic tasks. So measured disagreement is the better instrument. The caveats are theirs: the comparison is against single-response verbalised confidence, those particular results are on GPT-3.5, cost grows linearly with samples while "the improvement becomes marginal as the number gets larger", and consistency can be ineffective when the model is poorly calibrated or the query budget is small.

But do not throw the self-report away. Their section 5.4 finds that consistency-only aggregation is weaker than methods that also use verbalised confidence. The two signals combine better than either alone. And their broader conclusion is sobering for anyone hoping to buy this off a shelf: no single prompting strategy consistently outperformed the others, and "black-box approaches remain suboptimal".

On model upgrades: calibration does improve with capability — they measure roughly a 22.2% AUROC gain from GPT-3 to GPT-4 — but it remains "far from ideal performance", and prompting-based improvements "diminish as the model capacity scales up". A newer model will not retire this problem for you.

Two consequences follow, and they are the practical ones.

A confidence threshold is not a fixed threshold. If the number is partly a stylistic habit rather than a measurement, then "escalate below 0.7" means something different for a billing question than a returns question, different after a prompt change, and different after a model upgrade. You have built a control whose units drift underneath you. Teams discover this when a model version bump quietly halves their escalation rate and nobody can say whether quality moved.

Self-reported confidence and measured uncertainty are not the same signal. Asking the model is cheap and weak. Sampling the same question several times and looking at whether the answers agree costs more and tells you more, because disagreement is something you observed rather than something the model claimed. It is not a solved problem either — see the caveat above — but it is evidence rather than assertion.

What abstention actually buys, when it is built on something real

The case for escalation is not merely that customers dislike wrong answers. There is a measurable accuracy argument for letting a system decline.

Tomani et al., in "Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations", compared statistical uncertainty measures against a verbalised one across question answering, unanswerable questions, and safety. Their abstract says abstention can "improve correctness by 2% to 8%".

That range is worth unpacking rather than quoting, because the body of the paper tells you what it costs. On TriviaQA with an RLHF-tuned Llama 2 and semantic entropy as the signal, declining the most uncertain 5% of questions moved accuracy from 84.4% to 86.0%. Declining 25% moved it to 92.6%. So the "2%" is the 5%-rejection case and the "8%" is the 25%-rejection case, in percentage points.

That is not a cheap win, and it is not a steep early curve. The abstract's phrase "sacrificing only a few" is doing a lot of work: the paper never defines "a few", and its own examples reject between a twentieth and a quarter of all questions. Buying the top of that range means sending one request in four to a human. Note also the narrowness of the evidence — that accuracy curve is one dataset on one model family, and the same paper finds all its uncertainty measures perform near-randomly on GSM8K.

The honest reading: abstention genuinely improves the accuracy of what you do answer, the improvement scales with how much you decline, and therefore how much to decline is an economic decision, not a technical one. Which is the whole argument of this article.

The signal that is good at "I can't answer this" is not the one that is good at "I'm wrong"

The same paper contains a result that cuts against the simple story, and it is the most useful thing in it for designing an escalation rule.

Their verbalised measure — In-Dialogue Uncertainty, essentially counting hedging words — is close to useless at telling correct answers from incorrect ones (around 0.53 AUROC against roughly 0.69 for semantic entropy) and performs near chance on safety. The authors put it plainly: it "cannot effectively differentiate between correct and incorrect answers". But on separating answerable from unanswerable questions, it scored about 0.75 AUROC and beat every statistical measure they tested.

Two different questions, two different best signals:

• "Is this answer wrong?" Use measured disagreement across samples. Hedging language tells you almost nothing.
• "Is this answerable from what I have?" The model's own hedging is a surprisingly decent detector — alongside the cheaper structural check of whether retrieval returned anything above your relevance floor.

One more number from that experiment belongs in your planning, because it is the cost of escalating badly: catching close to 50% of unanswerable questions came with wrongly refusing about 10% of answerable ones. Every escalation rule has that second rate, and most teams never measure it. (The paper's abstract describes this as avoiding "50% hallucinations", which blurs a detection rate on unanswerable questions into an overall reduction in hallucination. It is the narrower claim.)

If you want the full landscape rather than two papers, the peer-reviewed survey is Wen et al., "Know Your Limits: A Survey of Abstention in Large Language Models" (Transactions of the ACL, vol. 13, 2025, pp. 529–556), which organises the work through three lenses: the query, the model, and human values. Note what it does not say — it treats abstention as a capability to study and tune per context, not as a setting that is always right to turn up. None of this literature establishes that handing work to a person is the correct operational choice; it establishes that declining to answer can raise the accuracy of what remains, which is a narrower and more useful claim.

The decision is a cost comparison, not a confidence reading

Here is the reframe that makes the rest of this tractable.

An autonomous answer has an expected cost: the probability it is wrong, multiplied by what being wrong costs you, plus the chance you cannot undo it. A human answer has a known cost: the handling time, loaded. You should escalate when the first exceeds the second.

The useful property of writing it this way: p(wrong) is the only term the model affects, and it is the term you have least control over. The other three are yours. Most teams spend their effort trying to push p(wrong) down with better prompts, and leave the three levers they actually own untouched.

A worked example, with the assumptions on the table

The figures below are invented to demonstrate the arithmetic. They are not measurements from any client engagement, not published benchmarks, and not our pricing. Nothing about your business can be inferred from them. They are placeholders in a formula, and the formula is the point:

Escalate when   p(wrong) × C(wrong)   >   C(human)   — or when the action cannot be undone, regardless of the result.

Suppose a human contact costs $6 fully loaded, and compare two intents handled by the same assistant:

Identical model, identical accuracy, identical confidence score — opposite correct decisions. A confidence threshold cannot produce that outcome, because confidence does not know what a cancellation costs. Only your routing table does.

The absolute numbers do not matter; the ratio does. What drives the decision here is that C(wrong) differs by 300× between two requests arriving at the same assistant. You do not need a precise figure to act on that — you need to know which of your intents are two or three orders of magnitude apart, which is usually obvious once someone asks. Where to get each term: p(wrong) from your own evaluation set per intent, C(human) from your existing cost per contact, and C(wrong) from whoever owns the budget that absorbs the mistake. If a term is genuinely unknowable, bound it: a worst-case estimate still sorts your intents correctly.

This is also why "what's a good escalation rate?" has no answer. A business whose traffic is mostly opening-hours questions and one whose traffic is mostly cancellations should have wildly different rates, and both can be right. Any benchmark quoted to you without that composition is noise. If you want the broader version of this arithmetic, including retry and context costs, we worked it through in the economics of agent cost per task.

The quadrant that decides it faster than a formula

In practice you rarely compute the expected cost per request. You classify intents once, during design, and route by category. Two axes do almost all the work.

Note what the right-hand column (B and D) does. Irreversibility is not a tiebreaker applied after the arithmetic — it overrides it. A $40 action you cannot reverse deserves a checkpoint even though the expected cost says automate, because the expected-cost calculation assumes you get to average over many attempts, and an irreversible action does not give you that. This is the same reason you confirm before a destructive operation in any system, model or no model.

When the assistant can act, escalation becomes an approval gate

Everything above assumes the assistant answers. The moment it can also do things — issue a refund, change a booking, write to a system of record — the escalation decision changes character. You are no longer choosing who replies. You are choosing who authorises.

That distinction has a regulatory edge worth knowing about, because it may turn this design work into a compliance requirement rather than a quality preference. Article 14 of the EU AI Act sets out human-oversight obligations for systems classified as high-risk. Its first paragraph requires that such systems "shall be designed and developed in such a way... that they can be effectively overseen by natural persons during the period in which they are in use." Paragraph 4 is more specific about what the people doing the overseeing must be able to do, including to:

• "decide, in any particular situation, not to use the high-risk AI system";
• "otherwise disregard, override or reverse the output of the high-risk AI system";
• "intervene in the operation" of it, or "interrupt the system through a 'stop' button or a similar procedure" that "allows the system to come to a halt in a safe state."

Read that against the quadrant above and the overlap is hard to miss: override, reverse and halt safely are the reversibility axis, written into law. Paragraph 4 also requires that overseers remain aware of the risk of automation bias — the tendency to defer to whatever the system proposed.

Two honest limits on that, both of which matter before you act on it. First, whether any particular support or intake assistant counts as high-risk under the Act's Annex III is a legal question about your sector and use case, not a technical one, and most customer-service assistants are not obviously in scope. Second, the timetable for high-risk obligations is genuinely contested: published dates conflict, and amendments to the phasing have been proposed and were unresolved when this was written. We are not your lawyers, and you should not take a date from an engineering blog — including this one. Treat Article 14 as a well-drafted specification for what oversight should look like, and get advice on whether and when it binds you.

The design consequence is the same either way, and it is the part teams get wrong:

• A stop button is not a stop path. "Halt in a safe state" means deciding what happens to work already in flight — the half-written record, the queued message, the partially applied change. That is design work, not a UI control.
• Approval that is always granted is not oversight. If your reviewer approves 99% of what arrives, you have built a confirmation dialog and called it a safeguard. Measure the rejection rate; if it is near zero, either the gate is in the wrong place or the reviewer has stopped reading.
• The agent should prepare the case, not just request permission. A gate that shows a proposed action with no evidence forces the reviewer to redo the work. Show what was found, which rule applies, and what the agent would do — then the approval is a judgement rather than a rubber stamp.
• Unanswered approvals are a failure mode of their own. Decide now what happens when nobody responds within the window. Defaulting to "act anyway" quietly removes the gate; defaulting to "stall forever" breaks the customer promise. Pick deliberately and alert on the queue.

Signals worth using, in the order they earn their keep

Once intents are classified, you still need request-level triggers for the cases that do not fit their category. These are the signals we actually wire up, cheapest and most reliable first.

Note that the table splits what most guidance merges. "Uncertainty" is two questions — is this answer wrong? and is this answerable at all? — with different best signals, and conflating them is why a single confidence slider never behaves the way teams expect.

The ranking also has an uncomfortable implication: the two highest-value triggers are ordinary application logic. Missing-field checks and threshold breaches are the kind of thing that has guarded financial software for decades. They escalate correctly because they do not involve the model's opinion at all. If you have a limited budget for this work, spend it here before you spend it on uncertainty estimation.

There is a reason consistency sampling has to be rationed, too. Identical inputs do not reliably produce identical outputs on hosted infrastructure, which means run-to-run disagreement partly reflects serving conditions rather than genuine ambiguity — we went through the mechanics in why the same prompt gives different answers. Treat the disagreement rate as a noisy instrument, and calibrate it rather than trusting a single reading.

Calibrate against your own traffic, because no benchmark transfers

You cannot pick a threshold from an article, including this one. But there is a sound method for finding yours, and it comes from the selective-prediction literature rather than from vendor guidance.

Jung, Brahman and Choi, in "Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement", build a framework where a model answers only when its confidence clears a bar, and that bar is set so agreement with human judgment is "provably guaranteed... to a user-specified agreement level." On a Chatbot Arena subset where GPT-4 rarely reaches 80% human agreement, their cascade holds above 80% agreement while covering 79.1% of samples — and 88.1% of those covered cases are decided by the cheaper models in the chain rather than the strongest one.

Three caveats, because this is borrowed machinery rather than a transferable result:

• Their "escalate" means a stronger model, not a person. The paper's rule is to "escalate to a stronger model only when the previous judge is not sufficiently confident." The cascade runs Mistral-7B, then GPT-3.5-turbo, then GPT-4-turbo. Nothing in it reaches a human.
• The domain is LLM-as-judge evaluation, scoring pairs of generated outputs — not customer support. No published guarantee covers your support queue.
• The guarantee assumes your calibration set is drawn from the same distribution as live traffic, and the formal bound does not cover distribution shift. The authors test a shifted setup empirically and report agreement holding while coverage drops. For a support queue — where a product launch or an outage rewrites your traffic mix overnight — that assumption is the one that will break first.

What generalises is the method's direction of travel: decide the agreement level the business requires, then discover empirically what coverage that costs you. That is the inverse of how most teams work, and it is the right way round, because the agreement level is a commitment you make to customers while coverage is merely its price. Treat a cheap intermediate step — a second model, a stricter retrieval pass — as a legitimate option before a human, since that is what their cascade actually demonstrates.

Concretely, over an afternoon:

• Build a labelled set per intent. 50–100 real requests with known-correct outcomes, grouped by the intent categories from the quadrant. Per-intent is the whole point — a single blended number hides the variation that matters.
• Sweep the threshold. For each candidate trigger, plot accuracy-on-answered against the share you answered. You are looking for the knee: the point where declining more stops buying accuracy. Tomani et al.'s 2–8% finding says to expect that knee early.
• Set the bar from the cost table, not the curve. The curve tells you what each accuracy level costs in escalations. The decision about which level you need belongs to whoever owns C(wrong).
• Re-measure after every model or prompt change. Treat a model upgrade as invalidating the calibration, because the Xiong finding says the units move. This is the step everyone skips, and it is why escalation behaviour drifts.

If this sounds like building an evaluation set, it is. The same evaluation set serves cost control and quality monitoring, which is the argument for building it before you need it rather than after an incident. The underlying problem — that correctness is expensive to establish even when answers are cheap to produce — is the one we called the oracle gap, and escalation design is the customer-facing version of it.

The handoff is where most of the value leaks away

Teams spend their effort on the trigger and almost none on the transfer. Then customers judge them entirely on the transfer.

A handoff that drops context is worse than no automation, because you have added a step and lost the information. The agent who picks up needs the intent, the facts already established, what the assistant already did or promised, and the unresolved question — not a wall of transcript to read while the customer waits.

Three failure modes are worth naming because each has a cheap fix:

• The silent handoff. The conversation changes hands and nobody says so. Announce it, and say what happens next.
• The cold handoff. The human opens with "how can I help?" after the customer has already explained everything. Pass a structured summary, not a log.
• The ping-pong. Work bounces back to the assistant after a human touched it. Once escalated, a conversation should stay escalated unless the customer starts something new.

Our own framing of that third path — route to a human with context rather than as a dead end — sits in the comparison of AI and rule-based assistants, where it is the path we flag as most often neglected. On voice the tolerances tighten considerably, because a caller cannot skim a summary and silence reads as a fault; the voice agent guide covers the latency budget that imposes.

What to measure, and the metric to stop reporting

Escalation rate on its own is close to useless as a target. It can fall because the assistant got better or because it got bolder, and those have opposite consequences. Reporting it as a number to drive down is how teams accidentally optimise for confident wrong answers.

Two of those require sampling conversations by hand every week. That is the cost of knowing, and it is lower than the cost of a confident system quietly doing the wrong thing. Budget a couple of hours weekly rather than treating it as a launch task.

When this is the wrong thing to build

We build intake and support automation, so take the following as a disclosure rather than modesty: the escalation design above only pays for itself in some situations, and we have told people not to proceed.

• Almost all your volume is high-cost and irreversible. If most requests land in the top-right quadrant, you are building an expensive form that still needs a human. Build the form, improve the queue, skip the model.
• You cannot price C(wrong). If nobody will say what a wrong answer costs, the arithmetic has no inputs and you will end up tuning a confidence slider by feel — exactly what this article argues against. Settle that first; it is a half-day conversation with finance.
• Your volume is too low to calibrate. Thresholds need labelled examples per intent. Below a few hundred repetitive requests a month you will not have enough to measure, and an uncalibrated threshold is a guess with a decimal point on it.
• The real problem is your content. If answers are not written down anywhere, retrieval has nothing to find and the assistant will escalate everything or invent things. Fixing the documentation is valuable on its own and is a prerequisite either way — more on that in what an AI assistant actually costs to build and run.

A sequence that works

1. List your intents and place each in the quadrant. Whiteboard exercise, an hour or two. Mostly argument about reversibility, which is the argument worth having.
2. Wire the deterministic triggers. Explicit requests, missing preconditions, policy thresholds, repeated failure. Ordinary code, highest value, no model involved.
3. Build the handoff payload before tuning any threshold. If the transfer is poor, better triggers only route people to a worse experience faster.
4. Price C(wrong) per category with whoever owns the number. Not engineering.
5. Build the labelled set and sweep. Per intent. Find the knee, set the bar from the cost table.
6. Add consistency sampling only where the arithmetic justifies the extra tokens.
7. Start sampling conversations weekly, both directions. Escalated and not.
8. Re-calibrate on every model or prompt change. Diary entry, not good intentions.

Steps one to three deliver most of the benefit and involve no uncertainty estimation at all. That ordering is deliberate: it front-loads the work that does not depend on the model behaving predictably.

Frequently asked

When should an AI chatbot escalate to a human?

When the expected cost of being wrong exceeds the cost of a human handling it, or when the action cannot be reversed. In practice that means always on explicit request, always when a precondition is missing or a policy threshold is crossed, and by default for anything high-cost and irreversible. Low-cost reversible requests should not escalate unless the customer asks.

Should I use the model's confidence score as the escalation trigger?

Not as your primary trigger, and never alone on an irreversible action. Xiong et al. (ICLR 2024) found that LLMs tend to be overconfident when verbalising confidence, and that sampling the same question several times and measuring disagreement beat the self-report on both calibration and failure prediction. Keep the self-reported score as one input rather than discarding it, since the same paper found consistency-only scoring was weaker than combining the two. Deterministic checks — a missing field, a policy threshold, a failed retrieval — outrank all of it.

What is a good escalation rate for an AI assistant?

There is no transferable benchmark, and a figure quoted without the traffic composition behind it is not usable. Two businesses with identical assistants should have very different rates if one handles mostly opening-hours questions and the other mostly cancellations. Measure escalation appropriateness and missed escalations instead of driving a single rate toward a target.

Is it better to escalate too often or too rarely?

It depends entirely on reversibility. For reversible, low-cost requests, over-escalating is the more expensive error because it removes the business case for automating at all. For irreversible actions, under-escalating is far worse, since the cost is not bounded by one annoyed customer. This is why the decision belongs per intent rather than globally.

How do I set the confidence threshold?

Empirically, per intent, from a labelled set of 50–100 real requests with known-correct answers. Plot accuracy on the requests you answered against the share you answered, which gives you the exchange rate between escalations and accuracy. Do not expect a convenient knee: in Tomani et al.'s TriviaQA experiment, declining 5% bought 1.6 points of accuracy and declining 25% bought 8.2, so the gains kept coming as the cost rose. Pick the operating point from what a wrong answer costs you, and re-run it after any model or prompt change.

Does escalating make the AI look like it failed?

Only if it is handled badly. A silent transfer, or one where the human asks the customer to start again, reads as failure. An announced handoff that arrives with full context reads as competence, and is frequently the interaction customers remember most positively.

Can I just escalate when the customer sounds frustrated?

Use sentiment, but not for this. Frustration tells you something true about the relationship and almost nothing about whether the last answer was correct. It is a good input to routing priority and tone, and a poor substitute for knowing your error rate per intent.

Working out where the line should sit? The useful starting point is a list of your request types and a rough cost of getting each one wrong — that is usually enough to place them in the quadrant and show which ones are worth automating at all. It is also where we occasionally conclude you should not build this. See how we build assistants, how escalation fits into an intake pipeline as a designed stage rather than a fallback, or send us the list.