Start by testing whether your categories are learnable
Take three hundred tickets from last month. Have two experienced agents label them independently, without discussing it, using your current category list. Then compare. This costs about a day of two people's time and it is the highest-value day in the entire project.
If the two agents agree on 90% or more, your taxonomy is sound and a model can learn it. If they agree on 70%, the taxonomy is the problem, and any model you build will hit roughly that same ceiling — because you are asking it to reproduce a judgment that people themselves do not make consistently. Nothing about a bigger model changes this. The disagreement is in the label definitions, not in the technology.
The disagreements are also the most useful output. Read them. They will cluster into a handful of causes, and every one is fixable: two categories that overlap, a category defined by topic when routing depends on something else, a bucket named “Other” that is quietly the second-largest class, or a ticket that genuinely contains two problems and the form only allows one label.
You are probably here because
- Volume grew and triage now takes a person most of a day
- Your reporting by category is not trusted, so nobody uses it to staff
- Urgent tickets sit in a queue for hours before anyone sees them
- A vendor quoted an auto-triage product and you want to judge the claim
The agreement test above is the first thing to do. The metrics section is what to hold a vendor to. The section on what not to automate is the one that prevents an incident.
Name categories after what happens next
Most taxonomies are organized by subject, because subjects are how people describe things. Operationally that is the wrong axis. A category earns its place if a different thing happens when a ticket lands in it — a different team, a different priority, a different first response, a different service commitment.
“Billing” is a subject. “Refund request over the approval limit” is an action. The first tells you where the ticket belongs in a report; the second tells you what to do at 9am. If two categories always route to the same person with the same urgency, they are one category with a reporting attribute, and merging them makes the model's job easier at no operational cost.
Three more rules that pay for themselves. Keep the top-level list short — most support organizations operate well with eight to fifteen routing categories, and anything past about twenty-five degrades human labeling before it degrades the model. Separate the axes: category, priority and product area are three fields, not one combined label, and combining them multiplies your classes and starves each one of examples. And give every category a written definition with two example tickets and one near-miss, because a category with no written definition means whatever the last person to use it thought it meant.
What the classification is for changes what accuracy has to be
The cost of a mistake is not uniform, and treating it as though it were is how projects get sized wrong.
| Use | Cost of an error | What accuracy has to be |
|---|---|---|
| Routing to a queue | Low. An agent reassigns it, losing minutes. | Moderate is fine. Errors are visible and self-correcting. |
| Reporting and staffing | Low per ticket, but systematic bias skews the aggregate. | Consistency matters more than accuracy. A steady error is manageable; a drifting one is not. |
| Priority and escalation | High. A missed urgent ticket is a real incident. | High recall on the urgent class specifically. Over-flagging is much cheaper than under-flagging. |
| Suggested reply for an agent | Low. The agent edits or discards it. | Useful more often than not is enough, provided it is fast. |
| Automatic reply to the customer | High. A wrong answer reaches the customer with your name on it. | Very high, and confined to a narrow set of clearly identified intents. |
| Automatic closure | Severe. The customer is ignored and does not know it. | Do not do this. See below. |
The practical consequence is that routing is a good first project and auto-reply is not. Start where errors are cheap and visible, earn the operational trust, and expand only into places where measurement supports it.
Which approach, and when each is worth its cost
Four options, roughly in order of cost. Try them in this order and stop when the numbers are good enough, because each step up costs more to build and more to keep running.
Rules on a handful of phrases. Unfashionable and frequently sufficient for the head of the distribution. A password reset ticket says “password” or “reset” or “locked out.” Two days of work over your top few categories often handles a large share of volume, costs nothing per ticket, and is instantly explainable to the team. Always measure this before proposing anything else, so you know what the model has to beat.
A classical text classifier. Trained on your own historical tickets, this handles the long tail that rules miss, runs in milliseconds, costs effectively nothing per ticket, and behaves identically every time. It needs labeled data — on the order of a few hundred examples per class to be decent, more for classes that are subtle — and you may already have it in the historical tags, provided those tags are trustworthy, which the agreement test tells you.
A language model with a written instruction and a few examples. The current default for a first build, and for good reason: it needs no training data, it handles categories you invented yesterday, it deals with phrasing nobody anticipated, and it can produce a short justification alongside the label. It costs a fraction of a cent per ticket at typical ticket lengths, adds a second or two of latency, and changes behavior when the provider updates the model unless you pin the version.
A small tuned model. Worth considering at high volume with a stable taxonomy and a few thousand labeled examples. Lower cost per ticket, lower latency, runs on hardware you control, and changes only when you change it. Rarely the right first step; often the right second one, once the label set has stopped moving and the corrections have accumulated.
Where a triage project actually spends its time — our read
Our judgment on projects of this shape, not a survey. The bottom row is the part that gets demonstrated.
Overall accuracy is the wrong number
Support volume is severely imbalanced. A large minority of tickets is usually one or two mundane categories, and a handful of categories that matter enormously are each a percent or two of volume. A classifier that predicts the biggest class every time can post an accuracy figure that sounds respectable and be operationally worthless.
Ask for the numbers per class, and ask for two of them. Of the tickets sent to this queue, how many belonged there — that is the cost borne by the receiving team. And of the tickets that belonged in this queue, how many arrived — that is the cost borne by the customer whose urgent issue went to the wrong place. The two trade off, and which one you favor should be a deliberate decision per category rather than a side effect.
Then look at the confusion pairs. In every system there are two or three specific mix-ups that account for most of the damage, and they are usually fixable by editing a category definition rather than by touching the model. Do that before anyone proposes a bigger model.
Finally, measure the model against the humans, not against perfection. If your agents agree 88% of the time, a classifier at 84% is doing well and further effort has poor returns. Comparing to 100% makes a good system look like a failure and pushes teams into spending months for two points.
Confidence, abstention, and the number to watch
A production triage system should not label everything. It should label what it is sure about and hand the rest to a person, which converts an accuracy problem into a coverage problem — a much better trade.
Set a confidence threshold per category, higher for the ones with expensive errors. Above the threshold, apply the label automatically. Below it, route to a human queue with the top two suggestions shown. Then track the number that actually matters: the share of tickets handled automatically at your required accuracy. A system that confidently and correctly handles 60% of volume and defers the rest is a genuine win. One that labels 100% at 78% accuracy is quietly creating work.
Confidence scores need calibration. A raw score of 0.9 does not mean right nine times in ten; it means whatever your particular setup happens to make it mean. Check it empirically on a held-out sample, pick thresholds from that evidence, and re-check quarterly.
The feedback loop nobody wires up
Every time an agent corrects a label, you have been handed a perfectly labeled example by someone with more context than any annotator you could hire. This is the cheapest training data in existence and most systems throw it away.
Capture the correction, the original prediction, the confidence and the ticket. Two things follow. Corrections become training data, so the system improves in the places it actually fails rather than the places somebody guessed it might. And the correction rate per category becomes your live quality monitor — a category whose correction rate doubles this month is telling you that something changed, usually in the world rather than in the model.
Make correcting a label take one click. If it takes four, agents will stop, and you will lose both the data and the signal.
Drift, and what causes it here
Ticket classifiers degrade for reasons that are specific and mostly foreseeable. A product launches and generates a kind of ticket that has never existed. A competitor's outage sends you a wave of unusual traffic. A price change alters the language customers use. A holiday changes the mix entirely. The model has not changed; the world has.
Three cheap monitors catch nearly all of it. Watch the predicted category distribution week over week, because a sudden shift is either a real event or a broken integration and both are worth knowing within a day. Watch the abstention rate, which rises when inputs stop looking like anything the system has seen. And watch the agent correction rate per category. Set alert thresholds on all three at levels that will fire a few times a year rather than weekly, because an alert people ignore is worse than none.
What not to automate
Two lines we hold, and recommend holding.
Never close a ticket automatically. Not on a confidence score, not after a timer, not because it looks like a duplicate. The failure is silent and it lands on the customer: they wrote in, nobody replied, and the record says it was handled. Deflection with an offered answer and a visible way to reach a person is fine. Closure without a human is not.
Never auto-reply on anything with legal, financial, safety or health content. Cancellations, disputes, complaints about billing, anything mentioning harm, anything from a lawyer. Build an explicit detector for these and route them straight to a person, before classification. This detector should be biased heavily toward false positives, because a human glance costs a minute and the alternative can cost a great deal more.
One more, less absolute: show the agent the predicted category and let them see it was predicted. Hidden automation gets blamed for everything that goes wrong in its vicinity, and visible automation gets corrected, which is how it improves.
Cost, timeline, and when to skip it
A first production system over a helpdesk with a usable interface generally takes three to six weeks: the agreement test and taxonomy work, the integration, the measurement harness, then a shadow period where the system predicts without acting so you can compare its labels against what agents actually did. Do not skip the shadow period; it is a week and it settles every argument about whether to turn it on.
Running cost is small. At typical ticket lengths, model-based classification runs at a fraction of a cent per ticket, so even substantial volumes land in the low hundreds of dollars a month; a trained classifier is cheaper still. The real recurring cost is a person spending an hour or two a month reviewing corrections and the distribution.
And the case for not doing it: below roughly a couple of hundred tickets a week, triage is not usually the bottleneck, and a better intake form with four required fields plus six routing rules will get most of the benefit in two days. It is worth being honest about this. If a form change solves it, the form change is the right answer, and it does not need us.
The mistakes we see most
- No agreement test, so the model is asked to learn a judgment humans do not share
- Forty categories where twelve would route the work
- Category, priority and product combined into one label, multiplying classes
- Overall accuracy reported on a distribution where one class is most of the volume
- No abstention, so the system guesses on the tickets it understands least
- Uncalibrated confidence, with thresholds chosen because the number looked high
- Agent corrections discarded, throwing away the best training data available
- Auto-close enabled, which fails silently and lands on the customer
- No shadow period, so the first real evidence arrives after go-live
- Trained once, with nobody watching the distribution afterward
Before you build
- Two agents labeled three hundred tickets independently and you know the agreement rate
- Every category has a written definition, two examples and one near-miss
- Each category corresponds to something different happening operationally
- Category, priority and product are separate fields
- A rules baseline exists and its score is written down
- Per-class numbers are reported, not overall accuracy
- A confidence threshold and a human path exist for low-confidence tickets
- Sensitive content is detected and routed to a person before classification
- Agent corrections are captured with one click and stored
- The system runs in shadow mode for a week before it acts
- Nothing closes a ticket without a human
Bottom line
Spend the first day testing whether two experienced people label tickets the same way, because that number is the ceiling on everything that follows. Cut the taxonomy down to categories that change what happens next, and write definitions for them. Build the rules baseline so you know what a model has to beat. Judge results per class rather than overall, set thresholds so the system defers what it does not understand, and measure the share it handles automatically at the accuracy you need. Capture every correction. Watch three numbers monthly. And never close a customer's ticket without a person seeing it.
Frequently asked questions
For a trained classifier, on the order of a few hundred per category is a reasonable target, with more for classes that are subtle or rare. For a language model given a written instruction and a handful of examples, you need almost none to start — which is a large part of why that is now the usual first build. You still need a labeled test set of several hundred tickets to know whether any of it works, and there is no way around that one.
Try it first. It is already integrated, which is a large share of the work, and for a standard taxonomy it may be sufficient. Evaluate it the same way you would evaluate a custom build: per-class numbers on your own held-out tickets, not the vendor's demo. The cases where custom work wins are an unusual taxonomy, categories that depend on account context the vendor cannot see, or a need to keep the data inside your own systems.
Reasonably well for explicit signals — outage language, revenue impact, an executive on the thread, a deadline mentioned. Poorly for implied urgency, and badly for a calm message describing something serious. Treat automated priority as a suggestion that can raise urgency but never lower it, and keep the rules that already work: account tier, contract commitment, repeat contact within a day.
Modern models classify across languages far better than they translate, so a single classifier over multiple languages usually works. Test each language separately, because average performance hides a language that is failing. And route to agents by language as a separate decision from category, since the routing constraint is the human, not the topic.
Review monthly for the first quarter, then quarterly. Retrain when the correction rate rises materially, when the taxonomy changes, or after a product launch introduces something new. Ad-hoc retraining on a fixed calendar with no evidence behind it is mostly wasted effort; the monitors tell you when it is needed, which is the reason to have them.