Automating email triage

What is actually in the inbox

Every company past about thirty people has one. Orders, or claims, or referrals, or service requests land in a mailbox that four people share, and somebody opens it at 7:30 and starts sorting. That sorting is real work, it is expensive, and almost none of it is written down. Which is why the first honest step is not building anything. It is spending two days finding out what the mail actually is.

The shared inboxes we get called into run between roughly 150 and 1,500 messages on a business day. A wholesale distributor's orders address. A property manager's maintenance address. A clinic's referral fax-to-email bridge. A freight broker's dispatch line. A benefits administrator's enrollment address. Different industries, and the distribution is remarkably consistent: somewhere between 55 and 80 percent of the volume falls into six or eight recurring shapes, and the remaining fifth is a tail of a hundred one-off things.

That tail is the part people forget when they scope the work. The six common shapes are the easy part — a purchase order, a shipping exception, a status question, an invoice dispute, a new-account request, a supplier newsletter. The tail is a customer replying to a two-year-old thread, a bounced delivery notification, a scanned handwritten note, a message that is entirely a 4 MB attachment with the subject line "see attached", a vendor's out-of-office bouncing off your own auto-reply, and a lawyer. You will not classify the tail. You need a design that survives not classifying it.

Measure before you build: the 300-message sample

Pull 300 consecutive messages from a normal week. Not a curated set, not last month's interesting ones — consecutive, including the newsletters and the bounces. Then have the person who actually does this work sort them into whatever buckets they use in their head, and write the bucket names down. That list is your taxonomy, and it will not match the folder structure in the mailbox, which is usually five years old and named after people who left.

Now do the part everybody skips. Have a second experienced person sort the same 300 messages, blind. Compare. On the inboxes we have measured, two people who both do the job agree on roughly 80 to 90 percent of messages, and the disagreements cluster in two or three specific pairs of categories that turn out to be genuinely ambiguous. That number is the most valuable thing you will produce all month, for a hard reason: it is a ceiling. A model trained on labels those two people produced cannot be more consistent than the labels are. If your reviewers agree 78 percent of the time and a vendor promises 97 percent accuracy, one of you is measuring something different.

The disagreement pairs are also a design instruction. If "billing question" and "invoice dispute" get confused by humans, they should be one queue with a sub-flag, not two queues. Collapsing a category is free. Discovering after launch that the model splits mail into two piles that go to the same person is a month you did not need to spend.

The unit of work is the thread, not the message

This is the single most common technical mistake, and it looks harmless right up until the pilot. A reply that says "yes, go ahead, ship it" is unclassifiable on its own. So is "attached". So is "per our call." Roughly a quarter to a third of the messages in a busy shared inbox are replies inside an existing conversation, and if the classifier sees them alone, its top prediction will be whatever the model does when it has no information — usually the largest class.

Thread the mail first. Match on the message-id reference chain where it exists, and fall back to normalized subject plus participant set plus a time window, because a surprising number of real replies arrive with the reference headers stripped by somebody's gateway. Then classify the thread, carry the decision forward, and treat later messages as updates to a thing that already has a home rather than as new arrivals needing a home.

While you are in there, strip the quoted history and the signature blocks before anything reads the text. A four-line question at the top of a message containing eleven rounds of quoted history plus two legal disclaimers is mostly disclaimer by character count. Naive text handling weights all of it equally, and every message from that one client with a 300-word confidentiality footer starts to look like every other message from that client.

Routing is the product. The classifier is a component.

The label is worth nothing on its own. What creates value is a message arriving in a place where somebody is expected to work it, with the context needed to work it, and with a record that it arrived. If the label is written into a log and a person still opens the inbox and sorts by hand, you have built an expensive metrics dashboard.

So decide the destination before the model. A queue in the helpdesk. A folder plus a category in the shared mailbox. A row in the order system with the message attached. A task assigned to a named person. Each of those has a different integration cost and a different failure mode, and the one you can actually write to is usually the deciding factor. We have had projects where the classification was solved in a week and writing into a fifteen-year-old order system took two months.

Destination	Works when	What it costs to build	Failure mode
Folder plus label in the mail system itself	People will keep working out of the mailbox and you want the smallest change possible	Days. Mail APIs are well-trodden	No assignment, no SLA clock, two people work the same message
Ticket in a helpdesk	You already own one and it has an API and an owner	One to three weeks including field mapping	Duplicate tickets when threading is wrong; a reply that opens a second ticket
Record in the system of record order, claim, policy, job	The mail is about a thing that already exists in a database	Weeks to months, driven by that system's age	Matching the message to the right record; a near-miss is worse than no match
Task to a named person	Small team, work is genuinely individual	Days	Assignment to someone on leave; nothing rebalances

The attachment is often the message

In order entry, claims intake and referrals, the body text is a courtesy and the content is a PDF. "Please see attached PO." Six words. Everything a router needs to know — customer, ship-to, urgency, whether this is a rush — is in a document that may be a clean export from an ERP or may be a phone photograph of a printed sheet taken at an angle in a warehouse.

Decide early how far you are going. Reading the attachment well enough to route it (what kind of document is this, whose is it) is a much smaller problem than reading it well enough to key it (what are the twelve line items and their quantities). Routing off filename patterns, sender domain and a coarse document-type classifier gets you a long way and costs a fraction. Full extraction is a separate project with a separate budget and a separate accuracy conversation.

What to do when the model is unsure

Every triage system needs an explicit "I do not know" path, and it needs to be a first-class outcome rather than a bug. Set a confidence threshold, send anything below it to a human review queue, and instrument the size of that queue as a headline number. A system that routes 70 percent of mail confidently and hands 30 percent to a person is genuinely useful. A system that routes 100 percent and is wrong on 15 percent is worse than the status quo, because now nobody knows which 15.

The correction has to be one click, and the correction has to become training data. If fixing a wrong route means opening a different tool, choosing from a 40-item dropdown and typing a reason, nobody will do it, and you will lose the only labelled data stream you were ever going to get for free. Build the correction into the queue view: right label, one click, logged.

What the numbers realistically look like

With eight to twelve well-separated categories and six months of history that has been cleaned rather than taken at face value, top-one accuracy in the high eighties to low nineties on the common classes is a reasonable expectation. Classes with fewer than a couple of hundred clean examples should be assumed to be unlearnable for now; do not promise them, and route them to the review queue on purpose.

The business number is not accuracy anyway. It is the share of mail that reaches the right destination without a person touching it, and time to first touch on the messages that matter. A distributor whose urgent shipping exceptions were being found an average of four hours after arrival cares about that four hours far more than about a percentage. Pick the two numbers before you start and measure them on the current manual process for a fortnight, or you will have no way of proving the thing worked.

When you do not need software for this

Three situations, and we say this often enough that it is worth writing down.

Low volume with a stable team. If two people clear the mailbox in ninety minutes a day and neither of them minds, the payback on a build is years. A better folder structure, three or four rules on sender and subject, and a written rota will get most of the benefit this month for the cost of an afternoon.

The sender could just tell you. A large share of "please classify this" is really "we never gave them a form." If 40 percent of your inbox is new service requests arriving as free text, a short web form that captures the six fields your team asks for anyway removes the classification problem instead of automating it. Customers complain about forms less than people expect when the form is genuinely short and the alternative is a slow answer.

The real bottleneck is downstream. If mail is sorted within twenty minutes and then sits in a queue for two days because nobody has capacity, faster sorting produces a longer queue and no improvement anyone can feel. Measure where the time actually goes before you assume it is the sorting.

The failures we get called in to fix

• Classifying messages instead of threads, so every "yes please proceed" lands in the largest category
• A taxonomy copied from the helpdesk dropdown, with 60 categories of which 11 are ever used
• No abstain path, so a low-confidence guess is routed with the same confidence as a certain one
• Corrections that go nowhere, throwing away the only free labelled data the company had
• Auto-replies to automated senders, generating a mail loop over a weekend
• Attachments ignored when the attachment was the entire content of a third of the mail
• Training on stored labels nobody checked, inheriting years of "close enough" sorting
• No before-measurement, so nobody can say afterwards whether it helped

A realistic first build

Cutting over one category at a time is the part people compress and should not. Start with the machine-generated mail, which is the easiest and the most boring, and let the team watch it be right for a week before anything customer-facing moves. Trust in a routing system is built exactly once, and a bad first fortnight means people keep checking the inbox by hand for a year afterwards, which costs you the entire benefit.

What to have in place before go-live

• Human agreement measured on a dual-labelled sample, and written down
• Threading verified against real mail, including replies with stripped headers
• A destination that a person actually works out of, with an owner per queue
• An abstain threshold and a review queue somebody is responsible for clearing
• One-click correction that writes back a label you can retrain on
• Baseline numbers from the manual process, taken over at least two weeks
• No outbound automated replies in version one
• An alert when the daily mix shifts, because a product change or a new client moves it fast
• Retention and access rules agreed, since the mailbox holds personal data
• A documented way to turn it off and go back to manual in an hour

Bottom line

Email triage is one of the few automation projects where the payback is easy to see and easy to overstate. It works when the mail has genuine repeated structure, when there is somewhere real to route to, and when the team accepts an explicit unknown pile. It fails when the taxonomy was invented in a meeting, when replies are classified out of context, and when the accuracy target was set without anyone checking whether two humans could hit it. Do the 300-message sample first. It costs two days and it will tell you either what to build or that you should not build anything, and both answers are worth having.

Frequently asked questions