Start with two hundred real questions
The standard way to start an internal assistant project is to gather documents. Somebody exports the handbook, the policy folder, the product manuals and a shared drive, points a retrieval system at it, and gets a demo working in a week. The demo is genuinely impressive, everybody is pleased, and the thing is quietly dead by the end of the quarter. The reason is simple and it is discoverable in advance: the corpus you had was never the same set as the questions people ask.
So do the unglamorous thing first. Pull two hundred real questions from the last ninety days. They live in more places than you expect: the support ticket queue, the operations channel where people ask the shift lead things, the inbox that human resources shares, the sales engineer who gets forwarded every hard product question, and the two long-tenured people whose desks everyone stops at. Copy them verbatim, into a spreadsheet, in the words the asker used.
Then sort them by hand. It takes an afternoon and it is the most valuable afternoon in the project. In most companies with a few hundred employees, sixty to seventy percent of those questions collapse into fifteen to twenty-five recurring shapes, and the rest is a genuine long tail that no system is going to handle. You now know what the assistant is for. You also know, before spending anything, roughly what fraction of the burden is even addressable.
You are probably here because
- An assistant went live, got a burst of use, and nobody has opened it since March
- Two experienced people are the bottleneck for everything anybody needs to know
- Somebody asked it a policy question and it confidently quoted a version from 2021
- Your board or your leadership asked what you are doing with AI and this was the obvious answer
The first three have one cause. The fourth is a bad reason to start, and a project begun for it usually produces the first three.
Half the questions are not document questions
When you sort the two hundred, a split appears that decides the architecture. Some questions have an answer written down somewhere: what is the bereavement policy, how do I request a badge, what is the return procedure for a damaged pallet. Others have an answer that lives in a system and changes by the hour: how much vacation do I have left, is part 4471 in stock at the Dallas warehouse, what is the status of ticket 88213, when does the Peterson job start.
Retrieval over documents answers the first kind. It cannot answer the second kind at all, and if you let it try, it will find a document that mentions vacation accrual and produce a fluent, plausible, wrong answer. The second kind needs a query against a real system, with the answer read back verbatim and no model paraphrasing in between.
This is where the honest advice sometimes is: you do not need a chatbot. If the top questions are all balance lookups, what you need is a link and possibly a better page. A chat interface over a single lookup is a slower version of a button. Where an assistant does earn its place is when the population of questions is wide and irregular enough that no menu covers it, and the answers are genuinely written down somewhere a person would have to hunt for.
| Question type | Example | What it actually needs |
|---|---|---|
| Written-down policy | “How much notice for unpaid leave?” | Retrieval over a curated, owned corpus, with a citation |
| Live record lookup | “What is my remaining balance?” | A direct system query, answer echoed without paraphrase |
| Procedure with a decision | “Customer wants to return an opened case—can they?” | Retrieval plus explicit rules, and a fast route to a person |
| Tribal knowledge | “Who do I call when the Denver line jams?” | Someone to write it down first. No system invents it |
| Judgment call | “Should we credit this customer?” | A person. Do not automate this one |
The corpus is the project
Everyone underestimates this and it is not close. The retrieval machinery is a solved commodity. What decides whether the assistant is trusted is whether the documents behind it are current, singular and owned.
In practice, when you go look, you find three versions of the expense policy, one on the intranet, one in a shared folder, one attached to an email from 2023, and the authoritative one is in a director's personal drive. You find product specifications that were superseded but never removed. You find a safety procedure that was rewritten after an incident, with the old copy still indexed. Point a retrieval system at all of it and it will faithfully return whichever one scores highest on similarity, which has nothing to do with which one is correct.
So before indexing, every document in the corpus gets three things: a named human owner, a last-reviewed date, and a decision that it is the single authoritative copy of that subject. Anything that fails those three is excluded, not included with a caveat. A small, curated, owned corpus outperforms a large one every time, in retrieval quality and in trust. Fifty good documents beat four thousand mediocre ones, and the fifty are achievable in three weeks.
The maintenance is the ongoing cost, and it is a real one. Budget for a person spending roughly two to six hours a week keeping the corpus honest — reviewing what changed, retiring what is stale, adding the thing the assistant did not know last month. If nobody has that time, the project will work for a quarter and then slowly become a source of confidently outdated answers, which is worse than nothing because people acted on it.
Permissions, and the failure that ends projects
An assistant inherits the union of everything it indexed. If a compensation band spreadsheet, a disciplinary file or an unannounced reorganization plan is in a folder that got swept in, the assistant will answer questions about it to whoever asks. This is the incident that gets these projects cancelled, and it is entirely preventable.
Two rules handle nearly all of it. First, filter at retrieval time by the asking user's actual permissions, not at indexing time by a guess about what is sensitive. Indexing-time filtering rots the moment someone's access changes. Second, start from an explicit inclusion list rather than sweeping a drive and excluding. Sweeping is how the compensation file gets in; it is always in a folder somebody forgot about.
There is a third rule that costs nothing and saves a lot: log every question, and have someone read the log. Not for surveillance — tell people it is logged — but because within two weeks the log will contain a question that reveals a document is reachable that should not be, and that is the cheapest possible way to find out.
What actually decides whether people keep using it — our weighting
Weights sum to 100. Our judgment from projects of this shape, not a measurement. The last row is where most of the enthusiasm goes.
Citations are not a nicety
An answer nobody can check is an answer nobody will rely on for anything that matters. The first time someone asks about a leave policy and gets a clean paragraph with no source, they will go read the handbook anyway, and at that point the assistant cost them time instead of saving it.
Every substantive answer should name the document, the section, and link to the exact place. Not the document root — the paragraph. This is more engineering work than people expect, because it means keeping character offsets through chunking and rendering, but it converts the assistant from something to be second-guessed into something that saves a search. In our experience it is the single feature most correlated with an assistant still being used six months later.
The related discipline: the answer must be grounded in the retrieved text and nothing else. If the retrieved passages do not contain the answer, the correct output is to say so. That is a prompt and evaluation problem more than a model problem, and it is testable.
Measure abstention and wrong answers separately
The usual first metric is deflection — how many tickets did not get filed. It is the wrong place to start, because you can drive it up by answering everything, including the things you should have declined.
Track three numbers instead. Answered and accepted: the user asked, got an answer, and did not then ask a person the same thing within a day. Declined: the assistant said it did not know. Wrong: it gave a confident answer that was incorrect, found by sampling. Declining is a healthy behavior with a real rate; something in the range of one in six to one in three questions is normal for a well-scoped assistant, and a system that never declines is one that is guessing.
Wrong answers are the only number that destroys trust, and they are invisible unless you go looking. The practical method is a weekly sample: thirty conversations, read by the person who owns that subject area, graded correct or not. It takes forty minutes. It is the difference between knowing how your assistant performs and hoping.
Build the question set from the log, and freeze it
Take one hundred to one hundred and fifty of the real questions you collected, write the known-correct answer beside each, and run that set against every change — new documents, a prompt edit, a model upgrade, a chunking change. Without it, every improvement is a guess and every regression is a surprise a user finds first. Add to the set when a new failure appears; never rewrite a question to make it pass.
Put it where people already are
An assistant on its own web page is an assistant people have to remember. One inside the chat tool the company already lives in gets used because it is in the path. The same is true of an assistant inside the ticketing system the support team already has open all day, suggesting an answer beside the ticket rather than in a separate window.
Two audiences will behave very differently and it is worth planning for both. New employees ask broad questions, are forgiving, and are the strongest early advocates because the alternative is interrupting someone. Experienced employees ask narrow, hard, edge-case questions, and a wrong answer to one of them costs you that person permanently. The evaluation set should be weighted toward the second group, because they are the ones whose trust you cannot rebuild.
Send us fifty real questions and we will tell you if this is worth building.
Fifty verbatim questions from your ticket queue or your operations channel, and a list of where the answers live today. Email to contact@precisionfederal.com and you get back a written split — which are document questions, which need a system query, which need somebody to write something down first, and what share an assistant would realistically handle. One business day, no charge.
contact@precisionfederal.comThe escalation path is part of the product
Every assistant needs one click that reaches a person, and the handoff has to carry the conversation. If a user has to retype their question to a human after the assistant failed, the assistant made things worse. That is the whole calculation from the user's side, and it is why the escalation button belongs in version one rather than in the backlog.
The same mechanism gives you your best improvement signal. Every escalation is a labeled example of something the assistant should have handled and did not. Route them to the corpus owner weekly. Within two months that queue tells you exactly which twelve documents to write, and they will not be the twelve anybody guessed.
Cost, timeline and the shape of the work
A first internal assistant, scoped to one department, over a curated corpus, with citations, permission-aware retrieval, an evaluation set and an escalation path, is typically eight to fourteen weeks of work and a build cost in the mid-five to low-six figures depending on how many systems it has to query and how bad the document situation is. Running cost per conversation is genuinely small — usually cents, sometimes a fraction of a cent — and is almost never the deciding factor. Do not let a vendor's per-query pricing dominate a decision where the real cost is content curation.
The line item that gets cut and should not is the ongoing corpus ownership. It is a few hours a week of a knowledgeable person, forever. If that person does not exist, the honest recommendation is to postpone the project and spend the same money getting the documents into one place with owners, which has value on its own and makes the assistant a much smaller job later.
When not to build one
Three situations where the answer is no, and saying so early saves everybody. If the recurring question volume is under roughly fifty a week, the payback will not cover the maintenance. If the answers genuinely are not written down anywhere — if the knowledge is in two people's heads — then the project you have is a documentation project, and no amount of retrieval creates text that does not exist. And if nobody will own the corpus, the assistant will become a machine for repeating last year's policy with great confidence.
The mistakes we get called in to fix
- Indexing everything, on the theory that more documents means better answers
- No citations, so every answer gets verified manually and the time saving evaporates
- Answering live-data questions from documents, fluently and wrongly
- Permission filtering at index time, which is correct on day one and wrong by day sixty
- A system that never says it does not know, tuned that way because declining looked like failure
- A standalone web page nobody has open, instead of the tool people already use
- Thumbs-up buttons as the only feedback, at a volume too low to mean anything
- No named corpus owner, so accuracy decays quietly and nobody notices for a quarter
Before you approve the build
- Two hundred real questions collected verbatim and sorted by hand
- Document questions separated from live-system questions
- An explicit inclusion list of documents, not a folder sweep
- Every document has an owner and a last-reviewed date
- Retrieval filters by the asking user's live permissions
- Answers cite a section a person can open
- Declining is an allowed, measured outcome
- A frozen evaluation set of real questions with known answers
- One-click escalation that carries the conversation to a person
- A named person with weekly hours to keep the corpus true
Bottom line
Internal assistants fail for reasons that have almost nothing to do with the model. They fail because the corpus was a folder instead of a curated set, because answers could not be checked, because the system guessed rather than declining, and because nobody owned it after launch. Every one of those is decided before any code is written. Start with the question log, keep the corpus small and owned, insist on citations, let it decline, and give it a fast path to a person. Do that and the usage curve stops falling off in week six, which is the only proof that matters.
Frequently asked questions
Far fewer than teams expect. Fifty carefully chosen, current, owned documents will outperform several thousand swept from a shared drive, both in retrieval quality and in trust. Every stale duplicate in the corpus is a candidate answer that competes with the correct one on similarity, and similarity has no opinion about which version is in force.
Only by querying the system that holds the answer and reading it back without paraphrase. If you let a document-retrieval system attempt it, you will get a fluent answer derived from the accrual policy rather than the person's actual balance. Separate the two paths explicitly and never let retrieval fill in for a lookup.
Something like one in six to one in three questions, for a well-scoped assistant over a curated corpus. A rate near zero almost always means the system is guessing rather than that it is unusually good. Track declines and wrong answers as separate numbers, because tuning one down usually pushes the other up.
Build from an explicit inclusion list rather than sweeping folders, and filter retrieval by the asking user's live permissions rather than by an index-time judgment about sensitivity. Then log every question and have someone read the log weekly — the first sign that something is reachable which should not be usually appears there within two weeks.
Typically eight to fourteen weeks and a build cost in the mid-five to low-six figures for one department, with citations, permission-aware retrieval, an evaluation set and a real escalation path. Per-conversation running cost is usually cents and rarely drives the decision. The recurring cost that matters is a few hours a week of a knowledgeable person keeping the corpus current.