Retrieval under the user's identity, not the service's

The pilot passed because one account could read everything

Someone created a connector account, gave it read access to the whole document store so the crawl would not fail halfway through, and pointed it at the wiki, the shared drive, the ticket system and the contracts folder. The index built cleanly. The demo was excellent. Then the assistant went to forty people instead of six, and within a fortnight somebody asked it a reasonable question and got back a paragraph summarizing a severance agreement, or a customer's unannounced renewal terms, or the compensation band for a role two levels above theirs. The model did nothing wrong. It answered from what it was handed, and it was handed everything.

This is the most common failure we are called in to fix on retrieval systems, and it is almost never described that way when the call comes. It arrives as "the security team blocked our rollout," or "legal wants an assessment before we expand," or "we need to add permissions to the RAG." The last phrasing is the telling one. Permissions are not a feature you add to retrieval. They are a property of how retrieval addresses the corpus, and if that was decided wrong during the pilot, you are changing the shape of the index, not adding a filter.

The mechanism is worth stating plainly. A vector index stores chunks with embeddings, and a similarity search returns the nearest chunks to the query vector. Neither operation knows anything about who asked. If nothing between the request and the index carries the asking user's identity and turns it into a constraint the search honors, every user of the assistant holds the union of every permission the ingestion account held. That is a straightforward privilege escalation, and it is the failure behind LLM02 in the OWASP Top 10 for LLM Applications, which names sensitive information disclosure through retrieval context as its own category rather than folding it into prompt injection.

Post-filtering is the wrong place and it is also wrong

The first fix teams reach for is to retrieve the top fifty chunks, then drop the ones the user should not see, then send whatever survives to the model. It feels like access control. It fails in two independent ways, and both matter.

The first is a quality collapse nobody predicts. Similarity search returns a fixed number of neighbors. If a user can see eight percent of the corpus and you ask for fifty chunks, you will typically keep three or four, and they will be the ones that happened to rank highly against a candidate set drawn mostly from documents that were never eligible. The user with the narrowest access gets the worst answers, and the assistant looks broken to exactly the people most likely to complain about it. Teams then raise the candidate count to five hundred, latency triples, and the recall problem is reduced rather than removed.

The second is a leak the filter cannot close. Chunks that get dropped still influenced what came back. Worse, most systems compute reranking, citations, and sometimes a summary before the filter runs. We have seen a system filter document bodies correctly and still return a source list naming files the user had no right to know existed. A filename is disclosure. So is a count.

The rule that follows is short. The permission constraint has to be inside the search, evaluated as part of finding the nearest neighbors, not applied to the neighbors after they are found.

Four architectures, and the one you probably need

There are four workable shapes. They differ in where the authoritative access decision is made and how stale it is allowed to be, and picking between them is a real decision with real tradeoffs rather than a matter of preference.

Architecture	How the constraint reaches the search	Staleness	Where it hurts
Query-time delegation	The retrieval service holds a token issued for the asking user and calls the source system's own search API on their behalf	None. The source system decides	You inherit the source system's search quality and rate limits, and every source needs its own integration
Access-list filtering	Each chunk carries the set of principals allowed to read it; the query carries the user's expanded group set as a filter predicate	Whatever your sync interval is	Access-list explosion on large corpora, and group expansion for a user in 300 groups
Namespace partitioning	The index is split by security boundary; a user's request is routed only to the partitions they hold	Low. Boundaries change rarely	Only works when the corpus really does partition. Documents shared across boundaries force duplication
Materialized visibility sets	A precomputed set of document identifiers each user may see, refreshed on change and joined at search time	Bounded by refresh lag, typically seconds to minutes	Storage grows with users times documents; needs a real invalidation path

Most enterprise builds land on access-list filtering with namespace partitioning above it, the only combination that survives a heterogeneous corpus without a per-source integration for every system you own. Query-time delegation is the most correct option, and we recommend it whenever the corpus lives in one or two systems with a decent search API, because it makes staleness disappear. Materialized visibility sets are right when access is driven by your own application's data model rather than a document store.

Group expansion is the part that breaks

Access-list filtering sounds simple until you write down what a principal actually is. A user in a mature directory belongs to nested groups, dynamic groups computed from attributes, and groups that exist only inside one application. A document's access list names some mix of individual users, groups, a group containing groups, and inherited permission from a parent folder shared with a distribution list.

To turn that into a query predicate, you must expand the user into the flat set of every principal identifier they effectively hold, and you must do it fast enough to sit on the request path. In organizations we have worked in, that set commonly runs from tens to a few hundred identifiers per person. A vector database filter with a three-hundred-term disjunction is not free, and some engines degrade sharply once the filter becomes selective enough that the approximate index has to be scanned rather than traversed.

Two things make this tractable. Cache the expanded principal set per user with a short lifetime, five to fifteen minutes, and accept in writing that a revocation takes that long to bite. Then define the small set of actions where that delay is unacceptable, offboarding above all, and give those a path that invalidates the cache directly. This is the same tradeoff as session lifetime in ordinary application security, and it deserves the same explicit decision rather than a default.

The second technique keeps the access lists small by hoisting the common case. If ninety percent of a corpus is readable by one broad internal group, tag those chunks with a single identifier and put per-document lists only on the remainder. The filter predicate becomes one cheap term plus a short list.

The permission copy goes stale, and staleness is the actual risk

Every architecture except query-time delegation keeps a copy of somebody else's access decisions. That copy is wrong from the moment it is written, and the question is only by how much and for how long.

Four events move permissions, at very different speeds. A person changes teams, and directory group membership updates within the hour. A folder is reshared, and everything beneath it changes at once, which is why permission changes arrive in bursts. A document is reclassified, usually without any file modification a crawler would notice. And a person leaves, which is the one that has to be fast.

A nightly full recrawl handles none of these acceptably. The design that works is a change feed. Most document platforms expose one, and the pattern is the same across them: subscribe to change notifications, treat a permission change on a container as a subtree invalidation, and reconcile with a full pass on a slower cadence. The reconciliation pass is not optional. Every change feed we have worked with loses events occasionally, and the failure is silent unless you look for it.

The measurement that matters here is permission lag: the elapsed time between an access change in the source system and the retrieval index honoring it. Instrument it, publish it, and set a target. A number you can state, like ninety-five percent of permission changes reflected within two minutes and all within thirty, is something a security reviewer can accept or reject on its merits. An unmeasured "we sync regularly" is what gets a rollout blocked.

Citations, counts and refusals leak too

Once retrieval itself is constrained, the leaks that remain are in what surrounds the answer.

Source lists must be filtered by the same constraint that filtered the chunks, computed from the same evaluated set rather than from a pre-filter candidate list. Result counts and "showing 5 of 47" affordances must be computed post-constraint, because a count reveals the size of what you cannot see. Query suggestions and autocomplete built from the whole corpus will happily complete a colleague's name against a document the asker cannot open. And the refusal message matters: "no results" and "you do not have access to the documents that would answer this" are different disclosures, and which one you can afford is a policy decision your security reviewer should make rather than your prompt engineer.

The subtler one is conversation memory. If earlier turns are summarized into a running context and that user's access narrows mid-session, or the conversation is later shared with a colleague, content that was legitimately retrieved travels to someone who was never eligible for it. Bind retrieved content to the identity that retrieved it, and re-evaluate rather than reuse when the identity changes.

Evaluation has to be per-user, or it proves nothing

A retrieval evaluation set that measures recall against the whole corpus tells you nothing about a permission-aware system, because it measures the behavior of an account that can read everything. What you need is a test harness with personas: several synthetic users at different access levels, a set of questions, and for each pair an expected answer that is either the correct content or a correct refusal.

Two metrics come out of it and both are needed. Authorized recall asks whether each user gets the documents they are entitled to, which is the quality number that catches the post-filter collapse described earlier. Unauthorized leakage asks whether any persona ever receives content, a citation, a count or an inference from material outside their access, and the only acceptable value is zero. Run the pair on every index change and every permission-model change, and run it as part of the release gate rather than quarterly.

Build the personas from real access patterns rather than inventing them. The interesting cases are awkward: the contractor with narrow project access, the person who changed departments last month, the assistant acting on someone else's behalf, the auditor with broad read and no write. Those four find more defects than fifty ordinary users.

What the frameworks actually ask for

If your buyer is federal, or sells to federal, the requirement is not novel and does not need a new vocabulary. NIST SP 800-53 AC-3 is access enforcement and AC-4 is information flow enforcement, and a retrieval layer that hands a user content from outside their authorization fails both plainly. AC-24 covers access control decisions and is the control that a reviewer will point at when asking where the decision is made and on what data. For a system handling controlled unclassified information, NIST SP 800-171 requirement 3.1.3 controls the flow of CUI, which is precisely what an unconstrained index does not do.

On the AI side, the NIST AI Risk Management Framework's Map function asks you to state the system's context and its data, and Measure asks for evidence about the risks you identified. A permission-aware retrieval design with a stated permission-lag number and a per-persona leakage test is a clean answer to both. ISO/IEC 42001 wants the same evidence in management-system form. The OWASP Top 10 for LLM Applications names sensitive information disclosure and excessive agency, and MITRE ATLAS catalogs the corresponding adversary behavior, which is useful when you need to explain to a non-technical reviewer that this is a known attack pattern rather than a hypothetical.

None of these frameworks tells you which architecture to build. They tell you what evidence your choice has to produce. That is the right division, and it is why we start by writing down the evidence the buyer will ask for and working backward into the design.

What a retrofit costs and how long it takes

Honest ranges, from doing this on real systems. A single-source retrofit, where the corpus lives in one platform with a usable permissions API and a change feed, runs roughly six to ten weeks of a small team and lands in the low six figures. That covers permission extraction, filtered search with recall validated per persona, a change-feed pipeline with reconciliation, the offboarding path, and the evaluation harness.

A heterogeneous corpus, four or more sources with genuinely different permission models, is a different project. Three to five months, dominated by the fact that each source needs its own extractor and its own semantics mapped into one internal representation. A file share's inherited access, a wiki's space permissions and a records system's role assignments do not mean the same thing, and reconciling them takes someone with authority to decide.

The number worth holding onto is the ratio. On these builds, permission plumbing is usually three to five times the effort of the retrieval and generation work it protects. Teams that budget for a retrieval project and discover a permissions project mid-flight are the ones whose rollouts slip by two quarters. Deciding the architecture before the pilot index is built costs a week and removes almost all of that risk.

How we would start on your system

The first artifact we produce is not code. It is a one-page map of every source in scope, what identity model it uses, whether it exposes a permissions API and a change feed, and what the worst realistic disclosure from that source would be. That page decides the architecture, and it usually takes two or three days of reading and a handful of short conversations with the people who administer each system.

Then the personas, then a thin end-to-end slice on one source with the constraint inside the search and the leakage test already running, then the remaining sources in order of disclosure risk rather than in order of how easy they are to connect. Connecting the easy source first is the habit that produced the pilot problem.

Sequencing it this way makes the security conversation an early one. A reviewer looking at a design with a stated permission-lag target, a per-persona test at zero leakage, and a wired offboarding path will engage with the specifics. A reviewer looking at a working assistant over an open index will stop the project, and be right to.

Bottom line

Retrieval that runs under a service account is a permission bug with a search box on it, and it is not fixed by filtering results after the fact. The constraint belongs inside the search, the permission copy needs a change feed and a measured lag, offboarding needs its own wire, and the citations and counts around the answer leak just as readily as the answer does. Get the architecture decided before the index is built and it is a week of design. Discover it after the pilot succeeds and it is a quarter, plus whatever the disclosure already cost.

Frequently asked questions