Multi-tenancy isolation for AI inference

What multi-tenancy means once a model is in the request path

Multi-tenancy in a conventional application is a shape everyone knows. Every row carries a tenant id, the data layer appends a predicate, and a reviewer reading the diff can see whether the predicate is there. An inference service breaks that shape. One request no longer touches one database. It touches a vector index, three or four caches, an adapter registry, a batching queue, a trace exporter, and tools that reach back into the same data through a different door. Every one of those is a place where one customer's text can reach another customer's response, and only one of them looks like a WHERE clause.

The failures are not exotic. Nobody extracts training data through a clever attack on the weights. What happens is a cache key computed from the prompt text and nothing else, a nearest-neighbor search that applies the tenant filter after ranking, an observability SDK whose content-capture default moved between minor versions, or a tool schema with a customer_id parameter the model fills in from whatever it just read. Each is a two-line bug and a disclosure you have to write a letter about.

This piece covers the leak surface in the order we review it, the one architectural rule that removes most of the class, four isolation tiers with what each one lets you commit to in writing, the performance half of the problem, and the test that proves the whole thing on every deploy.

The leak surface is not where your web app's was

When we review a shared inference platform, we look in a fixed order. The order is not arbitrary. It reflects how often the control is missing entirely rather than merely weak, and how far the data travels when it is.

Notice what is not on that list. The model is not a tenant boundary, and neither is the API gateway, which authenticates rather than authorizing access to data. The boundary lives downstream of both, which is why a security questionnaire answered by the API team is usually answered wrong.

Derive the tenant once, at the edge, and never again

The single change that removes most of this class is structural. Resolve the tenant exactly once, from the credential that authenticated the request, into an immutable context object. Then make every downstream client impossible to construct without it.

Concretely: the vector client, the cache client, the database repository and the tool executor all take a tenant context as a constructor argument with no default value and no setter. There is no VectorClient() in the codebase, only VectorClient(ctx). The factory that produces a context is the only exported way to make one, and it takes a verified credential rather than a string. A developer who wants to write the unsafe version has to add a parameter to a constructor, which shows up in review as an architecture change rather than as one missing filter argument buried in a 400-line diff.

Underneath that, use the database's own mechanism as a second layer. PostgreSQL row-level security works well, with two details people miss. Table owners bypass it by default, so the application role must not own the tables and FORCE ROW LEVEL SECURITY belongs on anything the owner touches. Any role carrying BYPASSRLS defeats the scheme, so audit grants and not only policies. The policy reads the tenant from a session variable your connection pool sets on checkout, which makes the pool configuration a security control.

Retrieval: filtering after the search is the wrong order

Approximate nearest-neighbor search returns the top k vectors by distance. If the tenant filter runs after that, two things go wrong. The mild one is recall collapse: with a thousand tenants in one index, a top-50 search returns mostly other tenants' chunks, the filter discards them, and the user sees three results where there should be twenty. Teams notice this one, because it looks like bad retrieval.

The severe one is that post-filtering puts correctness in your application code. A code path that forgets the filter returns another tenant's chunks with a plausible similarity score, and the model summarizes them without complaint. There is no error, no exception, no alert. The output looks like a good answer to a slightly different question.

Push the filter into the engine, or partition physically. Pinecone namespaces, Milvus partition keys and a Qdrant tenant payload index all restrict the search space before ranking rather than after. With pgvector the equivalent is a row-level security policy the planner applies, which is correct but degrades on a highly selective filter, because an HNSW scan that keeps discarding rows either loses recall or falls back to a sequential scan. Below a few hundred tenants a filtered shared index is usually fine. Above that, or when one corpus dwarfs the median, separate indexes earn their operational cost.

One more trap: hybrid retrieval runs a dense path and a lexical path, each with its own filter mechanism, which is two chances to get it wrong. Resolve the scope once in the retrieval service and hand the same object to both.

Caches are where tenant data actually crosses

Every cache in the request path is a store of customer data with no access control except the key. That is the whole model. The key is the ACL, and most cache-key functions were written in ten minutes by someone optimizing latency.

The canonical bug is a response cache keyed on a hash of the normalized prompt. Two customers ask the same common question, the second gets the first one's answer, and that answer carries the first customer's data because retrieval put it there. Nothing logs an error. The fix is trivial and the discipline is not: one key-derivation function, in one file, that every cache calls, taking tenant, model id, model version, prompt, retrieval configuration and tool versions. Unit test it with an assertion that two tenants sending byte-identical prompts get different keys. Four lines, and it catches the class on the day someone adds a seventh cache.

Semantic caches are the same bug with deniability. They do nearest-neighbor lookup over the prompt embedding, so the key is not exact by construction, and a filter applied after the lookup fails exactly the way post-filtered retrieval does. Partition them per tenant rather than filtering them. If that drops the hit rate below the point of running one, that is the real answer about whether you should have a semantic cache.

Cache layer	Key must include	What happens when it does not
Embedding cache	Text hash, embedding model id, model version, tenant	A retired model's vectors survive a migration, and the cache becomes a store of tenant text nobody has an owner for
Retrieval result cache	Tenant, index version, resolved filter set, query	Tenant B receives tenant A's chunk ids, then fetches the text behind them
Response cache, exact match	Tenant, model, model version, full prompt, tool versions	A common support question returns another customer's answer verbatim
Semantic cache, approximate	A per-tenant namespace, not a post-lookup filter	A near match crosses tenants and returns an answer to a question that was never asked here
Tool result cache	Tenant, tool version, resolved scope, arguments	Rows from another account's query, returned as if the tool had just run
Engine prefix cache	A per-tenant salt mixed into the block hash	No content crosses, but hit latency reveals whether a given string was recently processed

The KV cache: what it leaks and what it does not

Modern serving engines page the attention key-value cache into fixed blocks, sixteen tokens per block in a default vLLM configuration, and reuse blocks across requests whose token prefixes hash identically. This gets described as harmless and as a catastrophe, and it is neither. A hit requires an exact token match, so another tenant's content never appears in your generation. What crosses is timing: time to first token drops on a hit, so a caller who can send arbitrary prefixes can test whether a specific string was recently processed. That is membership disclosure rather than content disclosure, and it matters when prompts embed customer names, account numbers or document titles. Recent vLLM builds support a per-request cache salt so sharing can be scoped to a tenant. Set it, disable cross-request sharing, or write down what an attacker learns and accept it deliberately.

The bigger multi-tenant issue with the KV cache is capacity, not confidentiality. Work the arithmetic for an 8B-class model with grouped-query attention: 32 layers, 8 key-value heads, 128 dimensions per head, two tensors for K and V, two bytes per element in half precision. That is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KB of cache per token. A single 100,000-token request holds about 12.8 GB resident, so on an 80 GB card already carrying 16 GB of weights, two of them take most of the headroom every other tenant's throughput depends on. Maximum context length is a per-tenant quota with a dollar cost behind it, not a feature flag you hand out on request.

Adapters, fine-tunes, and the routing table

Serving many LoRA adapters against one base model is the standard way to give customers a tuned model without giving each one a GPU. It also turns a routing table into a security control, and routing tables do not get reviewed like security controls.

The rule is the same as everywhere else. Resolve the adapter identifier from the tenant context, never from the request. An API that takes model: "acme-support-v3" as a caller-supplied string is one enumeration away from serving one customer's fine-tune to another. Keep a tenant-to-adapter mapping, resolve on every request, and log the resolved id rather than the requested one.

Treat the adapter as customer data, because it is. It was trained on their corpus and belongs in the same retention schedule, backup policy and deletion path as the source documents. When a customer leaves, the adapter, its checkpoints and the training snapshot go with the index. Decide that horizon before anyone asks, and have the training pipeline enforce a per-tenant flag rather than a process people follow until a deadline.

Logs, traces, and the support tool nobody threat-modeled

In our experience this is where actual exposure happens, and it rarely involves the inference path at all. Prompts land in exception messages. Full request bodies land in an APM tool with a broad access group. Tracing SDKs capture message content behind a configuration flag whose default has moved between versions, so an upgrade quietly starts shipping prompts to a shared observability project. Prompt and response pairs get pulled into an evaluation dataset that lives in a repository with different access rules than the production database.

Four controls handle most of it. Content capture is off by default and enabled per environment, never globally. Redaction runs at the exporter rather than at each log call, because there is one exporter and hundreds of call sites. Traces carry the tenant as an indexed attribute with access enforced on it, or go to separate projects for customers who paid for that. And the support tool that replays a session requires a stated reason, writes an audit record naming the operator and the tenant, and shows the customer that record on request.

Agents and tools: never let the model choose the tenant

The moment a system does tool calling, the model composes arguments, and any argument it composes is attacker-influenced, because a retrieved document containing instructions is an input channel. So the tool schema should not carry a tenant, account or customer identifier at all. Remove the parameter and bind the scope inside the executor from the request context, where the model cannot reach it.

The same logic covers everything a tool touches. File paths resolve inside a tenant-rooted prefix rather than being concatenated. Database tools run parameterized queries against a connection whose role already carries the row-level policy, not raw SQL the model wrote. Outbound HTTP goes through a per-tenant allowlist, so a tool calling one customer's webhook cannot be redirected to another's. Log every invocation with the resolved scope, so an audit answers what was reachable rather than what was requested.

Four tiers, and what each one lets you promise

Isolation is a product decision with a cost curve, not a binary. Four tiers cover almost every platform we have worked on, and the useful discipline is naming which one you are at before a security questionnaire names it for you.

Tier	What is shared	Blast radius of one bug	Relative cost	What you can put in writing
1. Pooled with tenant tags	Index, all caches, GPUs, queues, traces	Every customer at once	1.0×	Little beyond "logical separation," and an auditor will press on it
2. Pooled compute, partitioned data plane	GPUs, queues, base model weights, control plane	The one path that failed, contained by the others	1.05–1.2×	Separate stores per tenant, scoped caches, deletion inside a stated window
3. Dedicated inference pool	Control plane, images, deployment pipeline	One tenant's pool	1.5–3× for those tenants	Dedicated compute, no shared cache or index, a capacity floor and a latency target
4. Dedicated deployment	The source code, and nothing else	One tenant	3–6×	Region residency, customer-managed keys, an independent upgrade cadence

The multipliers are planning ranges we reason to from GPU idle time, not measured benchmarks, and the dominant term is simple. A dedicated pool cannot amortize bursts across tenants, so utilization is set by one customer's daily traffic shape instead of the aggregate. A shared fleet at 55 percent average utilization against a dedicated pool at 20 produces most of the multiplier before any other line item.

Most platforms should run tier 2 for the long tail and offer tier 3 or 4 to the customers who will pay for it, on the same code path with a different deployment target. Two mistakes bracket that. Building tier 4 for everyone multiplies idle GPU hours by the customer count and makes the unit economics impossible. Answering a questionnaire as though you were at tier 3 while running tier 1 is worse, because it is discoverable and it is in writing.

Performance isolation is a token problem, not a request problem

The confidentiality half gets the attention and the availability half causes the outages. Rate limiting by requests per minute is close to meaningless here, because two requests can differ in work by three orders of magnitude. A 200-token classification and a 100,000-token summarization are both one request.

Limit the thing that costs money. Input tokens per minute, output tokens per minute, concurrent in-flight requests, and a maximum context length per plan. Output tokens matter separately because they occupy a decode slot for their entire generation, so one caller streaming 4,000-token answers holds capacity far longer than the input accounting suggests.

Continuous batching makes this a cross-tenant problem even when nothing leaks. A large prefill occupies the engine and delays decode for every sequence in the batch, so one customer pasting a long document raises time to first token for everyone on that replica. Chunked prefill splits a large prompt across several scheduler steps so decode keeps progressing, and it is worth the throughput it costs. Alongside it, run admission control with a per-tenant concurrency cap and separate queues by class, so an interactive request never sits behind a batch job.

Instrument queue time separately from service time. Total latency going up tells you something is wrong. Queue time going up while service time holds steady tells you it is a neighbor, and that distinction is the difference between a ten-minute diagnosis and an afternoon.

GPU-level isolation: what MIG, MPS and time-slicing actually buy

Three mechanisms get discussed as though they were interchangeable, and they isolate very different things.

Time-slicing interleaves work from several processes on one GPU. No memory partition, no fault isolation. One process allocating too much causes an out-of-memory failure in another, and a wedged kernel affects everyone on the device. It is an oversubscription tool, not a boundary. Multi-Process Service runs kernels from several processes concurrently rather than interleaved, which lifts utilization for small models, but memory is still one shared pool and the same failure modes remain.

Multi-Instance GPU partitions an A100 or H100 into as many as seven instances with hardware-separated memory, cache and streaming multiprocessors, and that is genuine isolation including fault isolation. The catch is fixed profile sizes, which strand capacity when a workload does not divide evenly, and small instances that cannot hold a mid-sized model's weights. For embedding services, rerankers and classifiers, MIG fits well. For serving a 70B model the boundary ends up being the process, the node or the cluster.

Proving it: the cross-tenant suite that runs on every deploy

Everything above is a claim until something tries to break it on a schedule. Seed two tenants in staging with canary strings that appear nowhere else, then write a suite that plays tenant B and tries every path to tenant A's canary.

• Direct retrieval, including a query engineered to rank A's canary chunk first
• A repeat of a prompt A already sent, to exercise every cache layer including the semantic one
• A request carrying A's tenant id in the body, header and query string while authenticated as B
• A tool call whose arguments name A's account, file path and webhook host
• An adapter or model identifier belonging to A, supplied directly in the request
• Trace, log, export and usage-report endpoints queried as B for A's records
• A prompt-injected document in B's corpus that instructs the agent to fetch A's data
• Assert the canary appears in zero responses and zero log lines, then assert every attempt was denied rather than merely empty

That last clause matters more than it looks. An empty result and a denial are different outcomes, and a suite that only checks for absence passes happily against a service that is silently broken and returning nothing to anyone. Fail the pipeline on any hit. Add the unit test asserting two tenants with identical prompts derive different cache keys, then a deletion test that removes a tenant and re-runs everything looking for surviving canaries in the index, the caches, the traces and the adapter store.

How this goes wrong

• A cache key computed from the prompt text alone, added during a latency push and never reviewed as a data-access change.
• The tenant filter applied in application code after the vector search returns, rather than by the engine before ranking.
• The application connecting to PostgreSQL as the table owner, which bypasses row-level security silently.
• A tool schema with a customer_id parameter, filled by the model from text it just retrieved.
• An adapter id accepted from the request body and passed to the serving engine without a permission check.
• Tracing content capture enabled globally after an SDK upgrade changed the default, shipping prompts to a shared project.
• Rate limits expressed in requests per minute, so one caller with 100,000-token prompts stays inside every quota while consuming the fleet.
• A residency commitment signed for one customer against a single pooled index that lives in one region.
• Tenant deletion that clears the primary store and leaves the vector index, the caches, the adapters and eighteen months of traces intact.

A two-week hardening pass

Two weeks is enough because none of this is research. The first step is the one people skip and the one that finds the surprises: draw the actual request path with every process it touches, and count the stores.

Bottom line

Tenant isolation in an inference stack is not one control. It is a property six subsystems have to hold at once, and five of them do not look like access control while you are writing them. Resolve the tenant once from the credential, make every client impossible to construct without it, push filtering into the engine, treat every cache key as an access decision, keep prompts out of shared observability, and never let a model compose an authorization argument. Then pick the tier you can actually operate, say so in writing, and run the canary suite on every deploy so the claim stays true after the next twelve refactors.

Frequently asked questions