The restriction is a data-flow problem, not a model problem
If you run data at a ratings shop, an analytics vendor or a market-data business, some meaningful share of what you hold arrived under terms that limit what you may do with it. A licence from an exchange or a data vendor. A client agreement covering material a customer sent you to process. An arrangement with a contributor who supplies the raw records on the understanding that they will be used for one purpose and not another. Somewhere in each of those documents is language about derived works, redistribution, or permitted use, and increasingly there is language about model training specifically.
Teams tend to hear that as a modelling constraint: do not fine-tune on this. That framing is comfortable because it points at a small number of deliberate acts by a small number of people. It is also why the restriction gets broken, because the actual risk is not deliberate. It is that restricted records get copied into thirty places over three years by ordinary engineering, and later something reads one of those places.
The useful reframing is that this is a lineage and boundary problem. You are not trying to stop your data scientists from doing something reckless. You are trying to be able to answer, on any Tuesday, the question where are the copies, and what reads them — and to have an answer that is enforced by the system rather than remembered by a person.
You are probably here because
- A customer asked you to warrant in writing that their data is never used for training
- A data vendor's renewal added language about machine learning that was not there before
- Your product team wants to build a feature on a corpus somebody licensed years ago
- Someone asked which of your datasets are in your vector index and nobody could answer
The section on what the word training has to cover is where most of the surprises are. The table of where copies land is the one to take into a meeting.
What the word “training” has to cover
If your control is a rule that says “we do not train on restricted data,” the first job is to decide what training means, because the ordinary technical meaning is narrower than the one a customer or a licensor has in mind. In practice a promise of this kind has to reach at least seven things.
Weight updates of any kind. Pre-training, continued pre-training, supervised fine-tuning, preference tuning, adapter training. This is the part everyone already has covered.
Retrieval indexes. An embedding index that returns a customer's text verbatim into a prompt is, from the customer's point of view, indistinguishable from a model that memorised it. Whether it is “training” in the technical sense is beside the point; if their content can be surfaced to another tenant, you have the problem the clause was written to prevent.
Few-shot examples and prompt libraries. The single most common real leak we encounter. Someone builds a prompt that works beautifully because it contains three real examples pasted from a live record, that prompt is committed to the repository, and now restricted content is in every request from every tenant.
Evaluation and test sets. Also built from real records, also committed, also shared with contractors, and rarely governed. Evaluation data has the added property that it tends to outlive the project that created it.
Caches. Prompt caches, response caches, embedding caches, feature-store materialisations. Each is a copy with its own retention behaviour, usually configured for performance by someone not thinking about data class.
Telemetry and debugging surfaces. Application logs, error-tracker payloads, request traces, screenshots in support tickets, a notebook on a laptop, a CSV in a shared drive from the week of the incident. This is where the largest number of copies live and where almost nobody looks.
Third parties who process on your behalf. Your model provider, your logging vendor, your error tracker, your annotation partner. Their default terms are their defaults, not yours.
Where the copies actually land
Draw this for your own system before you write a policy. The exercise takes a day and it is the highest-value day in the whole programme, because a policy written without it will describe a system you do not have.
| Where a copy lands | Why it happens | Control that actually works |
|---|---|---|
| Application and access logs | Someone logged a request body during an incident and never removed it | Structured logging with an allowlist of fields; no free-form payload logging in production paths |
| Error tracker | Exception context captures local variables holding records | Scrubbing configured at the client, verified by deliberately raising a test exception |
| Prompt templates and few-shot blocks | Real examples pasted in because they worked | Synthetic exemplars only, enforced by review and a repository scan |
| Evaluation sets | Built from production for realism, then copied and shared | Held in the governed store, referenced by identifier, never checked into a repository |
| Vector and search indexes | Everything was indexed because indexing everything was easier | Per-class indexes with tenant filters applied at query time and tested |
| Analytics warehouse | The pipeline copies everything; classification stopped at the source system | Class tags travel with the columns; restricted classes land in a separate schema |
| Backups and snapshots | They exist to be complete | Encrypt per class, plan deletion as key destruction, state the restore window honestly |
| Laptops and notebooks | A real extract was needed to debug a real problem | Query the governed environment; never move data to the person |
| Third-party processors | Defaults were accepted at signup | Named in a subprocessor register, with retention settings recorded and re-checked |
Classify at ingest, and use classes rather than datasets
The single structural decision that determines how hard everything else will be is where classification happens. If a record's class is attached at the moment it enters your systems and travels with it, every downstream control has something to act on. If classification is a spreadsheet maintained separately from the data, it will be wrong within a quarter, and you will discover it is wrong during a review.
Use a small number of classes — three or four — defined by what you are permitted to do, not by which vendor supplied the data. Something like: freely usable, usable for internal analysis but not for model development, usable for a named purpose only, and never leaves the boundary. The number of classes should be small enough that an engineer can hold them in their head, because engineers making hundreds of small decisions are the actual enforcement layer.
Then map every dataset to a class once, deliberately, with a person's name and a date against it, and treat that mapping as a reviewed artefact. When a new licence arrives, classifying it is part of accepting it. This is unglamorous, it is mostly reading, and it is the work.
Do embeddings and aggregates inherit the restriction?
This comes up constantly and there is no universal answer, so here is the honest version.
Technically, an embedding is a lossy transformation, and how much of the original can be recovered from it depends on the model, the length of the text and what the attacker knows. There is a substantial research literature showing that inversion is more effective than intuition suggests, particularly for short texts. So the engineering default should be that an embedding of restricted text is restricted text, stored in the same class, with the same access controls.
Contractually, whether an embedding or an aggregate is a “derived work” is a question about the words in your specific agreement, and those words differ. Some licences define derived works broadly enough to capture any statistic computed from the data; some carve out aggregates above a threshold; some are silent, which is the worst case because it will be argued later. Do not take a general answer from an article, including this one. Pull the agreement, find the definition of derived works and the permitted-use clause, and get counsel to read them against what you actually intend to build. Then write down the conclusion beside the dataset's class, so the next person does not have to re-litigate it.
One practical note. Where a licence permits statistics but not redistribution of records, a great deal of product value is still available: aggregate features, distributions, model coefficients, calibration curves. The mistake is deciding that because records are restricted, nothing derived from them can be used. Frequently the opposite is true, and the constraint is really about what leaves the boundary, not about what may be computed inside it.
Which controls survive being tested — our read
Our judgment from building and reviewing these systems, not a survey. The bottom row is the one most programmes start with.
Three kinds of control, and only one of them is evidence
Controls in this area come in three grades, and it is worth being clear-eyed about which you have.
Contractual. Your agreement with a processor says they will not use your content to train. This is necessary and it is what your own customer contract will mirror. It is not evidence of anything; it is a promise you can sue over.
Configured. The retention setting is off, the training flag is off, the scrubbing rule is in the client library. Better, because it changes behaviour. Its weakness is that configuration drifts, defaults change at renewal, and a new service gets added by a team that did not know the rule. Record every such setting in one register with the date it was last verified, and re-verify on a schedule.
Architectural. The data physically cannot reach the place you are worried about, because it is in a different account, behind a different key, with no network path and no credential. This is the only grade that is genuinely durable, because it does not depend on anyone remembering anything. It is also the most expensive, which is why it should be spent on the highest class and not on everything.
Add one thing that is not a control but is worth more than most of them: a detection. Seed a small number of distinctive synthetic canary records into each restricted class. Then search for them, on a schedule, everywhere output could appear — model responses, logs, the warehouse, exports, the search index. A canary that surfaces tells you a copy went somewhere you did not model. Nothing else on this list can do that.
What deletion honestly means
Deletion promises are made casually and are the hardest ones to keep, because a record that has flowed through a pipeline for two years does not have one location.
A defensible deletion story usually says something like this. Records are removed from the primary store immediately on request. Downstream copies in the warehouse and derived tables are removed on the next scheduled run, which is a stated interval. Search and vector indexes are updated on a stated interval, and the identifier is added to a suppression list so a stale index cannot resurface it. Backups are not selectively edited; they age out on a stated retention period, and until then they are encrypted with keys that are destroyed on that schedule. Logs carry a shorter retention than everything else, deliberately, and that number is stated.
The important word is stated. Customers who work with data are not surprised that backups exist; they are surprised when a vendor implies deletion is instantaneous and then cannot explain the restore path. A precise, slightly unflattering answer earns more trust than a clean one that does not survive a follow-up question.
The mistakes we see
- A policy written before the data-flow map, describing a system that does not exist
- Classification held in a spreadsheet rather than travelling with the data
- Real records in prompt templates and eval sets, committed to the repository
- One vector index for everything, with tenant separation applied only in application code
- Vendor defaults accepted at signup and never re-checked after a renewal
- Debug extracts on laptops, which is the copy nobody counts and everybody makes
- Deletion promised without a backup story, which collapses at the first serious question
- Treating all derived data as restricted, which quietly kills products the licence would have allowed
Where you do not need us
If you have one product, one database and one model provider, this is a policy, one conversation with counsel about your licence terms, two provider settings, and a scrubbing rule in your logging configuration. A competent engineer on your team can do all of it in a fortnight, and you will not get a better result by paying someone else to do it, because they would have to learn your system first.
The reading of your own licences is also not outside work. It is slow, it is specific to documents only you hold, and the person who does it acquires knowledge that should stay in your company.
Where an outside team is worth it is narrower. Building the data-flow map when nobody internally has the whole picture and the people who do are shipping. Separating a warehouse that grew without classification, which is a migration and a large one. Building the canary detection and the suppression-list plumbing. Or standing in front of a customer's technical reviewer and answering questions about your architecture, which is a different skill from having built it.
Before you make the promise
- A data-flow map exists and names every store, index, cache and processor
- Classes are defined by permitted use, and there are three or four of them
- Every dataset is mapped to a class, with a name and a date on the decision
- The class tag travels with the data, not in a separate spreadsheet
- Prompt templates and eval sets contain no real restricted records
- Logging uses a field allowlist, and scrubbing has been tested with a real exception
- Every processor is in a register, with its retention and training settings and a verified date
- Embeddings of restricted text are stored and controlled as restricted text
- Canary records exist and something searches for them on a schedule
- The deletion story includes backups, indexes and logs, with stated intervals
Bottom line
A promise not to train on someone's data is a promise about where copies go, and it is kept or broken in your pipelines rather than in your model code. Classify at ingest, keep the number of classes small enough to remember, and let the tag travel with the data. Assume that retrieval indexes, prompt templates, evaluation sets, caches and logs are all in scope, because the person you made the promise to certainly assumes it. Prefer architectural separation for the highest class and configured controls for the rest, record every configured control with a verified date, and put canaries in so that a copy you did not model announces itself. Then say what deletion actually means, including backups, with real intervals. The version of this that survives a serious review is not the one with the strongest language. It is the one where every claim maps to something a reviewer can be shown.
Frequently asked questions
Not technically, and usually yes in the sense that matters. The risk your customer is guarding against is their content reaching someone else, and a retrieval index that can surface their text into another tenant's prompt does exactly that. Treat indexes as in scope, separate them by class, apply tenant filters at query time rather than in application code, and test the filter with a deliberate cross-tenant query.
That depends on the definition of derived works in your particular agreement, and those definitions differ enough that a general answer is worthless. Pull the document, read the derived-works definition alongside the permitted-use clause, and have counsel opine against what you intend to build. Engineering should default to treating an embedding of restricted text as restricted, since inversion research suggests more is recoverable than intuition assumes.
It is one control of three, and the weakest grade. Check whether it is a default or something you must enable, whether it covers the abuse-monitoring path as well as the training path, what retention applies, and whether it survives your next renewal. Record the setting and the date you verified it, then re-verify on a schedule, because the failure mode is not a broken promise but a changed default nobody noticed.
Two passes. Structural: enumerate stores, indexes, buckets, warehouses and processors from your infrastructure definitions rather than from memory, and check each against the map. Empirical: take distinctive strings from real restricted records and search everywhere output can appear. Then add canaries so the second pass runs continuously rather than annually. Expect the first sweep to find copies in logs and in someone's analysis folder.
Frequently yes, and it is worth resolving deliberately because evaluation on synthetic data is much less useful. The pattern that tends to work is that evaluation runs inside the boundary, on records that never leave it, with only scores and aggregate statistics coming out. What breaks it is the eval set being copied into a repository or handed to a contractor, which is the most common way restricted records escape.