Telemetry that is safe by construction

The incident is almost never an attacker

The telemetry privacy problems we get called into look remarkably alike. Nobody broke in. An engineer added a field to a request object in March, a log statement three layers down serialized the whole object, and by June that field sat in a third-party log index with ninety-day retention, a search bar, and forty people who could query it. The data was not stolen. It was published by the system, on purpose, one struct at a time, and it kept being published until somebody noticed.

The standard response is a scrubber: something at ingest that reads every record and strips whatever looks sensitive. Every observability vendor sells a version, and most teams that have had this problem already own one, which should tell you something.

The alternative is a pipeline where a sensitive value has no route to the sink, because the code that would carry it there does not compile, the boundary that would forward it drops unrecognized keys, and the test that would catch it runs on every merge. Safe by construction means the property holds because of the structure of the system, not because everyone remembered.

A scrubber is a denylist, and denylists fail open

Pattern matching enumerates the bad. Whatever it does not recognize passes through untouched, and it passes silently, which is the part that matters. Whole categories have no pattern at all. There is no regular expression for a person's name, none for a street address that does not also match half your log lines, none for the free text of a support message. The categories with clean patterns, card numbers with a Luhn check and structured national identifiers, are the minority.

Then there are the shapes that defeat a matcher even when the pattern exists. A base64 blob. A JSON document nested inside a string field, so the quotes are escaped and the matcher sees different bytes than the ones it was written for. A gzipped payload. A protobuf dump rendered as hex. A UUID that means nothing in your system and is a customer's account key in the system you federate with. A scrubber reads all of these as noise.

The operational failure is worse than the technical one. A scrubber is written against the schema that existed the day it was written, and schemas change weekly. When a new field arrives nothing goes red: no test fails, no alert fires, no exit code changes. A control that stays green while it has stopped working is more dangerous than no control, because its greenness ends the conversation, and nobody re-opens a question a dashboard says is settled.

It costs money in a way that quietly makes it worse. A dozen regular expressions over every log line at ingest is real CPU on the hot path, billed by the vendor or paid for in your own collector fleet. The usual response, once someone reads the bill, is to sample.

What "by construction" actually means here

Three properties, and a pipeline that has all three does not need a scrubber for the categories it covers.

The value has no printable form. A secret or a direct identifier lives inside a type whose rendering produces a placeholder. Serializing it, formatting it, or dumping the enclosing struct all yield [redacted]. Reading the real value requires calling a method whose name shows up in a code review.

The boundary accepts an allowlist. The collector forwards attribute keys it recognizes, drops everything else, and counts what it dropped by key name. Adding a new field to a span becomes a visible event.

A test fails the build. A fixture carrying canary values goes through the real emitter and the real collector configuration into a test sink, and the assertion is that none of them arrive. This is the only part of the system that produces evidence rather than intent.

Everything else, every scrubber included, is a promise that somebody thought of all the fields. Here is how we rank the control points, scored on whether each fails open or closed.

A data-class table your pipeline can enforce

Classification only helps if each class maps to a mechanism. A policy that says "handle personal data appropriately" produces nothing. A table where every row names the thing in the code that enforces it produces a backlog.

Class	Examples	Rule in telemetry	Enforced by
Secrets	Bearer tokens, API keys, session cookies, signed URL parameters	Never present in any form, including hashed	Non-printing type, header allowlist, canary test in CI
Direct identifiers	Email, phone, full name, account number, national ID	Never raw; an HMAC token only where correlation is genuinely needed	Non-printing type plus collector allowlist
Payment data	Card number, verification code, track data	Never captured; a log store is storage, and the card rules treat it that way	Client SDK field masking, emitter policy, canary test
Content	Message bodies, uploaded document text, support tickets, prompts	Shape only: length, encoding, language, which validation rule failed	Emitter policy, no free-text passthrough anywhere
Quasi-identifiers	Precise timestamp, city, device model, build, IP address	Coarsen before storage and treat as identifying in combination	Collector transform: truncate, bucket, round
Operational	Request id, tenant id, route template, status, duration, retry count	Keep all of it; this is what debugging actually uses	Allowlisted by name in the registry

Make the value unprintable

Every mainstream language has a mechanism for this and most codebases use none of them. The idea is always the same: intercept the path the logging library uses to turn an object into text.

Go. Implement slog.LogValuer. LogValue() returns a placeholder and the structured logger calls it instead of reflecting over the struct. Delete the convenience String() method somebody added, because %+v on a struct without one prints every exported field and with one prints whatever it returns.

Rust. Do not derive Debug on a type holding a secret; write it by hand so it prints the type name and nothing else. The secrecy crate packages the pattern in a Secret<T> wrapper with no Debug or Display output.

Python. Pydantic's SecretStr renders as asterisks in both repr and str, and the value needs .get_secret_value(). Pair it with a structlog processor that raises on any unregistered key.

Java and Kotlin. The generated toString() on a record or a data class prints every component, and that string is what ends up in the log. Override it. A data class with a password field and a default toString() is one logger.debug away from an incident.

What this buys is not that leaking becomes impossible. It is that leaking becomes visible. A reviewer scanning a diff will not notice log.Info("checkout", "user", u). They will notice u.Email.GetSecretValue() inside a log call, because that expression has no other reason to exist.

Allowlist at the boundary, and count what you drop

The second control sits at the collector and is worth building even where the type work is half done. The OpenTelemetry Collector ships a redaction processor with an allowlist mode: reject all keys except the ones you name, and every attribute the code has not declared is removed before the record leaves your network. The processor also writes a summary recording how many keys it masked and what they were called.

That counter is the point. Default-deny turns "an engineer added a field" from an invisible event into a number that moves. A dashboard of dropped key names tells you on the day of the merge that user.email started appearing on the checkout span, and it tells you the key name without storing the value. Every other approach discovers the same fact from a customer.

Alongside it, the transform processor and its OTTL expressions do the coarsening: delete keys matching a pattern, truncate an IP address to its network prefix, round a timestamp to the minute, hash a value. The filter processor drops whole records matching a condition, which is how a noisy health-check endpoint stays out of the store entirely.

Keep the collector configuration in the application's repository, reviewed by the same people. A redaction rule configured through a vendor console has no diff, no history, no reviewer and no rollback. If the allowlist is worth having it belongs under version control, and if it is generated from a declared attribute registry, OpenTelemetry's Weaver tooling checks the code against that registry in CI.

Identifiers you can still debug with

The objection arrives immediately and it is fair: you have made production unreadable, and an incident now takes four hours instead of forty minutes. The answer is that debugging almost never needs the value. It needs correlation and shape.

Correlation comes from a request id, a trace id, a tenant id and a stable pseudonymous subject token. Shape comes from field length, type, encoding, error class, and which validation rule rejected the input. "Field email, 34 characters, failed the domain-part rule" reproduces the bug and does not tell you who the customer is. The rare case that needs the raw value belongs in a short-retention tier inside your own account with access logging on it.

The pseudonym has a trap worth stating plainly, because it shows up in most systems we review. A plain SHA-256 of an email address or a phone number is not de-identification. Ten-digit phone numbers are ten billion candidates, minutes of commodity GPU time. Email is worse, because the realistic space is names against a short list of common domains. Anyone with the hashed column and a wordlist recovers most of it.

Use an HMAC with a key held in a key management service the telemetry store cannot reach, and rotate it on a schedule. Correlation windows then end at rotation, which is a feature: it bounds how long any pseudonym stays linkable. Keep the legal framing straight too. Under the GDPR, pseudonymized data that can be re-linked is still personal data, so hashing changes your risk and not your obligations. For health data, the Safe Harbor method at 45 CFR 164.514(b)(2) lists eighteen identifiers to remove, and IP addresses, device identifiers and dates more precise than a year are all on it.

Quasi-identifiers deserve the same care. The widely cited result is that ZIP code, date of birth and sex uniquely identify most of the United States population, with replications landing between roughly sixty and ninety percent. A record carrying a millisecond timestamp, a city, a device model and a build number is the same kind of tuple, and it identifies a person without holding anything anyone would call personal data.

Where the values actually get in

The emitter. Somebody logged a whole object because that is one keystroke cheaper than picking fields, and the object held more than they were thinking about. Largest source by volume, and the easiest to close.

Client SDKs. Session replay, error reporting and analytics run in the browser, capture what the user typed, and send it over a path your backend never sees. Masking behavior differs by SDK and changes across major versions.

The rest arrives as defaults in automatic instrumentation, which nobody chose and nobody reviewed. Three account for most of it.

Database statements. If queries are parameterized, the statement on the span is the template and the values stay out. If any code path builds SQL by concatenation, the span carries the literals, which means it carries whatever the user typed into the search box. Parameterized queries are already the right answer for injection. This is the second reason.

Full URLs. Span attributes commonly record the URL with its query string, which is where password reset tokens, magic links, one-time codes and pre-signed storage URLs live. A pre-signed URL in a log index is a credential in a log index, valid until it expires and searchable by anyone with read access. Record the route template and the parameter names, not the values.

Headers. Capture is opt-in, a team opts in because it wants user-agent, and the setting takes a list. Somebody writes a wildcard, and now authorization and cookie ride every span. Enumerate headers explicitly and treat a wildcard there as a blocking review comment. Error reporting SDKs are the same story: they default to leaving identifying data off and expose a hook that runs before an event is sent. Read what your version does, not what the docs page says.

The only proof is a canary test

Design intent is not evidence. The test that produces evidence is small, and we have never found it in a codebase that had not already had an incident.

Build a fixture whose values exist nowhere else. An address like canary-7f3a@example.invalid, using the reserved TLD from RFC 2606 so it can never route. A card number from your payment processor's published test set. A random thirty-two character string committed once. Push it through the real emitter and collector configuration into a test sink, then assert no canary appears in the output in any encoding: raw, base64, URL-escaped, lowercased.

Run it on every merge, and again nightly against the staging collector configuration, because the configuration drifts on a different schedule than the code and the pair only breaks when they drift apart. Add a second assertion on the dropped-key counter: the dropped key names must be a subset of a reviewed exception list, so a new field forces a human decision instead of a silent drop.

Then run one control in production: a scheduled job that queries the production sink for the canary strings. They exist only in a test fixture, so a hit means something is copying data between environments, which is a finding worth having on its own.

Retention, and the deletion you will eventually be asked for

One question decides more of this architecture than any other. When a customer asks you to delete their data, can you do it in your telemetry store? Most teams find out during the request.

Telemetry stores are built to append and query, not to erase. Prometheus is the clearest example: label values land in the time series database and stay. Deleting a series needs the admin API, which is off unless the server was started with the flag that enables it, and the delete call only writes tombstones. Your backups are a separate problem, and your vendor's backups are one you do not control.

Do not put erasable data in a store that cannot erase. That one decision removes most of the difficulty, and it is a design decision rather than an operations one, which means it has to be made before the system ships.

Keep the raw tier short. Thirty days answers almost every debugging question anyone actually asks, and long retention belongs to aggregates with no subject-level rows. Two tiers is the shape that works: a narrow, short-lived, high-detail tier inside your own account, and a wide, long-lived, allowlisted tier that dashboards and vendors read.

Partition by tenant, so deletion is a partition drop rather than a scan across a store with no index on the thing you need to find. And use crypto-shredding where subject-level records must persist: encrypt each subject's records under a per-subject key held outside the store, then delete the key. It is the one erasure mechanism that reaches backups you cannot rewrite, and it has a limit worth stating, since it protects against future reads and not against a copy someone made while the key was live.

What we find on the first pass

• A String() or toString() on a user type that prints every field, called by a log statement nobody remembers writing.
• Header capture configured with a wildcard, so authorization and cookie arrive with the one header the team wanted.
• Password reset and magic-link URLs recorded whole, token included, in an index that forty people can search.
• Span names built from the raw request path, turning an email address into a permanent, unbounded-cardinality series.
• A metric label carrying an account identifier, which is an exposure and a cost line at the same time.
• Exceptions serialized with the request payload attached "for context" by a middleware written two years ago.
• Redaction rules configured in a vendor console, so no change was reviewed and nobody can diff last quarter.
• Session replay switched on with masking left at whatever the SDK default was two major versions back.

A four-week retrofit

Nothing here is research. The order matters more than the duration: inventory first, because teams are consistently wrong about how many sinks they have, and dry-run before enforcement, because the first honest allowlist always breaks a dashboard somebody depends on.

Own these regardless of which vendor you use

• The attribute registry, in your repository, reviewed like code
• Non-printing types for every secret and every direct identifier
• The collector configuration in version control, never in a vendor console
• The dropped-key counter, on a dashboard somebody reads weekly
• The canary fixture and the CI test that asserts it never lands
• HMAC keys for pseudonyms, held outside the telemetry store and rotated on a schedule
• A written retention number per tier, in a store that can honor it
• A deletion path exercised once before anyone asks you to use it
• An access log for the raw tier, because access is telemetry too

The argument that gets this funded

Privacy work pitched as risk reduction competes badly against feature work, because risk reduction has no due date and features do. This work has a second argument, and it lands with a finance team.

The fields that make telemetry dangerous are usually the fields that make it expensive. Observability vendors price on volume and cardinality. A span name built from a user identifier creates an unbounded number of series. Log lines carrying whole request bodies are, in most systems we have looked at, the largest single contributor to volume. Cutting them reduces the bill and the exposure in one commit, and the bill is the half that gets a quarter allocated to it.

A third argument applies to anyone selling to enterprise buyers. Security questionnaires ask what personal data your telemetry holds and how long you keep it. "We scrub it at ingest" invites a follow-up about fields the scrubber has not seen, and there is no good answer to that one. "The collector forwards a declared allowlist, unknown attributes are dropped and counted, and a CI test proves it" ends the thread. A SOC 2 or ISO 27001 assessor can read the allowlist, the test and the passing run. Intent is not auditable. A failing build is.

Bottom line

Stop trying to recognize sensitive data on its way out. Make it structurally unable to leave. Put secrets and identifiers in types with no printable form, allowlist the attributes at the boundary and count what gets dropped, coarsen the quasi-identifiers, key pseudonyms with an HMAC the store cannot reach, and prove all of it with a canary test that fails the build. Then set a retention number you can honor, and exercise the deletion path once on a quiet afternoon, before somebody asks for it in writing.

Frequently asked questions