Supporting a customer whose data you cannot see

The ticket that arrives with no evidence

A customer files a ticket at four in the afternoon. The whole report is one sentence: "The extraction is wrong on some invoices." Your software runs in their cloud account, under their identity provider, writing to their log store. The agreement you signed says your engineers do not read customer content. Your own logger enforces it, which is the point. So you cannot open an invoice, you cannot read a log line that quoted one, and you cannot ask them to attach a file, because not having to send you files is the reason they bought from you. The bug is real, it is yours, and every ordinary debugging move is gone.

This is the normal shape of enterprise software support now. Anything sold into healthcare, finance, insurance, legal or manufacturing lands in a deployment with no vendor read path to the data. Teams discover it three weeks after the first install and then run support by interrogation: ask a question, wait a day, ask another. Six sequential questions is a two-week ticket.

Blind support is not degraded normal support. It is a different discipline with its own instrumentation, artifacts and economics, and nearly all of it has to exist before the first install. What follows is the method we build into systems shipping into a boundary we do not control.

The four ways you end up blind

They stack, and each closes a different door. Naming which ones apply is the first thing to write down, because the remedies differ.

Contractual. The data processing agreement says vendor personnel do not access customer content except under a named, logged exception. It is the most common form, it applies even when the software runs in your own cloud, and it is the cheapest to work with because an exception path is already written into it.

Topological. The software runs in their account, their Kubernetes cluster, or a machine on a factory floor with no inbound route. Nothing you write reaches you unless a human moves it. Even your metrics are on the wrong side of the wall.

Categorical. Some fields are regulated in a way that makes looking at them an event rather than a decision. Protected health information under a HIPAA business associate agreement, cardholder data under PCI DSS, personal data where you are a processor under GDPR Article 28. Seeing it converts a support ticket into something a compliance officer writes up.

Practical. Nobody ever created you an account. No policy forbids it; there is a queue, an approver on leave, and an access process that runs monthly. A large share of blind deployments are this one, and it is the one teams mistake for the others.

Write the exposure ledger before you write the logger

The decision that matters is not what to redact. It is what may cross the boundary at all, agreed with the customer in writing before launch, in language their security reviewer can approve without a meeting. Every telemetry decision is downstream of this table.

Signal class	Example	Default disposition	What makes it safe
Record content	Invoice line text, a name, a card number, a free-text note	Never crosses	Nothing does. Do not build the path; one that exists gets called during an incident.
Derived values	Embeddings, extracted field values, model output text, cached summaries	Never crosses by default	Treat embeddings as content. Inversion work is good enough that calling a vector anonymous will not survive review.
Field shape	Length, character-class profile, null count, detected unit, encoding	Crosses	Carries the silhouette of a value, not the value. The workhorse signal.
Structural fingerprints	Schema hash, column order, PDF producer string, MIME type, page count, font subset names	Crosses	Identifies a format and a producing system, not a person.
Aggregates	Counts, histograms, percentiles, failure rates by class	Crosses above a floor	Set a minimum bucket size and suppress below it. A one-member bucket is a record.
Diagnostics	Stack traces, timings, config hash, dependency versions, resource counters	Crosses after message scrubbing	The leading leak channel in every codebase we have reviewed.
Correlation handles	Request id, tenant id, salted record fingerprint	Crosses	Lets both sides discuss "that one" without naming it.

Get this signed off during the security review that is happening anyway. A reviewer who has approved the ledger will approve the artifacts that follow it, and you stop renegotiating every time a ticket gets hard.

Instrument the shape, not the content

The central move is to log a description of a value instead of the value. A field becomes a record of its own properties: length, the kinds of characters it held and in what order, null-ness, detected unit, and a keyed fingerprint stable across occurrences.

That line describes 1 234.56 EUR without containing it. Two failing records sharing a class profile but differing in fingerprint tell you the problem is the format, not one bad row. A class profile that appears in the failure histogram and nowhere in the success histogram is a parser branch you have not written. It is enough to fix most extraction bugs, and boring enough that a privacy reviewer signs it in one pass.

Three rules make the fingerprints defensible. Use a keyed hash, HMAC-SHA-256 with a per-tenant key held on the customer side, so fingerprints from one customer cannot be joined against another. Size the truncation to the cardinality: twelve hex characters is 48 bits, and the birthday bound puts meaningful collision probability near sixteen million distinct values, fine for document identifiers and wrong at transaction scale. And never hash a low-entropy field and call it protected. A five-digit postal code has one hundred thousand candidates; a laptop enumerates the space in under a second and reverses your hash by lookup.

Cardinality is the other constraint, and it is a bill rather than a risk. Every distinct combination of label values is a separate time series. Tenant, endpoint, error class and version is a few thousand series and costs nothing. Add a document identifier and your metrics spend grows with your customer's business, the one scaling curve nobody budgets. Fingerprints belong in logs and traces; metrics get bounded labels only.

The support bundle is a product feature

The artifact that turns blind support from interrogation into engineering is a bundle: one command the customer runs, producing one file with everything you are allowed to have about a failure. Most teams write it as a shell script during their first bad incident, and it stays a script nobody trusts. Build it as a feature, version it, test it, document it.

What makes a bundle work in a customer environment is mostly trust, not content. One command, no arguments, running as the same user the application does. It prints a manifest of what it is about to collect, then collects exactly that. Redaction is on by default, and the flag that disables it is warned about and recorded inside the bundle so nobody hands you raw payloads by accident. A size cap with oldest-first truncation means it never fills a disk at three in the morning. Every file carries its SHA-256, so a month later both sides can prove nothing was edited in transit.

The property teams miss: the customer's security engineer reads the bundle before your engineer does. Make it human-readable. Plain text and newline-delimited JSON, a README explaining each file in ordinary language, no proprietary container. A reviewer who can open it approves the mechanism once and you get bundles on demand for the life of the account. A reviewer who opens an opaque archive escalates, and you spend two years asking permission per ticket.

Redaction that survives a security review

Two structural decisions carry almost all of the safety, and pattern matching is not one of them.

Allowlist the fields, do not denylist the patterns. A scrubber that strips things that look like card numbers and email addresses fails on the field nobody thought about, and the field nobody thought about is always the free-text one. Define which fields may be serialized into a log event and drop everything else. New fields are then invisible until someone deliberately adds them, which is the correct default direction.

Redact at write time, not at export time. If the value is never written to disk, no later mistake can leak it: not a misconfigured shipper, not a debug flag, not a bundle. Post-hoc scrubbing means the data existed in a file for some interval, and an auditor will ask exactly how long.

Then there is the leak in nearly every codebase we review, and it is not in the logging layer at all. It is in exceptions. A parser raises could not parse '4111 1111 1111 1111' and a card number is now in a stack trace, an error tracker and a bundle, having bypassed every field-level control you built. PCI DSS limits even the display of a primary account number to the first six and last four digits, so this is not theoretical. Wrap parsing and validation so raised messages name the field and the failure class, never the value. It is the highest-yield hour of redaction work available.

Mistakes that put content on the wrong side of the wall

• Regex scrubbing applied after a log line is formatted, which cannot tell a structured field from a customer’s sentence
• Logging the whole request object during an incident and never removing it
• Error trackers that capture local variables in stack frames, a default in several popular SDKs that exports every parameter in scope
• Debug or health endpoints that echo the request back, then get scraped into monitoring
• Metrics labels holding user-supplied strings, leaking content and detonating cardinality in one line
• Treating embeddings, hashes of low-entropy fields and truncated identifiers as anonymous without checking the value space
• Bundles emailed to a personal inbox, outside every retention rule you promised

Reproducing a bug you are not allowed to look at

Three steps: capture the shape, synthesize an input that matches it, confirm it fails the same way. For the invoice ticket that means the PDF producer string, page count, whether a text layer exists, font subset names, rotation, the character-class profile of the region the amount came from, the extraction schema version, and the model and prompt versions in effect. Then build a document with those properties locally and run your own extractor over it.

When it fails identically the customer is out of the loop for the rest of the fix, and the bug becomes an ordinary bug. You write a test from the synthetic fixture, the fixture goes in your repository because it contains nothing of theirs, and the regression is guarded permanently. That is worth as much as the fix: a fixture corpus generated from shapes can be shared, published and run in continuous integration, none of which is true of a customer file.

When it does not reproduce you have learned something rather than failed: the property separating failures from successes is not in what you captured. Add one dimension, ask for one more bundle, and the space narrows. Track the fraction of tickets you reproduce from the first bundle. That number is the honest score for whether your telemetry was designed or accumulated, and below about half the work belongs in the instrumentation.

Property-based testing does the searching for you. Encode the captured shape as a generator in Hypothesis, fast-check or whichever library your stack uses, and let it produce hundreds of inputs consistent with the fingerprint. It finds the failing case faster than reasoning about the class profile, and leaves a shrunk minimal example that becomes the test.

Rank your signals by value per unit of exposure

Not every signal is worth the same argument. Some are high yield and uncontroversial, some high yield and expensive to defend. Spend the review capital where it pays.

Match the failure class to the signal that names it

Most blind tickets fall into six classes. Knowing which one you are in tells you which signal to read first, and what to build so the class never costs a week again.

Failure class	What the customer reports	The blind signal that names it	What ships with the fix
Input format drift	"It worked last month"	A producer string or schema fingerprint new in the failure histogram	A parser branch plus an alert on unseen fingerprints
Environment	Intermittent, often one node	Image digest, clock skew, memory ceiling, disk pressure, CPU features	A preflight check that refuses to start and says why
Config drift	"One of our sites behaves differently"	Config hash differs from the shipped default; the diff is in the bundle	A startup warning naming every overridden key
Model or prompt regression	"The answers got worse"	Model id, prompt version, and score on an evaluation set both sides hold	Pinned versions and an evaluation gate before any rollout
Your own edge case	One class of record always fails	Class profile of failures compared against the population	A synthetic fixture and a permanent regression test
Upstream data quality	"Your output is wrong"	Invariants violated in their input, reported as checkable counts	A validation report they can act on without you seeing anything

The last row deserves more respect than it gets. A large share of blind tickets end in the finding that the input was wrong, and the only way to say so without a fight is to hand over counts the customer can check themselves. "Four hundred and six records last run had a total that does not equal the sum of their lines, here is how to list them on your side" ends an argument that explanation would not have.

Make every response describe itself

Put a small envelope on every response: request id, application version, config hash, model identifier and version, prompt or template version, the feature flags that evaluated true, and a trace id. Keep the map from config hash back to the full configuration on your side, so a hash is a lookup rather than a mystery.

A bug report then stops being a story and becomes a tuple you can reconstruct. The customer pastes an envelope, you know exactly what code, configuration and model produced that output, and the first three round trips disappear. Pair it with a rule that the system refuses to start when a pinned artifact digest does not match what shipped. Silent updates inside a boundary you cannot see are how you end up debugging a version that no longer exists.

Give them a diagnostic they can run without you

Ship a doctor subcommand. It checks every dependency the application needs, the permissions it actually requires rather than the ones the documentation asked for, clock skew, disk headroom, memory limits, artifact digests, and configuration drift against the shipped default. It prints a verdict in ordinary language and ends with what the operator should do next.

In a customer-run deployment the environment is the largest source of tickets you cannot see, and doctor closes those with no round trip. Round trips are the real cost of blind support: whoever can run a command is rarely whoever filed the ticket, so each question costs a business day and sometimes two. Batch every question into the first reply, or make the bundle answer all of them first.

Build the break-glass path before you need it

Eventually a bug arrives that shape cannot describe. Build the exception deliberately and early, because negotiating access in the middle of an outage produces either a bad process or a refusal, and both are expensive.

The path we build has six properties. It is customer-initiated through a ticket action or a written request, never a vendor asking a stressed operator for access. It is time-boxed with a credential that expires in hours, not a standing account. It is scoped to one namespace, index or bucket prefix, read-only wherever that will do. It is observed in the customer's own audit log, not only yours. Anything the engineer keeps leaves through an artifact the customer reviews first. And your side records the reason and the approver, which is what turns an access event into evidence of a working control instead of a finding.

There is a softer version that clears almost every review and costs one extra round trip. You write an exact script, they run it, they read the output, and they return what they are comfortable returning. Slower than access, faster than an argument, and for many tickets it is enough.

Price and staff for the difference

Blind tickets take longer, and pretending otherwise in a service agreement only means missing it in public. Commit to two clocks. Time to first meaningful signal is fast and fully in your control, because it is the bundle and the triage on it. Time to resolution is paced by the customer's round trips. Commit aggressively to the first and honestly to the second, and say in the contract why they differ. Customers running software in their own boundary already made that trade, and a vendor who says so plainly reads as competent.

Staffing changes too. A first-tier script does not work when the evidence is a set of fingerprints, because reading them means knowing what the parser does. Put support in the engineering rotation, budget hours in the sprint, and treat any ticket that took more than one bundle as an instrumentation defect rather than a bad day.

What we build in before the first customer-boundary install

• A signed exposure ledger listing every signal class and its disposition
• Field-shape logging with keyed fingerprints and bounded metric labels
• Allowlist redaction at write time, and parsers that never quote a value in an exception
• A one-command bundle with a manifest, per-file hashes, a size cap and an internal README
• A doctor subcommand that resolves environment tickets without a round trip
• A self-describing response envelope with version, config hash, model and prompt versions
• A synthetic fixture generator driven by captured shapes, wired into CI
• A break-glass procedure, drilled once before anyone needs it

Retrofitting this into something already deployed

Step six proves the work. Take tickets you already closed, ask whether the bundle you now produce would have named the cause, and count. Every miss names a signal you still owe yourself. Measuring against real history beats reasoning about what telemetry might be useful, and it takes an afternoon.

Bottom line

You cannot debug what you cannot see, so build a system that describes itself well enough that seeing it is unnecessary. Shape instead of content. A bundle instead of a conversation. Synthetic fixtures instead of customer files. Versions in every response, and a break-glass path written down before anyone is under pressure. Teams that do this before the first install run these deployments at close to normal speed. Teams that add it after a bad quarter spend that quarter learning it expensively.

Frequently asked questions