Screening one name against one sanctions list is a lookup. Screening fifty million entities every night, against lists that change without warning, through ownership chains that reach four hooks deep and cross six jurisdictions, and returning a result an analyst can defend to a regulator, is a distributed systems problem wearing a compliance costume. Most teams discover this after the first version ships, when the alert queue reaches numbers no staffing plan can absorb and the customer asks why a match fired.
This is written for the person who owns that system: the head of compliance products, or the engineering leader whose team carries it. The failure modes are not exotic. They are predictable, they are the same across firms, and they are almost all decided at design time by choices that looked minor when someone made them.
The three problems hiding inside one word
"Screening" names three separate engineering problems that share a customer-facing button. They have different data models, different latency budgets, different correctness definitions, and they fail in different ways. Teams that build them as one subsystem end up with a system that is slow at the fast part and wrong at the careful part.
Name matching. Given a string that a human typed or a file delivered, decide which entries on a watchlist plausibly refer to the same real-world party. This is a string and identity problem. It is fast, it is stateless, and it is where false positives come from.
Ownership traversal. Given an entity that is itself clean, decide whether a party on a list controls enough of it to make it restricted by derivation. This is a graph problem with arithmetic on the edges. It is slow, it is stateful, and it is where false negatives come from.
Evidence. Given a decision the system made at 03:14 on a Tuesday, reconstruct exactly why, using the list versions, the ownership snapshot, the thresholds and the code that were live at that moment. This is a versioning problem. It is where audits are won or lost, and it is the part most often bolted on last.
The right architecture separates all three and lets each one scale on its own curve. Name matching wants a horizontally scaled index and a lot of CPU. Traversal wants a graph store and memory. Evidence wants immutable storage and a clock. Fusing them into a single nightly job is the most common design error we see, and it is the one that caps throughput.
Where false positives actually come from
Every screening team believes its false-positive rate is a tuning problem. It usually is not. It is a data-modelling problem that tuning can only trade against recall.
Consider what a watchlist entry actually is. It is not a name. It is a bundle of aliases in several scripts, a set of dates that may be partial, a set of identifiers that may be absent, addresses that may be historical, and a program code that says which restriction applies. A screening engine that reduces this bundle to a single normalized string has thrown away most of the signal it needs to discriminate, and then it compensates with a lower similarity threshold, which floods the queue.
The teams that get this right treat matching as a two-stage decision. Stage one is recall-oriented and cheap: a blocking key generation step that produces candidate pairs. Phonetic keys, character n-gram keys, token-sort keys, transliteration keys for names that arrive in more than one script. Any candidate that survives any blocking key goes forward. This stage is tuned to miss almost nothing and it is allowed to be generous, because the cost of a candidate is milliseconds, not an analyst.
Stage two is precision-oriented and expensive: a scoring function over the full bundle. Name similarity is one feature among many. Date-of-birth agreement, where both sides have one, is enormously discriminative and is frequently ignored because the field is sparse. Identifier agreement is decisive when present. Country of registration, entity type, and the program code all carry weight. A scorer with fifteen features and a calibrated threshold produces a queue an order of magnitude smaller than a scorer with one feature and a permissive cut-off, at the same recall.
The second lever is negative memory. An analyst who clears a hit has produced a labelled example, and that label is worth more than any tuning parameter. A system that does not persist adjudications and suppress the identical pairing on the next run is asking its analysts to answer the same question nightly forever. The suppression must be keyed on the pair plus the list version, so that a change to the underlying entry re-opens the question rather than silently inheriting yesterday's clearance. That single rule is the difference between a queue that stabilizes and one that grows with the customer base.
What moves alert volume most, in our experience building these systems
Editorial weighting, illustrative rather than measured. The last row is deliberately low: the global threshold is the lever teams reach for first and it trades recall for volume rather than improving either.
Ownership traversal is where the real engineering lives
Name matching is a solved shape. Ownership is not: the graph is incomplete, the arithmetic is subtle, and the correct answer depends on rules that differ by regime.
Start with the data model. An ownership edge is not a percentage between two nodes. It is a percentage, as of a date, from a source, of a particular kind: direct equity, voting rights, beneficial interest, control by contract, control by board appointment. A model that collapses these into one weighted edge cannot express the case that matters most, where a restricted party holds a small equity stake and total voting control. Store the edge kind. Store the as-of date. Store the source document identifier. Everything downstream depends on it.
Then the arithmetic. The rule most regimes apply is aggregation of indirect holdings by multiplication along each path and summation across paths, with a threshold at which the target becomes restricted by derivation. That sentence describes a computation that is genuinely hard on a real graph, for four reasons.
- Cycles. Cross-holdings between subsidiaries are common and legal. A naive depth-first traversal loops forever or double-counts. The fix is either cycle detection with path memoization or an iterative fixed-point computation that converges on the ownership matrix.
- Path explosion. A large holding structure can have millions of distinct paths between two nodes. Enumerating them is not feasible at nightly scale. The practical approach is a sparse matrix formulation, where indirect ownership is the convergent sum of powers of the direct-ownership matrix, computed once for the whole graph rather than per query.
- Control overrides percentage. Once a party controls a node above the threshold, most regimes treat that node's own holdings as fully attributable rather than proportionally diluted. This turns a linear-algebra problem into a conditional one, evaluated in rounds: propagate, apply control rules, propagate again, until nothing changes.
- Missing edges. Real corporate registries have gaps. A traversal that treats absence as zero produces a confident wrong answer. Treat unknown ownership as unknown, carry it through the computation, and report a target's status as restricted, clear, or indeterminate with the fraction of the chain that could not be resolved.
Designing for throughput
The nightly window is the constraint that shapes everything. A portfolio of fifty million entities, screened against a few hundred thousand list entries and a corporate graph of several hundred million edges, inside a window of a few hours, is not achievable with a per-entity query loop no matter how fast the query is. The design has to be batch-first and incremental.
Incremental over full. On any given night, three things can change: the watchlists, the ownership graph, and the customer's own portfolio. A full rescreen of everything against everything is the easy design and it does not scale. The correct design computes the delta. If a list entry changed, rescreen everything that could match it. If an ownership edge changed, recompute only the subgraph reachable from that edge. If a portfolio entity was added, screen only that entity. Full rescreens still happen, on a schedule and after any code or model change, but they are the exception rather than the nightly path.
The graph computation is separable. Ownership does not depend on the customer's portfolio. It is a property of the world. Compute the restricted-by-derivation set once, for all customers, as a shared artifact, then intersect it with each portfolio. Firms that compute traversal per customer are doing the same expensive work many times and paying for it in cloud spend.
Match once, fan out many. The same legal entity appears in many customer portfolios. A screening result keyed on the resolved entity identifier rather than on the customer's row can be computed once and referenced everywhere. This depends on having a real entity resolution layer under the system, which is the single highest-return investment for a firm running this at scale.
Bound the tail, not the average. Screening latency distributions have long tails driven by entities with dense ownership structures. Design the interactive path with a time budget and a defined behaviour when it is exceeded: return the direct-match result, mark the derivation as pending, and complete it asynchronously. An interactive screen that hangs for forty seconds on a large conglomerate is worse for the customer than one that returns in two hundred milliseconds with an honest pending flag.
| Design decision | Common approach | What it costs later | The approach that holds |
|---|---|---|---|
| Match unit | The customer's row as delivered | Identical entities screened dozens of times; results disagree across portfolios | A resolved entity identifier, screened once, referenced by every portfolio |
| Ownership edge | A single percentage between two nodes | Cannot express small equity with full voting control; the case regulators care about most | Percentage plus edge kind, as-of date and source document reference |
| Traversal | Recursive query per entity at screen time | Cycles hang or double-count; latency tail unbounded; work repeated per customer | Batch fixed-point over the whole graph, computed once, intersected per portfolio |
| Missing data | Absent edge treated as zero ownership | Confident wrong answers; no response when an auditor asks about coverage | Three-valued result with an explicit unresolved fraction reported per decision |
| Adjudications | Stored in the case tool, not fed back | The same cleared pair re-alerts nightly; queue grows with the customer base | Suppression keyed on pair plus list version; a list change reopens the question |
| Evidence | Reconstructed from logs on request | Cannot reproduce a decision after list or code changes; audit findings | Immutable decision record pinning list version, graph snapshot, thresholds, code version |
Explainability is a data structure, not a report
The word "explainable" gets used loosely. In screening it has a precise operational meaning: given a decision identifier, the system can reproduce that exact decision, and can state in one screen why it fired in terms a reviewer accepts without reading code.
That requires a decision record written at the moment of decision, containing the resolved entity identifier, the list entry identifier and its version hash, the matched fields and each field's contribution to the score, the threshold profile applied and why that profile, the ownership path or paths with the percentage at each hop and the source for each edge, the unresolved fraction, and the versions of the model and the rule set. Written once, immutable, addressable by identifier.
Two properties follow that are hard to get any other way. Reproducibility: the decision can be recomputed from the pinned inputs and must produce the same output, which is a test the system can run against itself on a sample every night and alarm on any divergence. And ownership provenance: the analyst reviewing a derived hit sees the actual chain, hop by hop, with the filing that supports each hop, rather than a system assertion they have to trust.
What the auditors and the customers actually ask for
The evidence expectations converge across regulatory examinations and sophisticated customer diligence, and they are worth designing against directly rather than discovering.
- List currency. When did the system last ingest each source, what version is live, and what is the elapsed time between a publication and its appearance in production screening. This is a metric, it should be on a dashboard, and it should alarm.
- Coverage. Which portfolio entities were screened in the last cycle, which were not, and why. Silent skips are the finding that hurts most, because they suggest the control did not operate.
- Tuning governance. Every threshold change with a date, an owner, a rationale and the before-and-after effect on a held-out sample. A threshold changed by an engineer in a config file with no record is an unmanaged control.
- Effectiveness testing. A periodic exercise where known-positive and known-negative cases are run through production and the results compared to expectation. Build the test rig into the system rather than treating it as an annual project.
- Adjudication quality. Sampling of analyst decisions with a second reviewer, and the disagreement rate tracked over time. This is where a well-built system helps the compliance function argue for itself.
None of these are exotic engineering. All of them are much cheaper to build in than to add. A firm that has them can answer a customer security and compliance questionnaire in days rather than weeks, and that speed converts directly into shorter sales cycles.
What a sophisticated buyer weighs when evaluating a screening capability
Editorial weighting, illustrative rather than measured. The last row is deliberately low: a match rate without a named test set is not evidence.
Turning the control into a product customers pay more for
Screening built as an internal control is a cost centre. The same engineering, exposed deliberately, is a premium tier. The difference is a handful of product decisions.
Sell the chain, not the flag. A boolean restricted indicator is a commodity that several vendors supply. The traversed chain with sources, the unresolved fraction, and the as-of date is a differentiated asset, because reproducing it requires the graph and the computation the customer does not have.
Sell the history. Customers want to know what the answer would have been on a past date, because that is what their own auditors ask them. A system with versioned lists and graph snapshots can answer point-in-time queries. One that overwrites cannot, ever, retroactively.
Sell the integration. The customer's engineers need a stable identifier, a delta feed rather than a full file, a documented latency commitment, and a sandbox with realistic volume. These are the things that decide whether a technically capable buyer says yes, and they are frequently the weakest part of an otherwise good offering.
The federal variant of the same product
Agencies buy screening and ownership analysis, and the buying process rewards different things than a commercial sale. The data model is the same. The delivery around it is not.
An agency evaluating a commercial screening capability wants to know where it runs, what happens to the data it is given, how identities and access are controlled, how the system logs what it did, and how it is documented for a security authorization. A product that can deploy into a government cloud region or a customer-controlled environment, that handles controlled unclassified information according to the customer's rules, that meets accessibility requirements in its interface, and that carries a written security control description, is a candidate. A product that can only be reached as a multi-tenant service on the vendor's own infrastructure is frequently ruled out before anyone evaluates the matching quality.
None of that requires a separate product. It requires the deployment shape and the documentation to be treated as engineering work items with owners and dates, alongside the model quality work. Firms that discover this at the end of a procurement cycle lose the cycle. Building AI, data and cloud systems that go into production inside federal agencies is a large part of what we do, and the pattern is consistent: the technical merit is rarely the reason a good commercial product fails to land, and the deployment and evidence package almost always is.
How we work inside a firm building this
Precision Federal builds these systems and operates alongside the team that owns them. We are engineers, not advisors, and the deliverable is working software in your repositories.
The first two weeks produce three things. A written architecture for the screening path, the traversal computation and the evidence store, with the data model specified to the field. A measured baseline on your own data: current alert volume, the score distribution, the share of alerts that are repeat pairings already cleared, and the traversal coverage against a sample of known structures. And a running slice, deployed in your environment, that takes a subset of the portfolio end to end so the design is demonstrated rather than argued.
From there the work runs in defined increments, each with acceptance criteria written before it starts. Typical increments are the two-stage matcher with the calibrated scorer, the suppression store and its feedback loop, the batch traversal with cycle handling and the unresolved-fraction computation, the immutable decision record and its reproducibility test, the analyst surface, and the deployment package for whichever environments you need to reach.
What you keep is everything. The code is yours, in your source control, under a written assignment. The data never leaves your environment unless you decide otherwise. The models, the thresholds, the tuning history and the documentation are your assets. Your engineers work in the codebase with ours throughout, so that operating the system after we finish is the same activity as building it, not a handover event. If you have customer relationships involved, they stay yours and we stay invisible unless you want us named.
Pricing is fixed-price by milestone where the scope is defined, which is most of this work, or a committed team at a fixed monthly rate where you want a capacity you direct. We do not price by the hour, because that puts our interest and yours on opposite sides of every estimate.
The first step is one email with a one-page brief: what you screen, roughly how many entities, which lists and which registries, what the alert volume looks like now, and what the deadline is. We return a scoped, priced statement of work with the increments named and the acceptance criteria written out.
Bottom line
Screening at scale is decided by four design choices made early: whether matching is two-stage with a full-record scorer, whether analyst adjudications feed back as suppression, whether ownership is a batch fixed-point computation over a typed graph rather than a recursive query, and whether every decision writes an immutable record pinning the inputs that produced it. Firms that make those four choices get a queue that stabilizes, a traversal that finishes inside the window, and an evidence position that survives an examination. Firms that do not get a system that is expensive to run, impossible to reproduce, and staffed by analysts answering the same question every night. The engineering is well understood. It is mostly a question of building it in the right order.
Frequently asked questions
Separate recall from precision into two stages. The first stage generates candidates cheaply using several blocking keys, phonetic, n-gram, token-sort and transliteration, and is tuned to miss almost nothing. The second stage scores each candidate on the full record rather than the name string alone, using date agreement, identifier agreement, jurisdiction, entity type and program code as features with a calibrated threshold. Then persist every analyst adjudication and suppress the identical pairing until the underlying list entry changes. Lowering a global similarity threshold trades recall for volume and improves neither.
The usual rule is to multiply ownership percentages along each path from the restricted party to the target and sum across paths, comparing the total to a threshold at which the target becomes restricted by derivation. Real graphs make this harder than the rule sounds: cross-holdings create cycles, large structures have too many paths to enumerate, and control above the threshold generally makes a node's own holdings fully attributable rather than proportionally diluted. The practical implementation is an iterative fixed-point computation over a sparse ownership matrix with control rules applied between rounds, rather than a recursive query per entity.
Reproducibility above all: given a decision identifier, the system should recompute the same result from pinned inputs, meaning the list version, the ownership snapshot, the thresholds, the scoring model version and the code version. Beyond that, list currency with the lag from publication to production, coverage showing which entities were screened and which were skipped and why, a governed record of every threshold change with its measured effect, periodic effectiveness testing against known cases, and quality sampling of analyst adjudications with a tracked disagreement rate.
In a batch, for almost every firm operating at scale. The restricted-by-derivation set is a property of the world rather than of any one customer's portfolio, so computing it once and intersecting it with each portfolio avoids repeating the same expensive work per customer. Batch computation also lets you handle cycles with a converging fixed point rather than cycle-detecting inside a latency budget. Keep an interactive path for new entities with a time budget and an honest pending flag when a dense structure exceeds it.
Deployment shape and documentation, more often than matching quality. The buyer needs to know where the system runs, what happens to data it is given, how identity and access are controlled, what it logs, whether it can run in a government cloud region or inside an environment the customer controls, whether the interface meets accessibility requirements, and whether there is a written description of the security controls suitable for an authorization package. Products that can only be reached as a multi-tenant service on the vendor's infrastructure are often eliminated before anyone evaluates the model.