Every serious data business eventually discovers that its hardest engineering problem is not collecting data. It is deciding that two records refer to the same company. A filing says one thing, a sanctions list another, a news article a third, a customer's own vendor master a fourth, and all four are the same firm under four spellings, three legal forms and two countries of registration. Getting that right is what makes a corporate hierarchy trustworthy, an exposure calculation correct and a risk score defensible. Getting it wrong shows up as a wrong parent on a screen, a missed sanctions hit, or a customer quietly building their own mapping table and using yours for lookups only.
This is written for the person who owns that layer: the chief data officer or head of data engineering at a company whose product depends on knowing which company is which. The argument is that entity resolution should be built as a governed service with a lifecycle, not as a matching job inside a pipeline, and that once it is built it becomes a product line rather than a cost centre.
Why the naive approach reaches a ceiling
Most resolution layers begin the same way. Normalize names, strip legal suffixes, compare with a string-similarity measure, take the best score above a threshold. That works well enough to demonstrate and badly enough to erode trust over three years.
The reasons are structural. Company names are not identifiers. Two genuinely different firms share a name in different jurisdictions. One firm operates under a trade name that appears nowhere in its registration. Names carry transliteration variance, ligatures, ampersands, and abbreviations that differ per source. Corporate actions rename entities without changing them, and reorganizations change entities without renaming them. Address data ages and is entered inconsistently. And the tail of small, private and newly formed entities, which is exactly where risk questions concentrate, has the thinnest evidence.
A single similarity threshold also forces a bad trade. Set it high and the system misses true matches, which surfaces as an incomplete hierarchy and a missed exposure. Set it low and it merges two firms, which is worse: a merged record is loud, visible, and hard for a customer to unwind. The fix is not a better threshold. It is an architecture where different classes of evidence are handled differently, and where the system is allowed to say it does not know.
What a production resolution layer needs, weighted by how often it decides trust
Editorial weighting, illustrative rather than measured. The last row is deliberately low: the model is the part teams over-invest in relative to its effect on trust.
Three tiers, in strict order
The design that holds up puts evidence in tiers and never lets a weaker tier override a stronger one.
Tier one: deterministic keys. When two records share a registry identifier, they are the same entity, full stop. Registry numbers from a company register, Legal Entity Identifiers, tax identifiers where lawfully held, exchange listing identifiers, and identifiers assigned by regulators are all facts rather than signals. This tier should resolve the large majority of records in any reasonably sourced dataset, and it should be applied first so that nothing downstream can second-guess it. The engineering work here is not matching. It is building the crosswalk that connects identifier systems to one another, and validating each identifier's format and check digit before trusting it.
Tier two: probabilistic matching. For records with no shared identifier, compare on a combination of features and produce a score with an explanation. Name similarity is one feature among many and should not dominate. The features that carry real weight are the ones that are hard to share by coincidence: a registered address at building level, an incorporation date, a directors or officers overlap, a website domain, a phone number, an industry classification, and jurisdiction. A pair sharing a normalized name and nothing else is a weak match. A pair sharing an incorporation date, a jurisdiction and two officer names is a strong one even if the names differ.
Tier three: human adjudication. Everything in the uncertain band goes to a person, and that person's decision becomes training data and a permanent rule. This tier is not a failure of automation. It is the mechanism that keeps precision high in the tail, and the volume it receives is a design parameter you set, not an accident.
The ordering matters more than any individual technique. A system where a probabilistic score can override a shared registry identifier will produce merges nobody can explain, and explaining merges is most of what a data quality team does.
Blocking, or why this is a systems problem
Comparing every record to every other record is quadratic and impossible at any real scale. The standard answer is blocking: generate candidate pairs cheaply, then score only those. The quality of the blocking scheme sets the ceiling on recall, because a pair never generated can never be matched, and no amount of model quality recovers it.
Good blocking uses several independent keys in parallel and unions the results. Sorted name prefixes catch ordinary variants. Phonetic encodings catch transliteration. Token sets catch reordering. Address components catch renamed entities at a stable location. Domain and phone number catch entities whose names share nothing. Each key generates candidates, and a pair surviving any key gets scored.
Two operational facts about blocking deserve attention. First, blocking keys create hot blocks: common tokens produce candidate sets large enough to stall a job. Cap block size and route oversized blocks to a different strategy rather than letting one block define the runtime. Second, blocking recall is measurable and should be measured separately from match quality. If ninety-four percent of known true pairs appear in some candidate set, the model cannot exceed ninety-four percent recall no matter how good it is. Teams debug the model for months when the answer is a missing blocking key.
What precision and recall the downstream analyst actually tolerates
The right target is not a number from a benchmark. It is derived from what the customer does with the answer, and it differs sharply by use case within the same product.
| Downstream use | Cost of a false merge | Cost of a miss | Design posture |
|---|---|---|---|
| Sanctions and watch-list screening | Low. A reviewer clears it | Very high. A missed hit is a regulatory event | Favour recall hard; send everything plausible to review |
| Corporate hierarchy and ownership | Very high. A wrong parent corrupts every rollup | Moderate. A missing affiliate is visible and reported | Favour precision; require tier-one evidence for parent links |
| Credit or risk scoring | High. Blends two firms' financials into one score | Moderate. Thin file, flagged as such | Favour precision; never merge on name similarity alone |
| Customer master enrichment | Moderate. Wrong data on a real customer record | Low. The record stays unenriched | Balanced; expose confidence and let the customer set the bar |
| Market sizing and coverage analytics | Low at scale | Low at scale | Balanced; aggregate use tolerates both error types |
The design consequence is that one global threshold is wrong. The service should return a decision with a confidence and let each consuming product choose its operating point, or expose named policies with a documented posture. A screening product and a hierarchy product asking the same service the same question should be allowed to want different answers.
This also changes how quality is reported internally. A single accuracy figure for the resolution layer is close to meaningless. Report precision and recall separately, per tier, per source pairing, and per downstream policy, against a labelled set the team maintains.
Where recall is actually lost, by cause
Editorial weighting, illustrative rather than measured. The last row is deliberately low: the model is usually the last place recall is lost, and the first place teams look.
The labelled set is the asset
Nothing in a resolution layer can be improved without a set of pairs with known answers. Building and maintaining one is the least glamorous and highest-return work in the whole system.
Construct it deliberately rather than by sampling randomly, because random sampling produces overwhelmingly easy negatives and tells you nothing. A useful set is stratified: pairs sharing an identifier, pairs with high name similarity and different identifiers, pairs with low name similarity and shared address or officers, pairs from the same jurisdiction, pairs across jurisdictions, pairs involving recently formed entities, and pairs where a corporate action happened between the two observations. Every adjudicated decision from the review queue feeds back into it, which is why the review workflow should capture a reason code rather than just an outcome.
Freeze a holdout portion and never train against it. Re-measure on every release. Track the numbers over time, because the failure that matters is not a bad launch but a slow drift as sources change format and the model quietly stops working on a country or a segment.
Merges and splits, and the key nobody wants to change
The hardest part of running a resolution layer is not deciding matches. It is what happens after you publish a key and then change your mind.
Customers build on the key. They store it in their own systems, join to it, and write reports against it. When two entities you published separately turn out to be one, or one turns out to be two, every downstream customer is affected. Handled casually this is the single fastest way to lose the trust the layer exists to create.
The design that works treats identity as versioned and never destroys a key. When entities merge, both prior keys remain resolvable forever and redirect to the surviving key, with the merge event recorded, dated and explained. When an entity splits, the original key is retired to a tombstone that names both successors rather than being silently reassigned to one of them, because silently choosing a successor is how a customer's report changes meaning without anyone noticing.
Publish the change stream. Customers should be able to subscribe to merge and split events and reconcile their own systems on their schedule. That turns a disruptive correction into a routine operation and, incidentally, into a reason to stay on the product.
Rate-limit yourself as well. A resolution layer that reorganizes thousands of entities in an unannounced release will be blamed for every downstream anomaly for a quarter, whether or not it caused them. Batch structural changes, announce them, and give customers a window to see them in a staging feed first.
The data model underneath
A resolution layer that stores the answer loses the ability to explain it. Store the evidence and compute the answer.
The shape that holds up has four layers. Source records are immutable, one row per observation from one source at one time, retaining the source's own identifiers and raw values. Normalized records are derived from those by a versioned transformation, so a normalization change can be replayed rather than migrated. Match assertions link pairs of normalized records with a tier, a score, the features that drove it, the model version, and, when a human decided, who and why. Entities are the computed connected components over accepted assertions, published with a stable key and an as-of date.
Two properties follow from that shape and both are worth the cost. Every published entity can be explained by walking back to the assertions and the source rows that created it, which is what a customer's data quality team, and any reviewer, will eventually ask for. And the whole graph can be recomputed from a prior model version, which means a regression is diagnosable rather than mysterious.
One warning about connected components. Transitive closure over accepted matches is how a small number of bad assertions creates a giant merged blob containing thousands of unrelated firms. The classic cause is a shared placeholder value: a registered-agent address used by tens of thousands of companies, a default phone number, a generic name. Guard against it with component size limits that trigger review, with feature values blacklisted above a frequency threshold, and by requiring stronger evidence for assertions that would join two already-large components.
Two more properties are worth designing for early. Idempotence: reprocessing the same source record must not create a second observation or a duplicate assertion, which in practice means a deterministic content hash as the source row's own key. And separation of the online path from the batch path: the same scoring logic should serve a synchronous API call against a single record and a nightly recomputation over the whole corpus, or the two will diverge and customers will get different answers depending on how they asked.
Latency shapes the design more than teams expect. A resolution API answering inside a customer's onboarding flow has a budget measured in tens of milliseconds, which rules out generating candidates by scanning and requires precomputed blocking indexes held in memory or in a store built for it. A batch reconciliation over millions of records has no such constraint but does need restartability, because a job that fails at hour nine and starts over is a job nobody runs.
Turning the layer into a product line
Once the layer is governed, explainable and versioned, it stops being infrastructure and becomes something customers will pay for on its own.
A resolution API. Given a name, address and whatever identifiers a caller holds, return the key, confidence, explanation and the canonical record. Priced per call or per seat. This is the most common first product because every large customer has a vendor master or a customer master that needs it.
A batch reconciliation service. A customer sends a file of their own entities and receives keys, confidence and a review queue for the uncertain band. Priced per engagement or per record. It also functions as the most persuasive sales artifact you have, because it quantifies coverage against the buyer's own population rather than against your catalogue.
A hosted mapping table. The persistent crosswalk between the customer's identifiers and yours, maintained across merges and splits. Higher switching cost than anything else you sell, because unwinding it means rebuilding the mapping.
A change feed. Merges, splits, new entities, hierarchy changes and status changes, as an event stream customers integrate. This is what converts an annual licence into infrastructure.
A deployable resolution engine. For customers who cannot send data outward, the engine running inside their environment against their records with your reference data. This is what opens regulated buyers, and increasingly it is what government buyers require.
Each of those is the same core service with a different surface. Building the core once, properly, is what makes five products possible instead of one.
How we work inside a data business
Precision Federal is an engineering firm. We build data platforms, AI systems, APIs and full-stack applications, and we take them into production, including inside federal agencies where the bar for explainability and security is highest. Entity resolution sits at the centre of that work because it is where data quality, machine learning, systems engineering and governance meet.
What the first weeks look like. We start by measuring, not designing. We take a real slice of your source records and a real slice of a customer's population, and we produce the numbers the team probably does not have: blocking recall, tier-one coverage by source and jurisdiction, precision and recall in the uncertain band, review-queue volume at several operating points, and the size distribution of connected components with the top offenders named. That measurement takes two to four weeks and it usually reorders the roadmap, because the biggest gap is rarely the one the team assumed.
From there we build. Typically that means the crosswalk and tier-one resolver first, then the blocking layer with measured recall, then the scoring service with explanations, then the adjudication workflow with reason codes feeding the labelled set, then the versioned key with merge and split semantics and a published change stream. We work inside your repositories, your standards and your review process, and we write the tests and the runbooks alongside the code.
What you keep. Everything. The code is yours, assigned in writing, committed to your repositories from the first day. Your data stays in your environment and never trains anything of ours. Your customer relationships are yours; we work behind your brand unless you want us named. Our existing tooling is named and carved out with a perpetual licence to you inside what we deliver, so no future maintainer is blocked by anything we brought.
Pricing comes in two shapes. Fixed-price milestones with measurable acceptance criteria, which suits a bounded piece like a measurement study, a tier-one resolver or a deployable engine. Or a committed team at a monthly rate when the work is a sustained build with moving priorities. We will say which one fits the scope before you ask.
The first step is one email with a one-page brief: the sources you resolve across, the identifiers you hold, roughly how many entities and records, what the downstream products need, and the date that matters. We return a scoped, priced statement of work.
Failure modes worth naming
Tuning a threshold instead of adding evidence. When precision and recall trade against each other painfully, the answer is almost always a missing feature or a missing deterministic key, not a different cutoff.
No review queue, or a queue nobody owns. An uncertain band with no adjudication path means the system is forced to guess on exactly the cases where guessing is most expensive.
Reassigning a published key. The fastest way to break customer trust. Retire keys, never reuse them, and publish the event.
Measuring one accuracy number. It hides the case that matters. Report by tier, by source pairing, by jurisdiction and by downstream policy.
Ignoring the giant component. Placeholder addresses and generic names silently merge thousands of firms. Monitor component sizes as a production metric, not as an occasional investigation.
Treating resolution as a batch job. If it only runs nightly over the whole corpus, customers cannot ask it a question, and half the product surface described above is unavailable.
Bottom line
Entity resolution decides whether a ratings or risk-data business is trusted. Build it as a service in three strict tiers: deterministic identifiers first and never overridden, probabilistic matching second with explanations attached, human adjudication third with its decisions captured as durable evidence. Measure blocking recall separately from model quality. Set the operating point per downstream use rather than globally, because a screening product and a hierarchy product genuinely want different answers. Version identity so merges and splits are announced events rather than surprises. Store evidence and compute the answer, so every published entity can be explained. Do that, and the layer stops being a cost centre and becomes five products.
Frequently asked questions
Entity resolution is the process of deciding which records refer to the same real-world organization and assigning that organization a stable key. Fuzzy name matching is one weak signal inside it. A production layer applies shared registry identifiers first as facts, then scores the remainder on features that are hard to share by coincidence, such as registered address, incorporation date, officer overlap, website domain and jurisdiction, and routes an uncertain band to human adjudication whose decisions become permanent evidence.
It depends entirely on what consumes the answer, so a single global target is the wrong design. Watch-list screening should favour recall, because a missed hit is far more costly than a review that clears. Corporate hierarchy and credit scoring should favour precision, because a wrong parent corrupts every rollup and a false merge blends two firms' financials. The service should return a confidence and an explanation and let each consuming product choose a named operating policy.
Treat identity as versioned and never destroy or reuse a key. When two entities merge, both prior keys stay resolvable forever and redirect to the survivor, with the event dated and explained. When one splits, retire the original key to a tombstone naming both successors rather than silently reassigning it. Publish merges and splits as a change stream customers can subscribe to, batch structural changes, and let customers see them in a staging feed before they hit production.
Because identity is computed as connected components over accepted matches, so a small number of bad assertions joins unrelated firms transitively. The usual cause is a shared placeholder value: a registered-agent address used by tens of thousands of companies, a default phone number, or a generic name. Guard against it with component size limits that trigger review, by blacklisting feature values above a frequency threshold, and by requiring stronger evidence for any assertion that would join two already-large components.
Yes, and usually as five. A resolution API returning a key, confidence and explanation. A batch reconciliation service that maps a customer's own population and doubles as the most persuasive sales artifact you have. A hosted mapping table maintained across merges and splits, which carries the highest switching cost of anything you sell. A change feed of merges, splits and hierarchy changes. And a deployable engine that runs inside a regulated or government customer's environment when data cannot leave it.