The identifiers you already have are not keys
Every team that starts this work starts it in the same place: with a handful of identifiers that look authoritative, and the reasonable assumption that a join is available somewhere. It is not. A ticker is scoped to an exchange, gets reused after a delisting, and changes on a rename. An instrument identifier identifies a security, not the company that issued it, so a firm with senior notes, converts and two share classes is five rows. A legal entity identifier is genuinely good and covers only the subset of firms that had a reason to obtain one, so coverage in a portfolio of private mid-market borrowers is often under half. A vendor's internal id is stable inside that vendor's world and meaningless outside it. Your own CRM id is whatever the sales team typed in 2019.
So the join is a matching problem, and the matching problem is the easy half. Name similarity, address normalization, jurisdiction of incorporation, registration numbers — this is fifty-year-old craft with good open implementations. Teams get through it in a few weeks and then spend the next two quarters on the parts nobody scoped: which grain of entity they meant, what the answer was on a date in the past, and how to answer a portfolio manager who says this link is wrong.
This piece is about those three. If you take nothing else from it: choose the grain before you write the matcher, store the match as an assertion with a date range rather than overwriting an id, and publish a precision estimate with a sample size next to it.
You are probably here because
- Two dashboards report different exposure to the same borrower and both are defensible
- A backtest looks better than the strategy ever did, and nobody can say why
- An analyst keeps a private spreadsheet that maps vendor ids by hand, and it is the real system
- A merger closed and half your reports still show two counterparties
The first and last are grain and validity-time problems. The second is almost always a survivorship artifact in the entity graph. The third is what happens when the matcher has no way to record a human decision.
Decide what an entity is before you match anything
Ask five people in a financial firm to define "company" and you get five grains, all of them correct for their own use. The legal entity is the thing that signs a contract and files accounts. The issuer is the entity whose name is on a security. The obligor is who owes the money, which may be a subsidiary with a parent guarantee. The risk counterparty is the level at which limits are set, usually the ultimate parent, sometimes a ring-fenced sub. The reporting entity is whatever the consolidated statements cover, which changes with every acquisition.
These are not synonyms and a system that resolves to the wrong one produces answers that are wrong in a specific, expensive direction. Aggregating exposure at legal-entity grain understates concentration, because six subsidiaries of one group look like six names. Aggregating at ultimate-parent grain overstates it, because a non-recourse project company with a shared brand is not the parent's obligation. Both mistakes survive review, because the number is plausible either way.
The design that holds up is to resolve at legal-entity grain and model everything above it as a separate, dated hierarchy. Legal entities are the atoms: they have a registration number in a jurisdiction, which is the closest thing to a real key that exists in this domain. Ownership is an edge with a percentage, a start date and an end date. Rollups then become a query rather than a property of the record, and a team that needs a different rollup gets it without a re-resolution.
| Grain | Anchored by | What breaks if you pick it by default |
|---|---|---|
| Legal entity | Jurisdiction plus registration number | Nothing structurally; you must build the hierarchy on top or concentration is understated |
| Issuer | Security master linkage | Private companies and non-issuing subsidiaries have no row at all |
| Obligor | Contract terms, guarantees | Cannot be derived from public reference data; it is a credit judgment |
| Ultimate parent | Ownership graph traversal | Ring-fenced and non-recourse structures get consolidated when they should not |
| Reporting entity | Consolidation scope in the accounts | Changes every time the group buys or sells; not stable across periods |
Point-in-time correctness, or your backtest is fiction
An entity graph built today describes the world today. Used to reconstruct a portfolio as of three years ago, it silently rewrites history in a direction that flatters you. Companies that were acquired have been folded into the acquirer, so positions in them look like positions in a larger, healthier group. Companies that failed have often been removed from the vendor file entirely, so they are absent from the universe rather than present with a bad outcome. Tickers reused by a different issuer map old prices onto a new company. This is survivorship bias arriving through the reference data rather than through the price history, and it is harder to spot because everyone is watching the prices.
The fix is structural and cheap if you do it at the beginning. Every assertion in the resolution store carries valid_from and valid_to for when the fact was true in the world, and observed_at for when you learned it. Two timelines, not one. A merger effective in March that you learned about in May is retrievable both ways: what was true then, and what you knew then. Backtests use the second, reporting uses the first, and the difference between the two is a real number you can show.
Never mutate an identifier in place. When two entities merge, write a merge event and keep both rows resolvable. Downstream systems that stored your id three years ago must still be able to look it up and get a sensible answer, which is this id was superseded on this date by that id rather than a null.
Where the effort actually goes — our default split on a first build
Weights sum to 100. Our starting allocation, not a measurement. The scoring model is the part everyone budgets for and the smallest slice of the work.
Three stages, kept apart
Blocking, scoring and deciding are three different jobs and merging them is the most common architectural error in this domain. Blocking generates candidate pairs cheaply and its only metric is recall: a pair never generated can never be matched, and no amount of scoring sophistication recovers it. Scoring assigns a number to each candidate pair. Deciding turns numbers into links using a threshold that is a business choice, not a modelling one.
Blocking on a normalized name prefix alone is where recall quietly dies. Legal-suffix handling, ampersands, transliteration, and the habit of financial data of storing "The Coca-Cola Company" in one file and "COCA COLA CO" in another mean a single blocking key is never enough. Use several in parallel — normalized name tokens, name plus country, address, registration number, any shared identifier — and union the candidates. Measure recall by taking a set of pairs you know are matches and checking what fraction survives blocking. If that number is below the high nineties, nothing downstream matters.
Scoring can be a weighted rule set or a trained classifier, and for most financial reference data a well-tuned rule set gets to within a few points of a model. What earns its cost is the feature set, not the estimator: token Jaccard on normalized names, edit distance on the longest token, exact match on registration number, country agreement, industry agreement, address distance, and a strong negative feature for numeric or ordinal tokens that differ. "Series II" versus "Series III" and "Fund IV" versus "Fund VI" are different entities that every string metric calls near-identical.
Where embeddings help, and where they will hurt you
Vector similarity is genuinely useful for one thing here: recall in the blocking stage, where a semantic near-neighbour index surfaces candidates that lexical keys miss — the trading name that shares no tokens with the legal name, the transliterated form, the abbreviation nobody wrote a rule for. Used that way, with a generous cutoff and a real scorer behind it, an embedding index is a good investment.
Used as the decision function it is a liability, and specifically in corporate group structures, which is exactly the case financial data is full of. "Acme Holdings", "Acme Capital Markets" and "Acme Funding II" sit within a hair of each other in every general-purpose embedding space and are three distinct legal entities with different obligations. The property that makes embeddings good at recall — tolerance of surface variation — is the property that makes them dangerous at precision, because the variation that distinguishes group members is surface variation. Put the embedding in the candidate generator and put the discriminating logic in the scorer.
The two errors do not cost the same
A false merge and a false split are both errors and treating them as symmetric is a mistake. A false split — one company appearing as two — shows up as a smaller number in a report. Someone notices eventually, usually a relationship manager who knows the client. It is embarrassing and recoverable. A false merge combines two companies' data into one row: exposure to an unrelated borrower lands in your concentration figure, one company's financials contaminate the other's history, and the resulting number is not merely wrong but confidently wrong, with no internal contradiction to trip on.
So the threshold is asymmetric on purpose. Our normal starting policy uses three bands: auto-link above a high threshold, auto-reject below a low one, and a review queue in between, sized to the review capacity that actually exists rather than the capacity someone hoped for. Two more rules earn their keep. A conflicting strong identifier — two different registration numbers in the same jurisdiction — vetoes a link regardless of score. And any human decision, accept or reject, is stored as a durable assertion that overrides the scorer permanently, so the same pair never returns to the queue and an analyst's afternoon is never spent twice.
Store the match, not the merged record
The tempting design is a golden record: one clean row per company, with the best value from each source. It is the wrong primitive. It throws away which source said what, it cannot be corrected without a rebuild, and it cannot answer the question you will actually be asked, which is why these two rows are the same company. Store the pairwise assertion — source A id, source B id, score, features, decision, decider, timestamp, validity range — and let the merged view be a projection over assertions. The projection is cheap to rebuild. The lost provenance is not.
Send us a sample and we will tell you what the match rate really is.
Email a de-identified extract of two sources you are trying to link, plus the identifiers each carries, to contact@precisionfederal.com. You get back a written estimate of achievable coverage, the specific fields that will decide it, and the grain problem we see in your schema. Two business days. No charge and no meeting.
contact@precisionfederal.comMeasuring when nobody owns a truth set
Almost every project of this kind starts without labelled data, and the temptation is to report the only number available: how many records got linked. Coverage is not accuracy. A system that links everything to something scores wonderfully on coverage and is worthless.
Precision is estimable and you should estimate it. Draw a stratified random sample of accepted links — stratify by score band, because errors concentrate near the threshold — and have a person adjudicate each one against source evidence. Three hundred pairs is enough for a confidence interval a few points wide, and the work is roughly a day. Report the interval, not the point estimate. "Precision 97.2%" invites false confidence; "96 to 98 percent on a 300-pair stratified sample, drawn 12 August" invites the right kind of scrutiny.
Recall is harder and there are two honest routes. Build a small gold set by hand for a slice you care about most — the top 500 counterparties by exposure is a defensible slice and a week of work — and measure against it directly. Or use a capture-recapture design: run two independent matchers with different blocking strategies, and the overlap between their results supports an estimate of the pairs neither found. The second is cheaper and less precise. Both beat the usual practice, which is not measuring recall at all.
How much a linking signal is worth — our rating by evidence strength
How much confidence we give each signal on its own, before combination. Judgment from repeated builds, not a benchmark — the ordering is the useful part.
The sources keep moving, and that is the running cost
Entity resolution is not a project that finishes. Companies rename, restructure, redomicile, merge and dissolve, and every vendor reflects those events on its own schedule with its own conventions. A build that ignores this produces a graph that is excellent on delivery day and stale within two quarters.
Treat corporate actions as an event stream and process them as events. A merger produces a superseding assertion, not an edit. A rename produces a new name with a validity range and leaves the old one queryable, because your documents and contracts still use it. A demerger is the case that breaks naive designs: one entity becomes two, and any system that only knows how to combine has no operation for it.
Budget for the diff review. When a source publishes a new file, the useful artifact is not the file but the change list, and someone should look at the changes above a materiality line every week. In our experience an established graph over a handful of financial sources generates a queue that takes one analyst somewhere between two and six hours a week, spiking around quarter ends and any large deal. That is the honest running cost, and a project plan without it is a project plan that ends with the analyst's private spreadsheet coming back.
Explaining a link to someone who disputes it
The day a portfolio manager says the system has the wrong company, the system has to produce evidence in a form a person can read. Not a score. The source records side by side, the features that agreed and disagreed, the threshold in force, who decided and when, and the current validity range. If the answer takes an engineer half a day to assemble from logs, the system will lose that argument on schedule grounds no matter how good the matcher is.
This is also the cheapest possible feedback loop. Every disputed link that gets adjudicated is a labelled example, and a system that captures dispute outcomes as durable assertions builds its own gold set from the highest-value pairs in the book — the ones people actually look at.
The mistakes we are called in to fix
- A golden record with no provenance, so no link can be explained or corrected without a full rebuild
- Ids mutated in place on a merger, breaking every downstream system that stored one
- One current-state table, no validity dates, and a backtest quietly flattered by survivorship
- Coverage reported as accuracy, with no precision estimate and no sample
- Embedding similarity as the decision function, collapsing corporate group members into one entity
- A review queue larger than the review capacity, which becomes an auto-accept queue within a month
- Human decisions stored nowhere, so the same pair returns to the queue after every rerun
- Rollup baked into the resolved record, forcing a re-resolution when a second team needs a different one
A six-week shape that gets to a defensible number
First build, typical sequence
Step three is the one that gets cut and the one that determines whether anything after it is measurable. Building the gold set before the matcher also keeps it honest: a set assembled after you have seen the output is a set assembled around the output.
When not to build this
Two cases. If the population is small and stable — a few thousand counterparties that change slowly — a curated crosswalk maintained by two analysts with a decent review tool will beat a probabilistic system on accuracy, cost and trust, and it will be running next week. Build the tooling, skip the matcher.
And if the identifier coverage in your sources is genuinely high, the honest answer is that most of the work is a join plus exception handling. Profile first. A profiling pass that measures identifier overlap between the two sources costs about a day, and it regularly changes the shape of the project — a high shared-identifier rate turns a matching programme into exception tooling, which is a much smaller piece of work. Take that measurement before anyone commits a budget.
Before you call it done
- The grain is written down and every consumer knows which one they are getting
- Every assertion carries validity dates and an observation date
- Identifiers are never mutated; supersession is an event with a date
- Blocking recall measured against a gold set and stated as a number
- Precision published as an interval with the sample size and date beside it
- Thresholds asymmetric, with conflicting strong identifiers as a veto
- Human decisions stored as durable assertions that survive a rerun
- Any link explainable on screen in under a minute, with source evidence
- A weekly diff queue with a named owner and a materiality threshold
- Hierarchies are queries over dated edges, not fields on the entity
Bottom line
The matching algorithm is the part of this problem with the best literature and the smallest share of the risk. What decides whether the system is trusted in two years is whether it resolves to the grain people actually need, whether it can tell you what was true on a date, and whether it can defend an individual link to a person who is annoyed. Those three are schema and process decisions, they are cheap at the start and expensive to retrofit, and they are all made in the first two weeks — usually by default, usually without anyone noticing they were decisions.
Frequently asked questions
Use it wherever it is present — it is the strongest signal available. But coverage is the constraint, not quality. Registration is driven by particular reporting obligations, so listed and financial-sector firms are well covered while private mid-market companies, small suppliers and non-operating subsidiaries frequently are not. Profile your own population before assuming coverage; the answer varies enormously by portfolio.
It depends on what the wrong answer costs, so set the target per use. For a concentration or exposure figure the expensive error is the false merge, and a sensible policy holds precision on auto-accepted links in the high nineties while routing the ambiguous band to review. For exploratory analytics, lower precision with higher recall is often the better trade. What matters more than the number is that the number is measured on a sample and published with an interval.
Two timelines on every assertion: when the fact was true in the world, and when you learned it. That lets you reconstruct both what was true on a past date and what you knew on that date, which are different questions and both get asked. It also requires that identifiers are never overwritten — a merger is a superseding event, so old ids stay resolvable for systems that stored them.
In candidate generation, yes — they surface trading names, abbreviations and transliterations that lexical keys miss. As the decision function they are actively harmful on corporate groups, because members of one group differ by exactly the kind of surface variation embeddings are built to ignore. Put them in blocking, keep the discriminating logic in the scorer.
Precision by stratified sampling: draw a few hundred accepted links stratified by score band, adjudicate them against source evidence, report the confidence interval. Recall by a hand-built gold set over the slice that matters most, or by capture-recapture between two independent matchers. Both are a few days of work, and both are more informative than the coverage percentage most teams report instead.