Reference data and the golden-record problem

The quiet layer that owns every number

Reference data is the layer nobody presents at a board meeting and everything else depends on: the entity master, the instrument master, the product catalogue, the ownership hierarchies, the classification schemes, the calendars. It has no interesting logic and almost no visible cost, and when one of its values is wrong the failure does not look like a data problem. It looks like a report that quietly disagrees with another report, and it stays that way for months because both sides are reading their systems correctly.

The standard remedy is a master data programme that produces a golden record: one canonical value per attribute per entity, assembled from the sources by a survivorship rule, published as the single truth. It is a reasonable instinct and it works for a narrow class of attributes. For everything else it fails in a specific and repeatable way, and the failure is worth understanding before you fund the programme rather than in year two of it.

What the golden record throws away

Three things, and each is recoverable only if you never discarded it in the first place.

Time. A golden record answers what is true now. The questions that arrive in practice are about the past — what did we believe when this report was produced, what changed between then and now, and when did it change. A current-value table cannot answer any of them, and reconstructing the answer from backups is an archaeology exercise with a low success rate.

Provenance. Once a survivorship rule picks a winner and writes one value, the losing values and the rule that rejected them are gone. When someone disputes the number, the only available answer is that the master says so, which is not an answer. The whole cost of the dispute is then paid in re-derivation.

Legitimate disagreement. This is the one that sinks the programme. Two teams sometimes need different values for the same named attribute, and both are right. Country of risk is not country of incorporation is not country of primary listing. Ultimate parent under a majority-ownership definition is a different entity than under a lower control threshold. Industry classification depends on which scheme you are obliged to report under. A single golden value silently picks one consumer's definition and imposes it on everyone else, and the affected teams do not escalate — they build a shadow table, which is how you get two totals that never reconcile.

Keep the sources; publish views

The design that holds is a layered one. Land each source as it arrived, unmodified, with its own history. Resolve identity across the sources and store the match as an object rather than a merge, so it can be reviewed. Then publish views — named, versioned, documented resolutions of each attribute — instead of one truth.

A view is a profile: for this consumer, ultimate parent means majority ownership; for that one, it means a lower control threshold. Both are computed from the same underlying assertions by different rules, both are versioned, and both are documented with the definition in the words of the person who needs it. The disagreement stops being a conflict to suppress and becomes configuration, which is the correct shape for it, because the disagreement was real.

The objection is that this multiplies the number of published values, and it does. What it removes is the shadow tables, which existed anyway, were undocumented, and were maintained by whoever had time. Making the profiles explicit converts invisible divergence into a small number of governed, named differences, and that is the whole trade.

Survivorship belongs at the attribute, not the record

Record-level precedence — source A beats source B, always — is the default in most master data tooling and it is wrong at the first counterexample. The registry is authoritative on legal name and registered address and unhelpful on trading name. The commercial vendor is better on industry classification and stale on corporate actions. An internal operations system is the only place a client's servicing relationship is recorded and has no view on ownership at all.

So precedence is a table with a row per attribute, not a global ordering. Each row names the preferred source, the fallbacks in order, the freshness beyond which a value is no longer preferred, and the validation the value must pass to be eligible at all. Store that table as data with its own version history, never as code, because it changes far more often than the pipeline and every change is one someone will later need to explain.

Attribute	Usually best source	Where the naive rule fails
Legal name, registered address	The official registry for the jurisdiction	Registry lags a rename by weeks; the vendor is faster and less correct
Trading or brand name	Commercial vendor or internal sales system	No registry holds it; record-level precedence returns nothing
Ownership and control	Filings, plus a vendor for the chain	The right answer depends on the threshold, which differs by consumer
Industry classification	Whichever scheme the consumer reports under	Two consumers require two schemes; one golden value serves neither
Status: active, merged, dissolved	Registry, with a dated effective event	Stored as a flag, so the date it changed is unrecoverable
Servicing and relationship data	The internal system of record	Overwritten by an external file that never knew about it

Two clocks, and the retroactive hierarchy

Reference data needs the same bitemporal treatment as any history: valid time, when the fact was true, and assertion time, when you came to believe it. Store both, append rather than update, and make every read take an as-of pair. It is one extra predicate in a query and it is the difference between answering “what did the report see in March” and guessing.

Hierarchies are where the absence of this hurts most, because a hierarchy change restates history silently. An acquisition completes; the parent-child edge is updated in place; every aggregate that rolls up by parent now returns a different answer for every past period, including periods already reported. Nobody changed a report. Nobody changed a number. The rollup simply means something else now, and the first anyone hears of it is a comparison that no longer ties out.

Effective-dated edges fix it: the ownership link carries a valid-from and valid-to, and a rollup as of a past date uses the structure in force then. It costs a join. The alternative costs a quarter of investigation the first time someone notices, and the credibility of every historical comparison thereafter.

External identifiers are attributes, never keys

Every reference-data programme is tempted to key on an external identifier, because one already exists and it is free. Do not. Mint an internal surrogate id that means nothing, and carry every external identifier as a dated attribute with a status.

The reasons are all empirical. Identifiers get reused: a ticker released by a delisting is reassigned to a different company, so a join across a long history joins two unrelated firms. Identifiers lapse without becoming wrong — an entity identifier under an annual renewal regime can be marked lapsed while remaining perfectly valid as an identifier, and systems that treat lapse as invalid start rejecting live counterparties. Identifiers are also licensed, differently, and the licence sometimes governs where the identifier may travel inside your systems, which is a constraint you cannot discover from the data. And one entity legitimately carries several: an entity identifier, a national registration number, one or more vendor keys, and your own. Keeping them as dated attributes with sources makes all of that a mapping problem rather than a migration.

A reference-data change is a release

In most organizations a code change goes through review, testing, staged rollout and a rollback plan, and a reference-data change is a steward editing a field at four in the afternoon. The second one can move more numbers than the first. A reclassification applied across a category changes every aggregate that groups by it, in every report, with no deployment and no notes.

Treat it like the release it is. Batch changes into scheduled publications, except for corrections urgent enough to justify an exception. Generate a diff for every publication — entities affected, attributes changed, before and after — and send it to consumers rather than filing it. Give each publication a version identifier that consumers can pin to, so a downstream team can stay on last week's view while they check what moved. And make rollback possible, which follows for free from append-only storage and is impossible with in-place updates.

Stewards, and measuring the right thing

Every reference-data operation eventually depends on people making judgement calls: is this the same entity, which source is right about this address, does this hierarchy change apply from the announcement or the completion. That work is real and it does not automate away. What automation does is decide how much of it reaches a human and in what order.

Which means the metrics most programmes report are the wrong ones. “Percentage of records golden” measures how much of the data has been through a rule, which is a measure of coverage rather than correctness, and it goes up whether the rules are good or bad. The metrics that predict whether the layer is trusted look different.

Metric	What it tells you	What a bad number looks like
Dispute rate per thousand entities	Whether consumers believe the published values	Rising while coverage also rises: the rules are wrong
Time from report to correction	Whether the stewardship queue is staffed for the load	A long tail; the median hides a queue nobody drains
Downstream restatements per quarter	How often a change rewrites published history	Any number above zero that was not announced
Attributes with a named owner	Whether anyone can decide a contested value	Under half, which is the normal starting point
Consumers on the current version	Whether the release process is working	Teams pinned to an old view for months, silently
Shadow copies discovered	Where the published view fails a real need	Found by accident rather than reported

Migrating without a big bang

You rarely get to build this from nothing. There is a master, it is trusted unevenly, and dozens of consumers read it. The migration that works runs the new derivation beside the old one and makes the diff the acceptance test.

Publish both, compare them daily, and route every difference to a person until the differences are either explained or eliminated. That comparison is the most valuable artefact of the whole project: it finds the attributes where the old rule was silently wrong, the consumers who need a different profile, and the entities that were never resolved correctly. It also gives you something to show, which matters, because a reference-data programme has no visible output for a long time and this is the thing that keeps it funded. Only when the diff is understood does anyone cut over, one consumer at a time, with the old view still available and pinned.

The mistakes we are called in to fix

• One golden value where two consumers legitimately need different definitions, producing shadow tables
• Record-level precedence, so the best source for one attribute overrides the best source for another
• In-place updates, leaving no record of what was published on a past date
• Undated hierarchy edges, so an acquisition silently restates every historical rollup
• An external identifier as the primary key, which breaks when it is reused or relicensed
• Treating a lapsed identifier as invalid, rejecting live counterparties
• Survivorship rules in code rather than in versioned data, so nobody can explain a past value
• Reference-data changes published with no diff and no version to pin to

A twelve-week first phase

Step one is the one that gets cut and the one that determines the design. Every shadow spreadsheet you find is a specification for a profile somebody needs, written by the person who needed it badly enough to build it themselves. Teams that skip the inventory design for the consumers they know about and rediscover the others during cutover, which is the expensive place to find them.

Before you call it authoritative

• Every attribute value carries a source, a valid time and an assertion time
• Append-only history; nothing is updated in place
• Internal surrogate keys; external identifiers dated, with status
• Precedence defined per attribute and stored as versioned data
• Hierarchy edges effective-dated; rollups take an as-of date
• Named owner for every attribute that anyone can contest
• Consumer profiles published where definitions legitimately differ
• Releases versioned and pinnable, each with a diff sent to consumers
• Restatements announced with an effective date and an owner
• Dispute rate and time-to-correct reported, not percentage golden

Bottom line

The golden record is the right shape for the small set of attributes with one uncontested source and no meaningful history. For everything else it removes the information you need to defend a number: when it was true, where it came from, and which definition it answers. Keep the sources, put two clocks on every value, set precedence per attribute rather than per record, date the hierarchy edges, and publish views for the consumers whose definitions genuinely differ. It is more machinery than a single canonical table, and it is the machinery that lets you answer the only question that ever really matters here — what did we believe, when, and on what basis.

Frequently asked questions