Reasoning over a proprietary credit corpus without inventing a number

A research organization sitting on thirty years of issuer analysis has something no frontier model has: a private, internally consistent, expensively curated view of how credit actually behaves. The instinct is to put a language model on top of it. The instinct is right. The usual implementation is not.

Why this corpus is different from the documents in a demo

Public demos run on documents where being roughly right is fine. A summary that slightly misstates a policy is embarrassing. In credit work, a figure is either the figure in the record or it is a fabrication, and the distance between those two is not a quality gradient. It is a binary.

The corpus also has a property that breaks naive retrieval: it is longitudinal. The same issuer appears across years, under changed methodology, with restatements, under a name that changed twice. A system that retrieves three passages and reasons over them will confidently blend two accounting regimes and produce a trend that never happened.

The failure that survives every demo

Ask a retrieval system for a leverage figure and it will find a passage containing a leverage figure. It will not, on its own, establish that the passage refers to the right entity, the right period, and the right basis. Those three checks are what an analyst does automatically and what a similarity search does not do at all.

Similarity finds text that looks like the answer. It has no concept of whether it is the answer.

The result reads perfectly. It cites a real document. The number is real. It is simply the wrong number, and nothing in the output signals that.

What the architecture has to do instead

The corpus has to stop being a pile of text and start being a set of resolved entities with attached, dated, basis-tagged facts. That is unglamorous work and it is the majority of the project.

• Resolve every mention to a stable issuer identity across name changes, mergers, and subsidiaries
• Attach every extracted figure to a period, a basis, and the document span it came from
• Record methodology vintage so figures computed under different regimes are never silently compared
• Let the model select and parameterize a query rather than state a value
• Return the executed result with its span, so any sentence can be opened to the source paragraph

Once that layer exists, the model's job narrows to the thing it is genuinely good at: understanding an analyst's question, choosing what to compute, and writing the result in the house register.

Where the model earns its place

None of the above says the model is unimportant. Analysts do not ask questions in schema. They ask whether an issuer's cushion has thinned relative to peers since the last downturn, which is four operations, a peer definition, and a period alignment. Turning that sentence into those operations is real work and language models are unusually good at it.

They are also good at the reverse direction — taking a computed result set and writing the two paragraphs an analyst would have written, with the hedging conventions the house uses. Both ends are language. The middle is arithmetic, and the middle must never be generated.

How to tell in a demo whether you are being sold the middle

One question does it. Ask for a figure, then ask the system to show every record that produced it. A grounded system returns rows. A generative system returns a paragraph explaining the figure, which is not the same thing and is the tell.

The second question is harder and more revealing: ask something the corpus cannot answer. A system built correctly says so. A system that always produces an answer has told you its fabrication rate is not zero, it is merely unmeasured.

What this means for scoping

Projects framed as "put an LLM on our research archive" budget for model work and underrun on the entity layer by a wide margin. In practice the resolution, normalization, and provenance plumbing is where the schedule goes, and it is also the part that makes the result defensible to a committee, a regulator, or a client who disagrees with the conclusion.

That is not a reason to avoid the project. It is a reason to scope it honestly, because the version that gets built correctly is genuinely differentiating and the version that gets built as a demo will not survive its first disputed number.

A worked example: one ordinary question

An analyst asks whether an issuer's interest coverage has deteriorated relative to its peer group over the last three years. This is not an exotic request. It is close to the median question a research desk handles, and it is worth walking through what each architecture does with it.

A retrieval system embeds the question, finds passages mentioning interest coverage for that issuer, finds a few more mentioning peers, and hands roughly eight passages to a model with instructions to answer. The model writes three paragraphs. Every sentence is grammatical, the tone is right, and two of the four numbers are from the wrong fiscal period because the passages did not consistently state one.

A grounded system does something less impressive-looking and more useful. It resolves the issuer to an identity, resolves the peer group to a set of identities under a definition the analyst can inspect, retrieves the coverage series for all of them from the fact store with period and basis attached, computes the comparison, and returns the result with every input listed.

The second system will sometimes return less. It will say that peer coverage is available for six of the nine peers and that three have no reported figure for the most recent period. That admission is the product working correctly, and it is the sentence the first system will never produce.

The four layers, and where the effort actually goes

It helps to be concrete about the shape of the build, because the distribution of effort is consistently misjudged in planning.

The inversion is the lesson. The layer that gets the attention in planning is the one that takes the least time, and the layer nobody puts in the schedule is the one that determines whether the system is correct.

What the entity layer has to survive

A corpus spanning decades accumulates identity problems that no single heuristic handles. The company was acquired and the analysis continued under the parent's name. The subsidiary that issued the debt has a name that differs from the operating company by one word. A transliteration convention changed between 2011 and 2014 and the same issuer appears both ways. Two unrelated firms in different countries share a name exactly.

Handling these requires resolution on multiple independent signals — identifiers where they exist, jurisdiction, sector, date ranges of activity, and relationship evidence — with the match evidence retained rather than discarded on merge. Retaining the evidence is what allows a disputed link to be examined rather than argued about from scratch, and disputed links are a routine occurrence rather than an edge case.

It is also why the resolution has to be versioned. A conclusion produced last quarter was produced under a particular set of entity links. If those links change — correctly, because someone found an error — the prior conclusion still needs to be explicable in terms of what the system believed at the time.

Methodology vintage is the trap nobody sees coming

Research organizations revise their approaches. Ratios get redefined, adjustments get added, treatments of leases or pensions change. Each revision is documented and each one silently breaks longitudinal comparison for any system that treats a number as a number.

A leverage figure computed under the 2013 convention and one computed under the 2021 convention are not points on the same series, and a system that plots them together produces a trend that is partly an artifact of its own methodology history. Analysts know this and correct for it manually. An automated system that does not carry basis metadata cannot.

The fix is unglamorous: every extracted figure carries the methodology version in force when it was published, comparisons across versions are either adjusted or refused, and the refusal is visible. This is the sort of requirement that never appears in a pilot and always appears in production.

How to evaluate this without a benchmark

There is no public benchmark for a private corpus, which leaves teams either shipping on vibes or building their own evaluation. The second is not as expensive as it sounds.

• An answerable set. Fifty questions with answers verified by hand against the source documents, spanning the operations analysts actually run.
• An unanswerable set. Twenty questions whose answers are genuinely not in the corpus — a period not covered, an issuer not tracked, a field never collected. Correct behavior is a clear decline.
• A false-premise set. Ten questions that presuppose something untrue. Asking why an issuer was downgraded in a year it was not is the canonical form, and the failure it exposes is the most dangerous one.
• An ambiguity set. Ten questions the corpus answers more than one way. Correct behavior names the ambiguity rather than picking.

Ninety questions, a few days of analyst time, reusable forever. It will tell you more about deployability than any vendor benchmark, and it becomes the regression check that catches the day a model update makes the system more willing to guess.

The objection worth taking seriously

Experienced research leaders raise a fair point: analysts check everything anyway, so does the fabrication risk actually bite?

It bites in two places. First, checking is expensive, and if every figure must be verified by hand then the system has saved drafting time and added verification time, which is often a wash. The value of grounding is precisely that it removes the need to check the arithmetic.

Second, and more seriously, checking degrades against fluency. A number that appears in a well-constructed sentence, in the house voice, alongside three other numbers that are correct, does not attract scrutiny. That is not a discipline failure. It is how attention works, and it is why the control has to be structural rather than procedural.

What a build like this actually costs

Budgeting conversations go badly when the entity layer is discovered late, so it is worth naming the shape of the spend up front rather than after the first milestone slips.

The dominant cost is not compute and it is not model licensing. It is the analyst time required to adjudicate the resolution decisions the system cannot make confidently on its own — the ambiguous issuer matches, the methodology-change boundaries, the restatements that look like errors and are not. That time is unavoidable and it is what converts a text pile into a fact store.

The second cost is the evaluation set described above, which needs analysts rather than engineers to build and needs refreshing as the corpus grows. Organizations that treat it as a one-off deliverable end up with a benchmark that quietly stops describing the system in production.

The engineering itself — retrieval, query execution, the language interface — is the smallest line and the one most vendors quote against. That asymmetry is why quoted timelines and actual timelines diverge so reliably in this category.

Build, buy, or partner

The three options are not equivalent and the right answer depends almost entirely on how distinctive the corpus is.

The failure column matters more than the fit column. Every one of these can work; each fails in a specific, predictable way, and knowing which failure you are exposed to is more useful than a comparison of feature lists.

Who has to own it afterward

A fact store is not a delivered artifact. It is a living asset that degrades if nobody maintains the entity resolutions as the corpus grows and the methodology changes.

The organizations that get durable value from this assign an owner — usually someone from the research side rather than engineering, because the judgment calls are domain calls. That person adjudicates ambiguous matches, approves methodology-boundary rules, and owns the evaluation set. It is not a full-time role after the first year, and without it the system's accuracy drifts in a way nobody notices until a number is disputed.

This is worth deciding before the build rather than after, because it changes what gets built. A system designed for an owner has adjudication queues, override records, and a review interface. A system designed without one has none of those and cannot acquire them cheaply later.

Frequently asked questions