RAG is the wrong tool when your data is structured

Retrieval-augmented generation earned its reputation on unstructured text, where it works well. The failure begins when the same pattern is pointed at a database, an ownership graph, or a registry — the places where most enterprise value actually lives.

What retrieval does and does not do

RAG chunks a corpus, embeds the chunks, finds the ones nearest the question, and puts them in the prompt. The model then writes an answer from what it sees.

For a policy question answered by a paragraph, that is exactly right. For a question answered by a calculation, it is a trap: the retrieved chunks contain numbers, and the model performs the arithmetic in its head. Nothing computed anything.

Three questions retrieval cannot answer

• Total exposure across a filtered set of subsidiaries — requires aggregation, not recall
• Who ultimately controls this entity through five layers — requires traversal, not similarity
• Which of these records changed after a given date — requires a predicate, not a nearest neighbor

Each will produce an answer. The answer will be fluent. Whether it is correct is a coin flip weighted by how many of the right chunks happened to be retrieved.

The architecture that fits

Put the records in a system built to answer questions about records — relational for tables, graph for relationships. Give the model a set of deterministic operations it can call. Let it choose which to call, and in what order.

Retrieval makes the model read about the answer. Execution makes it compute the answer. Only the second one can be audited.

Where retrieval still belongs

Retrieval is the right tool for pulling the source document behind a structured fact — the filing, the contract, the report that a record summarizes. Even there, return the document with its identifier and do not let the model paraphrase figures out of it.

A mature system usually contains both. The discipline is knowing which question each one is allowed to answer.

How to tell which one you were sold

Ask the vendor to show you, for a single numeric answer, the exact operation that produced it and the records it touched. A system built on execution can produce that instantly, because it is a byproduct of how the answer was made.

A system built on retrieval will offer you the source chunks instead. Those are the inputs to a guess, not the derivation of a result.

Frequently asked questions

Is vector search useless then?
Not at all. It is excellent at candidate generation — finding things that might be relevant out of millions. The error is letting a similarity score decide a fact.

What about newer agentic RAG patterns?
Those move toward tool execution, which is the right direction. The question to ask is still the same: when a number appears in the output, what computed it?