The symptom is disagreement, not error
Almost nobody calls us and says our reporting is wrong. They say two numbers disagree and nobody can explain the gap. The board deck says 412 active customers. The CRM dashboard says 389. The billing export says 401. Three people built those three numbers, all three are competent, and all three can walk you through their work. Nothing is broken in the sense that anything failed. The company has simply been running on three definitions of the word customer for two years, and this is the first month anyone put the slides side by side.
That distinction matters because it determines who fixes it. If a pipeline is broken, an engineer fixes it in an afternoon. If three teams are counting different things on purpose, no engineer can fix it, and hiring one will produce a fourth number. Most reporting problems we are asked to look at are the second kind wearing the costume of the first.
What follows is the set of causes we actually find, in rough order of how often we find them, and a diagnosis you can run yourself in about a week. None of it requires new software. Most of it requires one person with the authority to decide what a word means.
You are probably here because
- Two dashboards disagree and both owners can defend their number
- A monthly figure changed after the month closed, and nobody edited anything
- Revenue on the operations report is higher than revenue in the accounting system, consistently
- Someone found the error, and the number had been in front of the board for four quarters
Those are four different causes with four different fixes. The section below separates them, and the diagnosis after it tells you which one you are looking at.
Four places a business number goes wrong
Definition drift. Nobody wrote down what the metric means, so each team implemented the version that fit their job. Sales counts a customer at contract signature because that is when commission is earned. Finance counts at first invoice because that is when revenue can be recognized. Support counts anyone with an open account, including the four trials and the two subsidiaries of a customer you already counted once. All three are defensible. None of them is wrong in isolation. Put them on the same slide and the company looks like it cannot count.
A silent partial load. The nightly job that copies orders from the e-commerce platform timed out on the largest region and moved on. Twenty-two of twenty-three regions loaded. The dashboard rendered normally, with a slightly smaller number, and nobody noticed because nobody knows what the number should have been. This is the failure that survives longest, because a partial load looks exactly like a slow week.
Join fan-out. Someone joined orders to order lines to compute revenue by customer. An order with four line items now appears four times, and its revenue is counted four times. The report is not off by a random amount, it is off by a multiple, and the multiple changes as your product mix changes. This one is common enough that it deserves its own reflex: whenever a number jumps and nothing else changed, count the rows before and after the join.
Moving boundaries. The source system stores timestamps in UTC and the report groups by local calendar day, so six hours of every December 31 land in the wrong year. Or the source system lets a user backdate a record, so the figure for June computed in July is not the figure for June computed in September. Or a currency conversion uses today's rate rather than the rate on the transaction date. Nothing failed. The number is simply a function of when you asked, and nobody told the reader that.
Why nothing turns red
When a website goes down, everybody knows inside a minute, because the failure is visible to the customer and the alert is wired to a phone. Reporting has none of that. The consumer of a wrong number is an executive who has no way to know it is wrong, and the system that produced it has no opinion about whether the answer is plausible. A pipeline that loads zero rows will happily draw a zero. A pipeline that loads half the rows draws half the bar.
Three things are usually missing, and all three are cheap. There is no freshness stamp on the dashboard, so a chart built from a feed that stopped on Tuesday looks the same on Friday. There are no row-count checks, so nobody notices that Tuesday's load was 40 percent lighter than every Tuesday before it. And there is no reconciliation against a system that has its own controls, which in almost every company means the accounting system. Accounting is reconciled monthly by people whose professional standing depends on it. Your operational reporting is not, and that gap is where the errors live.
Where the differences come from — our ranking across the reporting reviews we run
Our ordering, not a published statistic. The useful part is the rank: the top two are management problems and the bottom four are engineering problems.
Definition drift is a management problem in a technical costume
This is the finding people least want. When we trace two disagreeing numbers to the row level, the most common ending is not a bug. It is that the company never decided, and the decision got made by default, three times, by three people who each needed an answer that afternoon.
Look at revenue. There are at least five honest answers to what revenue was last month: the value of contracts signed, the value invoiced, cash collected, revenue recognized under the accounting standard, and bookings net of expected churn. A software company will use all five in the same week, in the same building, in documents that all say revenue at the top. Or look at a churned customer: is it a cancellation notice, the end of the paid term, ninety days of no usage, or the finance system marking the account closed? Each definition produces a different churn rate, and the spread between the loosest and the tightest is routinely a factor of two.
No tool resolves this. A data catalog will hold whichever definition you write into it. A new BI platform will display all three inconsistent numbers faster and in nicer colors. The resolution is a person with authority saying: for company reporting, this is what the word means, and here is the other number we will also publish for the team that needs it. That takes an hour of decision and about two weeks of implementation. The hour is the hard part, and it usually waits months for a meeting.
A diagnosis you can run this week
You do not need an engagement to find out which of the four you have. You need one afternoon a day for about a week, and the willingness to accept an unflattering answer.
Day one: write down the definitions, separately. Pick your three most-quoted numbers. Ask each person who publishes one to write the definition in plain English, without conferring, including what is excluded. Ask for the exclusions explicitly, because the exclusions are where the differences live. If the three definitions of one metric differ, stop. You have found it, and none of the remaining steps matter until it is settled.
Day two: ask as of when. For each dashboard, ask what time the underlying data last loaded and what happens when the load fails. If the answer is a shrug, or if the failure behavior is that the chart shows the previous day's data with no notice, you have found a second one. A dashboard that cannot tell you its own freshness should be treated as decoration until it can.
Day three: recompute one month by hand. Take a single closed month, export the raw rows from the source system, and add them up in a spreadsheet. Not the warehouse, the source. If your hand total and the dashboard agree to the cent, that number is probably sound. If they do not, you now have a specific, bounded difference to chase instead of a general feeling of doubt.
Day four: count the rows. Ask whoever wrote the query how many rows go into the metric before the joins and how many come out. If the number goes up, revenue or counts are being duplicated. This single question finds more real defects than any tool we have deployed.
Day five: reconcile to accounting. Put your operational revenue next to the general ledger for the same period and explain every difference. Some will be legitimate, such as timing or deferred revenue, and those become documented reconciling items. The rest are defects. A company that can explain the gap between its operational report and its books, line by line, has a reporting function. A company that cannot does not yet, whatever it has built.
| Symptom | Most likely cause | How to confirm in an hour |
|---|---|---|
| Two teams disagree, both consistent over time | Definition drift | Get both definitions in writing, compare the exclusions |
| Number is too high by a clean-looking multiple | Join fan-out | Count distinct source records versus rows in the report |
| One region or product is short | Partial load, one connector failing | Row counts per segment per day for the last 30 days |
| Last month's figure changed this month | Backdated source records, or a restated definition | Snapshot the report monthly and diff two snapshots |
| Month-end totals are slightly off, every month | Timezone or day-boundary handling | Compare the last and first day of adjacent months |
| Everything is right except after the 5th | A manual step that a person performs | Ask who touches the file, and what they do when travelling |
The manual step is almost always undocumented
In a large share of the reporting reviews we run, at least one number in the executive pack passes through a human hand each month: a file downloaded, a column pasted, an adjustment applied from memory. That person is competent and the adjustment is usually right. The problem is that the adjustment exists nowhere except in their head, it is skipped when they are on leave, and it silently encodes a business rule nobody else knows about. Finding it is not an accusation. It is the single highest-value hour of the whole diagnosis.
What actually fixes it, in order
The order matters more than the content, because doing these out of sequence is how reporting projects run for a year and change nothing.
First, write the definitions and name an owner for each. One page per metric: the plain-English meaning, what is excluded, the owner's name, how often it refreshes, and the known caveats. Ten to twenty metrics is enough for most companies. Not a catalog of four thousand fields, which is a tax nobody pays after month two.
Second, make failure visible. Every report carries a last-updated timestamp in the user's own timezone. Every load writes a row count, and a load that comes in materially below its recent range raises an alert to a person, not a log file. This is a day or two of work and it converts your most persistent class of error from invisible to obvious.
Third, kill the manual steps or write them down. If a step must stay manual, it gets a written procedure and a second person who can perform it. If it can be automated, automate it, but automate it after the rule is written, not instead of writing the rule.
Fourth, reconcile on a schedule. Monthly, against the books, with differences explained in a standing document. This is the control that catches the next problem, and it is the one most companies skip because it is clerical and unglamorous and nobody's promotion depends on it.
Fifth, and only fifth, rebuild anything. New warehouse, new BI tool, new pipeline. If you do this first, you will faithfully reproduce the same wrong definitions on better infrastructure and you will have spent six figures to be confused faster.
What does not fix it
- Rebuilding the dashboard in a different BI tool, which changes the rendering and none of the arithmetic
- Buying a data catalog before writing definitions, giving you an empty catalog with a subscription
- Turning on three hundred automated quality checks, which generates alert fatigue in six weeks and is then ignored
- Hiring one engineer to own the numbers when the disagreement is between two departments, not two queries
- Refreshing more often, which makes a wrong number wrong sooner and multiplies your compute bill
- Adding a third dashboard that is meant to settle the argument between the first two
What this costs and how long it takes
An honest range, because the question always comes and the answer is smaller than people expect. A focused reporting review, meaning definitions traced, one month recomputed from source, the pipeline read end to end and a written finding, is typically two to four weeks of part-time work by one or two people. At common mid-market consulting rates, that is a five-figure engagement, not a six-figure one. Some of it you can do yourself using the week above, and we would rather you did, because a client who has already run the diagnosis is a client whose engagement is half as long.
The remediation is where the range widens, and it widens on organizational grounds rather than technical ones. If the finding is definition drift, the fix is a decision and about two weeks of query changes. If the finding is that four source systems have no shared customer identifier, you are looking at months, because somebody has to decide how to match records and then live with the match rate. We can build that. We cannot shorten the part where your commercial team decides which of two customer lists is the real one.
The uncomfortable part
When a company fixes this properly, some published numbers move. Growth that looked like 31 percent turns out to have been 26 once double-counted line items come out. A churn rate improves because trials were being counted as customers and then as churn. Somebody has to say that out loud, to a board or an investor or a lender, and the instinct is to bury the correction in a footnote or to quietly change the definition without saying so.
Restate it plainly, once, with the old and new numbers side by side and a sentence about the cause. Every experienced board member has seen this. The version they hold against a management team is the one where the definition changed silently and they noticed the trend line bending for no reason. Publish the change, keep the old series available, and the credibility cost is close to zero. Hide it, and the cost arrives later at a much worse moment.
Before you call the number fixed
- Every published metric has one written definition and one named human owner
- Every report shows when its data last loaded, in the reader's timezone
- A load that arrives materially short of its recent range alerts a person
- Row counts are checked before and after every join that could duplicate
- Operational revenue reconciles to the general ledger monthly, with differences explained
- Manual steps are written down and a second person can perform them
- Reports are snapshotted, so you can answer what the number said on a past date
- Definition changes are announced, dated, and the prior series is retained
Bottom line
The question is not whether your reporting has errors. Every reporting stack we have opened has had at least one, including ones built by strong teams with good tools. The question is whether an error can survive for four quarters without anyone noticing, and that is decided by three cheap things: written definitions, visible freshness, and a monthly reconciliation to a system that has its own controls. Put those in before you buy anything. The buying decision gets much easier once you know which of the four problems you actually have, and in a fair number of cases it turns out you did not need to buy anything at all.
Frequently asked questions
Ask the two people who publish the disagreeing numbers to write their definitions down separately, including exclusions. If the definitions differ, it is a management problem and no engineering will fix it. If the definitions match and the numbers still differ, it is technical, and the next step is to recompute one month by hand from the source system.
Write the definitions before. Build after. A warehouse is a very efficient way to reproduce whatever logic you hand it, so migrating unresolved definitions gives you the same disagreement in a new place, plus a monthly bill. The definitions work is also the part that does not become obsolete when you change platforms.
For a company with a handful of source systems, a review that traces definitions, recomputes a month from source, reads the pipeline and produces a written finding is usually two to four weeks of part-time effort. That lands in five figures at typical mid-market rates. Remediation is the variable part, and it is driven by how many source systems have to be reconciled rather than by how much data there is.
Because a dashboard is a display layer. It shows whatever query sits behind it, and different dashboards usually have different queries written by different people at different times. Agreement comes from a shared definition applied once, upstream of the display, not from putting the charts on the same page.
Yes, and say it plainly the first time, with the old and new figures side by side and a one-sentence cause. Silent restatements get noticed later and cost far more in credibility than the original error did. Keep the old series available so anyone can reconstruct what was reported and when.