A number nobody can go outside and check
Most analytics work produces a number somebody could in principle verify by looking. Revenue last quarter. Units shipped. Sessions per user. Cost-of-illness work does not. When a client publishes that a condition costs the United States $47 billion a year, no meter anywhere recorded that figure. It was constructed: a prevalence estimate multiplied by a per-case cost, plus productivity losses derived from a wage series, plus caregiver time valued by a rule somebody chose, summed across categories that were defined by the analyst. Change three defensible assumptions and you get $31 billion or $68 billion, and every version is honest.
That property is not unique to health economics. It shows up wherever a client sells a constructed quantity. A ratings firm publishing a risk index. A manufacturer computing cost-per-failure to justify a maintenance program. An insurer estimating loss avoided. A software company reporting the productivity gain its product delivers. A defense supplier calculating readiness impact. In each case the deliverable is a number that exists only inside a method, and the buyer of that number will eventually be asked where it came from by someone who does not want it to be true.
The engineering problem this creates is different from a normal data problem. Accuracy is not available as a defense, because there is no ground truth to be accurate against. What is available is defensibility: the number reproduces, the assumptions are visible and individually arguable, the sensitivity of the result to each one is published, and a competent stranger can rebuild it. Firms that treat this as a writing exercise at the end lose the argument. Firms that treat it as an architecture requirement at the start keep publishing for years.
What Actually Sinks a Published Estimate — Our Ranking From Rebuild Work
Editorial weighting from rebuild and remediation work. Illustrative of where estimates fail, not a measured statistic. The last row is what teams worry about most.
The three questions that decide the whole design
Before any code, three decisions get written down, because every downstream choice inherits from them and changing one later invalidates the series.
First, prevalence or incidence. A prevalence approach counts all costs incurred in a single year by everyone who has the condition, which answers what it costs the economy this year. An incidence approach follows everyone diagnosed in a year across the remaining course of their illness, which answers what one new case is worth preventing. These produce very different numbers for the same disease and they answer different questions. Programs that want a budget-impact argument need the first. Programs evaluating an intervention need the second. Mixing them inside one model is the single most common structural defect we find.
Second, the perspective. Societal, payer, provider, employer, or government. The perspective determines which costs count at all. Sick leave is a real cost from a societal or employer view and invisible from a payer view. Out-of-pocket spending is a cost to households and a revenue line to providers. A model with no declared perspective will silently take the union of all of them, which double counts, and the double counting is nearly impossible to find after the fact because it hides inside category definitions rather than in the arithmetic.
Third, the treatment of indirect costs. Direct medical costs are the easy part. The fights are about productivity. The human capital method values lost work time at the person's expected earnings across the whole period they are unable to work, which produces large numbers and treats a permanently disabled 30-year-old as an enormous loss. The friction cost method values only the period until an employer replaces the worker, typically a few months, which produces numbers several times smaller. Neither is wrong. Publishing without saying which one you used, and without showing the result under both, is what gets an estimate attacked.
Where the double counting hides
Double counting is the defect that costs the most credibility per line of code, because it looks like rigor. A model with fourteen cost categories reads as thorough. It is also fourteen chances for the same dollar to appear twice.
The recurring patterns are specific enough to check for directly. Emergency department costs counted separately from inpatient costs when the visit led to an admission whose claim already bundles it. Prescription costs pulled from a pharmacy file and again from a total-cost-of-care field that includes pharmacy. Caregiver time valued as an indirect cost while the same hours appear in a home health line billed to a payer. Absenteeism and presenteeism drawn from two survey instruments whose recall windows overlap.
The engineering fix is a reconciliation layer that is part of the pipeline, not a review step. Every dollar that enters the model carries a source identifier and a category tag, the two are enforced as a unique pair per person-period, and the build fails if any person-period accumulates spending from two sources that a declared overlap matrix says are non-additive. That matrix is a real artifact, maintained by the domain lead, and reviewed when a data source changes. It is perhaps two hundred lines of configuration and it prevents the failure mode that most often forces a public correction.
Reproducibility has a precise meaning here
An estimate published in March will be questioned in November, often by someone comparing it against a competing figure from another group. The only useful response is to reproduce it exactly, then explain the difference. Teams that cannot do the first cannot do the second, and the conversation ends badly.
Reproducing a past estimate means recovering the exact source extracts as they stood at the moment of the run, the exact code version and the exact parameter set, and getting the identical number to the dollar. That is harder than it sounds because the inputs move underneath you. Federal survey files get reissued with corrections, claims databases restate as late claims arrive on a six to eighteen month lag, wage series revise and price indices rebase. A pipeline that queries live sources will not reproduce its own output six months later.
What works is unglamorous. Snapshot every input file at run time and store it content-addressed, so the file is identified by a hash of its bytes rather than by a path someone can overwrite. Pin every parameter, including the ones that feel like constants, in a versioned configuration file that is committed alongside the code. Store the run manifest with the published number: input hashes, code commit, parameter version, execution date, and the person who approved release. Retention has to outlast the argument, which for a published figure means five to ten years, not one.
Sensitivity analysis is the deliverable, not an appendix
A single point estimate invites a fight about whether it is right. A point estimate with a properly built sensitivity analysis changes the conversation to which assumptions a reader prefers, and that is a conversation the analyst wins because the answer under the reader's own preferred assumptions is already computed and printed.
Three layers do the work. A one-way analysis varies each input across its plausible range while holding others fixed, and produces a tornado diagram ranking inputs by how much the headline moves. This is the fastest way to find out that 70% of the variance in a $47 billion figure comes from one productivity multiplier, which is worth knowing before publication rather than after. A scenario analysis fixes coherent bundles of assumptions, conservative and central and expansive, because inputs are not independent and varying them one at a time understates the range. A probabilistic analysis assigns distributions to uncertain inputs and runs a Monte Carlo, typically 10,000 draws, producing a credible interval around the headline.
The probabilistic layer is where teams overreach. A credible interval is only as meaningful as the distributions fed into it, and analysts routinely assign a tidy normal distribution to a parameter whose real uncertainty is a disagreement between two literatures rather than sampling noise. When the uncertainty is structural, scenarios are the honest tool and a false confidence interval is worse than none. Say which kind of uncertainty each parameter carries and handle it with the matching instrument.
Where the Engineering Budget Goes on a Burden Model — Weight by Review Impact
Editorial weighting from our own project planning. The bottom row absorbs most of the internal debate and moves the published result least.
Attribution: what share of this cost is caused by the thing
Every burden estimate contains a causal claim wearing a costume. Saying a condition costs $47 billion asserts that $47 billion would not have been spent if the condition did not exist. People with a chronic condition also have other conditions, different age distributions, and different baseline spending, and simply totaling their claims attributes all of it to the one disease.
Two families of method handle this and both are defensible when stated. Attributable fraction methods apply an external relative risk from epidemiology to observed spending, which is transparent but depends entirely on a literature estimate the analyst did not produce. Matched-cohort methods build a comparison group without the condition, matched on age, sex, geography, comorbidity index and baseline spending, and take the difference. That is stronger on its own data, costs more to build, and still leaves unobserved confounding intact.
The design decision that matters most is where the comparison group comes from and whether the matching is auditable. Store the matched pairs, the covariate balance before and after matching, and the standardized mean differences. A reviewer who asks how you know the groups were comparable should get a table, not a paragraph. This is the same discipline that a regulator applies to a scoring model, arriving from a different direction.
The frameworks that apply, and the ones that do not
Teams building constructed estimates often ask which compliance framework governs them, and the honest answer is that no single one does, which is why the borrowed ones matter.
SR 11-7, the Federal Reserve and OCC supervisory guidance on model risk management from 2011, is written for banks and is the most useful text available for anyone whose product is a number. Its definition of a model covers any quantitative method turning inputs into an estimate, which is exactly a burden model. Its three-part validation structure of conceptual soundness, ongoing monitoring, and outcomes analysis maps directly onto this work, and its standard for documentation, that a party unfamiliar with the model can understand how it operates and its limitations and its key assumptions, is a build requirement worth adopting whether or not a supervisor will ever read it.
Where machine learning enters the pipeline, for imputation or risk adjustment or record linkage, the NIST AI Risk Management Framework supplies a vocabulary that federal and enterprise reviewers recognize, and ISO/IEC 42001:2023 is the certifiable management system if a customer wants third-party attestation of governance. Where the estimate feeds a federal deliverable, NIST SP 800-53 controls carry the authorization, SP 800-171 governs controlled unclassified information on your own systems, and FedRAMP governs any cloud service sold to an agency. Health data adds HIPAA and, for most claims sources, a data use agreement whose cell-suppression and minimum-cell-size rules constrain what the model may publish at a geographic or demographic breakdown. Design the output tables against those rules from the start; retrofitting suppression logic into a finished report is a rebuild.
What the build actually costs
A first serious burden model, built to be republished annually and defended publicly, is a six to nine month engagement for a small team. The distribution of effort surprises people. Roughly a third goes to data acquisition and conditioning, including license negotiations and reconciling a claims extract against its own documentation. About a quarter goes to the reproducibility infrastructure above. Perhaps a fifth is the estimation itself. The rest is validation, sensitivity and transferable documentation.
Data licensing is a real line item and frequently the largest one. Commercial claims databases, all-payer claims data from state agencies, and specialty registries carry annual fees ranging from the low tens of thousands into the mid six figures depending on scope, population and republication rights. Republication rights are the term to read carefully, because a license permitting internal analysis but not public dissemination of derived figures will stop a product cold after the model is built.
The second-year cost is where the architecture pays. A model built with pinned inputs, a parameter registry and an automated sensitivity harness refreshes in weeks. A model built as a chain of notebooks and spreadsheets is rebuilt every year by whoever still remembers it, at close to the original cost, with results that are not comparable across years because nobody can say exactly what changed. That comparability is often the product's actual value, and it is destroyed by the same shortcut that saved three months in year one.
Publishing a number you will have to defend
How the result is presented decides how it is attacked. A few practices separate figures that hold up from figures that get withdrawn.
Lead with the range, not the point. A headline of $38 to $56 billion with a central estimate of $47 billion is harder to dismiss than $47 billion alone, because the obvious counterattack, that the number is sensitive to assumptions, has already been conceded and quantified. State the perspective and the costing approach in the same sentence as the number, every time, including in the press summary, because the summary is what gets quoted and stripped of context.
Publish the parameter table. Every value, its source, its year, and the index used to bring it to the reporting year. This is the most effective defensive artifact there is, because most challenges resolve to a disagreement about one input, and a reader who can find that input argues about the input rather than the analyst's competence. Name the limitations in your own voice first, and be specific rather than ritual: which populations the data source undercounts, which categories were excluded and why, what the model shows if a contested assumption goes the other way.
Version the published figure itself. When a source file restates and the number moves, that is a normal event, and a client with a documented revision policy handles it as routine maintenance. A client without one faces a choice between quietly changing a published number and defending a figure they know is stale, and both of those damage the franchise more than the revision would have.
Bottom line
When the deliverable is a constructed number, the method is the asset and the number is its output. Estimates fail in production for reasons that have almost nothing to do with statistical sophistication: they cannot be reproduced, they double count, their assumptions are invisible, and their sensitivity to the one input that drives everything was never computed. Those are engineering problems with known solutions, and the solutions have to be in the architecture from the first week because retrofitting them costs more than building them and produces a weaker artifact. A team that builds this way publishes for a decade. A team that does not rebuilds every year and loses the first serious argument it has.
Frequently asked questions
A prevalence study counts all costs incurred in one year by everyone who has the condition, answering what it costs the economy annually. An incidence study follows people diagnosed in a year across the course of their illness, answering what preventing one case is worth. They yield very different figures for the same disease and should never be mixed in a single model.
Both are accepted. Human capital values lost work at expected earnings across the full period of incapacity and produces larger numbers; friction cost values only the period until an employer replaces the worker. Report the result under both, state which one drives the headline, and the choice stops being an attack surface.
Snapshot every input at run time and store it content-addressed by hash rather than by file path, pin all parameters in a versioned configuration, and store a run manifest with the published figure. Restatement then becomes a comparison between two known runs instead of an unanswerable question.
No. Probabilistic analysis is appropriate for sampling uncertainty with a defensible distribution. Where the uncertainty is a structural disagreement between methods or literatures, scenario analysis is the honest instrument and a fabricated confidence interval overstates precision.
Six to nine months for a small team on a first model intended for annual republication, with data licensing frequently the largest single line item and republication rights the contract term to read most carefully. The reproducibility infrastructure is roughly a quarter of the effort and is what makes year two cost weeks instead of months.