What labs and codes cannot see
A claims record shows that a visit happened. A lab result shows a value on a given day. Neither shows that the person spent the following three days unable to work, or that their fatigue now makes a grocery trip a planned event. For many conditions, including chronic pain, fatigue-dominant illness, mental health conditions and the long tail of recovery after acute illness, that unbilled, unmeasured experience is most of the burden. Patient-reported outcomes are the only data that capture it directly.
A patient-reported outcome is any report of health status that comes straight from the patient, without interpretation by a clinician. When it is collected with a validated instrument, it is a measurement in the full sense, with known reliability and a known scale. That makes it usable in models as an outcome, a feature or a label.
It also brings problems that clinical data does not have: who chooses to respond, how people's internal standards shift over time, and how to govern data that people typed into a phone at home. Each is manageable if it is planned for, and each will quietly bias a model if it is not.
The issues in one table
Every PRO project runs into the same short list of problems. This is the list, with the mitigation we apply to each.
| Issue | What goes wrong | Mitigation |
|---|---|---|
| Homemade questions | Scores with unknown reliability and no reference scale; results cannot be compared or trusted | Use validated instruments; adapt only with documented testing |
| Who responds | The sickest and the busiest stop answering, so the data look healthier than the population | Track response by subgroup; model dropout; run sensitivity analyses for data missing not at random |
| Response shift | People recalibrate what a 5 out of 10 means as they adapt, so change scores mislead | Anchor items, periodic retrospective ratings, and caution with long-horizon change scores |
| Mode and timing | App, phone and paper versions give different scores; time of day and recall period change answers | Keep one mode per analysis or test equivalence; fix the schedule and recall window |
| Linkage | Matching app data to claims or records exposes identity and can mismatch people | Consent that names the linkage; privacy-preserving matching; match quality reported |
| Free text | Diaries are rich but unstructured; models can add findings that were never written | Structured extraction with quoted evidence, abstention and human review |
Validated instruments, not homemade questions
The first decision is the instrument, and the right answer is almost always an existing validated one. Writing your own questions feels faster. It produces scores whose reliability, meaning and sensitivity to change are unknown, and no reviewer will accept them as a measurement.
PROMIS, the Patient-Reported Outcomes Measurement Information System developed with NIH support, is the most widely used family for general health domains. Its item banks cover physical function, fatigue, pain interference, sleep disturbance, anxiety, depression, social participation and more, calibrated with item response theory. Scores are reported as T-scores with a mean of 50 and a standard deviation of 10 relative to a reference population, which for many adult domains is the U.S. general population. A score of 60 on fatigue means one standard deviation worse than that reference, whatever the condition.
Two features make PROMIS particularly useful for models. Because every item sits on a common calibrated scale, short forms and computer adaptive tests produce comparable scores, so a five-item weekly check-in and a longer quarterly assessment land on the same metric. And because the scale is shared across conditions, models can compare burden across populations without converting units.
Condition-specific instruments still matter where they capture something general domains miss. The rule is to choose validated measures for the domains your model needs, document the version and recall period, and avoid changing wording without testing the change.
Who responds, and who stops
Missing data is the central technical problem in PRO work, and the usual assumptions fail. Most standard methods assume data are missing at random given what you have observed. With patient-reported data, people often stop responding because of the very thing you are measuring: a flare, a hospitalization, exhaustion, or a bad week. That is data missing not at random, and it makes the observed data look better than reality.
When a symptom diary goes quiet, the silence is data, and usually bad news about the person who stopped writing. Start by treating missingness as an outcome. Track response rates over time by age, language, disability, condition severity at baseline and access to technology. A response rate that falls faster among people with worse baseline scores tells you the remaining data are drifting toward the healthy.
Then model it. Multiple imputation under missing-at-random assumptions is a reasonable baseline. Pair it with sensitivity analyses that assume missing values were worse than imputed, by a stated amount, and report how far that assumption has to go before conclusions change. Where the design allows, a brief fallback check-in, a single item or a phone contact, recovers information from people who cannot complete the full instrument.
Response shift
People adapt. Someone who has lived with a condition for two years may rate the same level of fatigue lower than they did at onset, because their internal standard has moved, their priorities have changed, or their sense of what the question means has changed. This is response shift, and it can make a stable or worsening condition look like improvement in change scores.
It cannot be eliminated, but it can be managed. Anchor questions that ask about concrete activities, such as how many hours a person could work or whether they could climb a flight of stairs, are less prone to shift than global ratings. A periodic retrospective rating, asking people to rate their earlier state from where they stand now, can reveal recalibration. And models that predict long-horizon change from PRO scores should be evaluated against more objective anchors where any exist.
Collecting through apps
Smartphone collection makes frequent, in-the-moment measurement possible, which is exactly what conditions with fluctuating symptoms need. It also introduces design choices that change the data.
Mode matters: the same instrument can score differently on paper, phone interview and app. Professional good-practice guidance on moving instruments to electronic formats calls for testing proportional to how much the presentation changes. Timing matters: a symptom rated in the evening differs from one rated in the morning, and a seven-day recall differs from today. Fix the schedule and recall period and record the timestamp of each response.
Burden matters most. Every additional item lowers completion, and completion falls fastest among the people with the most severe illness. A short weekly measure that people complete beats a comprehensive daily one that the sickest abandon. Design for accessibility from the start: readable text sizes, low cognitive load, languages your population uses, and a way to complete check-ins on a bad day.
Linking PROs to claims and clinical records
PRO data become most valuable when linked to claims or electronic health records, which add diagnoses, utilization, cost and clinical measures. That linkage is also where identity risk concentrates.
Use privacy-preserving record linkage where possible: identifiers are transformed into tokens on each side, and only tokens are matched, so neither party sees the other's raw identifiers. Report match quality, including the share of records matched and estimated false match rates, because a mismatched person contaminates both their own record and the analysis.
Linkage should happen only with consent that names it specifically. A participant who agreed to track symptoms in an app has not agreed to have that data joined to their insurance claims unless the consent said so in plain words.
Consent and data governance
PRO data sit across several legal regimes at once, and the regime depends on who collects the data and why. Data collected by a health plan or provider is generally covered by HIPAA. Data collected by a consumer app that is not working on behalf of a covered entity generally is not, although the FTC's Health Breach Notification Rule can apply. Federally funded research falls under the Common Rule and IRB oversight. Federal agencies collecting identifiable data are subject to the Privacy Act. Programs should establish which apply before collection begins, because consent and governance have to be designed around them.
Whatever the regime, these are the elements we require before PRO data enter a model.
- Consent in plain language that names each use: care, research, model training and any linkage.
- A separate, revocable choice for any use beyond the participant's own care or the stated study.
- Data minimization: collect only the items the analysis needs, and say why each is collected.
- A named data steward and a written access policy with logged access.
- Separation of identifiers from responses, with linkage done through tokens.
- Retention and deletion rules, including what happens when consent is withdrawn.
- Participant-facing results: a commitment to share what was learned with the people who contributed.
The last item is not legally required. People who see what their data produced have a reason to keep responding, and response is the whole asset.
Using PROs in models
PRO data enter models in three roles, and each needs different handling.
As an outcome. The model predicts a future PRO score or a meaningful change in one. Define meaningful change in advance using published minimal important differences for the instrument where they exist, and account for the dropout pattern in how you evaluate.
As a feature. Recent PRO scores and their trajectory predict utilization, work loss and clinical events. Treat missing check-ins as informative features, not as gaps to fill silently.
As a label. PRO data can supply the ground truth for conditions poorly captured by codes, for instance identifying people with persistent, functionally limiting symptoms. A label built this way inherits the response bias of the people who answered, so check whether the labeled population resembles the population the model will score.
Frequent check-ins also change what a model can learn. With daily or weekly scores, the useful signal is often within a person rather than between people: the size of a dip after exertion, how long recovery takes, how variable a week is. Mixed-effects and other longitudinal models capture that structure; a model that averages each person into one row throws it away. Keep the timestamp of every response, and build features from trajectories, not snapshots.
Language models and free-text symptom diaries
Free-text diaries hold the richest information in a PRO program and the hardest to analyze. Language models can turn them into structured fields, such as symptom, stated severity, timing and trigger. They can also produce a tidy record of things the person never wrote, which is worse than no record at all. The design goal is extraction that is checkable at every step.
Structuring diaries without inventing findings
The verbatim quote check is the control that matters. It turns every extracted field into a claim that can be verified mechanically, so a hallucinated symptom fails automatically instead of reaching a model. Abstention matters nearly as much: a model that is allowed to say not stated is far less likely to fill the field with a guess. Evaluate extraction separately for entries written in other languages, by people using dictation, and by people writing on bad days in fragments, because those are where accuracy drops.
Evaluating a model built on PROs
Evaluate a PRO-based model on the people who will actually be scored, not only on those with complete data. Report performance separately for consistent responders and intermittent responders, and by the subgroups the program serves. Check calibration as well as discrimination, because a model that ranks well but overstates risk generates follow-up work nobody planned for.
Then validate against something outside the PRO data where possible: subsequent utilization, work status, or clinician assessment. A model that predicts future PRO scores well but not any outcome outside the app may be learning response habits rather than health.
Watch for leakage specific to PRO data. If the outcome is next month's fatigue score and a feature is this week's fatigue score, the model will look excellent and teach you little; report a baseline that simply carries the last score forward, and judge the model by how much it beats that. Split training and test data by person, not by response, so the same participant's habits do not appear on both sides of the evaluation.
Frequently asked questions
Because for many conditions the burden that matters most, such as fatigue, pain interference, lost function and lost work, never appears in a lab value or a billing code. Validated patient-reported measures capture that burden directly and on a known scale.
PROMIS is a family of validated patient-reported outcome measures developed with NIH support. Its item banks are calibrated with item response theory and scored as T-scores with a mean of 50 and a standard deviation of 10 relative to a reference population, so short forms and adaptive tests produce comparable scores.
Assume the data may be missing because of the person's health, not at random. Track response by subgroup, use multiple imputation as a baseline, and run sensitivity analyses that assume missing values were worse, reporting how far the assumption must go before conclusions change.
Yes, with controls. Use a fixed schema with a not-stated option, require a verbatim quote for every extracted field, reject fields whose quote is not in the entry, never infer what was not written, and review a sample by hand.