The claim that has to survive
Every education or workforce analytics project ends in the same room. Someone asks whether it worked, and the answer has to hold up in front of a person whose job is to doubt it: a state auditor, a legislative fiscal office, a school board, a program officer at the Employment and Training Administration, or the independent evaluator a grant required. By the time that question arrives, the measurement decisions have already been made. They were made months earlier, usually by whoever wrote the data-sharing agreement, and usually without anyone realizing a decision was being made at all.
We work these systems from the measurement side first. Before any modeling, our engineers write down the exact sentence the agency wants to be able to say at the end, then work backward: which records support it, who owns them, which legal exception permits the transfer, what the clock is, and what the comparison is. If that chain breaks anywhere, no amount of model quality closes the gap. A district can run a very good classifier and still be unable to say anything defensible about whether the program helped.
What follows is the chain: student and participant records, the privacy rules that shape architecture, the longitudinal plumbing that makes an outcome observable, what a valid outcome measure requires, why early-warning systems keep failing on rare events, and how the federal evidence clearinghouses decide what counts as proof.
Where outcome claims break — relative frequency in program reviews
Editorial weighting from public evaluation reports and practitioner reading — illustrative, not a measured statistic.
What the records actually are
On the education side, the working corpus is the student information system: enrollment and mobility, daily attendance, course marks by term, state assessment results, discipline referrals, English learner status, special education status under IDEA, and free or reduced-price lunch eligibility (increasingly set by direct certification rather than by application). Each has a custodian, a refresh cadence, and a history of definition changes nobody documented.
On the workforce side, the corpus is a state case-management system holding the elements of the WIOA Participant Individual Record Layout, a federal reporting schema running to several hundred fields: program entry, service records by date, credential and training provider, barriers to employment, exit, and follow-up. Employment and earnings come from somewhere else entirely, the quarterly unemployment insurance wage records held by the state workforce agency.
Those two corpora do not share a key. Wage records are keyed on Social Security number, and many K-12 and postsecondary systems deliberately do not hold SSN. The join is the whole ballgame in a P20W project, and it is a legal question before it is an engineering one.
FERPA is an architecture constraint, not a checkbox
FERPA (20 U.S.C. § 1232g, regulations at 34 CFR Part 99) sets consent as the default for disclosing personally identifiable information from education records, then carves out exceptions. Three carry almost all the weight in an analytics project, and picking the wrong one is the most common structural error we see.
The school official exception, 34 CFR § 99.31(a)(1)(i)(B). A contractor can be treated as a school official if it performs a service the agency would otherwise use employees for, is under the agency's direct control with respect to the use and maintenance of education records, and complies with the redisclosure limits at § 99.33(a). "Direct control" is where vendor AI arrangements fail. Training a general-purpose model on a district's records is not the service the district outsourced, and a terms-of-service clause reserving product-improvement rights breaks the exception outright. Agency counsel reads that clause.
The studies exception, § 99.31(a)(6). Disclosure to an organization conducting a study for or on behalf of the agency, to develop or validate predictive tests, administer student aid, or improve instruction. It requires a written agreement naming purpose, scope, and duration, restricting access to representatives with legitimate interests, and specifying a destruction date.
The audit or evaluation exception, § 99.31(a)(3) with § 99.35. Disclosure to an authorized representative of a state or local educational authority for the audit or evaluation of a federal- or state-supported education program. This is the exception that makes statewide longitudinal linkage legal, and it also requires a written agreement with designation, purpose, scope, and destruction terms.
De-identification is a test, not a field-deletion pass
Under § 99.31(b), releasing de-identified records requires a reasonable determination that a student's identity is not personally identifiable, including through single-out from small cells or other information available to the recipient. A linking code is permitted if it is not derived from student PII and the recipient does not hold the key. In practice: suppression thresholds on every published table, and a documented key-custody arrangement, both decided before the extract is built.
FERPA carries no private right of action (Gonzaga University v. Doe, 536 U.S. 273 (2002)); enforcement runs through the Department of Education's Student Privacy Policy Office, and the ultimate lever is federal funding. That does not make it soft, because state law layers on top with teeth: New York Education Law § 2-d, Illinois' Student Online Personal Protection Act, and California's SOPIPA (Cal. Bus. & Prof. Code § 22584) add contract terms, breach-notification duties, and bans on advertising and profiling uses. If any users are under 13, COPPA (15 U.S.C. § 6501; 16 CFR Part 312) applies as well.
Longitudinal data systems: the plumbing that makes an outcome observable
Statewide longitudinal data systems exist because of the SLDS grant program, authorized by the Educational Technical Assistance Act of 2002 (Title II of P.L. 107-279) and run by the National Center for Education Statistics since 2005. The America COMPETES Act of 2007 named twelve elements such a system should contain, including a unique statewide student identifier, transcript and assessment data, the ability to match records into postsecondary systems, and an audit capability. Most states have the elements. Fewer have the governance.
Governance is the constraint. A P20W linkage needs an interagency agreement, a body that can approve a research question, a defined match methodology, and a disclosure-review process. Our team plans that timeline explicitly, because it usually runs two to three times longer than the engineering.
Wage records deserve their own paragraph. They carry quarterly earnings by employer account and little else. No hours, occupation, job title, or full-time status, which means a reported "wage gain" is not a wage rate and never was. They miss the self-employed, independent contractors, federal civilian and military employees, and anyone working across a state line unless the state participates in the State Wage Interchange System agreement or obtains federal records separately. Write the measure the way it is actually computed: an employment rate is the share of exiters found in a covered wage record in a given quarter. Written that way, no auditor is surprised later.
What a valid outcome measure requires
An outcome measure that survives review has six properties, each fixed in writing before data collection, because each can be quietly tuned afterward to produce a better number.
A named population with an unambiguous entry rule. Who counts as a participant, on what date, based on what triggering service. Ambiguity here moves headline rates by several points without anyone touching the program.
An exit definition with a clock. WIOA fixes this: exit is the last date of service after ninety consecutive calendar days with no service and none scheduled. Every downstream quarter counts from there. If a program invents its own exit rule, its numbers stop being comparable to anything.
A source of record with documented coverage. Name the system, the refresh cadence, and the population it misses.
A pre-specified observation window. Second quarter after exit is a different measure from fourth quarter after exit, and program effects often reverse between them.
A missing-data rule written in advance. Missing counts as failure, or is excluded from the denominator. Pick one before seeing the data, and report both if the choice moves the result.
A counterfactual. What would have happened otherwise. Everything above is bookkeeping; this is the only part that turns a number into evidence.
WIOA's six indicators, and what each one hides
Section 116 of the Workforce Innovation and Opportunity Act (P.L. 113-128), with regulations at 20 CFR Part 677 and the joint Education rules at 34 CFR Parts 361 and 463, defines the primary indicators every state and local board reports.
| Indicator | Clock | What it does not capture |
|---|---|---|
| Employment rate, Q2 after exit | 2nd quarter post-exit | Hours, job quality, whether the job relates to the training |
| Employment rate, Q4 after exit | 4th quarter post-exit | Same gaps, plus a full year of attrition from the wage-record match |
| Median earnings, Q2 after exit | 2nd quarter post-exit | A quarterly total, not a wage rate; part-time and full-time look alike |
| Credential attainment | Within 4 quarters of exit | Labor-market value of the credential; issuer quality |
| Measurable skill gains | During the program year | Whether the gain persists or transfers |
| Effectiveness in serving employers | Program year, state-selected approach | Comparability across states, since approaches differ |
The statute is honest about its own limits. Section 116(b)(3)(A)(viii) requires a regression-based statistical adjustment model that resets negotiated performance levels for participant characteristics and economic conditions. Congress wrote into law that raw rates are not comparable between boards. Any product that ranks providers or offices on unadjusted rates is ranking caseload composition.
There is a harder lesson underneath. Research on the Job Training Partnership Act performance standards found that short-run measured performance had little relationship, and in some specifications a negative one, with longer-run experimental earnings impacts. A model that raises a measured rate by shifting who gets enrolled has improved nothing. Any indicator can be gamed by selection unless the population rule is fixed first.
Why early-warning systems fail when the base rate is low
Dropout, disconnection, and credential non-completion are rare at the individual level, and rare events break the intuition people carry about accuracy. Take a model with 80% sensitivity and 90% specificity, which sounds strong in a status meeting, and vary only the base rate.
| Base rate of the outcome | Flags per 1,000 students | Share of flags that are correct |
|---|---|---|
| 20% | 240 | 67% |
| 12% | 184 | 52% |
| 8% | 156 | 41% |
| 4% | 128 | 25% |
| 2% | 114 | 14% |
Nothing about the model changed across those rows. At a 4% base rate, three of every four flagged students are false positives, and the system produces 128 flags per thousand against a counseling capacity that might support forty. The threshold is wrong no matter how good the AUC looks, because AUC is a property of the whole ranking and the school operates at one cut point.
The fix is to report the metric that matches the operation: precision at capacity. If the response can serve the top forty students per thousand, evaluate the top forty. That number is smaller, less flattering, and the only one a principal can act on.
The comparison that keeps this honest is a simple rule. The Chicago Consortium on School Research's freshman on-track indicator, built on course failures and credits earned, separated four-year graduates from non-graduates far better than test scores did. Attendance, behavior, and course performance carry most of the predictive signal in K-12. Our engineers benchmark every model against that three-variable rule first, because a transparent rule staff will act on beats a stronger model that produces flags nobody trusts.
The system changes the thing it predicts
Once counselors act on flags, the observed outcomes are no longer the outcomes the model predicted. Evaluating a deployed early-warning system against realized graduation rates conflates prediction quality with treatment effect, and it makes a good system look bad: the students it flagged got help, so they graduated, so the model looks wrong.
Two designs separate the two. Hold the model constant and randomize the response, so some flagged students get the outreach and some get current practice. Or use the threshold itself: students just above and just below the flag cut are alike in everything except the flag, which gives a regression-discontinuity estimate of what the response did. That second design is often available in data an agency already holds.
Two habits belong here. Validate temporally, training on earlier cohorts and testing on a later one, never on a random split of rows within a cohort, which leaks cohort-level shocks into the test set. And check calibration and error rates by subgroup, knowing that when base rates differ across groups it is mathematically impossible to equalize calibration and error rates at once. Document which property you hold equal, and why.
Evidence standards decide what counts as proof
Federal funders do not evaluate claims informally. Two frameworks govern most of this work.
ESEA as amended by ESSA, at Section 8101(21)(A), defines four tiers: strong evidence from at least one well-designed and well-implemented experimental study; moderate evidence from a quasi-experimental study; promising evidence from a correlational study with statistical controls for selection bias; and a fourth tier requiring a logic model informed by research plus an ongoing effort to study effects. School improvement funds under Section 1003 require one of the top three.
The What Works Clearinghouse, run by the Institute of Education Sciences, applies the operational rules. Studies are rated Meets Standards Without Reservations, Meets Standards With Reservations, or Does Not Meet Standards. Trials with high attrition can still qualify with reservations if baseline equivalence is established, with the boundary at a quarter of a standard deviation and statistical adjustment required above 0.05. Multiple comparisons within an outcome domain get a Benjamini-Hochberg correction, and standard errors must account for clustering when assignment is at the school level and analysis at the student level. On the labor side, the Department of Labor's Clearinghouse for Labor Evaluation and Research (CLEAR) applies its own high, moderate, and low causal-evidence ratings.
The practical instruction is short. Find out which clearinghouse the funder reads, then design to that handbook. Retrofitting after data collection almost never works, because the failures are usually attrition and baseline documentation, and neither can be fixed later.
Most pilots cannot detect the effect they promise
For a two-arm trial with equal allocation, 80% power, and a 5% two-sided test, the minimum detectable effect is roughly 2.8 divided by the square root of the per-arm sample, in standard-deviation units. At 300 per arm, about 0.23 standard deviations. At 1,000 per arm, about 0.125. Clustering makes both worse, since a study randomizing thirty schools has effective power closer to thirty units than to the student count.
Set that against what education interventions actually produce. Syntheses of causal studies place most standardized-achievement effects well under 0.20 standard deviations, with a large share between 0.05 and 0.15. A 600-participant pilot is powered to detect an effect roughly twice the size of a genuinely good program. It will usually return a null, and that null will be read as failure.
There are honest responses. Pool cohorts and pre-commit to the pooled analysis. Choose a proximal outcome with more variance and a tighter mechanistic link, such as course completion rather than a distal earnings measure. Or declare the pilot a feasibility study up front and pre-register the confirmatory design that follows. Reporting an underpowered null as evidence of no effect is the one option that is not defensible.
Building the measurement plan before the model
Sequence we run before any modeling work starts
Before you sign the data-sharing agreement
- Name the FERPA exception in the document, and confirm the facts actually satisfy it
- State purpose, scope, and duration; attach a destruction date, not a destruction intention
- Bind redisclosure under 34 CFR § 99.33(a), including to subcontractors and hosting providers
- Prohibit training general-purpose models on the records unless the agency grants it explicitly
- Set suppression thresholds and small-cell rules for every table the project will publish
- Name who holds the crosswalk key for de-identified extracts, and confirm the analyst does not
- Define the wage-record match rule and what a non-match means in the denominator
- Stand up the disclosure log required by 34 CFR § 99.32 on day one, not at audit time
Bottom line
Education agencies and workforce boards are short on measurement designs that survive an auditor, an evidence clearinghouse, and a legislature. The work that makes an AI system valuable here sits upstream of the model: fixing the population, naming the source of record and its gaps, choosing the legal instrument that permits the linkage, pre-registering the outcome, and building a comparison that answers the question actually being asked.
Precision Federal builds these systems for education agencies, workforce boards, and the primes serving them. Our engineers do the pipelines and the modeling; our domain specialists write the measurement design and the data-sharing instruments that make the results usable. We would rather deliver a smaller claim that holds than a larger one that does not.
Frequently asked questions
Not under the school official exception at 34 CFR § 99.31(a)(1)(i)(B), which requires the vendor to be under the agency's direct control with respect to use and maintenance of education records and to use the information only for the outsourced service. A product-improvement or model-training clause in a terms-of-service document breaks that condition. If an agency wants to permit it, that has to be an explicit, separately negotiated term.
Because they are different measures. Employment in the second quarter after exit captures short-term placement; the fourth quarter captures retention after a year, and the wage-record match degrades as participants move across state lines or into work that unemployment insurance does not cover. Program effects can reverse between the two windows, so both should be reported with the window named.
Accuracy is the wrong frame at low base rates. What matters is precision at the capacity of the response: if staff can serve forty students per thousand, evaluate the top forty on that list. A model with strong overall discrimination can still put three false positives in front of a counselor for every true one, which is a scheduling problem before it is a statistics problem.
It depends on the program. ESEA Section 8101(21)(A) defines four tiers, and school improvement funds under Section 1003 require one of the top three. Confirm which clearinghouse the funder consults, the What Works Clearinghouse or the Department of Labor's CLEAR, and design against that handbook from the start.
Sometimes, with weaker claims. Credential attainment, skill gains, and program completion are observable inside agency systems; earnings and employment generally are not, which is why the linkage drives the schedule. When a linkage will not be approved in time, we design around a proximal outcome and say plainly what the study can and cannot conclude.