Agency data modernization programs: where the specialist fits

A data modernization program is sold as one thing and delivered as forty. The prime wins on scale, past performance, staffing depth and the ability to carry a five-year schedule with a government program office attached to it. Then the work decomposes, and a handful of the forty pieces turn out to decide whether the customer calls the program a success. Those pieces are almost never the ones the bid spent the most words on. They are entity resolution, data quality engineering, pipeline reliability, the two or three analytics products people actually open, and the governance layer that keeps all of it defensible. This is written for the program director who already knows that, and is deciding whether to grow those skills inside the program team or bring a specialist in against a defined scope.

The choice is not about capability in the abstract. A large integrator can hire anyone. The question is whether a program with a running schedule, a fixed staffing plan and a customer who reads the monthly status report can afford the ramp, and whether the failure modes in those specific pieces are the kind that respond to more people or the kind that respond to different people. They are usually the second kind.

The parts of a modernization program that fail

Programs of this shape rarely fail at the layer the schedule tracks. Servers get stood up. Data gets moved. The lakehouse exists. The failure shows up eight to fourteen months in, when the agency's analysts start using the thing and find that the numbers do not match what they had before, that the same person appears four times under three spellings, that yesterday's load is missing and nobody noticed, and that no one can say who is allowed to see which column. At that point the program is not behind on a milestone. It is behind on credibility, which is much harder to recover.

Five specific failures produce most of that. Each is an engineering problem with a known shape, which is why a specialist scope works on them and a general staffing increase usually does not.

Identity across systems that were never designed to agree. An agency running a grants system, a case system, a payment system and a legacy mainframe extract has four different notions of who a person or an organization is. Names are entered by hand. Addresses are inconsistent. Identifiers exist in some systems and not others, and where they exist they are sometimes reused. Joining these into one view is entity resolution, and it is a modeling problem with measurable accuracy, not a data-cleanup task that ends.

Quality that is asserted rather than measured. Most programs write a data quality plan and then have no instrument that tells them, on any given morning, whether today's data is good. When quality is a document, the first person to notice a problem is the customer.

Pipelines that are reliable until they are not. A pipeline that runs for six months and then silently produces a partial load is worse than one that fails loudly. Partial loads propagate into published numbers, and the correction is a retraction in front of the program office.

Analytics products nobody opens. A modernization program often ships dashboards counted as deliverables and used by no one. The count satisfies the contract line item. It does not satisfy the customer, and at recompete the customer remembers which one was true.

Governance discovered at the accreditation review. Access control, lineage, retention, privacy review and the record of who approved what are cheap when designed in and enormously expensive when reconstructed under a deadline.

Entity resolution, in the detail that decides it

This is the piece most often underestimated, so it is worth being concrete about what the work is. Precision Federal builds these systems, and the shape is consistent across agencies.

Start with the record model. A resolved entity is not a row. It is a cluster of source records with a stable internal identifier, a set of attributes chosen by rule from the contributing records, a confidence value, and a full record of which source rows contributed and why. That last part is what makes the result auditable. When an analyst asks why two organizations were merged, the system answers with the specific fields that matched and the score they produced, not with a shrug.

Then blocking. Comparing every record to every other record is quadratic and impossible at agency scale. Blocking generates candidate pairs cheaply, usually with several keys running in parallel: a normalized name prefix, a phonetic encoding, a postal code plus street number, a tax or entity identifier where present. Each key has a recall cost and a volume cost, and the engineering judgment is in choosing a set whose union catches the true matches without producing a candidate set nobody can score.

Then scoring. Field comparators produce similarity values, name similarity handles nicknames and transpositions, address comparison normalizes before comparing, and dates get tolerance windows. Those comparisons combine into a score, either by a probabilistic model of match and non-match distributions or by a supervised classifier trained on adjudicated pairs. Both need labelled data, and the labelled data has to come from people who know the domain.

Then clustering and the human loop. Pairwise decisions have to become clusters, and transitivity causes trouble: A matches B, B matches C, A clearly does not match C. Graph clustering with a threshold and a split rule resolves it. Above the upper threshold, auto-merge. Below the lower, auto-reject. Between them, a review queue with an interface designed for speed, where a trained reviewer sees the two records side by side with the matching fields highlighted and makes a call in seconds. Every adjudication becomes training data.

Then measurement, which is the part that turns this from craft into engineering. Build a labelled evaluation set drawn from real records, stratified so it includes the hard cases rather than only the easy ones. Report precision and recall separately, because the cost of a wrong merge and the cost of a missed merge are different and the program office cares about that difference. On a benefits or grants system a wrong merge can put two people's records together, which is a privacy event. A missed merge is a duplicate, which is a data quality complaint. Those get different thresholds, and the threshold is a policy decision the customer should make with the numbers in front of them.

Data quality as running tests, not a document

The useful reframing is that data quality is a test suite that runs on every load and blocks publication when it fails. Concretely, that means expectations written next to the pipeline code and versioned with it: row counts within an expected band for this day of the week, null rates per column with thresholds, referential integrity between fact and dimension tables, value domains for coded fields, distributional checks that catch a source system quietly changing units or a category being retired, freshness checks that assert the newest record is recent enough to matter, and reconciliation totals against the source of record.

Severity matters as much as coverage. A failing check that stops the load has to be one worth stopping for, or the team learns to override it and the suite becomes decoration. Three tiers work: blocking checks that hold publication, warning checks that raise a ticket and publish with a flag, and observational checks that only feed a trend. Getting the tiering right is the difference between a quality system people trust and one they route around.

The output the program office wants is a single page, refreshed automatically, showing which datasets passed, which failed, what the trend is, and how long the failures took to resolve. That page is also the best possible artifact at a program review, because it converts a claim into a record.

Pipeline reliability and the failure modes that hurt

Reliability engineering for data is different from reliability engineering for services, and program teams staffed with application engineers often carry the service assumptions over. The dangerous failures are not outages. They are quiet wrong answers.

Four patterns account for most of them. Partial loads, where a source delivers eighty percent of a file and the pipeline processes it happily. Late-arriving data, where records for a prior period show up after that period was published, and nothing recomputes. Silent schema drift, where a source adds or renames a column and the transformation keeps running against the old assumption. And non-idempotent reruns, where a retried job double-counts because the load was written as an append rather than as a replace of a defined partition.

The countermeasures are unglamorous and well understood: manifest checks with expected record counts before processing, partitioned loads with atomic swap so a rerun replaces rather than adds, watermarking with defined recomputation windows, contract tests against source schemas that fail the build rather than the load, and end-to-end lineage so a wrong number can be traced to the row and the job that produced it. What a specialist brings is not the list. It is having built them enough times to get the defaults right and the alerting quiet enough that people still read it.

Four ways to staff the specialist pieces, compared

The decision is usually framed as build or buy. It is really four options, and the right one differs by piece within the same program.

How the prime scopes a specialist piece so it can be held to it

The reason specialist subcontracts disappoint is almost always scoping. A statement of work that says "support data quality activities" cannot be enforced, cannot be evaluated at the customer level, and gives the program office nothing to point at. A scope written as outcomes can do all three.

The elements that make it work are few. A named dataset or system boundary, so scope creep is visible. A measurable acceptance criterion with a number and a method, such as a precision and recall pair on a stated evaluation set produced by a stated procedure, or a load-success rate with a defined detection window. A delivery form, meaning code in the program's repository, infrastructure as code in the program's pattern, tests in the program's pipeline, and documentation in the program's format. A named person on the program side who can approve a scope change. And a cadence that fits the program's existing rhythm rather than inventing a second one.

Price shape follows scope shape. Where the outcome is definable, fixed-price milestones transfer the estimation risk to the party that controls it and give the program a predictable line. Where the work is genuinely exploratory, a committed team for a defined period with a written stopping point is more honest than a fixed price with hidden padding. Precision Federal works both ways and says which one fits before the number is discussed.

Analytics products the agency actually uses

Delivered-and-unused is the quietest failure in this business, and it is measurable if anyone chooses to measure it. Instrument the products. Weekly active users by role, which views get opened, which filters get set, where sessions end, and how many exports happen. Then read the numbers honestly at the program review.

Three design habits move that number more than anything technical. First, build to a decision rather than to a data source. A view that exists because a table exists gets opened once. A view that tells a program analyst which cases to work today gets opened every morning. Second, make the product answer the question the user already asks their colleagues, in the words they use. Third, put the freshness and the quality state on the page. A user who cannot tell whether the number is current will verify it somewhere else, and once they build that habit the product is decoration.

Accessibility belongs in this section rather than in a compliance appendix. Federal systems have to meet Section 508 requirements, and an analytics interface that fails keyboard navigation or contrast is not deliverable regardless of how well the model underneath performs. Building to it from the first sprint costs a fraction of retrofitting it after a review finding.

The governance layer, described concretely

Governance in a modernization program means five running mechanisms, not a policy binder. A catalog that is populated automatically from the pipeline rather than by hand, because a hand-maintained catalog is stale within a quarter. Lineage captured at the job level so any published number can be traced back through transformations to source rows. Access control expressed as policy over attributes, so a rule about a sensitive column is written once and enforced everywhere rather than reimplemented per tool. Retention and deletion that actually execute, with evidence. And a change record that shows who approved a schema change, a threshold change, or a new data sharing path, with the date.

Programs handling personal information also need privacy work done alongside, not after: minimization decisions recorded, purpose limits enforced in the access layer, and documentation ready for the agency's privacy office in the form it expects. NIST SP 800-53 controls provide the vocabulary the agency's security staff will use, and mapping the platform's mechanisms to those control statements early makes the accreditation conversation short.

How we work inside a program

Precision Federal is a small business engineering firm. We build AI systems, data platforms, cloud infrastructure and full-stack applications, and we deliver them into production inside U.S. federal agencies. On a modernization program we work as a specialist subcontractor to the prime, under the prime's program management, on a scope the prime defines and the customer can see.

The first weeks look the same on most programs. Week one is access and grounding: environment credentials, repository access, a working session with the program's data engineers, and reading the data as it is rather than as documented. By the end of week two we deliver a written finding on the piece in scope, with the specific defect patterns found, a measurement of the current state, and a proposed target with the method for verifying it. Weeks three through eight produce a working increment in the program's environment, in the program's repository, with tests in the program's pipeline. Not a prototype on the side. The thing itself, small.

The prime keeps everything that matters. The customer relationship stays with the prime; we do not carry a separate line to the program office unless the prime asks us to. The code, the models, the tests, the infrastructure definitions and the documentation are delivered into the prime's repositories and assigned to the prime under the subcontract. Our pre-existing tooling is named and carved out with a perpetual license so nothing we bring can block a future maintainer. Data stays where the prime and the agency say it stays, we write the handling terms before the first extract, and none of it trains anything outside the program.

Pricing takes one of two shapes. Fixed-price milestones tied to written acceptance criteria, which is our preference when the outcome can be measured. Or a committed team at a defined allocation for a defined period, with a written stopping point, when the work is genuinely exploratory. Either way the key people are named with committed percentages and a substitution path, because a staffing plan that cannot survive a schedule slip is not a plan.

The first step is one email with a one-page brief: the program, the piece in question, what the customer has already been told, the environment and who grants access, the date that matters and the contract instrument. We return a scoped, priced statement of work with acceptance criteria written as tests. No call required.

Sequencing it against the program schedule

Timing decides how much the piece costs. Brought in at the design of a data domain, a specialist scope is a design contribution. Brought in after the analysts have lost confidence in the numbers, the same work carries a recovery narrative the program has to manage in front of the customer.

The practical windows are three. Before a major data domain is designed, when the entity model and the quality expectations can still be shaped. At the point where the first analytics product goes to real users, when quality problems become visible and there is still time to fix them quietly. And roughly a year before recompete, when a measured improvement the customer has personally seen becomes evidence rather than a promise. The third window is the one capture leaders care about, and it is the one most often missed by a month or two.

Bottom line

Modernization programs are won on scale and judged on a handful of specialist pieces. Entity resolution, measured data quality, pipeline reliability, products people open, and a governance layer that holds up under review are the pieces that decide whether the customer says the program worked. Each has a defined engineering shape, a measurable acceptance criterion and a scope that can be written and enforced, which makes them exactly the pieces a specialist subcontract handles well. The program keeps the customer, the code and the credit. What it buys is speed on the parts where a ramp costs a milestone, and a named capability in the bid the next time the work goes out.

Frequently asked questions