The position most buyers are actually in
A program office, a state agency, a hospital system, or a prime's capture lead ends up holding the same document: a vendor technical volume full of numbers nobody in the building can check. The accuracy figure has no denominator. The security claim is a logo. The architecture diagram has a box labeled "AI engine." Everyone in the room knows the proposal is persuasive. Nobody in the room can say whether it is true. That is not a failure of the buyer. It is a staffing reality, and it is the normal condition in AI procurement right now.
The instinct is to solve it by asking the vendor more questions. That rarely works, because the vendor answers in the same register the proposal was written in. The second instinct is to hire an advisor who also sells AI systems, which trades one problem for a worse one. The thing that works is a narrow, bounded, technical assessment done by people who can read the artifacts, who will run the numbers themselves, and who have no interest in what the buyer decides afterward.
This is a short discipline with a specific shape. It is not a market survey and not a management consulting engagement. It is a claim-by-claim examination of one vendor's technical representations, with a written record the buyer can put in a file and defend.
What "independent" has to mean before anything else
Independence is a structural condition, not a promise in an engagement letter. Federal acquisition has been explicit about this for decades. FAR Subpart 9.5 governs organizational conflicts of interest, and FAR 9.505-3 states the rule plainly: contracts for the evaluation of offers for products or services shall not be awarded to a contractor that will evaluate its own offers, or those of a competitor, without proper safeguards. FAR 9.505-1 keeps a firm that provides systems engineering and technical direction from later supplying the system. FAR 37.203(d) carries the same idea into advisory and assistance services.
Two practical consequences follow. First, the assessor cannot be a candidate for the follow-on work. Second, when the assessor gets access to another company's proprietary information, FAR 9.505-4 requires an agreement to protect it, and the disclosure limits on contractor bid or proposal information under FAR 3.104 apply the whole way through. Agencies that bring non-government advisors into an evaluation say so in the solicitation and put those advisors under written non-disclosure obligations before a single page moves.
Our rule is simpler than the regulation and stricter: we do not bid the work we assess. If we read a vendor's technical volume for a buyer, we are out of that competition and out of the follow-on. The fee is fixed and does not move with the finding. A recommendation that pays us more than the opposite recommendation would is not a recommendation, and everyone reading the report knows it.
The assessor must have nothing to gain from the answer
Fixed fee, written scope, no bid on the follow-on, no reseller relationship with the vendor or its competitors, and a conflict screen run before the first document is opened. Everything else in an assessment depends on this holding.
The first move: turn the volume into a claim register
Before any testing happens, the technical volume gets read line by line and every technical representation is extracted into a numbered register. Not themes. Claims. "Processes 4,000 documents per hour." "Deployed in a FedRAMP Moderate environment." "Fine-tuned on 1.2 million labeled records." "No customer data is used for training." "99.4% precision on entity extraction." A forty-page volume usually yields somewhere between sixty and a hundred and twenty discrete claims.
Each claim then gets a status, and the statuses are boring on purpose: demonstrated (we saw the artifact and reproduced it), documented (evidence exists in the public record or the data room, unreproduced), asserted (stated with no supporting evidence offered), ambiguous (the sentence cannot be tested as written), and contradicted (another part of the record says something different). The last two categories are where most of the value is. Ambiguity in a technical volume is a decision the buyer is being asked to make without knowing it, and internal contradiction is the single most reliable signal in the whole exercise.
Where technical claims most often fail to survive a read
Editorial weighting of common gap categories from public sources and practitioner reading. Illustrative, not a measured statistic.
The evidence standard: what would actually settle each claim
Most disputes about AI vendors are not disputes about facts. They are disputes about what counts as evidence. Settle that first and the rest of the assessment becomes mechanical. For every claim class there is an artifact that would end the argument, and it is almost never the artifact the vendor volunteers.
| Claim in the volume | What is usually offered | What would actually settle it |
|---|---|---|
| "94% accuracy on real-world data" | One number on a slide | The held-out set, the split protocol, per-class results, the sample size, and the baseline it beats |
| "Runs in a FedRAMP environment" | A logo on the last page | The FedRAMP Marketplace listing and its exact status, plus a boundary diagram naming which components sit inside it |
| "We own the model" | A sentence in the executive summary | Base-model license, fine-tuning artifacts, and the asserted-rights table under DFARS 252.227-7017 |
| "Sub-second response" | A demo on a laptop | p95 latency at contract-level concurrency, on the target infrastructure, with the cost per thousand calls |
| "The system explains its decisions" | A confidence score in the interface | One decision traced end to end from stored inputs through the log record and reproduced on demand |
| "Our team has done this before" | Client logos | Named individuals, committed hours, and a key-personnel clause that keeps them on the contract |
Benchmarks, and why one number is never a result
The accuracy figure is where the most money gets lost, because it is the easiest number to produce and the hardest to interpret. Four questions do most of the work. Who held the test set? If the vendor built both the training data and the evaluation data, the number measures internal consistency. Was the split done by record or by entity? Splitting a document corpus randomly by page, when pages from one case file land in both halves, inflates every metric. What is the baseline? A model at 94% is unimpressive against a rules engine at 93% and remarkable against a human floor of 71%. And what is the sample size? A 94% on 200 examples carries an interval wide enough to include 90%, which may be the whole decision.
Class balance is the fifth question and the one that hides the most damage. On a task where 96% of records are routine, a model that always predicts "routine" scores 96% and is worth nothing. Aggregate accuracy on an imbalanced problem is a number designed to be misread. The per-class table, including the rare and expensive class, is the number that matters, and it is almost never the one in the volume.
Data provenance, rights, and what you are allowed to keep
The technical question and the legal question are the same question here. What data trained the model, who owned it, and what does the buyer end up holding at the end of the contract? On federal work the answer is governed by the data-rights framework rather than by the vendor's preference: DFARS 252.227-7013 for technical data, 252.227-7014 for noncommercial computer software and documentation, and 252.227-7017 for the pre-award assertion of restrictions. If a vendor plans to deliver a fine-tuned model as restricted software, that assertion belongs in the proposal, before award, in the table the clause requires.
The provenance side is separate and just as consequential. If customer records, controlled unclassified information, or licensed third-party corpora went into training, the assessment traces the permission chain for each. CUI carries handling obligations under 32 CFR Part 2002 and the NIST SP 800-171 control set, and those obligations do not stop at the boundary of a training pipeline. A vendor who cannot produce the license terms for a training corpus has given the buyer a liability with an unknown maturity date.
The practical test we apply: describe, in one page, what the buyer holds on the last day of performance if the vendor disappears. Weights, or an endpoint? Training code, or a container? OMB Memorandum M-25-22, issued April 3, 2025 on AI acquisition, puts avoiding vendor lock-in and preserving interoperability near the front of what agencies are expected to manage. That page answers whether the contract does it.
What actually runs in production
Architecture claims are worth checking early, because a large share of AI proposals describe a system whose intelligence lives at somebody else's API endpoint. That is not disqualifying. It is often the right engineering choice. It changes the risk profile completely, and the buyer should know before signing rather than at the first price change or model deprecation.
The questions that resolve it: which model, which version, pinned or floating? What happens when the provider retires that version, as providers do on published schedules? What is the unit cost at contract volume, and who absorbs a price change? Is there a fallback path if the endpoint is unavailable, and does the workflow degrade safely or stop? Is any of the buyer's data leaving the boundary in the prompt, and does the provider's retention policy match what the vendor's proposal promised? Every one of those has a documentary answer.
Engineering maturity gets read the same way. A repository with tests, pinned dependencies, a reproducible build, and a deployment history is a different asset from a demonstration notebook. Neither is disqualifying. They are different things, priced differently, and the buyer is entitled to know which one is being sold.
Security posture, checked against the record and not the slide
Security claims are the easiest part of the assessment, because most of the record is public. FedRAMP status is on the Marketplace, and the distinction between "In Process" and "Authorized" is the entire claim. DoD Impact Level authorizations sit in the DISA record. For contractors handling covered defense information, DFARS 252.204-7012 requires NIST SP 800-171 implementation, and 252.204-7019 and 252.204-7020 require a current self-assessment score posted in the Supplier Performance Risk System, which the government can look up directly.
What takes judgment is the boundary question rather than the certificate. A vendor can hold a genuine authorization for a platform while the component being sold sits outside its boundary. The System Security Plan, the boundary diagram, and the plan of action and milestones together say which is true. An assessment that reads all three and maps the delivered component onto the authorized boundary answers a question that a certificate on a slide cannot.
Governance and documentation, against current federal expectations
Federal expectations for AI documentation have firmed up considerably. OMB M-25-21 and M-25-22, both issued in April 2025, set the governance and acquisition frames: agencies are directed to track AI performance after award, manage risk on high-impact uses, and engage acquisition, technical, and program staff together rather than sequentially. The NIST AI Risk Management Framework (AI 100-1) and its Generative AI Profile (AI 600-1) supply the vocabulary that most agency documentation now borrows.
For the assessment, that translates into concrete artifacts. Is there model documentation naming intended use, known limitations, and out-of-scope uses? Is there a logging design that would let someone reconstruct a contested decision months later? Is there a monitoring plan with a defined trigger for retraining, and does anyone own it? Is there a human review path on the decisions that carry consequence for a citizen, a patient, or a claimant? A vendor with good answers here is usually a vendor with good answers everywhere, because this documentation is expensive to fake.
How the work is scoped so the neutrality holds
The scope is written before the work starts and it is short. One vendor, one document set. The output goes to the buyer. The fee is fixed, the finding does not move it, and we do not compete for the resulting contract. That plainness is what makes the report usable in a file.
How an assessment runs
The reproduction step is what separates an assessment from a document review. Where a claim is measurable and the buyer can supply representative data, we rebuild the measurement ourselves: our own tooling, our split, our baseline, our error bars. That is also where the schedule flexes. A retrieval-quality claim on a document corpus can be reproduced in days. A latency-under-load claim against infrastructure we cannot touch cannot be reproduced at all, and the report says so rather than pretending otherwise.
What the deliverable contains
The report is written to be read by two audiences at once: the technical staff who will have to live with the system, and the decision maker who has twenty minutes and a signature to give. It runs long in the middle and short at both ends.
- A one-page summary a non-technical decision maker can act on
- The full claim register, every claim numbered and status-marked
- The evidence table, mapping each claim to the artifact that supports or fails to support it
- Reproduction records for every measurement we ran, including the ones that matched the vendor's numbers
- A risk register ordered by consequence, with the cost of being wrong stated in plain terms
- A written question set to put to the vendor, in language that admits only a specific answer
- Contract-language recommendations covering data rights, key personnel, model versioning, and exit
- An explicit list of what we could not check and why
That last item matters more than it looks. A report that claims to have verified everything has verified nothing, because no outside party gets full access. Naming the unexamined ground is how a reader calibrates the rest, and it is the fastest way to tell a real assessment from a rubber stamp.
Using it in a source selection
In a competitive procurement the assessment feeds the evaluation record rather than replacing it. FAR 15.305 requires the agency's technical evaluation to be documented, and the government retains the judgment. What outside technical reading supplies is the substance underneath the adjectives: a finding written to the definitions in FAR 15.001, so that a gap is characterized correctly as a weakness, a significant weakness, or a deficiency, and each characterization is tied to something in the proposal a reader can point at.
It also sharpens the exchange with offerors. Under FAR 15.306, clarifications and discussions are distinct instruments, and a question drafted from evidence lands very differently than a general request for more detail. "Provide the per-class precision and recall on the held-out set described at page 14, with sample sizes" is a question with one answer. "Please elaborate on your accuracy methodology" invites another page of the same prose.
The last benefit is durability. A protest at the Government Accountability Office runs under 4 C.F.R. Part 21 with a 100-day decision clock, and the agency defends the record it built at the time. Reproducible technical findings are far easier to defend than impressions, whichever way the award went.
Using it at a renewal or an option exercise
The second common trigger is quieter and often more valuable. A system has been running eighteen months, the vendor reports that it is working, and nobody inside the organization can independently tell. Under FAR 17.207 an option is exercised only after the contracting officer determines that exercise is the most advantageous method of fulfilling the requirement, and that determination is easier to make with measurement than with a status deck.
The renewal version of the assessment is anchored differently. It starts from the original proposal claims and asks what actually happened. What did the system predict, what did people do with those predictions, what was the outcome, and how far has the input data drifted from what the model was trained on? Where the buyer has retained logs, that question is answerable with real numbers, and it usually reframes the negotiation. A vendor whose system genuinely performs comes out of it stronger. A vendor whose system does not has a hard time explaining the gap in writing.
The commercial and board version
Outside the federal file the same method serves a different set of decisions. A board weighing an eight-figure platform commitment wants to know whether the technology is a durable asset or a wrapper. An investor in diligence wants to know whether the moat described in the deck is in the model, the data, the distribution, or the deck. An acquirer wants to know what transfers with the code and what walks out with three engineers.
State, county, and municipal buyers have the sharpest version of the problem, because the procurement office is small, the timeline is short, and a bad AI purchase becomes a public story quickly. The federal frameworks are not binding there, but they are useful anyway. A city that asks a vendor for the same artifacts a federal evaluator would ask for gets better vendors and better prices, and the request costs nothing to make.
What this assessment will not do
It will not render a verdict on the vendor. The report says which claims are supported and which are not. Whether the residual risk is acceptable is the buyer's call, made against a mission and a budget we do not own.
It will not reverse-engineer proprietary technology. Everything examined is either provided under agreement or already in the public record. That boundary is what keeps the work clean under FAR 9.505-4 and keeps the report admissible in a procurement file.
It will not price the contract. Cost and price analysis is a separate discipline with its own regulatory frame. The assessment feeds it by establishing what is actually being delivered, which is usually the hard part.
It will not substitute for the contracting officer. The record belongs to the agency. We produce technical findings that hold up when someone hostile reads them, and we produce them fast enough to be useful before the decision date.
Who does the reading
Our team builds the systems it assesses. Precision Federal is an SBIR and STTR shop that ships production AI, ML, data, and cloud systems for federal, state, and commercial customers, as prime or as subcontractor. The practice is led by a former professor in technology who ranks in the Kaggle Top 200 of more than 200,000 competitors, holds seven cloud certifications, and has spent twenty years building production systems for federal agencies across five consulting firms, three of them federal. Behind that sits a standing bench of named engineers, licensed professional engineers, and domain specialists across defense, health, energy, transportation, and public-sector data. We are SAM.gov active, CAGE 1AYQ0, and JCP / DD-2345 certified.
That matters for one reason. The people reading a vendor's benchmark have built the same benchmark on the same class of problem, and they know where the number came from before the vendor explains it.
Frequently asked questions
A document-only read of a technical volume with a claim register and question set runs about a week. Adding reproduction of the measurable claims on real data extends it to three or four weeks. When a decision date is fixed, the scope is cut to fit the date rather than the date moved to fit the scope.
Yes, under conditions. FAR Subpart 9.5 governs organizational conflicts of interest, FAR 9.505-3 restricts evaluating offers where the assessor has an interest, and FAR 9.505-4 requires protection of proprietary information. Agencies notify offerors in the solicitation when non-government advisors will have access, and those advisors sign non-disclosure agreements before receiving proposals.
At minimum, the vendor's technical volume and the decision date. Reproduction work needs a representative sample of real data and, where possible, access to the vendor's evaluation artifacts. Everything else, including public-record checks on security and authorization status, we gather ourselves.
A document-and-public-record assessment proceeds without vendor participation and still produces a useful result. Reproduction of specific claims usually needs the vendor's evaluation set or test code. A vendor's willingness to hand over those artifacts is itself one of the more informative findings.
Then the buyer has a documented basis for proceeding, which is worth as much as a negative finding. A report confirming that a vendor's numbers reproduce, that the data rights are clean, and that the authorization boundary covers the delivered component is a strong record for the file and a shorter path through review.