An architecture review before you commit to a platform

Five decisions you do not get to make twice

Most of what a team argues about in the first month of an AI build is reversible. Prompt templates, chunk sizes, the model version, the dashboard library, whether the orchestration lives in Python or in a workflow engine: change any of them on a Tuesday and nothing outside the repository notices by Friday. Five decisions do not behave that way. The data platform, the serving pattern, the evaluation infrastructure, the security boundary, and the degree of vendor coupling are load-bearing. Once application code, a contract, and an authorization package have been built on top of them, reversing one costs quarters rather than sprints.

Expensive to reverse does not mean technically impossible. Anything can be migrated. It means the cost lands in places nobody priced at the moment of the decision: a re-authorization, a renegotiated data rights clause, an evaluation set that has to be rebuilt from scratch because the old one was generated from the system's own outputs, a second team kept on payroll for eight months to run the old path while the new one catches up. We are asked to look at architectures after that bill has arrived often enough that we started offering the conversation that belongs before it.

This is a description of that review. Our engineers run it as a fixed two-week engagement against whatever exists: a slide deck, a vendor's proposed design, a half-built pipeline, a statement of work that has not been signed. The output is a written decision record, not a verbal opinion. The rest of this piece is what we look at and why each item earns its place.

The data platform

The data platform decision is really four decisions wearing one name: where the bytes physically sit, what format they sit in, what catalog describes them, and who is allowed to read them. Teams usually discuss only the first one, because that is the one with a logo attached.

Format matters more than brand. Data written as Parquet under an open table format such as Apache Iceberg or Delta Lake can be read by a different engine next year without a migration project. Data written into a warehouse's proprietary internal tables cannot, and the cost of finding that out is a full extract, transform and reload of everything, plus the rewrite of every query and every downstream job that assumed the old dialect. We treat open storage format as the default recommendation and require a written reason to depart from it.

The federal and state versions of this question add residency. If the workload touches controlled unclassified information, it inherits NIST SP 800-171 Rev. 3 expectations on the contractor side, and if it runs in a cloud that touches covered defense information it inherits the FedRAMP Moderate equivalency requirement under DFARS 252.204-7012. Tax data brought in from a state revenue agency carries IRS Publication 1075, which constrains where the data may reside and who may see it down to the individual. Health data carries HIPAA and its business associate chain. Student records carry FERPA. A platform that cannot run inside the region or partition your data class requires is not a platform choice at all, and that check takes an afternoon, not a quarter.

Egress deserves one paragraph of arithmetic. The major clouds now waive data transfer fees for customers leaving the platform entirely, on request, which sounds like the lock-in problem solved. It is not the case that matters. The expensive case is the hybrid split, where one system stores the corpus in one provider and a model runs in another, and several terabytes cross the boundary every day at standard outbound rates near nine cents per gigabyte. A design that moves 20 TB a month across providers is quietly spending roughly $21,000 a year on nothing but transport. We price that line in the review, because it is a design decision, not a bill.

The serving pattern

There are four serving patterns in common use, and picking the wrong one is the single most common reason a working prototype does not become a working system. Batch scoring runs on a schedule and writes results to a table. Synchronous request and response answers a user or a service inside a latency budget. Streaming consumes events continuously and emits derived events. An agent loop calls tools repeatedly until a stopping condition, with unbounded worst-case cost.

These are not interchangeable, and the code written for one is largely wasted on another. A team that prototypes against a hosted model API in a notebook has implicitly chosen synchronous serving with an internet dependency. If the production path turns out to be a nightly batch job running inside an accredited boundary with no outbound internet route, that team is not tuning a system, it is writing a new one. We have seen that discovery arrive in month seven.

The numbers we ask for are specific. Not "fast" but the p99 latency the calling system will tolerate, because the mean tells you nothing about the user who leaves. Not "scalable" but peak concurrent requests and the tokens or records per request. Not "cost effective" but cost per thousand calls at peak, and what happens to that number when the volume triples after the second office adopts it. Then the fallback: what the system returns when the model is unavailable, and whether that path has been tested rather than merely diagrammed.

Decision	Stays flexible	What locks it in
Data platform	Query engine, compute tier, catalog tooling	Proprietary storage format, residency commitments, a year of downstream jobs
Serving pattern	Model version, prompt, caching layer	The application contract, the SLA, the cost model, the network route
Evaluation	Metric thresholds, report formatting	The labeled golden set and who adjudicated it
Security boundary	Individual control implementations	The authorization package, the SSP, the inherited control story
Vendor coupling	Anything with a documented export path	Managed state, proprietary formats, embeddings you cannot regenerate

Evaluation infrastructure

Evaluation is the item most often cut from the first budget and the item whose absence is hardest to fix later. A real evaluation setup has five parts: a labeled golden set built from production-representative inputs, a regression suite that runs on every change in continuous integration, per-field metrics rather than one aggregate accuracy number, provenance from every output back to the source span that produced it, and an adjudication workflow so a human disagreement becomes a labeled example instead of an argument in a meeting.

The reason this is expensive to reverse is subtle. If the golden set is built after the system exists, it is almost always built by having humans review the system's output and mark what looks wrong. That set now encodes the system's blind spots as ground truth. Errors the system makes consistently look correct to the reviewer, because the reviewer is anchored on what the screen already says. A golden set built before or independently of the system does not have that defect, and there is no cheap way to retrofit independence into one that does.

The metric that changes behavior is not accuracy. It is the rate at which the system produces a confident, well-formatted, wrong value that a downstream human accepts without checking. Aggregate accuracy hides that number completely. For federal work there is also a documentation dimension: NIST's AI Risk Management Framework and its generative AI profile define what evidence of testing looks like, and OMB M-25-21 sets expectations for how agencies document and monitor high-impact AI uses, with M-25-22 covering the acquisition side. A contracting officer asking how you know the system works is asking for artifacts, and artifacts come from infrastructure that existed while the work was happening.

The security boundary

The boundary is a line drawn on a diagram that says which components are inside the authorization, which are outside, what data crosses the line, and who owns each control on each side. Teams routinely defer this to a security group who arrives after the build. That sequence is backwards, and it is the reason more AI pilots die in review than die in benchmarking.

The specifics depend on the customer. FedRAMP Moderate carries a control baseline in the low three hundreds derived from NIST SP 800-53 Rev. 5; High is a larger set with tighter parameters. Defense workloads add the Cloud Computing SRG impact levels, where IL2 covers public and low-sensitivity data, IL4 and IL5 cover controlled unclassified and higher-sensitivity national security information, and IL6 covers classified up to Secret. DFARS 252.204-7012 obligates covered contractors to report cyber incidents to DIBNet within 72 hours and to flow that clause down. CMMC is phasing into DoD solicitations under 32 CFR part 170. State work substitutes StateRAMP or a state-specific program such as TX-RAMP, and the underlying logic is the same.

One failure recurs across nearly every architecture we are handed. A design routes some step through a commercial model endpoint that sits outside the boundary, and nobody has written down that the request payload is the customer's controlled data leaving the accredited environment. The fix is available: an inference path inside the boundary, a self-hosted model on accredited infrastructure, or a documented and authorized connection. The cost of the fix in month two is a week. The cost in month fourteen, after the authorization package is drafted, is a redraw of the package.

The second thing the review checks is control inheritance. A large fraction of a system's controls are supposed to be inherited from the cloud service provider's own authorization, and the rest are yours to implement and describe. Teams that have not read their own responsibility matrix carry hundreds of controls they thought someone else owned. That is a documentation problem discovered at the worst possible time, and it is visible in an afternoon if someone looks.

Vendor coupling

Coupling is not a moral question. Every serious system depends on vendors, and building everything yourself is usually worse. The question is whether anyone has priced the exit. Our review produces a coupling inventory: every third-party component, what it holds that you cannot regenerate, and an estimate in engineer-weeks of what leaving would cost.

Four things generate real coupling. Managed state, where a vendor holds derived artifacts with no documented export. Proprietary storage or index formats. Orchestration expressed in a vendor-specific language, so the business logic lives in a dialect nobody else can execute. And embeddings, which deserve their own sentence: if the corpus is embedded with a model you cannot self-host or reproduce, then changing providers means re-embedding everything, which is fine if you kept the raw text and the chunk boundaries and ruinous if you did not.

Then the contract layer, which is where federal work differs sharply from commercial work. Data rights in defense contracts run through DFARS 252.227-7013 for noncommercial technical data, 252.227-7014 for noncommercial computer software, and 252.227-7018 for work developed under the SBIR and STTR programs, where the data protection period now runs twenty years from award. Civilian agencies generally use FAR 52.227-14. The trap is a build that promises the government rights in a deliverable while a component inside that deliverable carries a license the prime never read. Rights have to be asserted and markings applied before delivery, not argued about afterward. Our team checks the license inventory against the promised rights as part of the same pass, because the two questions are the same question.

What the two weeks actually produce

The engagement is ten business days and a fixed price. It runs against documents and interviews, and it does not require access to production systems, although read access shortens it.

Six artifacts come back, and they are written to be forwarded to people who were not in the room.

• A decision record covering the five load-bearing choices, each with the recommendation, the alternatives considered, and the estimated cost of reversing it in month twelve.
• A reference architecture at two levels of detail: one page a program manager can read, and a component diagram an engineer can build from.
• A security boundary memo naming what sits inside the authorization, what crosses it, and which controls are inherited versus owned.
• An evaluation suite specification: golden set size and sourcing, per-field metrics, the continuous-integration gate, and the adjudication workflow.
• A coupling inventory with an exit estimate in engineer-weeks for every third-party component.
• A 90-day sequencing plan ordered so the choices that are hardest to reverse get made first, with evidence.

The review is run by engineers who build these systems rather than by an advisory practice that does not. Our team has twenty years of production federal delivery behind it across five consulting firms, three of them federal, led by a former professor in technology who ranks in the top 0.1 percent of a global data science field of more than 200,000 and holds seven cloud certifications. We bring the specialists a given review needs from a standing bench of named engineers, licensed professional engineers, and domain specialists across defense, health, energy, transportation and public-sector data. When the honest finding is that a design is sound and the team should keep going, that is what the decision record says, in writing, and the engagement ends there.

When it is worth doing

There are four moments where two weeks buys back a quarter. Before a platform contract signs, because the terms are still negotiable and the exit clause is still writable. Before the authorization package starts, because the boundary drawn on day one is the boundary you defend for years. Before the first production data load, because format decisions harden the moment real volume lands on them. And before a proposal prices an AI workshare, because a capture manager who can name the serving pattern, the boundary, and the evaluation plan writes a better technical volume than one who cannot.

The people who ask us for this are usually one of five. A prime capture manager pricing an AI or ML workshare on a bid with a close date approaching. A system integrator whose customer just added a model requirement to a program that was scoped without one. A university team moving research code toward something a sponsor will accept as a system. A state or county agency comparing two vendor proposals with no in-house way to judge either. And a board or investor who needs a technical read on a company whose engineering story has to hold up.

Bottom line

Architecture review is cheap relative to what it protects. Ten days of senior engineering time against a build that will run for years is a rounding error next to one re-authorization, one storage migration, or one evaluation set that has to be rebuilt because the first one was assembled from the system's own answers. The decisions worth pausing on are few and identifiable, and every one of them is easier to get right on a whiteboard than in month fourteen.

Frequently asked questions