The sentence that arrives after the security review
It usually comes from a platform architect who was not on any of the earlier calls. Legal is done, the security questionnaire came back clean, and then one line lands in the thread: our data does not leave our account, so your software has to run in ours. Nothing about the model changes. Almost everything about how you build, release, support, price and staff the product changes, and most of that cost shows up in quarters two through six rather than in the sprint you are about to plan.
The first error is treating that sentence as one request. It is at least four different requests with wildly different costs, and the person who wrote it often has not distinguished them either. Someone in security wants data residency. Someone in platform wants the workload visible in their own monitoring. Those are satisfied by different topologies, and the cheapest one that answers the real concern is what you want to find before writing any code.
You are probably here because
- A deal you thought was closed came back with one line: our data does not leave our account.
- The install passes in your own cluster and stalls in theirs, on a proxy or a hostname nobody wrote down.
- You have installs on three different versions and no published policy saying which ones you support.
- Per-request pricing stopped making sense the moment you could no longer see the requests.
The topology table below is where to start, and the sections on egress, version skew and pricing take the rest in order; all four symptoms usually come from one root cause, which is that the product was built as a service you operate and now has to be an artifact somebody else installs.
Four topologies, and picking the wrong one costs a year
Multi-tenant hosted. What you have now. One deployment, shared infrastructure, logical isolation by tenant ID. Cheapest to run, fastest to fix, and the thing the customer just rejected.
Dedicated single-tenant in your account. A separate stack per customer, still yours to operate, optionally in a region they choose. This answers residency and noisy-neighbour concerns and costs you a multiple of your infrastructure bill. It does not answer "your company should never hold our data," so check which objection you are facing before you build it.
Customer-hosted data plane with your control plane. The workloads that touch customer data run in their account. Provisioning, license state, release metadata and aggregate health run in yours. This is where most vendors land, and it is the topology that demands the most engineering discipline because you now operate a distributed system across a trust boundary you cannot debug through.
Fully self-managed. You ship an artifact. They install it, run it, upgrade it and page themselves at 3am. Lowest ongoing infrastructure cost to you, highest documentation and support cost, and the only option that works for a customer with no outbound connectivity at all.
| Topology | Cloud credentials | Who upgrades | What you can see | Where it hurts |
|---|---|---|---|---|
| Multi-tenant hosted | Yours | You, continuously | Everything | Rejected by the customer asking this question |
| Dedicated single-tenant | Yours | You, per stack | Everything | Infrastructure cost multiplies with logo count |
| Customer-hosted data plane | Theirs | You, with their approval window | Health, versions, metering | Cross-boundary failure modes and change control |
| Fully self-managed | Theirs | They do, on their schedule | Only what they send you | Version sprawl and blind support |
One rule saves more time than any other on this table: pick one customer-hosted topology and make every customer use it. Two topologies means two install paths, two upgrade paths, two support runbooks and two sets of environment-specific defects. Vendors who let the second customer talk them into a variant spend the next three years paying for it.
Draw the control-plane and data-plane line on paper, first
Before any packaging work, write down every byte that crosses the boundary in each direction. Not a paragraph of assurance. A list, where each line has a hostname, a port, a payload, a trigger and a reason. The platform team reviewing you will ask for exactly this, and the vendors who produce it in the first meeting get through review in days instead of months.
Reasonable outbound. Container image pulls from a registry you name. A license or entitlement check with an offline grace period. A version manifest so you know what is running. Aggregate operational metrics with no customer content: request counts, latency histograms, error rates by class, queue depth, GPU utilisation. Crash reports with stack traces and no payloads.
Never outbound. Prompts, documents, retrieved chunks, embeddings, model outputs, user identifiers, filenames, database rows, and any log line that concatenates a user string. The last one is where this gets violated by accident, usually by a debug log added during an incident and never removed.
Inbound. Ideally nothing. If your control plane needs to push a command into their environment, invert it: the agent in their account polls yours over an outbound connection it opens itself. Nobody wants to punch an inbound hole for a vendor, and asking for one converts a two-week review into a two-month one.
First customer-hosted deployment — where the engineering hours go
Relative share of effort on a first deployment. Model quality work is not on this list, because none of it moves.
The artifact is not your repository. It is a signed bundle.
Your build has to produce one thing a stranger can install without talking to you. In practice that is a release manifest naming every container image by digest, the images themselves, an install chart or module, a schema migration set, an SBOM, and signatures over all of it.
Pin by digest, not by tag. A tag is a mutable pointer. If a customer installs v3.4.1 in March and reinstalls it in September onto a different node pool, digest pinning is the only thing that guarantees the same bits. It also makes the mirroring conversation trivial: they copy exactly the digests on your manifest into their internal registry with skopeo copy or oras, and nothing else.
Assume registry mirroring. Most enterprises above a certain size will not let a cluster pull from a public registry. Publish to one namespace, keep the image count small, and document the mirror procedure as a supported path rather than an exception. If you have forty images because every microservice got its own, that is now forty things to mirror and forty things to re-mirror at each upgrade.
Ship an SBOM and a signature. CycloneDX and SPDX are the two formats worth producing, and SPDX is standardised as ISO/IEC 5962. Sign images with cosign and publish the verification command in your install guide. This is not decoration. A security team that can verify provenance without contacting you removes a gate from your deal.
Make the offline bundle a first-class build target. One tarball with images, chart, migrations, SBOM and signatures, plus a checksum file. Even customers with connectivity often prefer it, because it turns your install into a reviewable object they can archive.
Egress is what actually breaks the install
The install rarely fails on your code. It fails on the network, and it fails the same way every time.
There is no default internet route. Outbound goes through a NAT gateway with a domain allowlist, and every hostname you touch needs an entry approved by someone who is not in the room. Enumerate them in your documentation, including the ones you forgot: your registry, its CDN backend, the model weight host, your telemetry endpoint, your license endpoint, and any package index a container touches at runtime, which should be none.
The proxy inspects TLS. Corporate proxies terminate and re-sign, so every container needs the customer's private certificate authority bundle mounted and every runtime reads a different variable to find it. Plan for SSL_CERT_FILE, REQUESTS_CA_BUNDLE, NODE_EXTRA_CA_CERTS, CURL_CA_BUNDLE, and a Java truststore if any component is on the JVM. Give operators one place to supply the bundle and wire it to all of them yourself.
NO_PROXY is the silent killer. When HTTP_PROXY and HTTPS_PROXY are set cluster-wide and NO_PROXY does not include the service CIDR, the pod cluster domain and the node CIDR, your internal service-to-service calls get routed to the corporate proxy, which refuses them. The symptom is a component that starts, passes its liveness probe and cannot reach the database. Check this in preflight and print the exact value it should be.
Model weights are a logistics problem. Parameter count times two bytes gives you the bf16 footprint: about 14 GB for a 7B model, about 140 GB for a 70B. Pulling that at pod start over an inspected proxy is slow, fragile and will get you blamed for the network. Either bake weights into a versioned image layer, or have the customer stage them once in their own object storage and mount from there. Never download weights on every restart.
GPUs you do not own
In your own account you choose the instance type. In theirs you inherit whatever their cloud team has approved, and quota is the constraint that surprises people. A large enterprise can easily have zero accelerator quota in the region they want you in, and raising it is a support-ticket process measured in weeks. Ask on day one.
The compatibility questions that matter: which instance families are approved, how many, what host driver version is installed, whether the device plugin or GPU operator is already deployed, whether nodes are tainted and what toleration you need, and whether they use multi-instance partitioning or time-slicing to share cards. Your container's CUDA runtime must be supported by the host driver, and assuming the customer runs a current driver is how an install slips two weeks.
Design the inference layer behind an interface from the first commit, because you will need at least three implementations. Self-hosted weights on their GPUs is the default. Quantized CPU inference is the fallback for small workloads and for the pilot that has to start before quota arrives. And a managed model endpoint inside the customer's own cloud account is often the fastest path to a signature: the data stays inside their boundary and inside their existing agreement with their cloud provider, and you need no accelerator quota at all. Making that a configuration value rather than a fork is worth a week of design.
Identity, secrets, and the permissions document
Long-lived access keys are a decline in most reviews now, and they should be. Use the platform's workload identity mechanism so your pods assume a role with no static credential: IAM roles for service accounts on EKS, Workload Identity on GKE, workload identity federation on AKS. Pull secrets from the customer's own manager rather than a values file, and let them supply their own encryption key so that revoking the key revokes your access without a conversation.
Then write the permissions document before anyone asks for it. Every permission, the resource pattern it applies to, the product feature that requires it, and what breaks if it is removed. Reviewers approve faster when they can delete a line and know the consequence, and wildcards read as "we did not do the work."
The install-time role and the runtime role are not the same role
Installation legitimately needs to create namespaces, storage classes, custom resources and network policies. Steady-state operation needs almost none of that. Split them, hand back the elevated role when the install is done, and say so in writing. A vendor that keeps cluster-admin permanently is a finding in the customer's next audit, which means it becomes your problem at renewal even if nobody objects today.
What a platform team scores you on in architecture review
Weighting varies by industry. In regulated environments the top two swap places with the SBOM row.
Version skew becomes your product
The day you have three customer-hosted installs you no longer have a version. You have a distribution, and every distribution needs a published support policy on day one, because it is impossible to introduce one later without a fight.
Borrow the shape from the ecosystem you sit on. Kubernetes ships a minor release roughly three times a year and supports each for about fourteen months, and the managed services mirror that: Amazon EKS gives fourteen months of standard support followed by twelve months of extended support at a higher price. Customers already understand that shape. Publish an end-of-support date with every release, support the current minor and the two behind it, and require a customer to reach a supported version before you debug an issue. Writing that into the order form is far easier than enforcing it later against an account that is nineteen months behind.
Schema migrations need a different discipline than the one you use in your own environment. You cannot roll back a database you do not back up, and you will not be there when it goes wrong at 2am. Three rules cover most of it. Every migration is idempotent, so a partial run can be re-run. Every release runs correctly against the previous release's schema, which means additive changes ship one release ahead of the code that uses them and destructive changes ship one release behind. And no upgrade ever requires an operator to run manual SQL, because that instruction will be followed incorrectly at least once.
Send it over and we will tell you what we would change.
Email your install chart or Terraform module, the image list it pulls, and the egress hostnames it needs, to contact@precisionfederal.com. You get back a short written note naming the three things we would change and why. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comThe support bundle is part of the product
What you will receive is a message saying it is broken, from a person who cannot show you their screen, cannot paste logs into a ticket because of policy, and does not know which component failed. Build the tool that closes that gap before your first install rather than after your first outage.
A support bundle command should collect, into a single archive the customer can inspect before sending: the release manifest with image digests, pod and node status, recent cluster events, your configuration with secrets redacted, resource utilisation and limits, the last few thousand log lines per component, migration state, a GPU inventory, and the results of a connectivity probe against every endpoint on your egress list. Redaction on by default, with the redaction rules visible in the output so nobody has to trust you about it.
The most valuable thing on that list is the connectivity probe, and it should also exist as a standalone preflight command that runs before installation. Preflight converts a week of email into a forty-second failure message that names the unmet condition: this Kubernetes version is below the minimum, this storage class does not support the access mode you need, this hostname does not resolve, this proxy is intercepting TLS and no CA bundle was supplied, this node pool has no allocatable accelerators. Every check you add there is a support ticket you never receive.
Patterns that turn a two-week install into a two-quarter one
- Mutable image tags, so nobody can prove what is actually running.
- Installing packages or downloading weights at container start, which fails the moment egress is restricted.
- Requiring cluster-admin at runtime because the install script needed it once.
- Telemetry that fails closed, so the product stops working when the customer blocks your endpoint.
- Secrets in a values file, which then lives forever in the customer's configuration repository.
- A separate code path for hosted and customer-hosted builds, which guarantees the second one is under-tested.
- No preflight, so every environmental assumption is discovered by an operator reading a stack trace.
- Forward-only migrations with no tested rollback, in an environment where you have no database access.
- Agreeing to a customer-specific fork to close one deal.
- A support process whose first step is asking for access you were never going to be granted.
Pricing something you do not operate
Per-request pricing does not survive contact with a customer VPC. You cannot count what you cannot see, self-reported counts turn every renewal into an audit argument, and a meter that phones home becomes the most scrutinised line in the security review. Move the price to something both sides can observe without trusting each other.
| Pricing model | How it is counted | Where it fails | Use it when |
|---|---|---|---|
| Per node or per accelerator | Capacity the customer provisions, declared in the license | Customer over-provisions and feels punished for it | Workload scales with hardware and the customer controls the fleet |
| Per named user | Seats in their identity provider | Automated or service-account usage is invisible | The product has a human interface with distinct users |
| Per environment | One license per install, production priced above staging | Growth inside one environment is unpriced | Deployments are few, large and long-lived |
| Declared volume tier | Annual commitment on documents, records or requests | Requires an honest true-up conversation | Value tracks data volume more than headcount |
| Usage metered by the vendor | Counts reported by your agent | Becomes the focus of the security review | Rarely. Only where the customer has already accepted the telemetry. |
Two things belong in every customer-hosted agreement regardless of model. An annual minimum, because your cost of supporting an install is roughly fixed whether they use it heavily or not. And a separate implementation fee for the first deployment, which is honest work: the network review, the permissions design, a rehearsal in your own replica of their environment, and a dry run in their staging with their operator.
Make license enforcement degrade rather than detonate. A signed offline license with an expiry date and a generous grace period, checked locally, is the pattern that survives review. A license check that can stop production inference during a network incident is a pattern that produces one very bad phone call and a lost renewal.
Where the recurring cost lands after go-live
Steady-state effort once installs are in the field. Budget it as headcount, not as overhead.
Ask these before you agree to a date
- Cloud provider, region, and whether the account is dedicated to this workload
- Cluster type and version, or whether you are being handed bare virtual machines
- Accelerator instance types approved, current quota, and the driver version installed
- Egress policy: allowlist, proxy address, and whether TLS is inspected
- Internal registry, and whether image mirroring is mandatory
- Identity mechanism, and who approves the permissions policy
- Where the database lives, who provisions it, and who owns the backups
- Named operator on their side, and whether that person carries a pager
- Change-management window and the approval lead time for a production install
- Whether a staging environment exists and genuinely mirrors production
The tenth item is the one people skip and the one that decides the schedule. An install rehearsed in a staging environment that matches production is a routine change. An install that meets its first real proxy, its first real storage class and its first real quota limit on production night becomes a week.
What a realistic first deployment looks like
First customer-hosted install
Step three is the one that gets cut and the one that pays for itself. Building a hardened replica of the customer's environment in your own account, with a proxy that inspects TLS, a private certificate authority, a domain allowlist, no public registry access and a node pool that matches theirs, costs a few days and catches most of what would otherwise be found in their change window. Keep it and run every release against it.
The second deployment should cost between a third and half of the first, and the fifth should be close to routine, but only if you resisted forking and only if the preflight tool absorbed each new environmental surprise as a permanent check. Teams that skip that discipline find the fifth deployment costs the same as the first, because it is effectively the first deployment of a fifth product.
Bottom line
Running inside a customer's account is a product decision wearing an infrastructure costume. The model, the prompts and the evaluation set are unchanged. What changes is that you now build a shippable artifact instead of a running service, you support a fleet of versions instead of one, you debug through a keyhole, and you price against something both parties can see. Decide the topology once and hold it. Write the egress list and the permissions document before you are asked. Put the hours into preflight and the support bundle, because those two tools determine whether each new deployment gets cheaper or stays expensive forever.
Frequently asked questions
The components that touch customer data run inside the customer's own cloud account and network, while the vendor keeps a control plane for provisioning, licensing and aggregate health. The customer owns the infrastructure bill and the security boundary. The vendor still owns releases, upgrades and support, which is what makes it harder than either pure hosted service or pure self-managed software.
Usually not. You cannot observe the count, self-reporting turns renewals into audits, and a meter that reports usage back to you becomes the most examined component in the security review. Price on capacity, named users, environments, or a declared annual volume, and add a minimum because your support cost is fixed regardless of how much they use it.
The current release and the two behind it is a defensible policy, with an end-of-support date published at release time. Kubernetes and the managed cloud distributions run on roughly a fourteen-month support window per minor version, so enterprise platform teams already plan around that shape. Put the policy in the order form rather than trying to introduce it later.
Have two alternatives ready. Quantized inference on CPU covers pilots and lighter workloads. Routing generation to a managed model endpoint inside the customer's own cloud account keeps data within their boundary and needs no accelerator quota at all. Both should be configuration changes rather than code branches, which means designing the inference layer behind an interface from the start.
Roughly eight to ten weeks from environment questionnaire to production for a team that already has containerized components and a Helm chart or Terraform module. Longer when accelerator quota has to be requested, when a proxy inspects TLS, or when change management requires a fixed window. The second deployment typically costs a third to half of the first, provided you did not create a customer-specific fork.