# How private LLM hosting choices fail in practice

> Compare private LLM hosting through real cost, data custody, operations, and sovereignty tradeoffs before choosing GPUs or managed endpoints.

Private LLM hosting is usually chosen for the wrong reason. Founders hear "private," assume the compliance question is settled, and then compare a GPU hourly rate with a token price. That skips the decisions that determine both risk and cost: who operates the model, where prompts and metadata travel, which people can access the control plane, what happens during a capacity shortage, and who carries the pager.

The right choice follows from the control you must prove, not the infrastructure you would enjoy owning. vLLM on your GPUs gives you the widest technical control and the largest operating burden. A managed private endpoint removes much of that burden but keeps the provider inside your trust boundary. A sovereign cloud may restrict geography, personnel, and legal jurisdiction more tightly, but it does not necessarily give you the model choice or economics of self-hosting. Treating these as three price tiers produces expensive architecture and weak evidence.

I have watched teams buy GPUs to avoid a procurement conversation, then discover that model serving had become a new production system with poor utilization and no owner. I have also watched teams approve a managed endpoint because traffic stayed on a private network, even though nobody had checked stored conversations, support access, or cross-region processing. Both failures begin with an imprecise definition of private.

## How the three hosting models actually differ

The three options allocate responsibility differently, and that allocation matters more than the label on the invoice. You need to separate four controls that teams routinely blur: network isolation, data custody, model custody, and operational sovereignty.

Network isolation controls the route between your application and the inference service. A private endpoint can keep traffic off the public internet while the provider still runs the service. Data custody covers prompts, outputs, embeddings, logs, fine-tuning files, and backups. Model custody asks who stores and can change the weights. Operational sovereignty covers who can administer the service, where the control-plane metadata lives, which law governs access, and whether the service can run without staff or systems outside the required jurisdiction.

With vLLM on infrastructure you control, your team owns the serving process, weights, logs, network, scaling, and incident response. The hosting provider may still control the physical hardware and virtualization layer unless you run it on premises. "Self-hosted" therefore does not automatically mean "outside cloud-provider access."

A managed private endpoint puts a private network interface in front of a provider-operated model service. Amazon Bedrock documents that PrivateLink can route calls without an internet gateway, NAT device, VPN, or public IP. That statement describes the network path. It does not say that your team operates the model or that every optional service feature is stateless.

A sovereign cloud adds jurisdictional and operational controls to a cloud environment. Depending on the offer, it may constrain data residency, support personnel, control-plane operations, billing metadata, or connectivity to a global parent cloud. Those controls vary by service and region. The word "sovereign" is a prompt to inspect a contract and architecture, not evidence by itself.

Use this responsibility map before discussing products:

- Your team patches a vLLM stack; a managed provider patches its service; a sovereign service uses the provider or an approved local operator.
- Your team usually chooses self-hosted weights; managed services limit you to a catalog; sovereign catalogs may be narrower still.
- Your budget absorbs idle GPUs; a managed provider pools idle capacity unless you reserve it; sovereign contracts handle this differently.
- Your architecture proves self-hosted data location; managed choices combine provider evidence with your configuration.
- Your on-call staff handles a vLLM outage; managed and sovereign services split incident work with your team.

This map is deliberately uncomfortable. No option wins every line.

## vLLM buys control by transferring the operating burden

Run vLLM when model custody, custom weights, deterministic upgrade timing, or complete local processing justifies owning an inference platform. Do not run it because an OpenAI-compatible endpoint makes the first demo easy. Compatibility at the HTTP boundary does not operate GPUs, secure containers, or keep latency stable under mixed workloads.

vLLM exposes common APIs such as Chat Completions, Completions, Responses, and Embeddings, subject to model support. Its documentation also warns that a model repository's `generation_config.json` can override sampling defaults. That small detail causes real migration bugs: two servers accept the same request and produce materially different behavior because temperature or other defaults changed below the application.

A minimal serving configuration can look like this:

```yaml
model: /models/acme-instruct
host: 127.0.0.1
port: 8000
generation-config: vllm
tensor-parallel-size: 2
```

Load the configuration through your release system, inject the API credential from a secret store, and pin the container and model by digest. Bind the raw server to a private interface or loopback address, then place an authenticated gateway in front of it. The gateway should enforce workload identity, request size, timeouts, rate limits, and log redaction. A server API key is only a basic gate. It does not provide per-service authorization, tenant isolation, or an audit trail that identifies the calling workload.

Before production, capture a response shape and make it a contract test:

```json
{"id":"cmpl-...","object":"chat.completion","choices":[{"index":0,"message":{"role":"assistant","content":"..."},"finish_reason":"stop"}],"usage":{"prompt_tokens":18,"completion_tokens":42,"total_tokens":60}}
```

Check the fields your application truly consumes, including tool calls, structured output, token counts, finish reasons, and streaming chunks. Do not assert the generated prose. Assert the protocol and run a separate quality evaluation against a pinned model digest.

Multi-GPU serving adds another boundary. vLLM supports tensor and pipeline parallelism, but cross-node tensor parallelism depends on fast interconnects. Its scaling guide tells operators to inspect NCCL logs: `NET/IB/GDRDMA` shows the high-speed path, while `NET/Socket` means traffic fell back to raw TCP sockets. A deployment can be healthy by Kubernetes standards and still miss its latency target because the GPU communication path is wrong.

Your ownership list includes more than the serving container:

- Validate the model license and preserve the exact weight digest.
- Scan container images and model artifacts before promotion.
- Benchmark prefill and decode latency with your prompt-length distribution.
- Plan capacity for failures, deployments, and traffic bursts.
- Patch drivers, CUDA libraries, vLLM, tokenizers, and the base operating system.

The model weights may remain inside your boundary, yet prompts can still leak through application traces, exception collectors, evaluation datasets, or a careless gateway log. Self-hosting narrows the number of trusted operators. It does not repair the rest of your data flow.

## Managed private endpoints reduce work, not vendor trust

A managed private endpoint is the sensible default when you need private connectivity and enterprise controls but do not need exclusive custody of the model weights. You buy faster model access, managed capacity, patching, and an existing compliance evidence package. In exchange, you accept the provider's service architecture, model catalog, quotas, policy changes, and contract terms.

Private connectivity is useful. Amazon Bedrock's PrivateLink documentation says instances can call the service without public IP addresses and can use private DNS so normal regional names resolve through the interface endpoint. Google Cloud documents private endpoints and Private Service Connect for Vertex AI workloads. These controls remove public routing and can simplify egress policy.

They do not answer what the service stores. Microsoft states that prompts, outputs, embeddings, and training data for Azure Direct Models are not available to other customers or model providers and are not used to train foundation models without permission. The same documentation explains that optional stateful features, including stored conversations and some agent or response features, create persistent data. That is the level of detail your review needs: not a general privacy sentence, but the behavior of the exact API mode you enable.

Ask the provider to identify every processing path for your chosen model and feature set. Cover inference, content filtering, abuse monitoring, support diagnostics, batch jobs, fine-tuning, embeddings, evaluation, and stored conversation state. A contract that protects prompts during ordinary inference may say something different about files you upload or content retained for a stateful tool.

Managed services also shift performance risk rather than eliminating it. On-demand capacity can throttle at the moment your application succeeds. Reserved or provisioned throughput improves predictability but turns part of the bill back into fixed capacity. Model versions can reach end of support, regions can have different catalogs, and safety controls can change observable behavior. Your application needs timeouts, bounded retries, load shedding, and a tested fallback just as it does with self-hosted inference.

The strongest reason to choose this option is organizational. A small team that already trusts a hyperscaler and needs two approved models will usually create less risk with a managed endpoint than by becoming a GPU operations team. The weakest reason is "our data never leaves our VPC." The request leaves your VPC for a provider-operated service even when it travels over private connectivity. Precise diagrams prevent precise people from approving a false claim.

## Sovereign clouds answer jurisdictional questions

Choose a sovereign cloud when regulation, public-sector policy, or a customer contract requires controls over jurisdiction, operator location, or administrative independence that an ordinary regional cloud cannot prove. Data residency alone is a narrower requirement and may be satisfied by a standard region with the right service configuration.

This distinction changes procurement. A European region can keep selected customer data in Europe while global personnel, control-plane systems, or parent-company legal obligations remain in scope. A sovereign offer may add EU-resident operations, locally held metadata, separate identity or billing systems, and technical restrictions on remote administration. AWS describes its European Sovereign Cloud as physically and logically separate, operated by EU-resident staff, with customer-created metadata kept in the EU. That is a claim about a particular environment, not a definition that transfers to every product using the same adjective.

Sovereignty requirements should name the protected asset and prohibited dependency. "All prompts must stay in Germany" is testable but incomplete. "Production prompts, outputs, account metadata, encryption keys, backups, and privileged support access must remain in Germany, and service operation must not require a non-German administrator" gives architects something they can trace. It may also reveal that no available managed LLM meets the requirement.

Expect a smaller service catalog, slower access to new models, different support procedures, and higher prices. Isolation has a cost because providers cannot pool every team, control plane, or hardware fleet globally. Some sovereign environments support customer-managed models on approved infrastructure; others offer a limited managed catalog. Verify the inference service itself, not the cloud's general certification list.

Disconnected or on-premises systems sit at the strict end of this spectrum but introduce their own dependencies. You still need a controlled path for model weights, container updates, vulnerability data, license records, and hardware service. If an engineer carries an unverified model file across the boundary on removable storage, an air gap has changed the transport mechanism without creating trustworthy provenance.

Do not use sovereign infrastructure as a ceremonial answer to a vague concern. Write the jurisdictional requirement, map it to technical controls and contract language, and record the residual dependency. If ordinary regional hosting passes that test, pay for ordinary regional hosting.

## The cost model must price idle GPUs and people

Compare annual cost per usable workload, not a GPU rate against a token rate. Self-hosting looks cheap when a spreadsheet assumes full GPU utilization, ignores replicas, and assigns no cost to the engineers who keep it running. Managed inference looks cheap when the model uses an introductory workload and nobody prices provisioned capacity, network endpoints, logs, or vendor-specific development.

Use this equation for self-hosting:

```text
annual_self_hosted = gpu_capacity + cpu_memory_storage + network + facilities_or_cloud_margin
                   + platform_engineering + on_call + security_compliance
                   + spare_capacity + upgrade_and_failure_testing
```

For managed service:

```text
annual_managed = input_tokens + output_tokens + reserved_throughput + private_networking
               + storage_and_logs + support + integration_engineering
               + expected_overage_and_fallback
```

For a sovereign service, start with the managed equation and add the price of restricted capacity, specialized assurance work, and any duplicate environment needed for disaster recovery. Do not add a generic "compliance premium." Obtain the actual offer and identify which requirement creates each extra cost.

Measure at least four workload properties before requesting prices: input tokens per request, output tokens per request, concurrency by minute, and latency objective by request class. Daily token totals hide bursts. A support assistant used mostly during business hours and a background document pipeline with the same token count impose different capacity needs.

GPU economics turn on utilization, but utilization needs a strict denominator. Report useful model-busy time against paid accelerator time, then include headroom and unavailable replicas. A two-GPU service that averages 45 percent useful utilization and keeps a second two-GPU replica ready for resilience consumes eight paid GPU-hours for every wall-clock hour while using less than one GPU-hour for productive work. Batching can improve throughput, but waiting to fill a batch can violate interactive latency.

Model size can move the result more than hosting mode. A smaller model that passes the task evaluation may fit on one accelerator, answer faster, and support more concurrent requests than a larger model selected by reputation. Quantization can reduce memory and cost, but it may also change quality, tool selection, or numerical output. Measure the quantized artifact as a different release instead of assuming that a weight format change is operationally invisible.

Separate experimental demand from production demand. Researchers may load several models, use long contexts, and tolerate queues, while a customer workflow needs one pinned model with predictable latency. Sharing one fleet between them can lower apparent idle time but lets experiments evict production cache and distort capacity forecasts. Charge each workload for the capacity and reliability it actually requires.

Run a replay with scrubbed or synthetic prompts that preserve length and concurrency. Record time to first token, output tokens per second, queue time, error rate, GPU memory, and cost for the whole test window. Repeat after enabling realistic context lengths, tool schemas, and structured outputs. A benchmark made of short prompts will flatter every option and size the wrong system.

People cost often decides the result. If self-hosting needs even part of an experienced platform engineer plus an on-call rotation you do not have, the apparent token saving must exceed that organizational cost. At high, steady utilization with a stable open model, owned capacity can win decisively. At low or volatile volume, the provider's pooled fleet usually wins even with a high unit price.

## Compliance starts with the data flow, not the provider badge

Compliance approval should trace data and control paths for one concrete workload. A provider certification helps, but it cannot certify your prompt construction, logging, retention, access roles, or choice of deployment region.

Draw the path from source data to the final user, including retrieval, prompt assembly, inference, safety filters, caches, traces, human review, and backups. Mark the legal entity, region, encryption boundary, retention period, and privileged roles at every store or processor. Include metadata. User IDs, document names, billing tags, and support tickets can reveal sensitive facts even when the prompt body is encrypted or deleted.

Then classify the workload rather than the entire LLM program. Public marketing copy, internal source code, patient data, export-controlled designs, and legal advice do not belong in one tier. Separate endpoints and policies are often cheaper than forcing the strictest control onto every request.

A compact approval record should answer five questions:

1. Which data classes may enter prompts, retrieval results, and outputs?
2. Where does each class transit and persist, including provider-operated filters and logs?
3. Which identities can read data or change retention, networking, models, and keys?
4. What contract, policy, or technical evidence supports each residency and usage claim?
5. How do we delete, export, suspend, and investigate the workload?

Regulated does not always mean self-hosted. A managed provider may supply stronger physical security, access reviews, audit evidence, and incident procedures than a startup can build. Self-hosting can still be mandatory when a contract bans third-party processing or the approved model must run inside a fixed enclave. The decision comes from the requirement and evidence, not from a general preference for control.

Keep legal conclusions with counsel and turn them into engineering assertions. Counsel may decide that a transfer mechanism or business associate agreement is acceptable. Engineering must still prove which endpoint, feature, region, and log setting the application uses. A policy document without deployed configuration is an intention, not a control.

## Security controls stay necessary in every model

Every hosting choice needs workload identity, least privilege, egress control, redacted observability, supply-chain checks, and incident response. Hosting changes who implements each control. It does not remove attacks at the application layer.

Prompt injection can make a model disclose retrieved content or misuse connected tools whether inference runs on your GPU or behind a sovereign endpoint. Treat model output as untrusted input. Enforce tool permissions in deterministic code, scope retrieval to the calling user, validate structured arguments, and require separate authorization for consequential actions.

Put the inference gateway in its own trust zone. It should authenticate workloads, attach tenant and policy context, reject unsupported models, cap context and output sizes, and emit metadata that is useful without copying raw prompts. Give developers a controlled debug mode with short retention and explicit access rather than letting every service log full request bodies.

For self-hosted models, verify weight digests and container provenance. A model file is executable behavior even when it is stored as data. Pin revisions, block runtime downloads, scan deserialization paths, and promote artifacts through the same environments as application releases. For managed models, pin an offered version where possible and subscribe to retirement notices. Run quality and safety evaluations before changing aliases.

Test isolation with two tenants and adversarial requests. Put a unique marker in tenant A's retrieval corpus, then attempt to obtain it through tenant B, logs, caches, batch output, and support tooling. The expected result is absence everywhere outside A's authorized path. This catches shared cache keys and trace exports that an architecture diagram misses.

Incident preparation should identify who can stop inference, revoke credentials, preserve sanitized evidence, contact the provider, and switch to a degraded mode. A kill switch that depends on the same unavailable control plane will disappoint you. Practice the action and record its observed propagation time.

## Portability requires tests, not an OpenAI-shaped API

An OpenAI-compatible API reduces client changes but does not make models or hosting services interchangeable. Providers support different subsets, streaming events, tool-call formats, error codes, token accounting, context limits, and safety behavior. The model itself will interpret prompts differently after a move.

Build a thin internal inference contract around what your product needs. Keep provider-specific request fields behind an adapter. Store model and tokenizer identifiers with evaluation results, and reject silent model substitution in production. If you use server-side conversation state, hosted file search, or proprietary agent tools, record that dependency as an intentional portability cost.

Your migration suite should contain protocol tests, workload replays, quality evaluations, and failure tests. Protocol tests check JSON fields and streaming sequences. Replays measure latency and capacity with representative prompt shapes. Quality evaluations score task outcomes. Failure tests cover throttling, timeouts, truncated streams, unavailable regions, and a model that refuses a request your previous model accepted.

Data portability needs equal attention. Keep the authoritative prompts, retrieval documents, policy configuration, and evaluation cases outside the serving provider when requirements allow. Define how to export fine-tuning data and adapters. Check whether provider-generated logs or conversation state can be exported and deleted on your schedule.

Avoid promising a live multi-provider fallback unless you operate it continuously. An untested secondary model produces different answers, may lack capacity, and often breaks tool calls exactly when the primary fails. A deliberate degraded mode, such as queueing background work or disabling generation for sensitive actions, is safer than a decorative provider switch.

Portability has a price, so spend it where concentration risk matters. A short-lived internal summarizer may tolerate a proprietary API. A customer-facing workflow embedded in core operations deserves a tested exit path and a model-independent data layer.

Set an exit trigger while procurement still has attention. Examples include a model retirement notice, a price increase beyond an agreed threshold, a residency requirement the service cannot satisfy, or repeated capacity failures. Give the owner a deadline to run the migration suite when a trigger occurs. An exit plan that never names an event or owner will not survive the first urgent renewal.

## How to choose without buying the wrong control

Choose vLLM on controlled GPUs when you must hold model weights, need a custom or fine-tuned open model, require local inference, or have enough steady volume and operating skill to justify fixed capacity. Choose a managed private endpoint when private routing, provider compliance evidence, fast model access, and low operations overhead satisfy the requirement. Choose a sovereign cloud when you can name a jurisdictional or operational-autonomy rule that ordinary regional service cannot meet.

Do not force one answer across the company. Many teams should use managed inference for low-risk product features, a tightly controlled self-hosted model for sensitive stable workloads, and no LLM at all where deterministic software is safer. Route by data class and business consequence, not by whichever vendor completed procurement first.

Make the decision in this order:

1. Write the prohibited data movements and administrative dependencies.
2. Map the full data flow for the exact API features and model.
3. Eliminate options that cannot produce evidence for those controls.
4. Replay the workload and price useful capacity, resilience, and people.
5. Approve a versioned architecture with an owner and an exit test.

This sequence also prevents a common founder mistake: using infrastructure to solve an unresolved policy dispute. A Team & AI Audit from oleg.is can turn the workload, staffing, and control requirements into a costed operating decision before you commit to GPUs or a long capacity contract. The useful deliverable is not a favorite vendor. It is a record showing why this workload belongs in this boundary, what it costs under real demand, and how the team will know when the decision stops being true.