# Do you need an LLM gateway yet?

> An LLM gateway can centralize routing, caching, quotas, and audit trails. Learn when that control pays off and when a proxy adds needless work.

An LLM gateway earns its place when several applications or teams need one enforceable policy for model access. It does not earn its place merely because your architecture diagram has an LLM box. For one product, one provider, and a team that can still review every call site, a small internal client often gives you all the control you need with fewer failure modes.

The decision changes when provider credentials spread across services, bills have no reliable owner, fallbacks behave differently in each application, or nobody can reconstruct why a request reached a particular model. At that point, the extra network hop is usually cheaper than letting policy drift. The hard part is not installing a proxy. It is defining the identities, routing rules, cache boundaries, quota semantics, and audit record that the proxy must enforce.

## The threshold is policy drift, not request volume

A gateway becomes useful when the same rule must apply to multiple independent callers and you can no longer trust every caller to implement it correctly. Request volume can be tiny while the governance problem is already expensive. A founder with three AI features, two providers, background agents, and customer-specific budgets has a better gateway case than a company sending millions of calls from one well-contained service.

I use four signals. First, at least two applications need the same provider credentials or routing policy. Second, finance or product owners need usage attributed to a tenant, feature, or team rather than one provider account. Third, an outage or rate limit must trigger a controlled fallback without redeploying callers. Fourth, security or support needs a defensible record of what happened to a request. Two signals usually justify a short gateway evaluation. Three mean the team is already paying for the missing control, even if that cost hides in incident work and invoice reviews.

Do not confuse a shared SDK with a gateway. A shared SDK reduces duplicated code, but each deployed copy can run a different version, expose credentials, skip metadata, or apply stale policy. A gateway puts the decision on the request path and can reject a caller that omits required identity. The trade is equally concrete: you create another service that must stay available, scale streaming connections, protect secrets, and preserve provider-specific behavior.

The cleanest early boundary is an OpenAI-compatible endpoint owned by your platform or application team. Callers send a logical model name, tenant identity, feature name, and idempotency key. They do not choose a provider account or carry a provider secret. The gateway resolves that logical request against centrally versioned policy. If your team cannot name who owns that policy and who gets paged when the gateway fails, you are not ready to put it in production.

## Routing needs an explicit contract

Good routing maps an application intent to an approved target under stated conditions. Bad routing sends traffic to whichever model looks cheap or healthy without preserving the behavior the application depends on. Models are not interchangeable CPUs. A fallback can change tool-call syntax, context limits, safety behavior, structured output reliability, latency, and data residency.

Start with logical classes such as `support-summary`, `invoice-extraction`, or `coding-agent`, not marketing model names scattered through application code. Each class should define an ordered set of eligible targets and hard constraints. A support summary may tolerate a cheaper fallback. An extraction pipeline that validates a strict schema may only fall back to a model that has passed the same contract tests. A coding agent with a long context cannot fall back to a target that truncates its prompt.

Retries and fallbacks also need separate rules. Retry the same target only for failures that may clear, such as a transient connection reset or a provider response that explicitly permits a retry. Use bounded attempts, exponential backoff, jitter, and a total deadline. Do not retry authentication failures, invalid requests, or content-policy rejections against the same provider. A fallback selects another approved target after the route policy decides that the original target is unavailable or unsuitable.

Streaming makes careless fallback dangerous. Once the gateway has sent tokens to the client, switching models can duplicate text, break JSON, or repeat a tool call. The safe default is to allow fallback only before the first response byte. If an agent performs side effects, require the caller to attach an idempotency key and record tool execution separately. The gateway can make model requests more reliable, but it cannot make an arbitrary tool call safe to repeat.

Route on signals you can explain later: logical model class, tenant plan, region, measured health, context size, required capabilities, and an approved cost ceiling. Avoid opaque scoring until simple rules fail. When support asks why customer A received a different answer from customer B, the audit record should say `route_policy=v17`, `selected=extractor-primary`, and `reason=region_and_schema_capability`, not merely `smart_route=true`.

## Caching is safe only when equivalence is explicit

A response cache should return a previous answer only when the application can define which requests are equivalent and which data boundaries must never cross. This sounds obvious, yet teams routinely hash the prompt text, ignore hidden context, and call the result a cache key. The missing fields are where leaks and stale answers begin.

Separate three mechanisms that vendors often place under one caching label. Exact response caching stores a completed model response and reuses it for an identical application request. Semantic response caching treats similar inputs as equivalent according to an embedding or classifier threshold. Provider prompt caching lets a provider reuse computation for repeated prompt prefixes, while the model still generates a new completion. Prompt caching can lower latency or input cost, but it does not return an old answer and does not remove output variability.

An exact cache key should cover every input that can affect the result: tenant or security domain, logical model and policy version, system prompt version, normalized messages, tool definitions, response schema, temperature and sampling parameters, retrieval snapshot or knowledge-base version, locale, and any user permissions reflected in context. Include the gateway transformation version if it rewrites prompts. Store the selected target and output format with the entry so a future policy change can invalidate it.

Never share a semantic cache across tenants by default. Similar wording does not imply equivalent authorization, current data, or intent. A question such as "What is my contract renewal date?" looks semantically identical for every customer and must produce a different answer. Tenant namespacing is the minimum. For personal data, live inventory, financial decisions, tool-using agents, and responses whose truth changes quickly, disabling response caching is often the right policy.

Cache only successful, validated responses. Do not cache provider errors, partial streams, moderation failures, malformed structured output, or a fallback response that violated the requested capability. Set a short, domain-specific TTL rather than one global duration. Invalidation should follow the source data: a knowledge-base publication event can increment a content version, while a prompt deployment increments a prompt version. Time alone cannot tell you whether an answer remains valid.

Semantic caching deserves a separate risk review because a similarity threshold is an application decision, not an infrastructure constant. Test it with pairs that look close but require different answers, including negation, dates, account identity, product tier, and locale. Track cache decisions by cohort and inspect false hits. A high hit rate can be evidence that the threshold is too loose, not that the cache works well.

## Quotas require identity, cost, and concurrency

A useful quota system answers who spent the resources, what limit applied, and when the request should be rejected. Counting requests per API key rarely answers any of those questions. One request may contain a short classification prompt, while another carries a long conversation and produces thousands of tokens. Agents can also open many slow streams and exhaust connections before they hit a daily request cap.

Give each request an authenticated workload identity, then attach business dimensions supplied or verified by trusted infrastructure. Typical dimensions include tenant, environment, application, feature, team, and end user. Do not trust a public client to declare its own billing team or premium tier in an unsigned header. The gateway should derive those fields from the credential or verify a signed internal token.

Treat rate limits, budgets, and concurrency limits as different controls. A rate limit restricts activity during a short window and protects provider capacity. A budget caps measured or estimated spend over a billing period. A concurrency limit caps requests that are currently in flight, which matters for long streams and agent loops. You may also need a token ceiling per request to stop a single call from consuming the remaining allowance.

Enforcement before a call relies on estimates because the gateway does not know output tokens yet. Reserve an amount based on input tokens plus the configured maximum output, reject if the reservation exceeds the remaining budget, and reconcile after the provider reports actual usage. Release reservations on a verified cancellation or terminal failure. Without reservation, ten simultaneous requests can all pass a budget check against the same remaining balance and overspend together.

Define the rejection behavior as part of the product contract. Return a stable machine-readable code such as `tenant_budget_exhausted`, a retry time where one exists, and the policy scope that rejected the request. Do not silently route to a much cheaper model after a budget breach unless the application explicitly allows that model class. A wrong answer can cost more than the tokens you saved.

Provider-reported usage should remain the billing source where available, while tokenizer estimates help with admission and anomaly detection. Record both when they differ. Pricing tables change, so version the price data used to calculate internal cost. Otherwise last month's audit will change when somebody reruns it with today's price.

## Audit records must explain decisions without hoarding prompts

An audit trail should reconstruct access and policy decisions, while observability should help engineers operate latency, errors, and capacity. The two overlap, but they are not synonyms. A trace may be sampled and deleted quickly. An audit event usually needs controlled access, a defined retention period, and stable fields that can support an investigation.

For each request, record a generated request ID, timestamp, authenticated actor or workload, tenant, feature, logical model, policy version, selected target, selection reason, cache status, quota decision, input and output token counts, latency, provider request ID, response status, and fallback chain. Record administrative changes to routes, credentials, budgets, and logging policy as separate events with the actor and before-and-after version. If nobody records policy changes, the request log can tell you what happened but not why the rules changed.

Full prompts and completions are not mandatory audit data. They often contain customer content, personal information, source code, retrieved documents, or secrets that a user pasted by mistake. Logging all of it creates a second sensitive data store with broader access than the original application. Prefer metadata by default. If a team needs content for debugging or evaluation, use explicit sampling, redact known sensitive fields, encrypt storage, limit access, set a short retention period, and let the application mark requests that must never capture content.

Centralizing provider credentials changes the threat model as well. The gateway becomes a privileged broker that can spend every connected account's budget and see traffic from many applications. Give its runtime a separate identity, expose only the provider secrets required by each route, and keep administrative access away from request-processing credentials. Rotate a provider key through the gateway without restarting callers, then confirm that the old key no longer works. Restrict outbound network access to approved provider endpoints and the gateway's required state stores, because a compromised proxy with unrestricted egress can quietly forward prompts elsewhere. Protect internal bypass endpoints with stronger controls than the normal route, not a shared emergency token stored in a runbook. Finally, decide what happens when the policy database, cache, or audit sink is unavailable. Authentication and quota checks should normally fail closed. Audit delivery can use a bounded local buffer if the risk policy permits it, while cache failure should degrade to an uncached request rather than block all traffic. Write these choices down before an incident, because a generic `fail_open=true` switch combines unrelated risks into one dangerous decision.

Hashing a prompt is useful for exact correlation only when the normalization and secret-handling rules are clear. A plain hash of a predictable prompt can be guessed with a dictionary attack. An HMAC with a protected, rotated key gives better resistance and still lets you correlate identical content inside a defined period. Rotation breaks correlation across periods by design, which can be a privacy benefit.

Audit the gateway itself. Secret reads, configuration changes, log exports, bypass routes, and failed authentication attempts deserve records. Send logs to storage the gateway cannot rewrite after an incident. Then test a reconstruction: choose one production request ID and ask an engineer who did not build the gateway to identify the caller, applied policy, target, cost, cache decision, and outcome. Missing answers expose the schema gaps faster than a dashboard review.

## One innocent fallback can create an expensive incident

A common failure begins with two customer-facing services using the same logical model. The primary provider starts returning `429` responses. Service A retries three times in its SDK, the gateway retries twice, and a service mesh retries once because it sees a reset. Nobody multiplied the layers. One user action now creates many provider attempts, some continuing after the client has given up.

The gateway then falls back to a second model. That model accepts the text prompt but does not follow the same tool schema reliably. It emits a tool call with a missing account identifier. The application retries the whole agent turn, this time through the primary provider, which has recovered. Both turns reach the tool layer because the system has no idempotency key tied to the user action. One produces an error; the other performs the action. Support sees a slow request and a successful final response, while finance sees an unexplained burst of tokens.

Caching makes the diagnosis worse. The cache key includes the visible user message and model alias but omits the system prompt version and tenant. A later request from another tenant matches the normalized text and receives the earlier fallback response. The cache has converted a reliability incident into a data-isolation incident. The team disables the entire gateway, returning credentials and policy to application code during the worst possible hour.

The prevention is mechanical. Set one retry owner for each failure class and disable overlapping automatic retries. Cap total attempts and total elapsed time at the gateway. Allow fallback only before streaming starts and only between targets that passed the same contract tests. Require an idempotency key for an agent turn that can reach side-effecting tools, then let the tool layer enforce it. Namespace response caches by tenant and include every policy and context version that changes the answer.

The investigation also needs one request tree. The original request ID should parent every retry, provider attempt, fallback, cache lookup, and tool execution, with an attempt number and cause. Counting each attempt as an unrelated request hides amplification. Flattening them into one final status hides the failure path. Preserve both the user-visible result and the ordered attempts that produced it.

## A minimal gateway contract is deliberately boring

The first production contract should specify identity, logical routes, failure limits, cache boundaries, quotas, and an audit event. It should avoid autonomous cost optimization or semantic caching until the team can test those decisions. The following vendor-neutral fragment is small enough to review and strict enough to prevent the failure above:

```yaml
policy_version: 17
identity:
  required_claims: [workload, tenant, feature, environment]
routes:
  invoice-extraction:
    targets: [extractor-primary, extractor-approved-fallback]
    fallback_before_first_byte_only: true
    required_capabilities: [json_schema]
reliability:
  max_total_attempts: 2
  total_timeout_ms: 30000
  retry_owner: gateway
cache:
  mode: exact
  namespace: tenant
  key_fields: [route, policy_version, prompt_version, messages, tools, schema, locale]
  ttl_seconds: 300
quotas:
  reserve_max_output_tokens: true
  dimensions: [tenant, feature]
  concurrency_per_tenant: 8
audit:
  capture_content: false
  required_fields: [request_id, actor, tenant, route, policy_version, target, reason, cache, quota, usage, status]
```

A request should carry only fields the caller owns. Trusted middleware should add or sign identity. One possible request envelope looks like this:

```json
{"request_id":"req_01J...","route":"invoice-extraction","tenant":"tenant_42","feature":"invoice-import","prompt_version":"p8","idempotency_key":"import_948:extract","messages":[{"role":"user","content":"..."}]}
```

The corresponding event must describe the decision, not dump the whole request. Keep field types stable so security, finance, and operations can query the same stream:

```json
{"request_id":"req_01J...","attempt":1,"tenant":"tenant_42","route":"invoice-extraction","policy_version":17,"target":"extractor-primary","route_reason":"capability_match","cache":"miss","quota":"reserved","input_tokens_estimated":1840,"input_tokens_reported":1827,"output_tokens":211,"status":"ok"}
```

Test this contract with failure injection, not only happy-path unit tests. Force a timeout before the first byte, a disconnect after streaming starts, a provider `429`, an invalid schema response, an exhausted tenant budget, a concurrent reservation race, and a cache request with the same text under another tenant. Verify both the client response and the emitted audit sequence. A gateway that routes correctly but produces an ambiguous record has failed half its job.

## Open-source gateways solve different first problems

Open-source choices overlap, but their operational centers differ. Select the smallest system that satisfies your current contract, then verify which capabilities exist in the open-source edition you will run. Product pages often discuss hosted, enterprise, and open-source features together, so a feature matrix copied from marketing is not a deployment plan.

LiteLLM Proxy is a practical candidate when the main need is one OpenAI-compatible interface across many providers with centralized authentication hooks, logging, cost tracking, budgets, and rate limits. Its official getting-started documentation presents the proxy around those controls and supplies CLI and container deployment paths. It has a broad configuration surface, which is useful for a platform team and heavy for a startup that only needs one provider wrapper. Pin versions, review database requirements, and test provider-specific fields that your callers use.

Portkey's open-source gateway emphasizes a universal API plus routing strategies such as fallbacks, conditional routes, retries, circuit breaking, load balancing, cache, timeouts, budget limits, and rate limits. Its official documentation shows a local `npx @portkey-ai/gateway` entry point. This makes it worth evaluating when route composition and guardrail hooks drive the requirement. Confirm the boundary between the gateway framework, hosted observability, and paid controls against the version you plan to deploy.

Helicone AI Gateway focuses on a lightweight unified interface, provider selection, rate limits, response caching, and OpenTelemetry-compatible tracing, with a natural connection to Helicone's observability product. Its repository documents self-hosting and routing strategies based on latency, weights, or cost. It fits teams that put request visibility and routing together. As with any gateway that makes dynamic decisions, capture the selected strategy and reason in your own event schema rather than depending only on a dashboard.

Kong AI Gateway makes sense when the company already operates Kong or wants AI traffic governed through its wider API gateway model. Kong's documentation describes provider proxying, rate limiting based on token usage, standardized AI analytics, metrics, and semantic cache capabilities through plugins. Some advanced plugins and analytics belong to particular editions, so inventory the exact plugin licenses and dependencies before committing. Adopting a general gateway stack solely for one experimental LLM call is hard to defend.

Do not run a feature bake-off with twenty checkboxes. Implement one route from the minimal contract, replay a sanitized workload, inject the failure cases, and compare operational facts: p95 gateway overhead for your streaming pattern, configuration rollback, audit completeness, secret rotation, per-tenant isolation, behavior when its state store fails, and upgrade effort. Use your own traffic shape. Vendor benchmark numbers rarely include your authentication, logging, network path, or long-lived streams.

## A proxy is premature when the policy is still local

Skip the gateway when one application calls one provider, credentials already live in a proper secret store, usage has one owner, and a provider outage can be handled by a controlled application error. Put a narrow adapter around the provider SDK instead. Emit structured usage events, attach request IDs, set timeouts, cap output tokens, and keep the adapter interface expressed in application terms. That preserves a migration path without operating another service.

A gateway is also premature when the team has not decided its routing or data policy. Centralizing undefined rules does not create governance. It creates a shared place for ad hoc exceptions. Write the route contract, data classification, retention rule, and budget owner first. If those fit on one page and one service can enforce them locally, keep them local until a second independent caller appears.

Avoid building a custom gateway because the first provider SDK feels untidy. Proxy work expands quickly: streaming backpressure, disconnect propagation, tokenizer drift, credential rotation, state-store failure, retry storms, schema compatibility, regional routing, log redaction, and provider API changes all become your responsibility. A thin adapter is code. A gateway is production infrastructure. Treating those as the same category produces under-owned systems.

When several teams already need shared policy, run a two-week proof with one read-only or otherwise reversible workload. Keep provider-specific escape hatches visible, but require an owner and expiry for each one. Measure added latency, route accuracy, cache correctness, quota races, audit reconstruction time, and the number of application changes needed. Do not start with semantic routing or cross-tenant caching. Deterministic policy gives you a baseline you can reason about.

In a Team & AI Audit, I look for this ownership boundary before recommending another layer. The deciding evidence is usually mundane: duplicated secrets, inconsistent retry code, invoices nobody can allocate, and incidents nobody can reconstruct. If those problems are absent, spend the week improving the application. If they are present across independent callers, choose a gateway against a written contract and make one team responsible for its behavior.

A gateway should remove policy from places where it will drift, not hide decisions behind a new box. Deploy it when central enforcement has a named owner and a tested failure model. Until then, a disciplined adapter is the more mature architecture.