# How an AI pipeline fallback path protects throughput

> Design an AI pipeline fallback path that survives rate limits and outages while controlling cost per accepted result and preserving customer commitments.

An AI pipeline without a priced fallback path is a single dependency with a nicer interface. It may look dependable while traffic is light, then rate limits, exhausted quotas, or a provider incident turn every queued request into a more expensive version of the same failure.

The fix is not to buy access to every model and call that resilience. You need to decide, before an incident, what each class of request is allowed to become, what that alternative costs, and what counts as a delivered result. Teams that skip this work usually discover their policy while a backlog grows and someone is approving an emergency spend increase.

## A fallback route needs a delivery contract

A fallback route is a complete way to finish a request under changed constraints. It has an input rule, a provider or non-model action, an output contract, a time limit, and a cost limit. "Try another model" is an intention, not an operating policy.

Start by naming the result your customer actually receives. A support reply may require correct account facts, a helpful tone, and a human review flag when confidence is low. A document extraction job may require every required field, source citations, and a machine readable schema. A coding request may require a patch that passes tests. Those are different delivery contracts, so they deserve different fallback routes.

Many teams confuse three events that demand different behavior:

- A transient failure means the same route may work after a bounded wait.
- A capacity failure means the route has no room for this request now.
- A quality failure means the route answered, but the result does not meet the contract.

Treating all three as errors to retry wastes money. A timeout might merit one retry with jitter. A rate limit should move the request to a queue or another capacity pool. A malformed extraction should use a repair pass only if that repair pass has a lower expected cost than running the whole job again.

Write the route as an explicit contract. For example, an invoice extraction fallback can accept only PDFs with legible text, use deterministic OCR plus field rules, return `needs_review: true` for missing tax values, and cost less than the normal model path. It does not pretend it can resolve every ambiguous scan. That limitation is what makes it safe to use.

The useful distinction is between availability and equivalence. A fallback can keep a service available without producing an equivalent result. If you promise equivalence where you only have availability, your team silently hands lower quality work to customers. State the quality tier in the internal result and expose any customer facing delay or review state where the product contract calls for it.

## Posted token price is the wrong number

Choose the order of routes by cost per accepted result, because a cheap call that fails validation is not cheap. The number includes every attempt required to produce work that passes the delivery contract.

Use this calculation for each task class:

```text
cost per accepted result =
(provider and tool spend + retry spend + review cost + repair cost)
/ accepted results
```

Keep the denominator narrow. A response that arrived is not automatically an accepted result. Count an extraction only after schema validation and required fields pass. Count a generated patch only after its test gate passes. Count a customer answer only after it passes the checks you already use to decide whether an agent may send it.

Suppose Route A costs $0.18 per attempt and passes 92 out of 100 requests without review. Route B costs $0.07 per attempt but passes 55 out of 100, and the rejected work needs another $0.18 attempt or human review. Route B can still be useful for simple work, but it is not the automatic cheaper fallback for the same contract. This is where an attractive pricing page turns into a bad engineering decision.

Separate fixed overhead from marginal cost. A warm worker pool, an OCR license, and observability tooling may be worth paying for if they keep a high volume workflow moving. Do not assign those costs to a low volume experiment and declare the route unworkable. At the same time, do not hide a human review queue in a shared operations budget and call the model route economical.

Use a small scorecard for every route: accepted-result cost, p95 completion time, acceptance rate, and the reason requests leave the route. The fourth field matters. A route that loses work because of capacity needs a different response from one that loses work because it cannot follow a schema.

## One provider account is not a capacity plan

A provider outage and quota exhaustion can look identical to your caller, but their remedies differ. Both require a route that does not depend on the same exhausted limit.

A common failure begins with an application that sends every request to one model deployment. The team adds exponential backoff after seeing HTTP 429 responses. Workers keep retrying, jobs stay active, and the queue fills. When the quota window finally resets, every waiting worker wakes up and sends another wave. The recovery traffic creates another rate limit, while customers see long delays without a clear state.

RFC 6585 defines HTTP 429 as Too Many Requests and allows a server to send `Retry-After`. RFC 9110 describes `Retry-After` for responses including 503 Service Unavailable. Respecting that header is basic protocol behavior, but it does not create capacity. It only tells you when the failed route says it may be worth trying again.

Your controller should make a capacity decision after a small retry budget. It can delay a nonurgent request, move an eligible request to an independent provider, use a smaller model for a lower quality tier, or send the work to review. The controller must also know when none of those outcomes are acceptable. A contract generation flow that needs a particular model's reviewed output should report a pending state rather than quietly send an inferior draft.

Independence deserves scrutiny. Separate API credentials with one shared organization quota do not give you meaningful protection. Separate regional deployments may help with a regional incident but not a global quota rule. A different provider can protect against a provider outage, while shared upstream tools, shared databases, or shared network egress can still stop both routes. Draw the dependency chain before you claim redundancy.

## A queue is where graceful degradation becomes real

Synchronous requests make every capacity event feel urgent. A queue lets you distinguish "cannot answer in two seconds" from "cannot complete the work." That difference gives you room to apply a fallback order without letting callers pile onto a failing provider.

Classify requests when they enter the system. Use a small set of classes that match customer commitments, such as interactive reply, background enrichment, batch extraction, and internal research. Each class needs a maximum wait, a cost ceiling, and permitted output tiers. Do not let every caller declare itself urgent. That is how a routine batch job steals capacity from a paying customer waiting for a reply.

A practical routing record can be this small:

```json
{
  "job_type": "invoice_extract",
  "quality_tier": "complete_or_review",
  "deadline_seconds": 300,
  "max_cost_cents": 35,
  "attempts_by_route": {
    "primary": 1,
    "secondary": 0,
    "ocr_rules": 0
  }
}
```

The worker reads this record before it calls any provider. The router rejects a route if its expected cost would exceed the remaining budget or if it cannot satisfy the requested tier. It writes the selected route and reason back to the job record. That history is what lets an engineer explain why a result took longer or needed review without guessing from scattered logs.

Put backlog age into the decision. A background task that has waited five minutes may be cheaper to process on a secondary provider than to keep in the queue for an hour. An interactive task that has already missed its response target may need a concise lower tier reply plus an explicit follow up rather than a perfect answer delivered too late.

Do not use one queue for work with radically different economics. If a million cheap enrichment jobs sit ahead of a small number of time sensitive customer requests, no model choice will rescue the experience. Partition by commitment, then set route policies inside each partition.

## The fallback order should be a policy, not a prompt

Put fallback decisions in code and configuration that operators can inspect. A prompt that says "use another method if unavailable" gives the model authority over spend, quality, and provider choice without an audit trail.

This example policy makes the order visible. The names are placeholders; bind them to your own routes.

```yaml
invoice_extract:
  primary:
    route: structured_model
    max_attempts: 2
    retry_on: [timeout, 503]
  fallbacks:
    - route: secondary_structured_model
      when: [429, quota_exhausted, provider_unavailable]
      max_cost_cents: 28
      requires: [json_schema]
    - route: ocr_and_rules
      when: [primary_quality_failure, secondary_quality_failure]
      max_cost_cents: 12
      output: needs_review
  terminal:
    route: review_queue
    when: [deadline_exceeded, missing_required_field]
```

The failure this prevents is subtle. Without the `requires: [json_schema]` condition, a router may select a cheap text model after a structured output provider fails. The text looks plausible in logs. Downstream code then guesses at field boundaries, inserts an amount into the wrong record, or fails much later with an error that appears unrelated to the fallback. The route succeeded from the provider's view and failed from the business's view.

Keep a terminal state. Every request must end as delivered, delivered with review, deferred, or failed with an actionable reason. "Still retrying" is not an outcome. A terminal state protects the queue from permanent residents and lets customer support give an honest answer.

The policy should also have a version. When someone changes the order after a cost spike, tag every result with the policy version. Otherwise a week of route data becomes hard to compare, and a good change can look bad because it inherited traffic from an earlier outage.

## Circuit breakers protect the recovery window

A circuit breaker stops a route after evidence shows that more calls will probably fail. It is not a substitute for health checks or rate limiting. It is a guard against turning a provider problem into your own queue and spend problem.

Open a breaker on a signal you can defend: a sustained run of provider failures, a documented quota event, or a sharp acceptance failure in a route that normally passes. Do not open it because one request produced a poor answer. Quality incidents often belong to an input class, a prompt version, or a tool dependency rather than the provider as a whole.

When the breaker opens, stop dispatching new work to that route for a defined interval. Send eligible work to the next route, defer work that cannot degrade, and keep a small probe budget. One probe at a controlled interval tells you whether the route recovered. Releasing the full queue at the first successful probe recreates the overload that opened the breaker.

Track provider health separately from task health. A model may return HTTP 200 while a schema pass rate collapses after a prompt or model version change. The provider is available, but your route is unhealthy. A useful breaker can open on acceptance rate for a specific job type, while other job types continue to use the same provider.

This is why a single global "AI is down" flag causes damage. It sends inexpensive, low risk classifications to manual work even when they are still completing correctly. Scope the breaker to the dependency and contract that failed.

## Smaller models belong behind a clear boundary

Falling back to a smaller model works when the task is bounded enough that capability loss is visible and controlled. It works poorly when the same request hides difficult judgment behind a short prompt.

Good candidates include intent classification with a fixed label set, language detection, straightforward extraction with strict validation, deduplication, and routing. Give the smaller route fewer choices. Ask it to return a schema, select from approved values, or explain uncertainty through a review flag. Do not expect it to compensate for missing context by sounding confident.

The popular recommendation to send every request to the cheapest model first is wrong for customer work. It is popular because the invoice drops quickly and the demo still looks fine. The hidden bill appears in rework, support contacts, inconsistent outputs, and senior engineers reading traces to determine which model made a decision. Put cheap models first only after your accepted-result data shows they satisfy a defined class.

Use input gates. A simple heuristic or classifier can distinguish a routine request from one that needs the primary route. For example, an extraction may use a lower cost route only when the document has one language, readable text, and a known template. The moment those conditions fail, route upward or to review. This avoids using an expensive model to inspect every easy item while keeping difficult work out of an unsuitable fallback.

Keep the quality tier in the response object. A downstream system that needs a final answer must reject a `draft` or `needs_review` result. That one field blocks a frequent failure: a temporary degradation path becomes permanent because another team treats all successful HTTP responses as equally trustworthy.

## Tool calls need their own degradation plan

A model fallback does not help if the model can only finish through a failed tool. Many AI workflows depend on search, retrieval, code execution, databases, email, or internal APIs. The model may be available while the workflow is effectively blocked.

List tool dependencies beside model routes. For each one, decide whether the request can continue with cached data, a reduced answer, a deferred job, or human review. A customer support agent may answer account policy questions from a dated knowledge snapshot with a date marker, but it should not claim a current order status when the order system is unavailable.

Make tool failure visible in the route result. Store a reason such as `customer_api_unavailable` rather than a vague `model_failed`. That separation stops people from swapping models during an internal database outage. It also gives incident responders an accurate count of customers who received a degraded response.

Cache only information whose staleness you can explain. A cached product description can be acceptable. A cached balance, inventory level, or access permission can be wrong in a way that costs money or trust. In those cases, defer the action or require a person to check the source system.

## Test the failure paths before customers do

A fallback design is unproven until you deliberately break each route. Provider sandboxes rarely reproduce quota exhaustion, malformed output, slow responses, and partial tool failures in the exact mix your application sees. Inject those conditions at the router boundary.

Run a small drill with representative jobs. Return a synthetic 429 with `Retry-After`, make the primary route time out, force a schema violation from the secondary route, and make a required tool unavailable. Inspect the final job state, the selected fallback, the spent budget, the customer visible response, and the queue age. If an engineer cannot read that story from one trace, an incident will be slower than it needs to be.

Test the ugly case too: the primary route recovers while the secondary route is handling a backlog. The policy must decide whether queued jobs stay on the selected route or move back. Switching every in-flight job can create duplicate work and unexpected cost. In most cases, let an accepted job finish on its chosen route and send only new work through the recovered primary path.

Review these drills after every meaningful change to routing, model versions, or tool contracts. A fallback that worked against an old JSON schema may be unsafe after a downstream team adds a required field. Reliability work decays when the main path changes and the degraded path does not.

## Spend limits should shape the customer promise

Cost controls fail when they exist only in a finance dashboard. A route needs a request budget before it starts work, and the product needs an honest behavior when that budget runs out.

Set a ceiling per task class and a broader ceiling for a customer or tenant. The per request ceiling stops one pathological prompt from consuming unlimited retries. The tenant ceiling stops a noisy integration from using capacity needed by everyone else. Let the router choose a lower tier, defer the job, or require review when either ceiling applies.

Do not bury degraded outcomes in generic success metrics. Report the percentage of requests that used a fallback, the route chosen, acceptance rate by route, queue delay, and cost per accepted result. Segment by job type. A healthy aggregate can hide a broken contract workflow if easy classification traffic dominates the total.

A founder should ask one blunt question in the weekly engineering review: if the preferred provider disappears for a day, which customer commitments keep moving and which ones change? If the answer is a vague promise of retries, the pipeline has no priced fallback path.

For an outside review, a Team & AI Audit can map those dependencies, route costs, and failure contracts before a growth spike turns them into an emergency project.