Which LLM cost optimization techniques pay off?

LLM bills rarely need an exotic fix. Most teams can remove a large share of spend by charging each request correctly, cutting avoidable output, reusing stable input, and moving patient work off the synchronous path. Routing comes after those changes. Distillation comes much later.

I have watched teams spend weeks building a clever model router while every request still carried a 12,000-token policy manual and generated 900 tokens that nobody read. That reverses the economic order. Optimize the tokens and service level first, then decide whether the remaining volume justifies new model infrastructure.

The ranking below uses effort, likely payoff, and operational risk. The dollar example uses 100,000 monthly requests, each with 6,000 input tokens and 500 output tokens. For a concrete reference, it uses the published standard rates for GPT-5.6 Terra: $2.50 per million input tokens, $0.25 per million cached input tokens, and $15 per million output tokens. Prices move, so keep the formulas and replace the rate card before approving a project.

Rank savings against your own token mix

The usual order is prompt diet, caching, batching, routing, then distillation, but the correct rank depends on what the bill actually contains. A support assistant with a long shared policy prefix should favor caching. An overnight extraction job should favor batching. A short interactive classifier may have little to cache and plenty to gain from a smaller model.

Use this ranking as an investment queue, not as a universal benchmark. Prompt diet and output controls require low effort and save money on every removed token, but they must preserve the context needed for a complete answer. Prompt caching takes low to medium effort and pays when a stable prefix dominates input; exact reuse and provider rules constrain it.

Batch processing also takes low to medium effort. It offers a published 50% token discount on eligible asynchronous work, in exchange for a wait that may last hours. Routing takes medium effort and can save more when many requests pass on a cheaper model, but it needs labeled evaluations and fallback logic. Distillation takes high effort and belongs at the end because training, serving, drift, and retraining can consume the apparent gain.

First compute spend from metered token classes, not invoice totals. A request can contain uncached input, cached input, visible output, hidden reasoning tokens, tool charges, and retrieval charges. The provider may bill those classes at different rates. If the usage event collapses them into one number, no optimization team can explain a saving or catch a regression.

For the example workload, the monthly baseline is straightforward:

input_cost  = 100,000 * 6,000 / 1,000,000 * $2.50 = $1,500
output_cost = 100,000 *   500 / 1,000,000 * $15.00 = $750
total_cost  = $2,250
cost_per_successful_request = total_cost / successful_requests

The last line matters more than cost per call. A cheaper call that fails and gets retried can cost more than one good call. Record at least request_id, task class, model, input tokens, cached tokens, output tokens, latency, retry count, and an outcome score. A finance dashboard without an outcome denominator rewards broken systems.

Do not use average tokens alone. Keep p50 and p95 input and output counts by task class. Long conversations, giant retrieved documents, and runaway agent loops hide in the tail. I have seen a small fraction of requests create most of a cost surprise while the average looked respectable.

Allocate shared costs with the same care. A retrieval index, cache storage charge, gateway, or evaluation service may support several task classes. Put those charges in a separate pool and allocate them by a stated driver such as requests, stored tokens, or model spend. Do not hide them inside one convenient task. The allocation will never be perfect, but a consistent rule lets a founder compare months and prevents one route from appearing artificially cheap.

Tag experiments as well. When a team changes a prompt, model, cache prefix, or queue policy, attach the configuration version to every usage event. Without that version, analysts compare mixed cohorts and attribute noise to the latest change. Keep the previous path available long enough to form a control group under the same traffic conditions. This costs a little during the test and avoids approving a saving that disappears after rollout.

Prompt diets pay before architecture changes

A prompt diet should remove context that does not change the answer and cap output at the length the product can use. This is usually the fastest saving because it touches configuration and content assembly rather than model infrastructure.

Start with output. In the example, output costs six times as much per token as fresh input. Removing 100 unnecessary output tokens from every request saves $150 a month, while removing 100 input tokens saves $25. Teams often polish the system prompt and ignore a model that writes five paragraphs into a UI slot built for two sentences.

Set an output limit by task, not one global maximum. A category label may need 10 tokens. A customer reply may need 250. A code change can need thousands. Then tell the model the expected shape in plain terms or a schema. The limit catches a failure; the instruction prevents it.

Input needs a harsher edit. Delete duplicated rules, examples that teach the same behavior, verbose role prose, raw tool catalogs that are irrelevant to the current step, and retrieved chunks with weak relevance. Keep security constraints, domain definitions, format requirements, and evidence the task actually needs. Shorter is not automatically better. A missing constraint that increases retries wipes out the saving.

Suppose the team removes 1,000 input tokens and 100 output tokens per request. The new mix is 5,000 input and 400 output tokens:

input  = 500M * $2.50 = $1,250
output =  40M * $15.00 = $600
total  = $1,850
saving = $400, or 17.8%

That is a real reduction with almost no runtime complexity. It also lowers latency because the provider reads and generates fewer tokens, although the exact latency change depends on model and load.

One popular recommendation is to compress every instruction into cryptic shorthand. It is wrong. Tokenizers do not price characters uniformly, engineers cannot safely review dense prompt code, and the model may need clearer language to follow a constraint. Remove irrelevant meaning before shortening useful meaning. Test each edit against the same evaluation set and revert any edit that lowers the acceptance rate beyond the agreed tolerance.

Prompt diets also include conversation control. Do not resend an unlimited chat transcript. Preserve the recent turns required for reference resolution, keep durable user facts in structured state, and summarize older material only after testing the summary for lost obligations. A summary that drops a refund promise or a security exception is expensive in a way the token bill will never show.

Caching wins on stable prefixes

Prompt caching pays when many requests share a long, byte-stable prefix. Put stable instructions, tool definitions, examples, and shared reference material first. Put the user's message, current retrieval results, timestamps, and request-specific state afterward.

This distinction matters: application response caching reuses a final answer, while prompt caching reuses the provider's processing of input tokens. A response cache is suitable for deterministic, repeatable questions with the same authorization context. A prompt cache still runs the model for each request, so it can answer a new question against reused context. Confusing the two creates either poor hit rates or dangerous cross-user answer reuse.

OpenAI reports cached tokens under the input token details in its API usage object. Anthropic splits cache creation and cache reads, with a five-minute cache write priced at 1.25 times base input and a cache hit at 0.1 times base input for the models in its published pricing table. Google documents implicit caching for recent Gemini models and advises placing large common content at the beginning. The mechanics differ, but all three reward stable, repeated prefixes rather than merely similar ideas.

Assume 3,000 of the dieted prompt's 5,000 tokens become cache hits on every request. Using the example rates:

cached_input = 300M * $0.25 = $75
fresh_input  = 200M * $2.50 = $500
output       =  40M * $15.00 = $600
total        = $1,175

That is 47.8% below the original $2,250 baseline and 36.5% below the prompt-diet result. The result is large because cached input costs one tenth of fresh input in this rate card and the shared prefix is long.

Cache arithmetic needs the write cost and hit rate. With a provider that charges extra to create a cache entry, a prefix used once costs more than an ordinary request. The break-even reuse count for one cached block is:

break_even_reuses = cache_write_premium / (fresh_read_price - cached_read_price)

Use actual usage events to measure cached_tokens / cache_eligible_tokens. Do not infer hits from lower latency. Changes as small as a timestamp, reordered JSON property, dynamic tool description, or per-user banner placed before the stable block can break prefix identity. Canonicalize serialized objects, keep volatile values at the end, and version the shared prefix deliberately.

Security boundaries still apply. Never make cache reuse broader than the provider's isolation rules and your own tenant model allow. Prompt caching is safe only when account boundaries, retention settings, and sensitive-data handling meet the application's requirements. A discount does not change the classification of the cached content.

Batch work that can honestly wait

Batch processing is the cleanest fixed discount when users do not need an immediate answer. OpenAI's Batch API documentation says jobs complete within a 24-hour window for a 50% discount. Anthropic publishes a 50% discount on batch input and output tokens. Google publishes the same 50% reduction for its Batch API. Those are rate-card terms, not performance estimates.

Good batch candidates include nightly classification, document extraction, embedding backfills, evaluation runs, catalog enrichment, report generation, and queued email drafts that a human reviews the next morning. Live chat, interactive coding, fraud decisions, and a user waiting on a page are poor candidates even if finance likes the price.

On the untouched example baseline, moving every request to a provider's half-price batch tier would reduce $2,250 to $1,125. If only 60% of requests can wait and their token mix matches the whole workload, the saving is 30%:

batchable_share = 60%
discount        = 50%
portfolio_saving = 60% * 50% = 30%
new_total        = $2,250 * 70% = $1,575

Do not stop at submitting a JSONL file. A production batch path needs stable custom IDs, idempotent result ingestion, per-item error handling, expiration handling, and a dead-letter queue. The batch can complete while individual items fail. If the importer treats missing rows as empty answers, the business gets silent data loss at a discount.

Batching and prompt caching can coexist on some platforms, but pricing modifiers and cache behavior vary. Check the chosen model's current table and run a small billed sample. Never add a 50% batch saving to a 90% cache saving as if percentages were dollars. Apply each rate to its eligible token class and show the equation.

The operational compromise is usually two queues. A standard queue handles deadlines measured in seconds. A batch queue handles deadlines measured in hours. Put the deadline in the job payload, promote a job only when its business deadline approaches, and measure how often promotion happens. If half the jobs get promoted, the product does not truly tolerate batch latency.

Route by measured difficulty, not prompt length

Routing saves money when a cheaper model passes the task's quality bar on a substantial share of traffic. The router should predict that pass condition, choose the cheap path, and escalate uncertain or failed cases to the stronger model.

Prompt length is a weak routing rule. A long extraction can be simple, while a two-sentence request can require subtle legal or technical judgment. Route first by task class and observable risk: required tools, schema complexity, language, document type, user tier, safety category, and confidence from a cheap classifier. Add prompt features only when an offline evaluation proves they help.

For a transparent estimate, route 70% of the dieted workload to GPT-5.6 Luna at its published rates of $1 per million fresh input tokens, $0.10 per million cached input tokens, and $6 per million output tokens. Leave 30% on Terra. Keep the assumed 3,000 cached and 2,000 fresh input tokens plus 400 output tokens per request:

Luna, 70%: 210M*$0.10 + 140M*$1.00 + 28M*$6.00  = $329.00
Terra, 30%: 90M*$0.25 +  60M*$2.50 + 12M*$15.00 = $352.50
routed total = $681.50

The routed estimate is 69.7% below the original baseline. That number is not a forecast for another product. It assumes a 70% acceptance rate on the cheaper model, no extra classifier charge, identical output length, stable cache hits, and no retries. Change any of those inputs and the answer changes.

An escalation can also erase savings. If 20% of cheap-model calls get repeated on the expensive model, those requests pay twice for input and output. Track first-pass acceptance, escalation rate, total cost after escalation, and quality by route. The router has succeeded only when cost per accepted result falls.

Amazon Bedrock's intelligent prompt routing documentation makes a useful limitation explicit: its router predicts quality between models in the same family, and AWS says application-specific performance data does not directly adjust the routing decision. That convenience may fit a general workload. For a narrow product, I prefer a small explicit router trained and tested on the product's own acceptance labels because the team can explain why a request moved.

Start with rules before building a learned router. Fixed task classes are easy to debug and often capture most of the gain. A learned router earns its maintenance burden only after the rule-based confusion matrix shows enough ambiguous volume.

Distillation is a product commitment

Distillation pays when a narrow task has high, steady volume and a smaller model can learn the required behavior from curated teacher outputs. It is not a switch on a rate card. The team takes responsibility for training data, evaluation, deployment, capacity, monitoring, and retraining.

The field often blurs fine-tuning and distillation. Fine-tuning adapts a model to examples, format, tone, or domain behavior. Distillation specifically uses a stronger teacher to produce labels or probability targets for a smaller student. A fine-tuned frontier model may improve quality without lowering inference price. A distilled small model aims to preserve enough quality at a lower serving cost. The business case depends on that serving difference.

Use a break-even calculation before collecting data:

break_even_requests = (data + training + evaluation + deployment + retraining)
                      / (teacher_cost_per_accepted_result - student_cost_per_accepted_result)

Consider a hypothetical project with $12,000 in total initial and near-term maintenance cost. If the teacher costs $0.010 per accepted result and the student costs $0.002 including hosting, retries, and fallback, the saving is $0.008 per result. Break-even arrives at 1.5 million accepted results. At 100,000 results a month, that takes 15 months before the model produces net savings. If the task changes every quarter, the project never reaches its own break-even point.

That hypothetical also explains why teams overestimate payoff. They compare a teacher API token price with raw GPU time and omit idle capacity, autoscaling headroom, observability, engineer time, and the teacher fallback. Include every recurring cost. A self-hosted student with 20% utilization can be more expensive than a managed API that charges only when called.

Distill tasks with objective acceptance criteria: extraction into a schema, classification with stable labels, reranking against judged pairs, or repeated transformations with clear invariants. Avoid distilling broad advisory work whose acceptable answer changes with context and policy. A small model can imitate yesterday's confident phrasing while missing today's boundary condition.

Teacher output also needs review. Sampling a large model does not create ground truth. Filter malformed responses, remove private data, balance rare cases, and keep a human-labeled holdout set that no teacher generated. Otherwise the student can score well by reproducing the teacher's blind spots.

Stack techniques with one cost model

The techniques multiply across eligible traffic; their headline percentages do not add. Prompt diet changes the token base. Caching changes the price of part of the remaining input. Routing changes the rate card for a share of requests. Batching changes the service tier for work that can wait. Distillation replaces a route only after its full cost beats the alternative.

The worked sequence moved the example from $2,250 to $1,850 with a prompt diet, then to $1,175 with caching, then to $681.50 with routing. If 60% of that routed workload receives a true 50% batch discount across its billed token classes, the arithmetic would be:

interactive_share = $681.50 * 40% = $272.60
batch_share       = $681.50 * 60% * 50% = $204.45
combined_total    = $477.05
cumulative_saving = 78.8%

Treat $477.05 as a sensitivity case, not a budget promise. It depends on provider rules allowing the assumed discounts together, equal token mixes across queues, a 70% cheap-model route, and unchanged quality. The useful artifact is the spreadsheet logic behind it.

Build the cost model with rows for task classes and columns for request count, fresh input, cached input, output, tool calls, retry rate, model rate, batch share, and acceptance rate. Finance should be able to change one cell and see the monthly effect. Engineering should be able to trace each cell to usage telemetry or a published rate.

Watch for interactions that increase spend. A shorter prompt may change its prefix and lower cache hits. A small model may produce longer answers or call more tools. A batch job may retry an entire file instead of failed rows. A distilled model may need a costly teacher fallback on rare cases. Measure the combined system after every change rather than multiplying laboratory savings.

Commit rate cards and optimization assumptions alongside the service configuration. When a provider changes model names or prices, rerun the model. A route that saved money last quarter may become pointless when the stronger model drops in price or the cheaper model's output rate rises.

Quality guardrails make savings real

Cost optimization is complete only when the cheaper system meets the product's acceptance bar. Define that bar before changing prompts or models. Otherwise every regression becomes an argument and the cheapest configuration wins by exhaustion.

Use task-specific measures. Structured extraction needs field accuracy, schema validity, and abstention behavior. Support replies need policy compliance, factual grounding, resolution rate, and human edit distance. Agentic coding needs tests, static checks, security review, and the number of repair loops. One generic judge score cannot cover all of them.

Keep a frozen evaluation set with common cases, expensive tail cases, and failures already seen in production. Add fresh production samples on a schedule, but do not replace the frozen set, or trend lines lose meaning. Where mistakes carry financial, legal, security, or safety consequences, require human review and price that review into cost per accepted result.

Online controls need hard budgets. Set maximum input and output tokens by task, a maximum number of tool calls, a maximum retry count, and a per-request dollar ceiling. When an agent reaches a limit, stop with an explicit error or send it to a deliberate fallback. Quietly allowing another loop teaches the system that budgets are optional.

Use a small canary before full rollout. Compare the old and new paths on acceptance rate, p95 latency, total tokens, cache-hit tokens, escalations, and cost per accepted result. Segment results by task and language. A routing policy that works on English support tickets may fail on another language even when the overall average looks good.

Savings claims should survive reconciliation. Sum metered cost from usage events, compare it with the provider invoice, and explain the gap from rounding, tool fees, storage, or credits. Then compare the optimized cohort with a control cohort over the same demand mix. A falling invoice during a quiet month is not an engineering win.

Run the optimization in dependency order

A disciplined rollout can produce evidence inside a month without attempting five changes at once. Sequence the work so each measurement supports the next decision.

1. During the first week, instrument token classes, outcomes, retries, latency, task class, and per-request cost. Reconcile one full day with the provider's usage report.
2. During the second week, cap outputs and remove irrelevant prompt material behind a feature flag. Run the frozen evaluation and a production canary.
3. During the third week, stabilize shared prefixes, measure cache eligibility and actual cache hits, then move truly patient tasks to a batch queue.
4. During the fourth week, evaluate a cheaper model by task class and deploy simple routing rules with an expensive-model fallback.
5. Write a distillation proposal only if the measured residual volume, task stability, and break-even period justify it.

Assign one owner to the cost model and one business owner to the acceptance bar. They can be the same person in a small company, but both decisions need names. A committee-owned threshold changes whenever a bad example reaches the meeting.

For founders who lack clean telemetry or an evaluation set, a Team & AI Audit through oleg.is can map the token flows, task mix, and engineering work before a larger transformation. The fixed offer costs $5,000, takes five business days, and guarantees at least $50,000 a year in identified savings or it is free. That only makes sense for a company with enough engineering and AI spend to support the target.

Do not approve a router because its demo picks the small model on easy prompts. Approve it when thirty days of production evidence show lower cost per accepted result, stable tail latency, and a fallback rate the team can explain. If prompt cleanup, caching, and batch processing already captured most of the gain, stopping there is good engineering.