GPU cost optimization starts with workload shape

GPU cost optimization fails when a startup treats every accelerator hour as interchangeable. A cheap hour that restarts a training run, an expensive card that sits idle behind a quiet endpoint, and an API call that removes an operations burden belong in different ledgers. Compare the cost of a completed unit of work, not the price printed beside a GPU name.

I have seen founders negotiate a respectable cloud discount while ignoring the larger waste: notebooks left running overnight, replicas sized for a launch spike that never returned, and training jobs that could not survive an interruption. The fix starts with workload shape. Training, batch inference, and interactive inference need different capacity, recovery, and latency choices.

Price each completed result, not each GPU hour

A useful GPU cost model includes every resource required to produce an accepted result. The GPU rate matters, but so do CPU and RAM on the host, storage, data transfer, startup time, failed work, idle replicas, engineering time, and any capacity commitment that goes unused. A team that reports only accelerator charges cannot tell whether a cheaper instance improved the business.

Choose one business unit for each workload. Training might use cost per successful model candidate or cost per million training samples. Batch inference might use cost per thousand accepted outputs. An interactive service might use cost per request that meets its latency target. Keep rejected outputs and retries in the denominator calculation. They consumed money even if the product discarded them.

Use this basic equation for a measured window:

fully_loaded_cost = gpu_compute + host_compute + storage + transfer + api_fees + operator_time
useful_unit_cost = fully_loaded_cost / accepted_units

Operator time belongs there when the choice changes operational work. If an engineer spends four hours each week nursing unavailable Spot capacity, estimate that labor at its real loaded rate. If an API removes driver upgrades, autoscaling work, and on-call pages, credit it for those avoided hours. Do not pretend labor is free because payroll sits in another budget.

Tag costs by workload, environment, model, owner, and purchase type. A shared GPU node without job attribution produces an invoice, not evidence. At minimum, emit a job ID into the scheduler, application logs, and cost export. Then join completed outputs and errors back to the same ID.

A monthly total also hides timing. Record queue time, startup time, active compute time, checkpoint time, and idle time separately. A batch system can look busy for ten hours while kernels execute for three. An online endpoint can show high utilization for five minutes and remain empty for the rest of the day. Both need a timeline, not an average alone.

Memory fit and compute use answer different questions

Right sizing starts with memory capacity, then tests throughput. Teams often read 70 percent memory use as 70 percent GPU use. That conclusion is wrong. Model weights can occupy most device memory while the compute units wait on request arrivals, CPU preprocessing, storage reads, or synchronization between workers.

Capture utilization during representative traffic, not during a handpicked benchmark. NVIDIA documents that nvidia-smi GPU utilization measures the share of the recent sample period in which one or more kernels ran. Memory utilization measures time spent reading or writing device memory, while memory.used reports allocated frame buffer memory. Those three numbers describe different constraints.

This command gives a readable record that a startup can collect before changing instance types. An example response has the same field hierarchy that the tool prints:

$ nvidia-smi -q -d MEMORY,UTILIZATION,POWER
GPU 00000000:01:00.0
    FB Memory Usage
        Total : 24564 MiB
        Used : 9180 MiB
    Utilization
        GPU : 37 %
        Memory : 21 %
    GPU Power Readings
        Average Power Draw : 118.42 W

Run it on an interval throughout a normal workload window and have the collector add a timestamp plus the job or replica ID. The output separates the device, total memory, allocated memory, compute activity, memory activity, and power. A single row proves little. Percentiles across full jobs expose sustained load, bursts, and idle gaps.

Also record application metrics: batch size, input and output lengths, tokens or images per second, request latency, queue depth, errors, and completed units. GPU metrics explain the machine. Application metrics tell you whether customers received useful work. The two sets must share timestamps.

Treat averages with suspicion. Two replicas can both average 45 percent compute activity, yet one may run steadily while the other alternates between saturation and silence. The first could fit a smaller card. The second may need better batching or autoscaling, because shrinking it could make the saturated periods miss latency targets. Inspect the time series and latency percentiles before making the change.

Cold starts deserve their own measurement. Loading weights, compiling kernels, warming caches, and downloading model files may dominate short jobs. If a five minute task spends two minutes preparing, a faster GPU may not help. A prepared image, local model cache, or longer job grouping can save more than a cheaper accelerator.

Right size the model before the fleet

Pick the smallest accelerator that fits the model, peak working memory, and a tested safety margin at the required batch size. Then benchmark the complete request path. A catalog comparison based on nominal memory and theoretical arithmetic throughput misses host bottlenecks, kernel support, precision behavior, and model architecture.

Start with one representative corpus and an acceptance test. Run the same container, model revision, precision, batch policy, and output checks on two or three candidate GPU classes. Measure accepted units per dollar and the latency percentile your product promises. If a cheaper card needs twice as long and holds a worker twice as long, its lower hourly rate may buy nothing.

Reduce the workload before buying a larger card. Quantization can lower memory use and improve throughput, but only an output quality test can approve it. Shorter context limits, smaller image sizes, fewer generated candidates, and a smaller model for easy requests can remove work the customer never valued. These changes need product decisions, not an infrastructure switch hidden from the team.

Batching improves arithmetic efficiency when the workload tolerates waiting. Set a maximum batch size and a maximum queue delay. Without the delay limit, a quiet service may wait too long to fill a batch. Without the size limit, a traffic burst may trigger an out of memory failure. Measure both throughput and tail latency after every change.

Kubernetes does not automatically make a whole GPU economical. Its official scheduling documentation says GPU resources normally appear as extended resources such as nvidia.com/gpu, and a pod limit also becomes its request. A pod that asks for one GPU reserves one schedulable unit even when its process uses a fraction of the device. More pods do not fix fragmentation unless the device plugin and hardware support a sharing method.

For compatible NVIDIA data center cards, Multi-Instance GPU can divide one physical GPU into isolated instances with assigned compute and memory. Time slicing can increase concurrency too, but it provides a different isolation and performance model. Test noisy neighbor behavior, memory limits, failure handling, and scheduler labels before consolidating production services. Do not describe either method as free capacity. It trades operational complexity for higher occupancy.

Keep interactive and batch traffic in separate queues even if they eventually share hardware. Online requests need bounded waiting and spare capacity. Batch work can fill valleys and yield when demand rises. A single first in, first out queue lets a large offline job block customer traffic, which turns a utilization win into a product incident.

Spot capacity is safe only for disposable workers

Spot GPUs make sense when the worker can disappear and the job can resume or restart cheaply. They do not become safe because the cloud console offers a large discount. The state model determines safety. Put durable inputs, checkpoints, queue ownership, and output commits outside the instance.

The provider warnings are short. AWS EC2 documentation gives a Spot interruption notice two minutes before interruption. Google Cloud describes a best effort shutdown period of up to 30 seconds for a default Spot VM. Azure documents eviction with up to 30 seconds of notice and no high availability guarantee. Treat every notice as an opportunity for cleanup, never as your primary recovery mechanism. A process can crash or a host can fail without a polite countdown.

Training works on Spot when it writes usable checkpoints often enough. The right interval comes from a cost comparison, not habit. Let C equal the time to write a checkpoint, I the checkpoint interval, and R the expected recomputation after a loss. More frequent checkpoints increase storage and pause overhead; infrequent checkpoints increase repeated compute. Measure checkpoint duration and interruption history for each capacity pool, then choose the interval that minimizes checkpoint overhead plus recomputation.

Batch inference is usually easier. Make each queue item small enough to retry, but large enough that startup and model loading do not dominate it. Give every item an idempotency key. Write results to a temporary location, validate them, then commit completion atomically. If a worker dies after writing output but before acknowledging the queue, the replacement must recognize the existing committed result instead of charging twice.

Interactive inference needs a stable base. Run the minimum capacity needed for ordinary latency on on demand or committed instances, then let Spot workers absorb asynchronous work or traffic that the stable pool can take back. Never route a customer request to Spot unless another healthy path can retry it within the latency budget. Capacity that may vanish cannot own the only copy of an in-flight request.

Test interruption behavior deliberately. AWS recommends using its Fault Injection Service to initiate a Spot interruption, which supplies the same two minute notice in the test. The useful part of that advice is the forced failure, not the vendor button. Kill a worker during model loading, during checkpoint upload, after result write, and before queue acknowledgment. Verify both correctness and the final bill.

A resumable worker needs an explicit contract

An interruption tolerant system defines what survives, who retries, and when work counts as complete. If those answers live only in an engineer's memory, the next capacity event will discover a different interpretation. Put the contract in job metadata and worker code.

A compact job record can carry the required state:

job_id: train-2026-08-09-017
input_version: corpus-42
model_revision: candidate-118
checkpoint_uri: object-store/checkpoints/train-2026-08-09-017/latest
checkpoint_every_steps: 800
max_attempts: 6
output_uri: object-store/results/train-2026-08-09-017
commit_marker: object-store/results/train-2026-08-09-017/COMPLETED

The worker first checks the completion marker. If it exists and matches the expected input and model revision, the worker acknowledges the queue item without rerunning it. Otherwise it loads the latest valid checkpoint, processes work, writes versioned output, validates it, creates the completion marker, and only then acknowledges the item. The marker prevents a retry from treating a partial upload as a finished result.

Do not keep the only checkpoint on an instance disk. An attached persistent volume may survive some stop events, but it can tie recovery to one zone or leave storage charges after eviction. Object storage gives the scheduler more placement choices. Upload asynchronously only if the local checkpoint remains valid until the remote write confirms completion.

Pool diversity matters more than a clever maximum bid. A request restricted to one GPU model in one zone waits whenever that exact pool dries up. Support several accelerator classes, zones, and batch sizes behind a capability label. AWS Spot best practices explicitly recommend flexibility across instance types and Availability Zones and favor capacity aware allocation. That advice fits GPU fleets particularly well because individual accelerator pools can be thin.

Keep an on demand escape path with a budget ceiling. When Spot retries exceed the allowed delay or attempt count, move the job to stable capacity, postpone it, or fail it clearly. An infinite retry loop can spend less per hour and more per completed job. Expose retry cost and queue age so the scheduler can make the choice before a customer asks where the result went.

APIs win when they sell utilization you cannot create

An API usually beats rented GPUs when demand is low, bursty, uncertain, or changing faster than the team can operate models. The provider pools requests across customers and charges for use rather than your idle endpoint. That advantage shrinks when traffic stays high, predictable, and compatible with a model you can run efficiently.

Compare measured unit economics with the same quality and latency bar. For a token service, calculate input, cached input, and output charges for the actual distribution of request lengths. For an image or audio service, use accepted outputs or processed duration. Include retries, moderation or routing calls, data transfer, and minimum provisioned capacity on the self hosted side.

Use a break even equation rather than a monthly hunch:

api_monthly = accepted_units * api_cost_per_accepted_unit
self_hosted_monthly = gpu_hours * blended_hourly_cost + storage + transfer + operations
break_even_units = self_hosted_monthly / api_cost_per_accepted_unit

The equation needs scenarios because gpu_hours does not always stay fixed. Build at least a low, expected, and high demand case. In the low case, include the replicas required for availability even if traffic barely uses them. In the high case, include queue growth, extra replicas, and any rate limits on the API. Compare cash cost and service risk separately so a cheap option does not conceal an unacceptable dependency.

Run a shadow sample before switching. Send a fixed evaluation set through the API and the hosted model, then compare acceptance rate, latency, and cost. Human review may matter for subjective output. Keep the prompt, sampling settings, and postprocessing equivalent. A price comparison between different quality levels has no decision value.

APIs also buy access to model improvements and remove driver, framework, scheduler, and capacity work. They introduce provider dependency, privacy review, regional availability, rate limits, and model change risk. Contract terms and data handling can disqualify an API even when the spreadsheet favors it. Conversely, vague fear of lock-in does not justify running a fleet that consumes scarce engineering time. Keep an internal request schema and an evaluation set so you can test another provider or a hosted model later.

Asynchronous provider modes can change the comparison. OpenAI's official Batch API documentation, for example, describes a 24 hour completion window with a 50 percent discount. That specific service may or may not fit the workload, but the broader point holds: do not price delay tolerant jobs at an interactive API rate. Ask each provider about batch, cached input, reserved throughput, and minimum commitments, then model only the options the product can actually use.

Owned GPUs need sustained demand and an exit plan

Buying hardware can win when a startup has stable demand, high occupancy, suitable facilities, and people who can operate it. Purchase price alone does not prove the case. Amortize servers, networking, spare parts, warranties, rack space, power, cooling, installation, and the value of cash tied up. Add failure coverage and staff time.

Calculate an effective hourly cost across conservative useful hours:

effective_owned_hour = (purchase + facility + power + support + operations - resale_value) / useful_gpu_hours

Useful hours exclude installation, repairs, idle periods, and time after the hardware no longer meets the workload. Run the model at several occupancy levels and resale values. If the decision works only at near perfect occupancy for three years, it does not work. Startup roadmaps and model requirements change too quickly for that assumption.

Cloud commitments carry a similar trap. A one or three year discount saves money only when the workload consumes the committed shape. Commit the stable base, not a launch forecast. Leave uncertain growth on flexible capacity until measurements show a floor. A discount on an idle reservation is still waste.

Owned hardware also changes failure planning. A cloud region can offer another instance when one host fails, though scarce GPU capacity may delay it. A small local cluster may have no compatible spare. Price at least one failure domain, replacement lead time, backups, and remote access. Decide whether the product can wait through a hardware repair before approving the purchase.

Data gravity can favor ownership when large private datasets already live near the servers and repeated cloud transfer or compliance controls add friction. It can also favor cloud or API use when customers require specific regions or rapid geographic expansion. Write the constraint plainly. Teams get into trouble when they turn a regulatory requirement into a vague preference and buy hardware before counsel or customers confirm it.

An exit plan names the event that changes the decision. Examples include a new model that no longer fits memory, occupancy below the approved floor for two months, support expiry, or API unit cost falling below the owned cost. Decide who reviews the trigger and what happens to the hardware. Resale value belongs in the original approval, not in a surprised conversation two years later.

Cost controls belong in the scheduler and product

A spreadsheet can approve a GPU strategy, but software must enforce it each day. Put maximum run time, retry count, queue age, instance class, purchase type, and job budget into scheduler policy. Stop idle development instances automatically after a short grace period, while preserving notebooks and data outside the machine. Require an owner and expiry date for every exception.

Give product teams cost feedback at the unit they control. Show cost per accepted image beside generation settings, cost per completed training candidate beside experiment configuration, or cost per customer request beside model routing. A cloud invoice arrives too late and aggregates away the cause. Daily unit cost catches a changed context limit, batch size, or retry bug before the month closes.

Set two alerts. The first catches absolute spend against a daily or weekly budget. The second catches unit cost moving away from its recent baseline. Absolute spend alone punishes healthy growth, while unit cost alone misses a runaway workload with stable efficiency. Both alerts need a named owner and a response, such as pausing a batch queue or moving overflow to a capped pool.

Development capacity needs a separate policy from production. Engineers need fast iteration, but a notebook session does not need an accelerator through lunch or overnight. Store the environment definition, code, and data outside the instance; warn before shutdown; then stop idle capacity automatically. Give a developer a simple way to request more time with a reason and a new expiry. This preserves flow while making forgotten machines visible.

Put a maximum dollar cost on every experiment before it enters the queue. The scheduler can estimate that cap from instance price, maximum run time, retry allowance, and expected storage. Reject or pause a job whose estimate exceeds the owner's remaining experiment budget. A cost cap will not predict the final model quality, but it prevents a malformed configuration from running across a fleet until someone notices the invoice.

Promotion between experiment stages should require evidence. A small sample can catch broken data, an unsupported precision mode, exploding memory, or an output validator that rejects everything. Only jobs that pass the sample should receive a full dataset and larger budget. This gate saves compute without limiting serious experiments, because it removes runs that had no chance of producing an accepted candidate.

Model routing belongs in the same cost system. If a smaller model meets the acceptance test for routine requests, route those requests there and reserve the larger model for cases where measured quality improves. The router needs its own evaluation because a bad confidence rule can create retries and cost more than one direct call. Record the chosen route, final acceptance, latency, and total cost under one request ID.

Separate allocation from utilization when reporting shared infrastructure. Allocation says which team reserved the capacity. Utilization says whether work occupied it. A platform group may own the cluster bill while a product configuration causes excess context length or duplicate generation. Chargeback does not need to become an internal tax system, but every cost anomaly needs an identifiable workload owner who can change the cause.

Delete stale artifacts with the same care used to stop compute. Old checkpoints, duplicate model images, unattached disks, verbose logs, and cached datasets can remain after a GPU disappears. Define retention by artifact type and protect release models or audit records with explicit holds. Storage often looks small beside accelerator spend, which is why it grows quietly and restricts later placement choices.

Finally, track capacity failures separately from application failures. An unavailable Spot pool, an out of memory error, a rejected output, and a customer timeout demand different fixes. Combining them into one retry count hides whether the team should add instance diversity, change batch size, repair quality, or protect latency. The scheduler should attach a reason code to every attempt and include wasted cost in the completed job record.

Review decisions on a fixed cadence and after workload changes. Record model revision, traffic distribution, acceptance test, hardware classes, prices, and operational assumptions. Current cloud and API rates belong in the decision record, not hardcoded into architecture folklore. Rebenchmark when a model, driver, precision, or customer latency target changes.

For founders who lack reliable attribution, a Team & AI Audit from oleg.is can map engineering costs and identify savings before the company commits to a fleet. The advertised engagement costs $5,000, runs for five business days, and guarantees at least $50,000 a year in identified savings or it is free. That review should still produce the workload measurements and decision record described here, because no advisor can substitute for production evidence.

Assign one person to own cost per accepted unit and give that person authority to change queue policy, instance selection, and API routing within agreed limits. GPU cost stops drifting when every job carries its economic contract: what it may spend, how it survives failure, and when it has done enough useful work to count.