Self-host inference on one GPU: limits, costs, privacy

Why teams look at a model gateway

Teams usually start thinking about a model gateway when model use stops being a side experiment. One app uses a local coding model, another needs a chat model, and someone wants embeddings for search. Without a gateway, each tool talks to each model in its own way. That gets messy fast.

A model gateway gives the team one entry point for all that traffic. Apps send requests to one place, and the gateway decides which model should handle them. That sounds minor, but it removes a lot of day-to-day friction. Developers do not have to rebuild the same integration every time they test a new model or swap one model for another.

Control is the other big reason. Teams want one place to set limits, inspect usage, and keep rules consistent. If a product manager wants request caps for internal tools, or an engineer wants to block prompts with secrets, the gateway gives them one control point instead of settings scattered across several apps.

Small teams also worry about lock-in. When everything depends on one vendor API, changing direction later gets expensive. Costs rise, model quality shifts, and terms change. A gateway makes testing easier because the app can stay mostly the same while the team compares models behind the scenes.

In practice, the gateway usually does four basic jobs: it routes requests to the right model, applies rate limits by user or app, keeps logs and usage records in one place, and hides model-specific quirks behind one API.

That does not mean teams want a heavy platform. Usually they want the opposite. They want self-hosted inference to feel boring: one box, one entry point, clear logs, and very few moving parts. If the gateway needs constant babysitting, it defeats the point.

A five-person product team is a good example. They might run a local chat model for internal support, a coding model for developer tools, and an embedding model for search. A gateway lets them manage that mix without turning every app into a custom AI project.

What one GPU box can really handle

A single GPU server can feel fast in a demo and slow in real work. The gap usually comes down to three things: model size, prompt length, and how many people hit it at the same time.

Start with memory, not speed charts. If the model barely fits in VRAM, every other comparison gets muddy. Longer context uses more memory, and so does the cache the model builds while it answers. A 7B or 8B model often fits much more comfortably on a 24 GB card than a 13B model. Bigger models usually force tougher trade-offs like shorter context, heavier quantization, or slower offloading.

That is why raw tokens per second can mislead. A box may look quick with short prompts and one user, then crawl when someone pastes a long document. On self-hosted inference, context length, batch size, and response time pull against each other. You can push throughput up with batching, but users may wait longer before they see the first token.

Long requests make this worse. If two people send large prompts at nearly the same moment, the slowdown is rarely graceful. One request can hold the GPU long enough that the second request sits in a queue, and both users feel the lag. Teams often plan around daily averages, but the real limit shows up during short bursts after a meeting, a release, or a support spike.

The model is not the whole system either. A gateway still needs CPU time and RAM for routing, tokenization, logging, auth, and other background work. If you size the machine so tightly that the model consumes almost everything, the small jobs around it start adding delay.

A simple way to think about one box is this: it handles steady light traffic well, short bursts with some pain, and concurrent long requests badly. If several people need fast answers at once, plan for headroom instead of hoping average usage will save you.

Where batching helps and where it breaks

On a single GPU server, batching is often the first thing teams try because it can raise throughput quickly. If five short requests arrive within a tiny time window, the GPU can usually process them more efficiently together than one by one. That works especially well for jobs like tagging, classification, or short summaries where nobody is waiting on each token.

The catch is simple. Users do not feel throughput. They feel delay. In chat, people notice the pause before the first token appears. A bigger batch can improve total tokens per second while making each person wait longer in the queue. This is where many self-hosted setups start to feel slow, even when GPU use looks great on paper.

A practical rule helps: batch short requests that land close together, but keep the waiting window small. If you hold requests too long to build a larger batch, the queue becomes the real bottleneck.

Good use cases for batching

Batching works best when the work is predictable and not urgent. Short prompts with short outputs, background summaries, tagging, moderation, and overnight processing all fit well. These jobs can wait a few hundred milliseconds, sometimes longer, and users will never notice.

Where it breaks

Interactive chat is different. Long prompts, long answers, and tools that call other services make batch behavior uneven. One slow request can hold up several others. A batch that looked efficient at the start can hurt response time for everyone inside it.

Teams should separate traffic early. Put live chat and other interactive requests in one lane, and send offline work like summaries, labeling, or document cleanup to another. That one change stops background jobs from eating the queue when real users need fast replies.

You also need to watch queue time, not just generation time. Measure how long each request waits before the model starts producing tokens. If queue delay keeps climbing, your batch settings are too aggressive, your server is too small, or both.

Set a maximum queue age and enforce it. If an interactive request waits past your limit, reroute it to a smaller model, send it to another worker, or drop it and return a clear error. Stale work is worse than failed work because it wastes GPU time and still leaves users annoyed.

What to monitor from day one

Most teams look at average latency first. That helps, but queue length usually warns you sooner. If requests start lining up on a single GPU server, users feel the slowdown before the dashboard looks dramatic.

Track two timing numbers on every request: time to first token and full response time. Time to first token shows how long the user waits before anything appears. Full response time shows how long the whole job took. In a chat tool, the first number often matters more because people forgive a slower finish if the reply starts fast.

Resource use matters just as much as latency. Watch GPU memory, GPU use, CPU use, and disk fill from day one. GPU memory spikes can block new requests even when GPU use looks moderate. CPU bottlenecks also show up more often than people expect, especially when tokenization, logging, and routing all happen on the same box.

Count prompt tokens, output tokens, and failed requests for each model. Token counts show which workflows are expensive and which teams send giant prompts by habit. Failed requests need a reason code, not just a total. A timeout, an out-of-memory error, and a routing mistake are three different problems.

If you route across more than one model, log which model answered each request and why the gateway picked it. That one detail saves hours later. When users complain that answers got worse or slower, you can check whether the router sent traffic to a smaller model, hit a fallback rule, or retried after an error.

At the start, simple alerts are enough. Watch for queue length staying above its normal range for several minutes, time to first token jumping past your limit, free GPU memory dropping too low, inference processes crashing or restarting, and disk usage climbing because logs or cached files keep piling up.

If you already use Prometheus, Grafana, Loki, or Sentry, add inference metrics there instead of building a separate stack. Keep the first dashboard boring and clear. One screen should tell you whether the box is fast, full, failing, or quietly building a queue.

What privacy improves and what still stays risky

Running models on your own server changes where data goes. Prompts, files, and outputs can stay inside your network, which often makes legal review and customer commitments easier. If policy says certain data cannot leave your environment, self-hosted inference can solve a real problem.

The gain is straightforward. Fewer outside companies handle customer messages, internal notes, bug reports, contracts, and product drafts. That lowers third-party exposure and cuts the number of systems that can keep copies of sensitive text.

This matters most when teams work with support tickets, sales notes, health data, finance records, or private source code. A cloud API adds another party to every request. A local gateway removes that extra hop.

But self-hosting does not erase privacy risk. It shifts more of the risk onto your own setup and your own people. If you keep raw prompts in app logs, proxy logs, backups, analytics tools, or copied test files, you can still leak data inside the company.

Small teams usually get burned by ordinary mistakes. Too many admins can read raw prompts and outputs. Backups keep sensitive logs for months. Dashboards show customer text in plain form. Engineers test with real records instead of masked samples.

Redaction helps more than many teams expect. If a staff member does not need a full email address, phone number, account ID, or payment detail to get a useful model answer, strip it out before inference. A ticket summary with names and IDs removed often works just as well.

Access rules need to be written down, not assumed. Decide who can view prompts, who can view outputs, who can open raw logs, and who can export data from the gateway. Keep that list short, review it often, and remove access when roles change.

Privacy gets better when you keep data close, collect less of it, and limit who can inspect it. It gets worse when logs grow quietly in the background. That is the trade: more control, but more responsibility every day.

How to test self-hosted inference before you commit

Pick one job that happens every day and has a clear success line. Good test cases include support reply drafts, ticket tagging, short document summaries, or internal search answers. Avoid broad pilots like "all assistant traffic." You need one use case with known daily volume, rough prompt size, and a response target the team can judge.

Write down the numbers before you run anything. How many requests arrive per hour? How many can wait two seconds, and how many need an answer in under 500 ms? If 80 people use the tool at once after lunch, your single GPU server has to survive that burst, not just look good at midnight.

Run the same workload twice: once during quiet hours and again during the busiest hour you expect. That gap matters. A setup that feels fast with five parallel requests can feel broken with thirty, even if the model itself stays stable.

Track a small set of metrics from the first day: queue time, time to first token, total response time, output quality on a fixed sample set, and cost per day, including power and ops time.

Quality needs a fixed check, not gut feel. Keep 20 to 50 real prompts, score the outputs, and compare them with your current API result. If the self-hosted model is cheaper but creates more editing work, the savings can disappear in a week.

Break one thing at a time on purpose. Fill the queue and watch what users see. Kill the model process and see how fast it comes back. Drop one dependency, like the gateway or tokenizer service, and check whether requests fail cleanly or hang until users refresh.

The side-by-side comparison matters more than most teams expect. Run the same prompt set through your current API and your gateway. Compare speed, answer quality, refusal behavior, and daily operating cost. Use the same prompt format for both, or the test will lie to you.

A small team can finish this in a week. One engineer can script the replay, one product person can score outputs, and a CTO or advisor can review the failure logs. That is usually enough to tell whether self-hosted inference is a real fit or just an interesting lab project.

A simple example from a small team

A support team with eight agents sends ticket summaries through one model gateway on a single GPU server. During work hours, they do not ask the model to write long reports or scan old tickets in bulk. They keep it focused on short tasks: summarize the thread, suggest a reply, and pull out the next action.

That choice matters because agents need answers in a few seconds while a customer waits. If the team mixes live traffic with big batch jobs at 2 pm, the queue grows fast and everyone feels it. A single box can look fine at low volume, then slow down hard when several agents paste long tickets at the same time.

After 6 pm, the team changes the pattern. They batch backlog work such as old unresolved tickets, satisfaction follow-ups, and weekly tag cleanup. Those jobs can wait a minute or two per item, so the gateway can pack requests together and use the GPU better. Daytime traffic stays live, and evening traffic does the heavy lifting.

Managers do not add another model just because people ask for one. First they watch a few basic signals for two weeks: queue length during peak hours, average and worst reply time per request, GPU memory use and crashes from oversized context, and token volume by task so they can spot prompts that waste capacity.

That tells them whether they need another model, a second GPU, or just shorter prompts. In many cases, prompt cleanup solves more than new hardware.

Privacy is also a practical reason they chose self-hosting. Their tickets include names, billing notes, account history, and sometimes refund disputes. Keeping that data inside their own system cuts outside exposure and makes review easier. It does not remove risk, though. The team still needs access rules, log filtering, retention limits, and a clear rule for who can inspect prompts and outputs. One GPU in one office is not private by default. The setup only works if the team treats the gateway like any other internal system that handles customer data.

Mistakes that cause trouble early

Many teams start with the biggest model they can fit on the card. That usually backfires. If the job is support replies, draft summaries, or light code help, a smaller model often feels better because it answers fast and stays available. On one GPU, a model that responds in one second beats a larger one that makes everyone wait twelve.

A gateway can make this mistake harder to spot because the interface looks neat even when the box is struggling underneath. A clean demo hides queue growth, memory pressure, and long warm-up times. The model may look fine with two people testing it, then stall when the whole team opens it after standup.

Queue design

Another early problem is mixing short chat requests with long batch work in one queue. A few document jobs, embeddings, or large prompt runs can block interactive users for minutes. People read that delay as "the model is bad" when the real issue is scheduling. Split traffic by job type, set time limits, and protect chat latency before you add more use cases.

Logs also get ignored for too long. Teams often wait until users complain that answers feel slow or disappear. By then, you have guesses instead of evidence. Start with basic monitoring on day one: request count, queue time, tokens per second, GPU memory, error rate, and timeout rate. You do not need a huge observability stack, but you do need enough data to spot trouble before support messages pile up.

Privacy mistakes are quieter, but they last longer. Self-hosted inference does improve privacy because data stays on your own box. That does not mean you should keep every prompt forever. Full prompts can hold customer data, passwords pasted by mistake, contract text, or internal plans. Set a retention rule early, mask sensitive fields where you can, and decide who can read logs.

A good Friday demo proves very little. Monday morning traffic is the real test. Ten people asking short questions at once can break a setup that looked perfect in a quiet room. Run load tests with realistic prompt sizes, response lengths, and concurrency before you commit to a single GPU server.

Quick checks and next steps

Before anyone orders hardware, write down the traffic you expect during a busy hour, not an average day. A single GPU server can look fine in a demo and still choke when a few long prompts arrive at once.

Start with numbers your team can test against. Note your peak requests per minute, the longest prompt you expect, and how long users will wait before the product feels slow.

Privacy needs the same plain rules. If prompts may include customer data, decide now what can stay in logs, what needs redaction, and who can read raw traces when something breaks.

A short pilot plan is usually enough:

• Write down peak request rate, longest prompt size, and expected response length.
• Set redaction rules for logs, error reports, and support screenshots before the first real test.
• Pick a stop point for the pilot, such as p95 latency, timeout rate, or answer quality dropping below an agreed level.
• Assign ownership for monitoring, on-call coverage, model updates, and rollback if a new model behaves worse.

Teams often underestimate the boring work. Dashboards, alerts, model updates, disk cleanup, and late-night restarts take time, even on one box.

That matters because self-hosted inference can improve privacy, but it does not remove risk by itself. Sensitive text can still leak through logs, tracing tools, copied prompts in chat, or crash dumps if nobody sets firm rules.

A small team can keep the pilot simple. Run one model, one gateway, one dashboard, and one alert path for a week under realistic traffic. If latency climbs too fast, batching stops helping, or users notice weaker answers, stop there and rethink the setup before you buy more hardware.

If you want a second opinion, Oleg Sotnikov at oleg.is can review the rollout as a Fractional CTO. He works with startups and smaller companies on AI-first development, infrastructure, and practical automation, which can be cheaper than learning these limits after users hit them in production.