Self-hosted LLM for coding in a real engineering team

A self-hosted model is worth running for coding only when you can name the work it will own, measure its failures, and keep enough memory free for the context your developers actually send. Buying a large GPU first and choosing a model afterward reverses the decision. You end up defending a machine instead of improving delivery.

Local inference has three honest advantages: code can stay inside a boundary you control, marginal usage can become predictable, and you can keep a fixed model available when a vendor changes its service. It also gives you patching, capacity planning, model evaluation, access control, and an on-call problem. For many teams, a strong hosted coding subscription still produces better code for less money. Self-hosting wins in narrower conditions than hardware enthusiasts like to admit.

Define the job before choosing the model

A useful self-hosted deployment replaces a specific slice of work, not an abstract category called "coding." Autocomplete, chat about one file, repository search, test generation, pull request review, and an agent changing twenty files put very different pressure on the model and the serving stack.

Write down the request shape before looking at GPU listings. Record the languages, median prompt size, largest useful prompt, acceptable first-token delay, expected simultaneous users, and whether the model may edit files or run commands. A model that feels quick for one developer in a terminal can become unusable when four editor plugins continuously submit completion requests.

Concurrency deserves its own number. Interactive chat produces bursts: a long prompt is processed, tokens stream back, and the GPU may sit quieter while the developer reads. Autocomplete sends short requests far more often. Agent loops alternate inference with file reads and tests. A server that supports two agents may still choke on ten completion clients, even if the daily token totals look identical. Size for the busiest useful minute, then cap admission so overload becomes a visible queue rather than a cascade of out-of-memory restarts.

I split coding work into two service levels. The first is assistive: completion, explanation, small refactors, unit tests, and questions with a few relevant files supplied. A good 7B to 14B coder model can handle much of this work. The second is agentic: locating the right files, planning a change, editing across modules, running tests, reading failures, and correcting its own patch. That work rewards stronger reasoning, reliable tool use, and enough context more than raw token speed. A 32B model is a practical local starting point, but it still will not consistently match the strongest hosted systems on unfamiliar repositories.

Do not treat privacy as a complete job definition. "The code cannot leave our network" explains a constraint, not what the server must deliver. You still need a workload and an acceptance test. If the constraint applies only to two repositories, route those repositories locally and let ordinary work use the better hosted model. Selective routing is often the sensible architecture.

The awkward question is whether local coding is safe. It can reduce disclosure to an external model provider, but it does not make prompts safe. The inference server sees source code, secrets accidentally included in files, user identity, and sometimes command output from production. Protect it like a source-code system: authenticate every client, encrypt traffic across machines, restrict logs, pin model files, and separate model output from command execution. A private model with an unauthenticated HTTP port is not private.

Model size sets a ceiling on useful work

Parameter count is an imperfect but useful capacity signal within a model family. Small coder models are cheap and responsive, yet they lose track of constraints sooner and produce plausible local edits that conflict with code elsewhere. Larger models usually plan better and recover from test failures more often, but architecture, training data, instruction tuning, and the agent wrapper can overturn a simple size comparison.

Use model families with explicit coding training and an instruct variant for chat or agents. Qwen2.5-Coder is a useful reference family because its model card exposes 0.5B, 1.5B, 3B, 7B, 14B, and 32B sizes. The 7B instruct model is suitable for fast explanations, completion, and contained edits. The 14B tier is the minimum I would evaluate for daily mixed-language repository work. The 32B instruct model is the first tier in that family I would ask to own multi-file changes.

DeepSeek-Coder-V2-Lite-Instruct is another sensible evaluation candidate when your runtime supports its mixture-of-experts architecture. Its model card lists 16B total parameters with 2.4B active and a 128K advertised context. "Active parameters" helps explain compute per generated token; total weights still have to live somewhere. Teams see 2.4B, size the machine like a dense 3B model, and discover that memory follows the total model, not the active experts.

Do not confuse a published context window with a useful coding window. The Qwen2.5-Coder-32B card advertises 131,072 tokens but also explains that its default configuration is 32,768 and that longer input needs YaRN configuration. More context also consumes more key-value cache and may dilute attention. Repository retrieval that sends eight relevant files usually beats dumping eighty files into a nominally long window.

Licensing belongs in the model decision. Read the exact license attached to the exact repository and revision you plan to deploy. Check commercial use, redistribution, hosted-service restrictions, acceptable-use terms, and obligations for modified weights. "Open weights" does not automatically mean an OSI-approved open-source license, and a permissive runtime license tells you nothing about the model license.

Use benchmarks to make a shortlist, never to approve a model. HumanEval-style tasks measure small functions in controlled prompts. Your agent needs to understand your naming, migrations, generated files, test commands, and review rules. A five-point benchmark lead will not save a model that edits the wrong package or ignores a failing integration test.

VRAM sizing needs room for more than weights

GPU memory must hold model weights, the key-value cache, runtime workspaces, and overhead. Quantized weight size gives you the floor, not the purchase target. The rough calculation is simple enough to reject impossible configurations before you download anything:

weight_GiB = parameters_in_billions * bits_per_weight / 8 * 1.08
required_VRAM = weight_GiB + KV_cache + runtime_overhead + safety_margin

The 1.08 factor is a planning allowance for quantization metadata and mixed-precision tensors, not a physical constant. Runtime overhead varies. KV cache depends on architecture, context length, cache precision, batch size, and concurrent sequences, so measure it with the actual runtime after this first estimate.

For a dense 7B model at four bits, the formula gives about 3.8 GiB of weights. Plan on an 8 GB GPU for modest contexts and one interactive user, although 12 GB leaves far fewer compromises. A 14B model starts near 7.6 GiB, making 12 GB a tight configuration and 16 GB a healthier floor. A 32B model starts near 17.3 GiB; 24 GB can work for one user with controlled context, while 32 GB or more gives the cache and runtime space an agent needs. A 70B model starts near 37.8 GiB and normally pushes you into 48 GB, multiple GPUs, or aggressive compromises.

Those figures describe four-bit weights, not guaranteed file sizes or measured peaks. Six-bit or eight-bit quantization raises the requirement. Long context and concurrency can raise it again. Vision encoders, speculative decoding models, and embeddings loaded on the same GPU also take memory.

Prompt processing and token generation stress hardware differently. Reading a large context can use parallel compute efficiently, while generating one token at a time often exposes memory bandwidth limits. Ask vendors and internal testers for both prompt-processing speed and generation speed. One combined "tokens per second" figure can hide a painful first-token wait or flatter a server with fast prompt ingestion that writes code slowly.

Measure the serving process at the context lengths that matter. With llama.cpp, a controlled smoke test can look like this:

./llama-server \
  -m /models/coder-32b-q4_k_m.gguf \
  -c 32768 -ngl 99

nvidia-smi

Expect the second command to print a process table with columns for process ID, process name, and GPU memory use, such as 42137 llama-server 22184MiB. Run one realistic prompt, then two at once, and watch both peak memory and token latency. An idle reading misses cache growth. If the process fails only under the second request, you have a capacity problem rather than a mysterious model problem.

System RAM still matters. Keep enough to load or map the model, run the operating system, hold retrieval indexes, and survive GPU fallback without swapping. For a 24 GB GPU serving a 32B quant, I prefer at least 64 GB of system RAM. That is an operational margin, not a claim that inference consumes all of it.

Hardware choices carry different operational costs

An NVIDIA GPU remains the least surprising choice when you want broad runtime support and predictable deployment. CUDA receives first-class attention from the major serving projects, and used 24 GB cards can make a capable single-user workstation. Check power draw, physical size, cooling, power-supply connectors, and whether consumer cards may run continuously in your environment. Cheap VRAM stops being cheap after thermal throttling and random resets.

Apple silicon is attractive for a developer workstation because unified memory can hold models that do not fit on a similarly priced discrete GPU. llama.cpp treats Metal as a first-class backend. A machine with 64 GB or 128 GB of unified memory can run large quantized models quietly, but memory bandwidth and shared use affect speed, and it is not a drop-in multi-user server. Buy it when one or two people value local convenience, not because the memory number resembles data-center VRAM.

CPU-only inference works for evaluation, background summarization, and low-rate batch jobs. It rarely satisfies interactive agent work on 14B or larger models. Hybrid CPU and GPU offload lets llama.cpp run models larger than VRAM, but every layer moved across a slower memory path reduces generation speed. Hybrid mode is a compatibility valve, not free capacity.

Multiple GPUs solve capacity only when the runtime can split the model efficiently. They do not combine into one transparent pool. Interconnect speed, tensor parallel support, card topology, quantization kernels, and synchronization all matter. Two consumer GPUs may fit a larger model and still deliver worse latency per request than one smaller model on a single card. Test the exact topology before standardizing it.

AMD and Intel hardware can be reasonable when your team already operates it and the chosen runtime has a proven backend. llama.cpp documents HIP, Vulkan, SYCL, and other backends. vLLM's quantization documentation makes the harder point: each quantization method has its own hardware support matrix. "The runtime supports this GPU" does not mean your preferred quantization kernel works on it.

Renting a dedicated cloud GPU counts as self-managed inference, but not as on-premises data control. It can be the best pilot because you can resize or stop it after the test. Include attached storage, idle hours, data transfer, backups, and operator time in the cost. Hourly accelerator price alone is the hardware enthusiast's version of ignoring payroll tax.

Quantization trades quality for capacity unevenly

Four-bit quantization is the practical starting point for local coding, not a universal optimum. Good modern schemes preserve much of a model's capability, but the loss appears unevenly. Syntax may remain solid while long dependency chains, rare libraries, or exact instruction following degrade.

For GGUF models in llama.cpp, Q4_K_M is a sensible first build, Q5_K_M is the next comparison when memory permits, and Q8_0 is useful when you want a high-quality quantized reference. The llama.cpp quantization manual warns that requantizing already quantized tensors can severely reduce quality compared with quantizing from 16-bit or 32-bit weights. That warning matters. A conveniently repacked file may be smaller yet fail in ways you blame on the base model.

Keep the source and quantizer visible in your model inventory. Record the upstream model revision, quantization method, quantizer version, template, context settings, and file hash. Two files with nearly identical names may use different chat templates or calibration data. If a new download changes behavior, the inventory should tell you what changed.

Do not spend days debating four-bit formats before testing the 14B versus 32B decision. Moving up a capable model tier often matters more than moving one quantization step within the smaller tier. Compare both under the same memory budget. A 32B Q4 model may beat a 14B Q8 model on repository reasoning, while the smaller model can still win on latency and completion.

Quality also depends on prompt formatting. Instruct models expect a particular chat template and special tokens. A runtime that guesses the wrong template can make a good model ramble, ignore tools, or continue the user's text. Confirm the template from the model metadata or model card. Do not repair a template error with a page of system prompt.

The serving stack can ruin a good model

Choose llama.cpp when you want a compact deployment, GGUF quantization, broad hardware support, and a strong workstation experience. Choose vLLM when you serve supported models on GPUs to concurrent users and care about batching and throughput. Ollama can make local setup and model packaging convenient, but convenience does not remove the need to pin versions, authenticate network access, and measure memory.

Keep the inference API separate from the agent. The server should turn messages into tokens. The agent should select files, construct prompts, parse tool calls, enforce allowed commands, capture diffs, and stop runaway loops. Combining both layers makes failures hard to assign. When a patch is bad, you need to know whether retrieval omitted a file, the prompt lost a constraint, the model chose poorly, or the tool runner applied an unsafe action.

An OpenAI-compatible HTTP shape makes editor integration easier, but compatibility at the path and JSON level does not guarantee identical behavior. Models differ in tool-call syntax, stop tokens, system-message handling, and fill-in-the-middle support. Test the exact feature your editor uses. Chat success says little about autocomplete quality.

Version every dependency that can alter output: model revision, quantized artifact, runtime build, GPU driver, chat template, system prompt, agent version, and retrieval settings. Keep one known-good bundle available for rollback. Model servers are software systems, and an apparently harmless runtime upgrade can change tokenization, kernels, cache behavior, or stop handling. If developers report that the assistant "got strange," a version record turns that complaint into an investigation.

Put limits in configuration rather than team folklore. A small deployment policy might read:

model: coder-32b-q4_k_m
max_context_tokens: 32768
max_output_tokens: 4096
concurrent_requests: 2
request_timeout_seconds: 180
log_prompts: false
allowed_repositories:
  - billing-api
  - internal-tools
tool_policy: patch_and_test_only

This fragment prevents three familiar failures: an editor cannot silently request the full advertised context, a traffic spike cannot fill the cache with unlimited sequences, and the agent cannot turn a code assistant into a general shell. The names will differ in your stack, but the controls should exist somewhere enforceable.

Run the service under a dedicated identity. Bind it to a private interface, put authentication in front, and deny outbound network access unless model download or a tool explicitly needs it. Store prompts only when you have a retention reason. Logs that contain the proprietary code you kept away from a vendor defeat the main reason for self-hosting.

Test patches, not clever answers

A coding model passes evaluation when its patches pass your checks and survive review. Conversation quality is weak evidence. Build a small set of tasks from resolved work in your own repositories, remove the final patches, and ask each candidate to solve the same commit from the same starting state.

Use 20 to 40 tasks across the work you actually expect: a narrow bug, a multi-file feature, a migration, a flaky test, a refactor with a public API constraint, and a request that should be refused because information is missing. Keep secrets and customer data out of the fixture. Freeze model, runtime, prompt, retrieval method, token limits, and tool permissions for every run.

Score outcomes that affect delivery:

• Builds and tests pass without weakening the tests.
• The diff changes only necessary files and respects repository rules.
• The model finds and uses the correct existing abstraction.
• The agent recovers after a genuine test failure.
• A reviewer can accept the patch with limited correction.

Track time to accepted patch, not tokens per second alone. A fast model that generates three wrong patches occupies more developer time than a slower model that lands one usable change. Also record failure classes. Repeated wrong-file edits call for better retrieval; malformed tool calls implicate model or template; out-of-memory errors implicate serving; plausible but incorrect logic points back to model capability.

Include negative controls. Give the agent a task whose requested function already exists, one that conflicts with a documented repository rule, and one that lacks a required schema decision. A trustworthy result may be a short explanation or a request for missing information. Models that always produce a diff look productive in demonstrations and create dangerous review load in practice.

A reproducible runner does not need to be elaborate:

git checkout "$TASK_COMMIT"
git clean -fdx
./run-agent "$TASK_FILE" 12
./ci/test.sh
git diff

The expected output shape is your normal test report followed by a unified patch with diff, file headers, and changed lines. Run it only in a disposable clone or isolated worktree because git clean -fdx deletes untracked and ignored files. The command is deliberately blunt: evaluation should reset state instead of letting one model inherit another model's artifacts.

Have a senior engineer review blind when possible. Hosted and local candidates should use anonymous labels, because people forgive the system they wanted to buy. Re-run the set after model, quantization, runtime, agent prompt, or retrieval changes. Your benchmark then becomes a regression suite rather than a one-time purchasing ritual.

The subscription wins more often than the spreadsheet admits

A hosted coding subscription usually wins when developers need the strongest reasoning, usage is intermittent, repositories can legally leave your boundary, and nobody wants to operate inference. The monthly price includes access to expensive hardware, model upgrades, capacity management, and a polished agent. A local server has to beat that whole package, not just the API token line.

Calculate three-year cost with hardware depreciation, electricity, rack or desk space, cooling, storage, spare capacity, engineering setup, security work, upgrades, and incident time. Then divide by accepted developer hours saved, not generated tokens. If a $10,000 workstation occupies forty engineering hours in setup and maintenance, those hours may cost more than the GPU.

Self-hosting becomes credible when one or more of these conditions holds: policy forbids external processing, utilization stays high and predictable, latency to a remote service is unacceptable, the workload is narrow enough for a smaller model, or model stability matters more than receiving upgrades. It also works well as a fallback for outages and quota limits. None of these requires sending every request locally.

The common recommendation I argue against is buying hardware to "stop paying subscriptions." It is popular because the server is visible and the subscription feels endless. It is wrong when the local model reduces developer output or creates an unpaid operations role. Payroll dwarfs inference cost in most software teams, so a small quality loss can erase a large compute saving.

Use a routing policy. Keep sensitive repositories and routine high-volume tasks local. Send difficult architecture changes, obscure framework failures, or low-frequency work to a stronger hosted model when policy permits. Give developers a clear indicator of which route they are using, and prevent silent fallback that sends restricted code outside the boundary.

At oleg.is, a Team & AI Audit is the entry point I use when a founder needs this decision tied to team cost rather than GPU enthusiasm. It costs $5,000, takes five business days, and identifies at least $50,000 per year in savings or it is free. The useful output is a workload and operating model, which may recommend local inference, hosted tools, or both.

A pilot should make failure cheap

A self-hosted LLM for coding should earn expansion through a four-week pilot on real but reversible work. Use rented hardware or one workstation, one model from each practical size tier, a fixed task suite, and a small developer group. Do not begin with a rack purchase or a mandate that everyone abandon the hosted tool.

In the first week, classify the workload and establish the hosted baseline. In the second, deploy the local server with authentication, version pins, memory limits, and logging disabled by default. In the third, run blind patch evaluations and limited daily use. In the fourth, compare accepted patches, reviewer correction time, latency, failures, and full operating cost.

Set exit criteria before the pilot. Stop if the model cannot meet a minimum acceptance rate, if latency interrupts normal flow, if security controls remain exceptions, or if one engineer becomes the permanent model babysitter. Continue only when the team can state which requests belong locally and the measured economics support that route.

Do not fine-tune first. Retrieval, repository instructions, a correct chat template, better task decomposition, and a stronger base model usually deserve testing before training. Fine-tuning can help stable, repeated patterns, but it adds data preparation, evaluation, deployment, and model lifecycle work. It cannot teach a small model all the reasoning capacity you hoped to avoid buying.

The final decision is allowed to be "keep the subscription." That is not a failed pilot. A failed pilot is a server that stays because someone bought it, while developers quietly paste hard tasks into the hosted agent. Keep local inference when its boundary, cost, or stability advantage survives an honest patch test. Otherwise, spend the infrastructure budget on the tool that gets reviewed code into production.