On-premise AI coding assistants for regulated teams

Cloud models still set the quality ceiling for many coding tasks, but sending a regulated repository to a third party can turn a useful assistant into a procurement fight. On-premise AI coding assistants make sense when the organization can name the data boundary it must enforce and fund the people who will operate inference. They do not make sense merely because "local AI" sounds safer.

I have watched teams buy a GPU before they classified a single repository. Six weeks later, developers had a slow completion model, security still blocked half the codebase, and finance had acquired an unusually expensive space heater. Start with the control you need, then choose the smallest system that meets it. The answer may be a laptop runtime, a shared inference server, a dedicated cluster, or a cloud service with contract and retention terms your counsel accepts.

On-premise is a data boundary, not a box

An on-premise assistant is one whose prompts, retrieved code, generated output, logs, model weights, and operational metadata stay inside a boundary your organization controls. The server's physical location matters less than the complete data path. A rack in your office that downloads telemetry or crash dumps to a vendor is not fully on-premise. A dedicated environment in your own private cloud account may satisfy policy even though no server sits in your building.

Teams routinely blur three different arrangements:

• Local inference runs on each developer's workstation and may work without a network service.
• Self-hosted inference runs on shared infrastructure controlled by your team.
• Private managed inference runs in an isolated tenant or cloud account, but another party may operate part of the service.

The distinction changes who can inspect prompts, who patches the runtime, where logs land, and whether a support engineer can access memory or storage. Write those answers down before comparing models.

Regulation rarely says, in plain words, "buy a GPU and keep it in the office." GDPR, HIPAA, financial-sector rules, export controls, client contracts, and internal policies impose different duties. Some focus on processors and transfers, some on safeguards and auditability, and some prohibit particular data from entering any generative system. Counsel and the security owner must map the actual duty. Engineering should supply a precise flow of data rather than a vague promise that the setup is private.

Use five questions to draw that flow:

1. Which repository content can enter a prompt?
2. Can the assistant read tickets, logs, database schemas, secrets, or production traces?
3. What does the server record, for how long, and who can query it?
4. Where do model weights, embeddings, indexes, caches, and backups live?
5. Which outbound connections remain after deployment?

If nobody owns one of those answers, the boundary has a hole.

The viable options form four operating models

Regulated teams have four practical deployment patterns, and each makes a different trade between isolation, user experience, and operating work. Choosing a product name first hides that trade.

The simplest pattern is one runtime per workstation. An editor extension talks to llama.cpp, Ollama, or another local server on loopback. This can suit a small team, an air-gapped lab, or source code that cannot leave a managed endpoint. It also creates version drift, uneven performance, duplicated model storage, and a support problem on every laptop. Central policy becomes hard unless endpoint management locks the runtime and model files.

The second pattern is a shared completion service. Tabby packages a self-hosted server with editor integrations and repository context. Its manual describes code completion as a combined problem across the editor extension, prompt construction, and model serving. I agree with that framing. A raw chat endpoint is not a coding assistant: cancellation, fill-in-the-middle prompting, context selection, authentication, and latency all affect whether developers keep it enabled.

The third pattern separates the client from a general inference layer. Continue or an internal editor plugin can call an OpenAI-compatible endpoint served by vLLM, llama.cpp, or another engine. This gives the platform team more model freedom and lets several applications share inference. It also makes the platform team responsible for compatibility, request limits, routing, observability, and upgrades. vLLM is attractive for concurrent shared service because its documentation covers continuous batching, prefix caching, streaming, and parallel execution. Those features help throughput, but they add knobs that can fail under real editor traffic.

The fourth pattern is a controlled cloud service. It belongs in the comparison because "regulated" does not automatically mean "on-premise." A service may be acceptable if its contract, data retention, subprocessors, regional controls, access model, and incident process meet the organization's obligations. It often delivers stronger models and less operational work. Reject it because of a documented control gap, not because the word cloud feels uncomfortable.

Air-gapped deployment narrows the options further. You need an approved route for model weights, container images, editor extensions, license files, security fixes, and vulnerability data. Model installation is only the first transfer. If the environment cannot receive a patched runtime promptly, isolation can preserve known vulnerabilities as efficiently as it preserves source code.

Model licenses belong in this decision too. "Open weights" does not guarantee permission for every commercial use, redistribution, modification, or hosted service. Record the license version with the weight digest and have counsel review restrictions that affect your use. The editor extension, server, embedding model, and reranker may all carry separate licenses. A team that approves only the main model has reviewed one item in a small software supply chain.

Do not assume repository indexing is mandatory. Retrieval improves answers about private APIs when the index is current and access-aware, but it expands the sensitive data footprint. Start with open files and nearby symbols for completion. Add an index only after a task set proves that it improves outcomes enough to justify storage, deletion, and permission synchronization.

Hardware budgets begin with memory and concurrency

Model weights must fit somewhere, but the advertised parameter count does not tell you the full memory bill. Weight precision, context length, concurrent requests, key-value cache, runtime overhead, embeddings, and reranking all consume memory. Size the complete workload, then check it on the exact runtime and accelerator.

A rough weight-only estimate is useful for rejecting impossible plans:

weight_memory_bytes = parameters * bits_per_weight / 8

An 8 billion parameter model at 16-bit precision needs roughly 16 GB for weights alone. At 4-bit precision, the arithmetic floor is roughly 4 GB. Real use needs more because quantization metadata, buffers, cache, and the runtime take space. Do not turn the rough figure into a purchase order.

Quantization reduces memory and may improve speed, but it can also damage the behavior you bought the model for. vLLM's quantization guide says this plainly: lower precision trades model precision for a smaller memory footprint, and hardware support differs by format. The practical consequence is that an AWQ, GPTQ, GGUF, INT8, or FP8 checkpoint is not interchangeable just because the model name matches. Test the exact checkpoint on the exact serving engine.

For a solo developer or a proof of concept, a modern workstation with enough unified memory or one consumer GPU can run a small quantized coding model. This is the cheapest way to learn where local quality breaks. CPU-only inference can work for occasional chat or batch review, but interactive completion punishes delay. A suggestion that arrives after the developer has typed the next line is waste, even when its tokens cost nothing.

For a shared team, concurrency matters more than the largest model that fits. Ten editors create short, bursty requests. Agent tasks create long prompts and long generations. Mixing both on one queue lets an agent monopolize capacity while completions time out. Separate pools or at least separate priorities are usually worth more than one larger checkpoint.

Measure capacity with a trace that resembles the working day. A benchmark that sends one request at a time reports the best case and hides queueing. Replay bursts of completion requests, keep a few chat generations active, and then start the longest agent task you permit. Watch p50 and p95 time to first token, cancellations, rejected requests, memory pressure, and recovery after the burst. Add users until the service misses the response target, then plan below that point so an upgrade or traffic spike does not consume all headroom.

Multi-GPU support also depends on the server. Some engines split one model across devices; others expect one model instance per GPU. Tabby's FAQ, for example, says its server uses one GPU per instance and suggests multiple instances selected with device visibility settings. That can scale replicas for more users, but it will not make an oversized model fit across two cards. Confirm the serving pattern before buying several smaller GPUs.

Budget for the unglamorous parts:

• Accelerator memory and a second node or fallback path for maintenance.
• Host RAM, fast local storage, network capacity, and power.
• Cooling, rack space, monitoring, backups, and replacement parts.
• Engineering time for drivers, runtimes, model evaluation, access control, and incident response.
• Capacity lost to replicas, upgrades, and traffic peaks.

llama.cpp supports Metal, CUDA, HIP, Vulkan, SYCL, and CPU paths, plus several quantization levels. That breadth is useful for experimentation and edge deployment. It does not mean every backend has equal speed or identical numerical behavior. A benchmark from one laptop says little about a different GPU server.

Quality falls in specific places

Smaller local models usually fail unevenly rather than becoming uniformly less intelligent. They may complete routine syntax well and still miss cross-file invariants, misuse an internal API, weaken a test, or lose instructions deep in a long context. Those are expensive misses because the output often looks plausible.

Treat completion, chat, code transformation, review, and autonomous agent work as separate products. A fast small model can be excellent for inline completion because the task has narrow context and the developer reviews every token. The same model may be unsafe for repository-wide refactoring where it must plan, search, edit several files, run tests, and recover from failure. Routing every task to one model saves operational effort but hides quality differences.

Context length deserves particular suspicion. A runtime accepting a large token window does not prove the model uses distant instructions reliably. Repository retrieval can also make matters worse by inserting outdated generated files, vendored code, or a similarly named API ahead of the relevant definition. Measure whether retrieval selected the right evidence, not merely whether the server accepted a long prompt.

Cloud comparisons must use the same tools and constraints. If the cloud agent can search the repository, run tests, and retry while the local model receives one pasted file, you measured the test setup. Conversely, a local system with a tuned index and permissive tools may beat a cloud chat window on internal APIs. Record model, checkpoint, quantization, system prompt, context policy, tools, timeout, and random seed where supported.

There is also a quality floor created by latency. For inline suggestions, look at time to first token, cancellation behavior, and acceptance after normalization. For chat, measure whether developers finish the task correctly. For agents, measure the repository state and tests after the run. Tokens per second is a capacity measure, not a user outcome.

Do not dismiss developer preference as softness. If the local assistant interrupts typing, proposes stale patterns, or fails on the team's main language, people will disable it or route around policy. A compliant tool that nobody uses does not reduce unapproved AI use.

A local bake-off should use your rejected work

The best evaluation set comes from tasks your team actually struggled with, including outputs that looked good and failed review. Public coding benchmarks can screen models, but they do not contain your authorization layer, build system, migration rules, or review standards.

Build 30 to 50 tasks across the work you expect the assistant to do. Remove secrets and personal data from the evaluation set. Include routine completions, unfamiliar internal APIs, bug fixes with a failing test, dependency upgrades, security-sensitive changes, and one task where the correct action is to ask for missing information. Keep a hidden expected result or scoring rubric for each task.

Use a simple record that can compare local and cloud runs without pretending judgment is fully automatic:

{"task_id":"auth-017","system":"local-q4","success":false,"tests_passed":41,"tests_total":43,"review_minutes":18,"unsafe_change":true,"ttft_ms":620,"wall_seconds":94}

The fields expose several failure modes. success says whether the task met its acceptance criteria. Test counts catch partial completion. Review time captures cleanup cost. unsafe_change blocks a model that gets a high average by occasionally weakening controls. Time to first token matters for interactive use, while wall time matters for agents.

Run every system more than once on tasks with stochastic output, but do not average away severe failures. Report pass rate, median review time, tail latency, and unsafe changes separately. A model with slightly lower pass rate and zero control violations may be the correct choice for a sensitive repository.

Set the acceptance rule before looking at results. For example, require no unauthorized network access, no weakened authentication tests, a maximum review-time increase, and a completion latency that developers will tolerate. Then state which measures are gates and which are comparisons. Otherwise the team will move the threshold to favor the system it already wants, usually the new hardware or the familiar cloud assistant.

Have two reviewers score a sample independently. Disagreement reveals vague criteria such as "good code" or "minor cleanup." Replace those labels with observable conditions: accepted without changes, accepted after formatting, required logic repair, violated a repository rule, or introduced a security defect. The rubric will never remove judgment, but it should make arguments specific.

Keep the test runner boring. A request loop, a fresh worktree for each task, test commands, and a small result file are enough. The output shape above can feed a spreadsheet or a database later. Resist building an evaluation portal until the team has used the rubric and argued about real examples.

Re-run a stable subset when the model, quantization, runtime, prompt, retrieval settings, driver, or editor extension changes. "Same model" is not a reproducible version. Record a model file digest and container image digest so rollback has a concrete target.

Controls must cover prompts, tools, and output

Hosting the model internally removes one external data recipient, but it does not make the assistant safe. The system can still expose one team's code to another team, write secrets into logs, execute destructive tools, copy incompatible code, or generate a vulnerable change that passes shallow tests.

NIST's AI Risk Management Framework organizes work around govern, map, measure, and manage. The useful part for an engineering owner is the loop: assign ownership, describe the intended use, measure failures, and respond. A firewall rule covers only a fraction of that loop. OWASP guidance for large language model applications also treats prompt injection, sensitive information disclosure, excessive agency, and supply-chain risk as distinct problems. Keeping inference local does not erase them.

Start with repository classes and allowed capabilities. A policy can remain small enough for humans to review:

repositories:
  public:
    models: [local-completion, approved-cloud]
    tools: [read, search, test]
  confidential:
    models: [local-completion]
    tools: [read, search, test]
  restricted:
    models: []
    tools: []
logging:
  prompt_content: false
  metrics: [latency, tokens, status, model_digest]

This fragment prevents two common failures. A developer cannot silently send a confidential repository to the cloud route, and operational metrics do not require storing prompt content. It also states that some repositories get no assistant. That is a legitimate answer when isolation or review cannot reduce the risk enough.

Authentication should identify the developer and repository context, not a shared editor token copied through chat. Authorization should run at the gateway because client settings can drift or be modified. Use short-lived credentials where the environment supports them, restrict service accounts, and test that one group cannot retrieve another group's indexed code.

Agent tools need a second control plane. Read access, shell execution, package installation, network calls, database access, and production deployment carry different consequences. Default to a sandbox with no production credentials. Require human approval for effects that leave the worktree, and log the tool name, actor, repository, decision, and result without dumping sensitive command output.

Prompt injection can arrive through source comments, issue text, generated documentation, test fixtures, or a dependency README. The model cannot reliably decide which natural-language instruction has authority. The agent controller must separate trusted policy from untrusted repository content, constrain available tools, and validate effects outside the model. A local model obeying a malicious instruction can still delete files or expose one internal system to another.

Write an incident procedure for accidental disclosure before the pilot. It should identify who can disable the gateway, preserve minimal evidence, revoke credentials, remove an index, delete retained prompts, and determine which repositories or people were affected. Local hosting makes containment your job. It does not eliminate notification or contractual duties when sensitive data reaches the wrong internal audience.

Generated code still goes through tests, static analysis, dependency checks, secret scanning, and human review according to the repository's normal rules. Labeling every line as AI-generated adds little if the organization cannot act on the label. Preserve the request and model version for high-risk changes when policy needs traceability, but set a retention period rather than keeping prompts forever.

The operating burden lasts longer than installation

The first successful completion is the beginning of ownership. Drivers expire, model formats change, editor APIs move, vulnerabilities arrive, and usage grows. Assign a service owner before launch and give that owner a maintenance window, an incident path, and permission to reject unsupported models.

Model provenance belongs in the deployment record. Store the source, license, checksum, model card, quantization method, conversion command, approval decision, and evaluation result. Scan container images and pin them by digest. If an engineer downloads a convenient quantized file from an unverified account, your internal server can faithfully distribute a poisoned artifact to every developer.

Separate content logs from service metrics. Operators need request counts, queue depth, time to first token, generation rate, failures, cancellations, GPU memory, temperature, and model identity. They usually do not need source code in an observability vendor. Debug logging can capture prompts unexpectedly, so test log behavior with a recognizable canary string before admitting real repositories.

Backups deserve the same classification as the source material represented in indexes and prompts. Encrypt them, restrict restore rights, set expiry, and practice deletion. Tabby's FAQ warns against putting its root directory on NFS because SQLite depends on filesystem locking and some network filesystems can corrupt the database. That is the kind of dull operational note that should shape architecture before production.

Plan upgrades as controlled changes. Stage the new runtime and model beside the current one, replay the stable evaluation subset, compare latency and safety failures, then move a small user group. Keep the previous image and model digest until the observation window closes. A model rollback that requires downloading yesterday's weights during an outage is not a rollback plan.

Availability requirements should match the workflow. Inline completion can usually tolerate a temporary outage because developers can keep typing. An agent embedded in release or incident procedures may need redundancy and tested failover. Do not buy high availability for a convenience feature, and do not run a release dependency on a single workstation.

The cost model must include people

On-premise inference replaces a variable service bill with hardware, idle capacity, power, and engineering ownership. It can be cheaper at sustained utilization, but low token prices in a spreadsheet do not compensate for a platform engineer spending every Friday on drivers and editor bugs.

Compare annual cost over the expected hardware life:

annual_local_cost = depreciation + power + hosting + support + operator_time + expected_downtime
annual_cloud_cost = subscriptions + usage + network + security_review + vendor_management

Use loaded labor cost for operator time. Include evaluation and security review on both sides because cloud systems also need governance. Model the expected load by task class, concurrency, and peak period rather than multiplying headcount by an arbitrary token allowance.

Three breakpoints matter. First, local hardware sits idle when adoption is low. Second, a team that outgrows one server may need redundancy, networking, and scheduling, which changes the cost curve sharply. Third, better cloud quality may save review time or finish tasks the local system cannot. The expensive model can be cheaper if it avoids enough engineering cleanup.

A hybrid route often wins. Keep restricted repositories local, allow approved cloud models for public or lower-sensitivity work, and route small completions to inexpensive local capacity. The policy must make the route visible and enforce it at the gateway. Asking developers to remember which editor menu is compliant will fail.

Buying and renting solve different capacity problems. Hardware purchase suits stable baseline demand and environments that require physical custody. Reserved private-cloud accelerators can suit predictable demand when policy accepts the provider. On-demand capacity suits trials and peaks, but startup delay and regional availability can hurt interactive work. Compare the control evidence and exit cost alongside the hourly price.

Run a four-week pilot before a fleet purchase. Measure active users, accepted completions, completed tasks, review time, queueing, failure classes, and operating hours. Price the configuration that met the quality bar with headroom, not the biggest model a vendor demonstrated.

For founders who want an outside cost and control review before committing capital, the Team & AI Audit offered through oleg.is is a fixed $5,000 engagement completed in five business days. The decision still belongs in your risk register and budget, not in a sales deck.

Choose the smallest boundary that survives review

The right deployment is the least complicated one that satisfies the written data rule, clears the team's quality bar, and has an owner. That may be local completion on managed laptops, a shared GPU server, a private cloud endpoint, an approved cloud assistant, or different routes for different repositories.

Reject an on-premise plan if the team cannot patch it, evaluate it, or keep its logs clean. Reject a cloud plan if the provider's actual terms and data flow leave an unresolved obligation. Reject either plan if developers need to bypass it to finish ordinary work.

Take one representative sensitive repository and write its permitted data flow on a page. Add two real coding tasks and one security change to an evaluation suite. Price only the systems that pass both the boundary review and the tasks. Hardware selection becomes much easier once models have failed in front of the people who must approve them.

Do not let a purchase freeze the architecture. Models, runtimes, and rules will change. A documented interface between editor, policy gateway, inference service, and tools lets you replace one layer without reopening every repository. The useful asset is the controlled path from developer intent to reviewed code. The GPU is replaceable.