When does on-device AI belong in your product?

On-device AI earns its place when local execution changes the product, not when it merely changes the architecture diagram. It can make an interaction immediate, keep sensitive input off your servers, work without a connection, and remove inference cost from the marginal cost of each action. Those are product effects. A model that happens to run on a phone but adds a long download, drains the battery, or behaves differently across chipsets is an engineering demo.

I have watched teams choose the model first and discover the product constraints afterward. That order produces six weeks of compression work around a feature nobody priced, measured, or scoped. Start with the user promise and the weakest device you intend to support. Then choose the smallest model and runtime that can keep that promise under heat, low battery, poor connectivity, and ordinary multitasking.

Local inference also does not require an all-local product. Many good systems use a small model on the device for fast, private, frequent work and a server model for rare requests that need more knowledge or compute. The boundary should follow the interaction and the data, not a slogan about edge or cloud computing.

Local inference must buy a product advantage

A local model is justified when it creates an advantage users can notice or a cost structure the business can defend. Privacy alone can qualify, but only if the full data path supports the claim. Sending raw audio to analytics after transcribing it locally does not make the feature private. Caching camera frames for remote debugging can undo the main reason for running vision on the device.

Four advantages commonly survive scrutiny:

• The interaction needs a response while the user is still touching, speaking, or pointing.
• The feature must work in a tunnel, warehouse, aircraft, rural area, or other unreliable network environment.
• The input contains material that should not leave the device, such as personal audio, documents, or a live camera feed.
• The action happens often enough that server inference cost or network traffic changes unit economics.

Do not accept "lower latency" as a requirement until the team defines the clock. Time to first result, time to final result, frame rate, and input-to-feedback delay describe different experiences. A writing assistant might need the first suggestion quickly but can finish ranking later. A camera overlay cannot hide a slow frame behind a loading state. A background photo classifier may tolerate seconds and still benefit from offline processing.

There are equally clear reasons to keep inference on the server. The feature may need fresh public information, access to a large private corpus, a model too large for the supported hardware, centralized abuse controls, or results that must stay identical across clients. Server execution also makes model replacement easier. If a model changes every week, shipping hundreds of megabytes through app releases or asset downloads can become the dominant operational problem.

Write the decision as a falsifiable product statement: "This feature runs locally so it returns an initial result within our interaction budget, works without a connection, and keeps the raw input on the device." If testing disproves one of those effects and the remaining effects do not justify the complexity, move it back to the server. Architecture should not become a matter of pride.

The weakest supported device sets the budget

Model selection starts with a device envelope, not a leaderboard score. The envelope describes what the product can safely consume while the rest of the application and operating system continue working. Peak lab speed on a recent flagship tells you almost nothing about an older phone with less free memory, a warm battery, and another app playing audio.

Set explicit budgets for package or download size, peak resident memory, cold load time, steady latency, battery use, and sustained temperature. Add concurrency: a model that fits alone may fail when the camera pipeline, image decoder, database, and UI all allocate memory at once. Mobile operating systems can terminate an application under memory pressure without giving your inference code a polite error.

A one-billion-parameter model illustrates why weight size is only the opening calculation. At four bits per parameter, raw weights occupy roughly 0.5 GB. The runtime may also need quantization metadata, tokenizer data, temporary tensors, converted buffers, and a growing attention cache. Some backends copy or repack weights during loading. The application therefore needs materially more than 0.5 GB of available memory, and that memory competes with every other feature.

Use a checked-in capability envelope that product, mobile, and ML engineers can review together:

on_device_feature: compose_suggestion
supported_tier: mid_range_2022_and_newer
model_download_mb_max: 180
peak_rss_mb_max: 650
cold_start_ms_p95: 900
first_result_ms_p95: 250
sustained_run_minutes: 10
offline_required: true
fallback: server_or_manual_input

These values are examples, not universal targets. Their purpose is to force decisions. If the feature requires a 900 MB download but growth expects users on metered connections, the conflict appears before integration. If product requires support on entry-level hardware, the ML team can train for that tier rather than apologize after launch.

Thermals deserve a sustained test. Phones and fanless laptops often run a model quickly for the first minute, then reduce frequency as heat builds. Benchmark the actual usage loop for longer than a demo. Include charging, camera use, screen brightness, and ambient conditions that resemble the product. Measure energy per completed task as well as latency; a slightly slower CPU path can be preferable when a GPU path wakes expensive resources or fights the renderer.

Storage has its own trap. Bundling a model increases the initial application size for every user, including people who never touch the feature. Downloading it later saves that cost but adds consent, progress, retry, integrity, versioning, deletion, and low-storage behavior. Neither choice is free. Treat model delivery as a product flow with an owner.

Choose a model family before choosing a famous model

The right model family is usually the narrowest one that solves the job. Teams reach for a small language model because it gives an impressive demo across many prompts. They then spend months trying to make a general generator reliable at a task a classifier, ranker, encoder, or compact speech model could perform faster and with clearer acceptance criteria.

Match the model to the output. Use a classifier to detect a known state or intent because it has a small output space and clear thresholds. Use an embedding encoder to find similar text or images and reuse the vectors for retrieval and ranking. A tagger or compact encoder fits field extraction from constrained input, while a purpose-built audio model fits transcription or wake-phrase detection. Reserve a small generative model for rewriting or open text where the product actually requires generation.

A generative model can still be the correct choice, especially for rewriting, summarizing private documents, natural-language commands, or offline assistance. Judge it by the full feature, not parameter count. Tokenizer behavior, context length, attention cache growth, supported operators, quantization quality, and decoding speed all affect whether it fits. A smaller model with an unsupported operator can run worse than a larger model that the device accelerator handles completely.

For vision, input resolution often costs more than teams expect because intermediate activation memory grows with spatial dimensions. Cropping a region of interest, reducing frame rate, or running detection every few frames may save more energy than swapping architectures. For audio, streaming chunks can reduce perceived delay and memory use, but state management becomes part of correctness. For text embeddings, vector dimensions affect local index size and search time as the user's collection grows.

Model quality must be measured on the product's data. Public benchmark averages hide the errors that make a feature unacceptable: names in a transcription tool, small text in a document scanner, mixed-language input in a keyboard, or a rare safety state in an industrial application. Build an evaluation set from consented, properly handled examples that represent supported users and conditions. Separate model errors from preprocessing errors; incorrect resizing, normalization, tokenization, or audio sampling can make a good model look broken.

Licensing and redistribution belong in the first model review. Confirm that the model license permits the commercial use, modification, and distribution your application needs. Record the model source, exact version, training or fine-tuning lineage you know, evaluation result, and license text. A model embedded in a shipped binary is a distributed dependency, not a notebook experiment.

The runtime decides which hardware you actually use

A phone may contain a CPU, GPU, and neural accelerator, but your model uses only the units the runtime can target with supported operators. Marketing specifications for the chip do not guarantee acceleration for your graph. One unsupported operation can cause a partition to fall back to the CPU, add copies between memory layouts, and erase the expected gain.

On Apple platforms, Core ML is the native first choice for most product teams. Apple's Core ML documentation says the framework can select CPU, GPU, and Neural Engine resources, while MLComputeUnits lets an application restrict the allowed units. The useful detail is control, not the promise that all units are always faster. A background task may deliberately use the CPU, while an interactive foreground task can allow all units. Convert the model early and inspect its behavior on every supported operating-system and chip tier.

On Android, LiteRT provides a compact runtime with CPU execution and hardware delegates. Delegates are not interchangeable switches. Operator coverage, device drivers, model format, and vendor implementation determine what runs on a GPU or NPU. Google's LiteRT NPU documentation makes the vendor relationship explicit: chip vendors provide delegates for their hardware. That fragmentation is why an Android plan needs a tested compatibility matrix and a reliable CPU path.

ONNX Runtime Mobile is a reasonable choice when a team values one model representation across platforms or already exports through ONNX. Its execution-provider design can target different accelerators, but the same coverage rule applies. Reducing the runtime build to the operators a model uses can control binary size. The price is another compatibility layer to diagnose when exported graphs, providers, and platform versions disagree.

ExecuTorch fits teams whose training and export workflow already centers on PyTorch. Its documentation describes ahead-of-time export, lowering, and memory planning, with specialized backends and portable CPU fallback. It also recommends backend-specific program files because hardware support differs. That is honest engineering: "cross-platform" means a shared workflow and runtime contract, not one binary that performs identically everywhere.

For local language models, runtimes built around quantized transformer execution and formats such as GGUF can be practical on desktops and selected mobile tiers. They make it easy to try different quantized weights, but easy experimentation should not choose the production stack. Check application binary size, model loading, memory mapping, tokenizer parity, accelerator support, license obligations, and how the runtime behaves when the operating system takes memory away.

Pick one primary runtime per platform and keep a boring fallback. Supporting several runtimes to chase the best benchmark on each chipset multiplies packaging, observability, testing, and upgrade work. Earn that complexity with measured adoption and user impact, not a synthetic speed chart.

Quantization is a product trade, not a packaging trick

Quantization reduces the precision used for weights and sometimes activations. It can shrink downloads, reduce memory bandwidth, and enable specialized hardware paths. It can also change model output in ways a generic quality score misses. Treat each quantized artifact as a different model release with its own evaluation and device results.

Post-training quantization is the fastest place to begin. Integer or low-bit weight formats often preserve enough quality for classification, embeddings, and many generative tasks, but the safe level depends on the model and data. Quantization-aware training can recover quality when post-training conversion fails, though it adds training complexity. Mixed precision is often sensible: keep sensitive layers at higher precision and compress the rest.

Do not optimize raw file size while ignoring runtime representation. A four-bit file may be expanded, repacked, or partially dequantized after loading. Ask the profiler how much memory the process holds during initialization and inference. Measure first-run compilation or cache creation too; users experience that pause even if steady-state tokens per second look good.

The acceptance suite should compare the original and candidate artifact at three levels. First, evaluate task quality on representative inputs. Second, compare feature behavior, including thresholds, formatting, refusal behavior, and structured-output validity. Third, run performance and energy tests on physical devices. A model can pass mathematical similarity checks and still cross a product threshold often enough to annoy users.

For a classifier, inspect per-class precision and recall around the operating threshold rather than one average. For embeddings, test whether the same useful results remain near the top of real searches. For generative text, use deterministic fixtures where possible, then human review for cases where several outputs can be correct. Record the tokenizer, prompt template, decoding parameters, and stop conditions with the model version. Otherwise a prompt change will be mistaken for a quantization regression.

Pruning, distillation, smaller input dimensions, and constrained decoding can matter as much as quantization. Distillation is especially useful when a larger model can generate or label training examples for a narrow student model, provided the team reviews the data and does not copy the teacher's mistakes blindly. The goal is the smallest complete feature, not the smallest weight file.

The best product patterns hide the model boundary

Local inference works best when the product can use a bounded result immediately and recover cleanly when the model is uncertain. The user should experience a responsive feature, not manage a miniature model server. Designs that expose raw generation delays, model downloads, context limits, and unexplained confidence scores transfer engineering problems to customers.

Good patterns include continuous perception, private assistance, and local ranking. A camera application can detect document edges without uploading frames. A meeting tool can produce a live draft transcript locally, then let the user decide whether anything is shared. A writing product can rank a small set of suggestions near the cursor. A field application can classify equipment states where connectivity is unreliable. In each case, the model has a narrow job inside a larger interaction.

Progressive results often beat waiting for one final answer. Speech can display stable partial segments. Search can show local matches while a remote source continues. A document tool can extract obvious fields locally and request confirmation for uncertain ones. This structure turns variable inference time into a usable flow, but the UI must distinguish provisional and final output without flicker.

Confidence needs a product policy. Model probabilities are not automatically calibrated, and a score of 0.9 does not mean the prediction is correct nine times out of ten in your application. Use held-out product data to choose thresholds. Give the system a non-model fallback such as manual entry, ordinary search, or a server request. A low-confidence path should protect the user, not merely display a smaller number.

Personalization is tempting because the data already sits on the device. Start with local context selection, preferences, adapters, or a small user-specific index before attempting on-device training. Training raises new questions about battery, data corruption, reset behavior, migration, and support. Apple documents on-device update capabilities in Core ML, but framework support does not make continuous learning the right product decision. Predictable personalization usually beats a model that quietly changes after every session.

Avoid local AI for authoritative decisions that require a complete audit trail, centrally updated policy, or heavy cross-user context unless the architecture supplies those properties elsewhere. A local fraud signal can help prioritize a check; it should not become an invisible final authority that support staff cannot reconstruct. A local medical or safety feature carries an even higher verification burden. The ability to run offline does not remove accountability.

A hybrid design should fail in both directions

A good hybrid architecture defines what happens when the network disappears and what happens when the device cannot run the model. Teams often design only the happy path: local first, cloud when needed. Production adds low storage, model corruption, unsupported operators, thermal pressure, server timeouts, version skew, permission denial, and users who disable downloads.

Separate routing policy from model code. The policy can consider feature availability, model version, connectivity, user consent, device tier, request sensitivity, and current resource pressure. Keep the decision small enough to test. Do not let separate screens invent separate meanings for "offline capable."

A practical request can have four outcomes: local result, remote result, manual path, or explicit unavailability. Define which inputs may leave the device before implementing remote fallback. If the feature promise says raw audio stays local, a timeout cannot silently upload it. Ask for consent at the moment a new data path becomes relevant, and explain the consequence in product language.

Model updates need the discipline of application releases. Sign artifacts, verify integrity before activation, keep the previous compatible version, and switch atomically after validation. Store the model version beside derived data when reproducibility matters. If an embedding model changes, old and new vectors may not share a meaningful space; the client may need to rebuild its local index or keep versioned indexes during migration.

Server and device models can also disagree. If both generate structured fields, version the schema and validate each result before the rest of the application consumes it. If the server reviews or improves a local result, retain enough provenance to explain which model produced which part. Do not use device model output as trusted instructions on the server. Treat it as untrusted client input and apply the same authorization and validation you would apply to any request.

Privacy claims must describe the system, not the inference location. Document raw inputs, derived features, logs, crash reports, analytics events, remote fallbacks, backups, and support exports. Local processing can reduce exposure substantially, but only when telemetry does not recreate the sensitive payload. Prefer operational measures such as duration, model version, route, and error code over captured prompts or frames.

Shipping a model to the user device changes the security assumptions. Anyone who can read the application bundle or downloaded assets may eventually inspect, copy, or modify the artifact. Platform encryption and code signing raise the effort and help detect tampering, but they do not turn distributed weights into a server-side secret. If the model contains irreplaceable intellectual property, price the extraction risk before choosing local delivery. Legal terms and obfuscation may deter casual copying; neither can guarantee confidentiality on hardware controlled by another person.

Never place API credentials, private prompts, access rules, or confidential records inside model weights or adjacent configuration. A model does not provide a secret store, and determined analysis can recover memorized material or plainly bundled files. Keep authorization on a trusted system. If a local feature needs privileged data, give it a narrow, authenticated interface and return only the minimum result the user is allowed to receive. The same rule applies to tools invoked by a local language model: the application must validate each action rather than trust generated arguments.

Local output is untrusted even when the artifact came from your release pipeline. A user can patch a client, replace a downloaded model, hook the runtime, or construct an input that changes model behavior. Servers must not accept local classifications, extracted identities, safety decisions, or generated commands as proof. Send the original verifiable facts when policy permits, or repeat the sensitive decision on a trusted service. When offline operation makes server verification impossible, limit the consequence to what the local user is already authorized to do.

Protect model delivery with the same controls used for executable content. Pin an expected digest or signed manifest, download to a temporary location, verify before activation, and reject a model built for an incompatible runtime or schema. Record activation failure without recording the model input. Security testing should include a truncated download, a replayed older artifact, modified metadata, a missing signature, and a client clock that is wrong. A safe updater must keep the last known compatible model available until the new one has loaded and passed a small local health check.

Production testing starts where the benchmark ends

A benchmark tells you whether one artifact can execute under one set of conditions. A product test tells you whether the feature remains useful across the supported fleet and over time. Ship only after the team can connect model quality, runtime performance, resource use, and fallback behavior to a specific model and device tier.

Test physical devices that represent the floor, the median, and important accelerator families. Include clean installation, model download interruption, low storage, airplane mode, background and foreground transitions, camera or audio interruption, thermal load, and application upgrade. Emulators help with logic but cannot establish battery, accelerator, driver, or memory-pressure behavior. One old device on an engineer's desk often finds more truth than another afternoon tuning a desktop benchmark.

Keep inference telemetry narrow and non-sensitive. An event can support operations without recording user content:

{
  "feature": "compose_suggestion",
  "model_version": "2026-08-rc3",
  "device_tier": "mid",
  "route": "local",
  "load_ms": 412,
  "inference_ms": 138,
  "fallback_reason": null,
  "result_status": "accepted"
}

Aggregate distributions rather than celebrating a median. The slow tail shapes the experience, and failure rate can hide behind latency charts that exclude timeouts. Break results down by model version, application version, OS, device tier, runtime backend, thermal state if available, and route. Use coarse device classes when exact hardware identifiers create privacy or cardinality problems.

Roll out by capability and cohort. Start with employees or consenting testers, then a small production group on hardware you understand. Keep a kill switch for the feature or model version, but make sure disabling local inference does not violate a privacy promise by silently routing requests remotely. A safe shutdown might return the manual interface instead.

Ownership should be explicit. The ML engineer owns evaluation and artifact lineage; platform engineers own integration and resource behavior; product owns the user promise and fallback; operations owns release visibility and rollback. In a small company, one person may wear several hats, but the responsibilities still exist. A Team & AI Audit can expose where those responsibilities and costs have drifted apart before a local model becomes another permanent subsystem.

Approve on-device AI only when the feature passes on the weakest supported hardware, its data path matches the privacy language, and the team can update or disable it without breaking the product. If the local path cannot meet that standard, a well-run server call is the better engineering choice. Users care that the feature works when they need it; they do not care where the matrix multiplication happened.