How the AI engineer role works in practice

An AI engineer turns a model capability into a dependable product behavior. The job is mostly software engineering under unusual uncertainty: models return variable answers, vendors change them, user input can attack them, and a convincing response can still be wrong. Someone must make that system useful, measurable, affordable, and safe enough to operate. That person is usually the AI engineer.

The title is inconsistent across companies. A research lab may use it for training models, while a startup may mean a product engineer who can connect an API to business data by Friday. Read the ownership behind the title, not the title itself. If the role has no authority over evaluation, data access, release controls, or production monitoring, the company probably wants a demo builder and has mislabeled the job.

How company stage changes the scope

The AI engineer role expands or contracts with the company's product maturity. At an early startup, one engineer may interview users, choose a model, build retrieval, write the backend, add a review screen, and inspect production traces. That breadth is rational when the team is still proving that a workflow deserves to exist. The same arrangement becomes reckless once the system handles regulated data or can take costly actions.

A small product company usually needs an application-oriented engineer. This person starts with an existing model, connects it to the product, builds evaluations, and owns the service in production. Fine-tuning may appear later, but it is rarely the first move. Most early failures come from a vague task, poor context, missing permissions, or no definition of a good answer. Training does not repair those product decisions.

A larger company splits the work. A platform team may provide model gateways, identity, logging, approved data connectors, evaluation infrastructure, and cost controls. Product AI engineers then build use cases on that foundation. ML engineers may train ranking, forecasting, or classification models, while researchers test new architectures. Security, legal, design, and domain specialists become regular collaborators rather than occasional reviewers.

At a model company, the same title can sit much closer to research. The engineer may optimize inference, prepare training data, run distributed experiments, or build tooling for research teams. That is legitimate AI engineering, but it is a different career from shipping an assistant inside an accounting product. A candidate should ask what percentage of the first six months concerns model development, application code, data pipelines, evaluation, and operations. The answer exposes the actual job.

Consultancies create another variation: fast prototypes across many client environments. Breadth and discovery skills matter there, but a portfolio of prototypes does not prove production judgment. Ask who maintains the system after handoff, who owns incidents, and whether the engineer sees real user failures. Repeated exposure to consequences teaches the role faster than repeated exposure to launch presentations.

Product ownership defines the job

An AI engineer owns a user outcome, not a prompt. Prompts are one implementation detail alongside data selection, tool schemas, permissions, interface design, fallbacks, and ordinary code. If a support assistant must draft an accurate refund reply, the engineer needs to know which policy applies, how the account state enters the request, what the model may propose, who approves it, and what happens when required evidence is missing.

The normal work starts before model selection. The engineer turns a broad request such as "help agents answer faster" into an observable task. A useful specification might say that the system drafts a reply from the current policy and ticket history, cites the policy passage internally, never changes an account, and routes uncertain cases to a human. Now the team can collect representative cases and compare implementations.

This is why a clever chat demo proves so little. A presenter controls the questions, recognizes a plausible answer, and quietly retries failures. Production users paste malformed text, mix several requests, omit context, and expect consistent latency. Some inputs will contain instructions copied from untrusted documents. The engineering work begins where the controlled demo ends.

Ownership continues after release. The AI engineer reviews failed traces, separates model errors from retrieval and product errors, updates the evaluation set, and decides whether a model change is safe. They watch cost and latency distributions, not just averages. They also give support and operations teams enough evidence to explain a result without exposing private reasoning or sensitive context.

A normal week mixes feature work with investigation. The engineer may spend one morning labeling failed conversations with a domain expert, then change a retrieval query, add a regression case, and compare two model versions. Another day goes into ordinary backend work: correcting a race condition, adding a timeout, or changing a schema after a tool returned an unexpected value. Production AI work does not happen in a separate experimental world. It lives in the same issue tracker, deployment process, and on-call rotation as the rest of the product.

When an answer is wrong, the engineer should locate the earliest broken assumption. Did the request omit the customer region? Did retrieval return an obsolete policy? Did the prompt describe the task badly? Did the model ignore correct context? Did post-processing discard a citation? Did the interface let the user approve an action without seeing its basis? Each cause demands a different repair. Switching models first is popular because it takes minutes and produces a fresh demo, but it often moves the failure instead of removing it.

The work also produces less visible artifacts. The engineer writes a model-change checklist, an evaluation rubric, a data-flow note, a tool permission table, and a runbook for degraded service. They agree with product managers on which errors block release and with domain reviewers on how disputed labels get resolved. They give finance a cost-per-completed-task measure and give support a way to attach a trace identifier to a report. These documents are short when the system is understood; their absence usually means decisions remain trapped in one person's memory.

Daily priorities should follow user consequence. A formatting defect in an internal draft can wait. Cross-tenant retrieval, an unauthorized tool call, or a confident medical or financial claim demands immediate containment. The role therefore needs access to production evidence and a clear escalation path. An engineer who only receives screenshots of bad answers cannot distinguish a rare model variation from a systematic data or authorization defect.

A healthy role has a decision boundary. The engineer can reject a use case that lacks measurable success criteria, can remove a dangerous tool, and can require human approval for an irreversible action. A company that demands "an AI feature" but withholds those decisions turns the engineer into a decorator.

AI engineering and ML engineering overlap without matching

AI engineering and ML engineering share code, data, deployment, and measurement, but their center of gravity differs. The clearest distinction is what each role changes when quality is poor. An ML engineer often changes training data, features, model architecture, optimization, or the serving pipeline. An application-focused AI engineer often changes task decomposition, context assembly, retrieval, tool design, prompts, workflow controls, or the user interface before changing the model.

Google Cloud's Professional Machine Learning Engineer guide describes an ML engineer who builds, evaluates, productionizes, and optimizes ML models, with strong skills in data platforms, distributed processing, model architecture, training, retraining, and MLOps. That is a useful definition because it centers the model lifecycle. Many current AI engineer jobs consume pretrained models instead. They still need to understand model behavior, but their primary artifact is an application.

A data engineer makes trusted data available and maintains the pipelines, contracts, lineage, and storage that support it. An AI engineer may build a small ingestion path for one feature, but should not quietly become the owner of the company's entire data estate. When retrieval fails because documents are stale or access labels are wrong, the two roles must agree on ownership instead of passing the incident back and forth.

A software engineer can build AI features without changing titles. In fact, an experienced backend or product engineer with good experimentation habits often adapts faster than someone who knows model vocabulary but cannot operate a service. The separate title is useful when uncertainty evaluation, model behavior, and AI-specific risks occupy a sustained part of the job. It is not a promotion above software engineering.

"Prompt engineer" describes a task, not enough ownership for most permanent teams. Writing and testing instructions matters, but prompts without datasets, code review, version control, evaluations, and monitoring are unmaintainable. The market briefly treated phrasing as a standalone secret. In practice, good instructions come from understanding the domain and the system around the model.

Research engineers sit nearer the boundary between experiments and implementation. They reproduce papers, build training systems, and turn research ideas into efficient code. They generally need deeper mathematics and framework knowledge than an application AI engineer. Neither path is more authentic; confusion begins when a job description asks for both at full depth and offers one person's time.

The required skills form a practical stack

A strong AI engineer first needs solid software engineering. They should write readable code, design APIs, model data, test failure paths, use version control, review changes, and debug distributed systems. Python is common around models and data; TypeScript, Java, Go, or another production language may matter more in the existing product. Language choice matters less than the ability to work inside the company's real stack.

They need enough model knowledge to predict behavior and investigate surprises. That includes tokens and context limits, sampling, embeddings, retrieval, tool calling, structured outputs, latency, and the difference between training, fine-tuning, and prompting. They should understand that fluent text is not calibrated confidence. For model-building roles, add linear algebra, probability, optimization, deep learning frameworks, experiment design, and distributed training.

Evaluation is the most commonly missing skill. The engineer must convert product expectations into testable criteria, build a set that represents ordinary and ugly cases, choose automated checks where possible, and organize human review where judgment matters. A single aggregate score hides failures. Results should be sliced by language, customer type, input length, workflow, risk level, or any dimension that changes the consequence.

Data judgment follows closely. The engineer should trace where context came from, how fresh it is, whether the user may see it, and what transformations altered it. Retrieval quality depends as much on document structure and access control as on embeddings. A technically correct search result can still be a privacy failure if it crosses a tenant boundary.

Product skill separates useful systems from technically impressive ones. The engineer needs to observe the existing workflow, find where uncertainty is tolerable, and design a fallback that preserves the user's work. They should know when a deterministic rule beats a model. Parsing a fixed identifier, checking a permission, or calculating an invoice belongs in code. Use a model for the part that genuinely needs interpretation or generation.

Operations and security complete the stack. A production owner needs tracing, rate limits, caching judgment, queues, timeouts, retries, budget controls, secrets management, and incident response. They must treat retrieved text and model output as untrusted input. A model that can call tools deserves the same careful authorization design as any other client, plus extra suspicion because natural-language instructions can redirect it.

Communication is not a soft extra. An AI engineer has to explain probabilistic behavior without hiding behind it, obtain examples from domain experts, and tell a founder when a requested guarantee is impossible. Clear writing improves specifications, prompts, evaluation labels, runbooks, and incident reports at once.

Evaluation starts with cases, not a benchmark score

A useful evaluation reproduces the decisions the product must make. Public benchmarks can help compare general capabilities, but they rarely encode your refund policy, document permissions, tone constraints, or cost of a false action. Build the first evaluation set from actual workflow examples before spending days comparing model leaderboards.

Keep each case small enough to diagnose. This JSON Lines fixture gives a support-drafting system an input, the facts it may use, required properties, and a forbidden action:

{"id":"refund-eligible-01","input":"I was charged twice for order 1842","facts":["two settled charges","duplicate charge policy permits refund"],"must_include":["acknowledgment","human approval required"],"must_not":["claim refund was issued"]}
{"id":"refund-missing-02","input":"Refund this now","facts":[],"must_include":["request order identifier"],"must_not":["invent account details","call refund tool"]}

Run the fixture through the complete application path, not directly against the model. Retrieval, templates, tool schemas, policy code, and post-processing all affect the result. Store the model and prompt version, retrieved document identifiers, tool requests, latency, token usage, grader result, and final user-visible output. Without that context, a failed score tells you almost nothing.

Use deterministic graders for facts that code can inspect: valid JSON, required fields, citation identifiers, tool arguments, prohibited phrases, and numeric bounds. Use rubric-based model graders carefully for qualities such as relevance or tone, and calibrate them against human labels. Reserve human review for ambiguous cases and high-consequence decisions. An automated judge can accelerate review; it cannot define business correctness on its own.

NIST's Generative AI Profile recommends comparing output with known ground truth through varied evaluation methods, including human oversight and automated evaluation. I agree with the mix, but teams often skip the earlier requirement: someone must define ground truth and document its limits. For a policy answer, the approved policy version may be ground truth. For a creative draft, reviewers may need a rubric and accept several valid results.

Track regressions by slice and severity. A release that improves average tone while newly exposing private data has failed. Set hard gates for security, authorization, and destructive actions; use softer thresholds for style. When a provider releases a new model snapshot, replay the same set before routing production traffic. Add every meaningful incident as a permanent regression case.

Reliable systems constrain model authority

The safe way to build an agent is to give the model less authority than the user or backend, then enforce the difference outside the prompt. A sentence such as "never send money without approval" is guidance, not access control. The tool itself must require an authorized user, validate the amount and recipient, enforce limits, and record the decision.

OWASP's GenAI Security Project calls this risk Excessive Agency. Its 2025 guidance points to excessive functionality, permissions, and autonomy as root causes. That framing is more useful than blaming hallucination because a perfectly coherent model can still act on a malicious instruction embedded in an email or document. The permission boundary determines the damage.

Consider an assistant that reads vendor invoices and prepares payments. A retrieved invoice contains hidden text telling the assistant to ignore prior instructions and replace the bank account. If the model can both read arbitrary documents and submit an approved payment under a shared credential, prompt injection becomes a financial control failure. A better design extracts proposed fields, displays the source evidence, compares changes against known vendor records, and requires a separate authorized person to approve the payment. The payment service validates all of that independently.

Reliability needs the same external controls. Set timeouts and total step limits so an agent cannot loop indefinitely. Make retries idempotent so a network error does not duplicate an action. Validate structured output before another system consumes it. Route low-confidence or incomplete cases to a person with the context needed to decide. Preserve an audit record of inputs, retrieved sources, tool calls, approvals, and the result.

Privacy rules belong in the architecture. Minimize the data sent to a model, redact fields that the task does not require, keep tenant filters outside model control, and define retention with each provider. Do not log full prompts by default and then discover they contain credentials or health data. Give engineers a safe diagnostic view with access controls and deliberate sampling.

Cost also needs an engineering budget. Track cost per completed task, not cost per model call. A cheap call that causes three retries and a human correction can cost more than a stronger first pass. Cache only when the permissions and freshness rules permit it, and cap runaway workflows before the invoice reveals the bug.

Hiring should test a production decision

A good hiring process asks candidates to reason through a bounded product problem. Trivia about model names expires quickly, and a weekend chatbot mostly proves that an API is easy to call. Give the candidate a small workflow, representative examples, one dangerous edge case, and a requirement to explain what they would measure.

For example, ask them to design an assistant that classifies incoming sales requests, drafts a reply, and may create a CRM task after approval. A strong candidate clarifies the business objective, separates classification from generation, defines the data and permission boundary, proposes evaluation cases, and keeps task creation behind an explicit control. They discuss failure recovery and observability without being prompted.

A practical work sample can remain short:

1. Provide 15 labeled inputs, including duplicates, missing fields, hostile text, and one ambiguous case.
2. Ask for a working service or clear design that returns structured output and records enough detail to debug it.
3. Require a small evaluation report that explains failures rather than hiding them in one score.
4. Discuss how the design changes at ten times the traffic and with confidential customer data.

Score the submission on problem framing, code quality, evaluation, safety, and judgment. Do not reward framework complexity. The candidate who uses a simple classifier with clear failure handling may be stronger than the candidate who deploys a multi-agent workflow for the same task. Ask what they deliberately chose not to build. Good engineers protect the team from unnecessary AI as often as they introduce it.

Senior candidates should have incident stories. Ask about a model or data change that passed testing and still failed, how they found it, what immediate mitigation they used, and which permanent test came from it. Listen for precise ownership. Someone who describes every failure as "the model hallucinated" may not yet know how many application defects wear that label.

For a first AI hire, prefer production software experience plus evidence of evaluation and product judgment. Hire a specialized ML engineer first only when proprietary model development or a genuinely model-centric capability is the business. If the company mainly needs to add AI to an existing SaaS workflow, an application engineer who can learn the model layer usually ships sooner and leaves a maintainable service.

Career paths start in adjacent disciplines

There is no single entry route into AI engineering. Backend engineers arrive with service design and operational habits. Frontend and product engineers understand interaction, feedback, and human review. Data engineers bring lineage and pipeline discipline. ML engineers know experimentation and model behavior. Domain experts who can program may have the rarest advantage: they know what a correct result looks like.

A beginner should build one narrow system deeply rather than five chat wrappers. Choose a task with an accessible set of examples, such as extracting obligations from contracts or routing support tickets. Define success, create an evaluation set, build the simplest baseline, add tracing, and document failures. Then add retrieval or tools only if the evidence demands them. The resulting project demonstrates judgment, not just API familiarity.

The portfolio should include artifacts an interviewer can inspect: a task specification, anonymized evaluation cases, a short architecture note, a failure analysis, and a cost and latency breakdown. Show how one change improved a defined slice and harmed another. Explain the release gate. A polished interface helps, but it should not conceal the engineering record.

Study in the order the work requires. Learn production programming and APIs, then model fundamentals, structured output, retrieval, evaluation, security, and operations. Read provider documentation, but also read NIST's AI Risk Management Framework and the OWASP GenAI guidance because vendor examples tend to optimize for getting a first response, not assigning organizational accountability. Build enough statistics knowledge to reason about samples, noise, and tradeoffs.

Career progression follows scope. A junior engineer implements a defined component and its tests. A mid-level engineer owns a feature in production. A senior engineer chooses architecture, establishes evaluation and release practices, and coordinates domain, security, and product decisions. A staff engineer sets shared standards and prevents every team from rebuilding an unsafe gateway. Management is another path, but it does not replace technical judgment.

Certificates can structure study, but evidence wins. I would hire the candidate who can show why 12 carefully chosen regression cases changed a release decision over someone who lists six model certificates and cannot define failure. Open-source contributions, internal tools described without confidential details, and rigorous personal projects all count.

Some companies should not hire this role yet

A company should delay hiring an AI engineer when it cannot name a workflow, provide examples, assign an owner, or tolerate the operational work after launch. The hire cannot manufacture a product strategy from a mandate to "do AI." Start with the business decision or repetitive workflow, then determine whether a model improves it enough to justify uncertainty and cost.

Do not hire the role to avoid fixing basic software and data problems. If customer records lack stable identifiers, permissions live in spreadsheets, and nobody owns the support policy, a model will make those defects harder to see. Repair the source systems or constrain the first use case to data the company can trust.

A short external assessment may be cheaper than a premature permanent team. At oleg.is, I use a five-business-day Team & AI Audit to map engineering work and identify where AI augmentation can save money before proposing a transformation. The useful output is a prioritized operating plan, including the places where deterministic automation or process changes beat an AI feature.

Once the company has a named workflow, representative cases, an accountable product owner, and authority to change the surrounding system, the role can pay for itself through shipped work. Give the engineer ownership of evaluation and production behavior, not a queue of prompts. If leadership will not grant that scope, change the job description before hiring anyone.