Hiring engineers with AI needs a different interview

AI agents have made the traditional coding interview less honest, not because candidates can cheat, but because the work itself has changed. An engineer can now produce a convincing patch before an interviewer has finished reading the ticket. Measuring unaided syntax recall or typing speed tells you little about how that person will perform when an agent can write the first draft.

The scarce skill has moved upstream and downstream of code generation. Strong engineers decide what should change, give an agent useful boundaries, detect when a plausible patch violates the system, and know what evidence makes a change safe to ship. I hire for those decisions. I still expect coding fluency, but I treat generated code as raw material rather than proof of ability.

Stop using code production as the main signal

Code production is now a noisy proxy for engineering ability. A candidate who writes a clean function from memory may work well, but that exercise does not show whether they can own a change in a live product. A candidate who uses an agent may finish faster while missing a data migration, an authorization boundary, or an operational consequence. The interview must separate output from ownership.

This distinction matters: tool skill and engineering judgment are related, but they are not the same skill. Tool skill means giving context, splitting work, inspecting results, and recovering when an agent goes off course. Engineering judgment means choosing an acceptable tradeoff under incomplete information. A prompt can improve the first. It cannot supply responsibility for the second.

I no longer ask candidates to prove that they can reproduce library calls under observation. Documentation and agents are part of normal work. I ask them to explain which facts they need before changing a system, which assumptions they are making, and what could make a locally correct implementation wrong in production. Their answers reveal experience much faster than a puzzle does.

Keep a short unaided segment if the role requires reading code during an outage or working in restricted environments. Ten or fifteen minutes is enough to confirm that the candidate can trace control flow, name basic data structures, and reason without an assistant. Do not turn that check into the whole interview. Memorized syntax decays in value as soon as the editor opens.

The popular recommendation to ban AI during interviews is wrong. It is popular because it seems to restore a level playing field and makes the process easier to police. It actually creates an artificial job where candidates cannot use a standard work tool. Allow the tool, make its use visible, and evaluate the decisions around it.

Give the candidate a production-shaped task

A useful work sample has ambiguity, existing constraints, and observable consequences. It should be small enough to finish in roughly ninety minutes, yet broad enough that blindly generating a patch produces a defect. A tiny repository with one service, a database migration, tests, logs, and a short runbook is enough. You do not need a miniature copy of your product.

Give every candidate the same starting repository and let them use the agent they know. State that prompts, chat history, commands, and diffs will be part of the discussion. This removes the theater around hidden assistance. It also lets you see whether they use the agent as a typist, an investigator, or an authority.

A good exercise might ask the candidate to add account suspension to a small subscription service. The written requirement says suspended users cannot create new projects. The repository also contains an asynchronous import worker, an administrator endpoint, cached authorization decisions, and an audit event schema. The obvious controller check passes the visible acceptance test but fails the actual system.

Give them this task packet:

1. A one-page request with the user outcome and two explicit non-goals.
2. A repository map that may be incomplete but is not misleading.
3. Commands for tests, linting, and starting dependencies.
4. A support note describing one prior authorization incident.
5. A blank decision record asking for assumptions, rejected options, and release checks.

Do not score whether the candidate discovers every hidden detail. Score how they search for boundaries, update their model when they find evidence, and communicate uncertainty. A candidate who says, "I found three write paths and need to check whether there are more," is showing more ownership than one who confidently edits the first controller.

The exercise should use disposable credentials and isolated infrastructure. Never expose production data, secrets, private customer information, or proprietary code you cannot share with every candidate under the same terms. The point is to reproduce the shape of work, not its confidential contents.

Judgment appears before the first patch

Good candidates reduce uncertainty before they ask an agent to write code. They inspect the repository, restate the behavior in testable terms, identify the risky boundaries, and choose a change size that they can verify. This planning need not be long. Five disciplined minutes can prevent an hour of polished rework.

Listen for questions that change the implementation. Does suspension block scheduled work or only new interactive requests? Can administrators act on behalf of a suspended user? Must an existing project remain readable? How quickly must cached permissions expire? What event must support staff see? These are engineering questions because each answer changes code, data, tests, or rollout.

Weak candidates often ask many low-value questions, then claim they have clarified the task. Counting questions rewards performance. Score the effect of each question instead. Did it expose a missing invariant, narrow an unsafe assumption, or identify who owns a decision? If not, it is conversation rather than analysis.

Ask the candidate to write a compact plan before generating anything. This template is enough:

Behavior: Suspended accounts cannot create projects through any write path.
Invariants: Existing projects remain readable; admin recovery remains possible.
Boundaries to inspect: HTTP API, import worker, authorization cache, audit events.
Largest risk: A stale cached decision permits a write after suspension.
Evidence required: Path-level tests, cache invalidation test, audit event assertion.
Rollout: Migration first, guarded application change, observable rejection count.

The plan is not documentation theater. During review, compare it with the final diff. Strong candidates revise the plan when the repository contradicts them. Weak candidates quietly change direction and leave no account of why. In normal work, that missing reasoning becomes expensive when another engineer has to diagnose the change.

An agent can help search call sites or propose edge cases, but the candidate should decide which findings matter. If they paste the task into a chat and wait, ask what they expect the agent to miss. A useful answer names repository-specific risks. A vague answer about AI making mistakes does not.

Review taste finds plausible wrongness

Review taste is the ability to reject code that works in the narrow case but makes the system harder or less safe to change. It combines correctness, clarity, proportion, and awareness of local conventions. Agents produce enough polished, plausible code that this skill now deserves its own interview stage.

Give the candidate an intentionally flawed patch and ask for a review, not a rewrite. Seed four or five defects of different kinds: one visible logic bug, one missing authorization path, one needless dependency, one test that asserts implementation details, and one operational omission. Add a harmless style issue so you can see whether they spend limited attention on the wrong thing.

A strong review orders comments by consequence. It blocks the authorization gap, questions the stale cache, asks for evidence about the migration, and treats naming as optional cleanup. A weak review produces twenty comments about formatting while approving behavior that can corrupt data. The number of comments tells you almost nothing.

GitHub's documentation on reviewing AI-generated code recommends running tests and static analysis first, then checking intent, dependencies, maintainability, and AI-specific failures such as invented APIs or deleted tests. That sequence is sensible, but I add one requirement: the reviewer must reconstruct the change's blast radius before trusting automated checks. Tests only cover the paths somebody thought to encode.

Use a comment taxonomy when scoring the review:

• Blocking: the change can violate a security, data, money, or availability invariant.
• Required: behavior or evidence is incomplete enough that ownership would be unclear.
• Suggestion: the patch works, but a smaller or clearer design would reduce future cost.
• Question: the reviewer needs context and does not pretend otherwise.

Then ask the candidate to choose the first two comments they would send if the author were handling an incident. This forces prioritization. Mature reviewers adapt the depth and tone of review to the risk and the moment. They do not confuse exhaustive commentary with good judgment.

AI review can be a useful second set of eyes, but it cannot be the candidate's alibi. If an agent finds a bug, ask the candidate to validate it and explain the failure path. If the agent misses a seeded issue, ask what additional context or test would expose it. You are evaluating whether the person can turn a suggestion into evidence.

System thinking crosses repository boundaries

System thinking means tracing a change through state, time, interfaces, and operations. It is not the ability to draw a crowded architecture diagram. The best signal is whether a candidate notices that the requested code sits inside a larger sequence of events.

Walk through the suspension exercise. The candidate adds a status column and checks it in the project creation endpoint. Unit tests pass. Minutes later, a queued import creates a project because the worker loaded the account before suspension. A retry creates another because the idempotency token covers the HTTP request but not the job. Support sees no reason in the audit log, while the authorization cache continues to permit writes on another instance.

Each line of that failure is ordinary. Together they show why local correctness is weak evidence. The candidate should be able to move across several questions: Where is the source of truth? When does state become effective? Which callers bypass the main interface? What is retried? Which action must be idempotent? What can an operator observe? How can the team reverse the change?

Ask the candidate to sketch the sequence using boxes and arrows, then remove anything that does not affect the decision. A clean sketch might show the client, API, account store, cache, queue, worker, and audit sink. Watch where they place checks. If authorization exists only at the edge, internal callers may bypass it. If it exists only deep in storage, the user may get slow or confusing failures. Often the right answer has an early rejection for experience and an authoritative check near the write.

You can make system thinking testable with a small fitness check. For example, scan every project creation handler for a call to a shared authorization function, then keep behavioral tests for the API and worker. The static check catches a forgotten path; the behavioral tests catch a broken rule. Neither replaces the other. The distinction between a policy and its enforcement points is one teams routinely blur, and the consequence is a rule that exists only in the path used by the original developer.

Do not demand one architecture. Ask for tradeoffs. A candidate may prefer cache eviction, short expiration, versioned permissions, or a direct store read for high-risk writes. The useful signal is whether they connect the option to consistency needs, failure behavior, load, and rollback.

Run an interview loop that produces evidence

A compact loop can test these skills without consuming a full day. Use four stages: a work discussion, the production-shaped exercise, a review session, and a system reasoning conversation. Each stage should produce notes against defined behaviors. Avoid a final vote based on who felt senior.

In the work discussion, ask for one change the candidate owned after release. Follow the timeline: what did they know at the start, which assumption failed, what signal exposed it, who made the tradeoff, and what changed afterward? Candidates who did the work can usually distinguish their decisions from the team's. Polished generalities collapse under chronological questions.

During the exercise, let the candidate work for a period without interruption. Record only observable behavior: searched for all write paths, ran the existing tests before editing, accepted an agent patch without reading it, changed a test to fit the implementation, or documented an unresolved risk. Avoid labels such as "smart" or "not strategic." Labels invite bias and cannot support a hiring decision.

Score each dimension from one to four. A one means the behavior is below the role's needs, a three means independently effective, and a four means the candidate improves the method itself. Use these anchors:

• Problem framing: one implements the first interpretation; two notices ambiguity after coding; three defines behavior and meaningful assumptions first; four finds hidden stakeholders and resolves material ambiguity.
• Agent direction: one delegates the whole task blindly; two gives steps with weak context; three sets boundaries and verifies each result; four uses the agent to test alternatives while retaining control.
• Review taste: one focuses on style or accepts plausibility; two finds obvious bugs; three prioritizes correctness, risk, and maintainability; four improves the review strategy and identifies missing evidence.
• System thinking: one reasons inside one function; two follows one adjacent dependency; three traces state, asynchronous paths, failure, and recovery; four finds second-order effects and proposes proportionate controls.
• Verification: one treats passing tests as proof; two adds happy-path tests; three matches tests and checks to risks; four designs evidence that also helps operation and rollback.

A score of three should mean "independently effective at the level we need," not "better than average in this applicant pool." Define role-specific examples before interviews begin. Interviewers should submit scores before the debrief so a forceful opinion does not rewrite everyone else's observations.

One interviewer should own consistency across candidates. Rotate participants if needed, but do not improvise the task or raise the bar after meeting an unusually strong applicant. The process should compare evidence to the role, not people to the most memorable person that week.

Seniority changes the scope of judgment

Junior candidates should not be failed for lacking production history they have had no chance to acquire. Test whether they form a model, notice contradictions, read generated code carefully, and learn after feedback. Give them enough repository guidance that discovery does not depend on knowing your framework. Their strongest signal may be a precise explanation of what they do not yet know.

For a mid-level engineer, expect independent ownership of a bounded change. They should find relevant paths, choose useful tests, question dependencies, and explain a safe rollout. They may need help with organization-wide tradeoffs, but they should not need someone else to notice the worker, cache, or migration.

A senior engineer must control scope as well as code. Add a product constraint, such as support needing an emergency restoration path, and an operational constraint, such as the queue being hours behind during peak load. Ask what they would defer and how they would make that debt visible. Seniority shows in the ability to preserve the invariant without turning a small feature into a platform rewrite.

For staff-level roles, widen the decision. Ask how the team should encode the authorization rule so future services adopt it, how to detect drift, and when centralization would create a dangerous dependency. Expect them to discuss incentives and ownership, not merely components. They should be able to improve the environment in which other engineers and agents produce changes.

Do not use the same passing answer for every level. "Found the hidden worker" may be strong junior evidence, expected mid-level behavior, and insufficient staff evidence if the candidate never asks why the organization keeps creating bypass paths. Calibrate the bar before reviewing names or resumes.

Make AI use visible and safe

Candidates should be allowed to use AI under a clear interview policy. Tell them which tools are permitted, what data may be entered, what artifacts you will retain, and whether the session is recorded. Offer a company-controlled option for candidates who do not want to connect a personal account. Equal access matters more than forcing everyone to use the same interface.

Do not score prompt eloquence. Long prompts can hide weak thinking, while a short request can work because the candidate first assembled excellent context. Score whether the candidate chooses trustworthy inputs, constrains risky actions, reads commands before approval, and verifies output. Anthropic's Claude Code security guidance makes the user responsible for reviewing proposed code and commands before approval. That is the right boundary for an interview too: permission prompts do not transfer accountability to the tool.

Prompt logs are useful evidence only when candidates know they are being reviewed. Treat those logs as interview data with the same retention and access rules as notes. Remove secrets and personal identifiers from the repository. If local employment or privacy rules require consent or limit automated decision making, have counsel shape the process rather than copying another company's policy.

A candidate who declines AI should still be able to complete a smaller version of the exercise. You are hiring engineering judgment, not loyalty to a vendor. Compare the quality of decisions, review, and verification, while treating speed as context. For a role built explicitly around agent operation, tool fluency can be a stated competency, but name it in the job description.

Cheating becomes less useful when the interview asks for a live defense of decisions. A candidate may import a polished solution, but they still have to trace a failure, modify the patch under a changed constraint, and explain the evidence they would require before release. Identity checks and a short synchronous conversation handle the remaining concern without turning the process into surveillance.

Test recovery when the agent goes wrong

Recovery reveals more than a perfect first pass. Give the candidate an agent transcript that took a reasonable request in the wrong direction, then ask them to regain control without discarding useful work. Real teams spend plenty of time correcting partial changes, stale assumptions, and commands that behaved differently from the plan.

A simple setup works well. The agent has added account suspension by scattering checks across five handlers. It also changed a shared error type, regenerated a large fixture file, and skipped a failing worker test. The visible API behavior is correct. Ask the candidate to decide what to keep, what to revert, and what to investigate before another command runs.

Strong candidates first stabilize the workspace. They inspect the diff by concern, restore the skipped test, and separate generated noise from the behavior under review. They do not ask the same agent to fix everything in one prompt. They form a new hypothesis, reduce the change, and run the narrowest check that can disprove it. If the shared error change has unknown callers, they search those callers before accepting a supposedly harmless refactor.

Watch how the candidate handles sunk cost. Generated code can look expensive because there is a lot of it, even when producing it took seconds. Keeping a weak design to preserve that output is irrational. A candidate with good judgment will delete most of a large patch if a smaller enforcement point gives clearer behavior. They will also retain a useful test or repository note when it remains valid.

Add one failed command to the transcript, such as a migration running against the wrong local database. The candidate should inspect the environment and resulting state before retrying. Repeating the command with broader permissions is not recovery. It is escalation without diagnosis. For roles with production access, this distinction deserves serious weight.

Score the recovery on four observations: whether the candidate stops compounding the failure, reconstructs what changed, preserves verified work, and leaves the branch easier to review. Speed matters only after control returns. Someone who calmly spends ten minutes reducing uncertainty is safer than someone who produces a second large patch in two minutes.

This exercise also tests agent fluency without rewarding knowledge of one interface. Every coding agent can receive narrower context, inspect a diff, run a test, and revise a plan. The specific command changes across tools; the ownership pattern does not. A candidate who can explain that pattern can transfer between products as the tools change.

Hire for ownership after generation

The final decision should answer one question: would I trust this person to own the consequences of an AI-assisted change at the role's expected scope? The answer must come from observed behavior, not confidence, vocabulary, or the visual polish of generated output.

Look for a coherent chain across the interview. The candidate framed the behavior, found system boundaries, directed the agent in bounded steps, rejected a plausible but unsafe patch, and designed checks that matched the risk. One weak moment may be coachable. A repeated habit of outsourcing judgment is not.

Also examine the environment you are hiring into. If the team rewards large diffs, skips design context, and treats review as a merge queue, even excellent hires will struggle to use agents safely. Faster generation increases the need for small changes, explicit invariants, fast tests, and reviewers with time to think. GitHub's review guidance puts automated checks first; the organization must make those checks fast enough that engineers actually run them.

Do not average scores blindly when the evidence conflicts. A candidate who earns a four for agent direction and a one for verification is not a solid midpoint; that combination describes someone who can produce unsafe changes very quickly. Set nonnegotiable floors for the risks of the role, then discuss the observations behind any outlier. If two interviewers disagree, return to the recorded behavior or run a short follow-up focused on the disputed skill. Confidence in a debrief should never outweigh direct evidence.

This is where a Team

This is where a Team & AI Audit AI Audit can be useful: I examine the work, roles, and AI workflow together, then identify where a smaller AI-augmented team can operate without hiding risk. The hiring scorecard still belongs to you, because no outside advisor should decide which tradeoffs your engineers must own.

Keep the interview hard in the places the job is hard. Let agents write the routine code. Ask humans to define the right change, recognize the convincing wrong one, and defend the system after the patch ships.