Technical hiring in the AI era: judgment over boilerplate

What changed in technical hiring

A few years ago, routine coding work told you a lot. If someone built a small API cleanly, fixed bugs quickly, and finished a home task without much help, that often matched how they would perform on the job.

That match is weaker now. AI tools can produce scaffolding, tests, refactors, and common patterns in minutes. Two candidates can submit work that looks equally polished even when their skill levels are far apart.

Home tasks still show whether a person can finish something. They just reveal less about how that person thinks. A candidate can get to a decent draft with an assistant, then spend most of their time smoothing the surface.

The bigger difference now sits behind the code. When the first version is cheap, the real signal comes from decisions. Why this data model? What risk did they miss? Where would they add monitoring? What would they cut if the deadline moved up by a week?

That is why hiring now puts more weight on judgment than boilerplate. Good engineers still use AI. The difference is that they do not stop at the first workable answer. They question it, trim it, and reshape it for the real problem.

A simple example makes the shift obvious. Two candidates submit a small service with similar endpoints, tests, and structure. On paper, both look strong. Then you ask how each one would handle rate limits, bad input, or a partial outage. One gives vague answers. The other explains tradeoffs, spots failure points, and lays out a sensible plan. That gap matters more than who typed the first controller faster.

On lean teams that use AI heavily, this gap gets wider. One engineer with sound judgment can ship a lot. One engineer who accepts generated code without thinking can create weeks of cleanup.

Why boilerplate lost value

A clean CRUD app used to be a useful signal. It showed that a candidate could structure code, wire up a database, and finish a basic task. Now many candidates can produce almost the same result with AI help, even when their understanding is very different.

That does not make AI use a problem. It changes what the test measures. Fast output often tells you more about prompting habits than engineering judgment. Someone can generate a polished service, tests, and docs in 20 minutes, then struggle to explain why the schema looks that way or what breaks under load.

The gap shows up as soon as you leave the happy path. Boilerplate does not cover the messy parts teams deal with every week: partial failures, bad data, unclear requirements, ugly migrations, noisy logs, and rollbacks that need to happen now, not tomorrow. A smooth demo can hide weak understanding because demos live in clean conditions. Production does not.

Give two candidates the same task: build a small API with auth, validation, and database writes. Both return working code. One talks mostly about the generated files and how fast they finished. The other asks who owns the data, what happens if a client retries the same request, how the team will monitor errors, and how to undo a bad migration. The second candidate may write less code during the interview, but they usually think more like someone you can trust in production.

The best interviews look for signals boilerplate cannot fake for long. Can the candidate explain tradeoffs in plain language? Do they notice missing constraints before they code? Can they spot risk in a polished answer? Do they write down assumptions instead of hiding them?

That is the harder part now. Getting code on the screen is easy. Making choices that still look sensible a month later is not.

What strong candidates do differently

Strong candidates do not rush to code. They spend a minute shaping the problem first. They ask what success means, who will use the feature, what can fail, and what matters most: speed, cost, safety, or ease of change.

That first move tells you a lot. A weaker candidate often treats the prompt as complete. A stronger one notices that real work usually starts with unclear inputs and missing limits.

Give them a simple case such as an AI tool that summarizes support tickets. The better candidate asks a few pointed questions. Can the model see customer data? Does the team need perfect summaries or fast drafts? What happens when the model is wrong? Do agents review the output before anyone sends it?

Those questions are not stalling. They show judgment. The candidate is deciding which tradeoffs they can accept instead of pretending every goal fits together.

You can hear the difference in their answers. A strong candidate might say, "If support staff reviews every summary, I would ship a simpler version first. If summaries go straight to customers, I would slow down and add checks." That is better than a polished design with no sense of risk.

Strong candidates also bring missing constraints into the discussion. If your prompt says nothing about scale, privacy, ownership, rollout, or failure handling, they surface those gaps on their own. Most hard problems hide there.

Writing gives you one more clear signal. Ask for a short note afterward with their approach, assumptions, open questions, and what they would test first. Good candidates write plainly. They separate facts from guesses, explain why they chose a path, and make it easy for someone else to act on their thinking.

Perfect recall matters less than it used to. Assistants can produce standard code quickly. They cannot replace a person who spots the hidden constraint, makes a clear tradeoff, and explains it well enough for a team to move forward.

How to test judgment

Judgment shows up when the problem is messy, the facts are thin, and the candidate still finds a sane path forward. A clean coding task often hides that. Real work rarely does.

Start with a situation that feels unfinished. For example, a product team ships a pricing change, conversions fall, support tickets rise, and analytics look unreliable. Ask the candidate what they would do on day one, what they would ask for, and what they would avoid doing too early.

A strong answer starts with questions, not instant fixes. Good candidates separate facts from guesses, name the biggest risks, and explain what they need before they act.

A simple interview flow works well:

1. Give the candidate a messy scenario with a few gaps and a little noise.
2. Ask for two or three reasonable options, not one perfect answer.
3. Halfway through, add a new constraint.
4. Ask what they would measure after launch.
5. Score the reasoning first and look at code later.

The new constraint matters. Maybe the team cannot roll back. Maybe legal needs the feature live this week. Maybe only one engineer understands the payment service. Now you can see whether the candidate adapts or clings to the first idea.

When you ask for options, listen for tradeoffs. One candidate may push a fast patch and tight monitoring. Another may pause changes, audit the data, and run a smaller test. Either answer can be good if the logic is clear.

The measurement question is where shallow answers often crack. Ask what they would watch in the next 24 hours and the next two weeks. Good candidates name a few concrete signals: conversion rate, failed checkouts, support volume, refund rate, page latency, or error spikes. Better ones add guardrails so one fix does not create a new problem somewhere else.

If the candidate writes rough notes, outlines assumptions, or revises the plan after new information, give that weight. You are not testing who sounds most certain. You are testing who can think clearly under ordinary product pressure.

A neat solution is nice. Sound reasoning is what you keep.

How to test system thinking

Give the candidate a small service that feels real. One useful prompt is a SaaS app that lets users upload invoices, stores files, runs OCR, and shows results in a dashboard. Say it has 20,000 users, a few larger accounts, and a support team that reports slow imports every Monday morning.

That is enough to show how a person thinks. You are not looking for a perfect diagram. You want to see whether they notice that product choices, code paths, and operations affect each other.

Ask where delays, errors, and costs would appear first. Strong candidates usually ask a few clarifying questions before they answer. They want file sizes, peak traffic, retry behavior, and some sense of what users expect if processing takes more than a minute.

Then listen for the connections they make. A slow OCR job is not just a queue issue. It turns into support tickets, duplicate uploads, and refund requests. Keeping every intermediate file may help debugging, but it can raise cloud costs quickly. A retry rule can smooth over short failures, or it can create a storm that slows the whole service. One customer with huge files can hurt everyone else unless the system separates heavy jobs.

Strong answers move between the user view, the code view, and the operations view without getting lost. A candidate might suggest a clear progress message in the product, idempotent jobs in the worker, and separate queues or rate limits in production. None of that is flashy. It is the kind of thinking that keeps a service usable.

Good candidates also stay grounded on tradeoffs. They do not jump to five new services because the prompt mentions growth. Often the first fix is smaller: cap upload size, cache one expensive step, improve logs, split urgent work from bulk work, or remove a retry loop that burns money.

One useful twist is to change a fact halfway through. Tell them costs doubled last month, or uptime is high but complaints keep rising. People with real system thinking update their plan when the facts change.

Use writing in the interview

A short design note can tell you more than another live coding round. When AI tools fill in standard code quickly, clear writing often reveals clearer thinking.

The prompt does not need to be fancy. Give candidates a small, real problem: add audit logs to an internal tool, decide where to store user events, or roll out an AI assistant feature without exposing private data. Then ask for a one page note.

Do not grade it like an essay. You are not hiring a novelist. Look for structure that helps another engineer or manager make a decision.

A useful note usually covers five things: the goal, the main assumptions, one proposed approach, the biggest risks, and the next step a teammate could take tomorrow.

Assumptions matter more than polished prose. Strong candidates write sentences like "I assume event volume stays under 1 million per day" or "I would confirm whether support staff need read access before I choose the permission model." That is good because it shows they can see the edges of a problem.

The best notes are also actionable. Another teammate should be able to read the page and start work, challenge a decision, or spot a missing dependency. If the note sounds smart but leaves everyone asking "So what do we build?", it failed.

This is one place many teams miss strong people. A candidate may not speak smoothly in a live interview and still write a clean, practical note with sound judgment. I would trust that more than someone who talks fast, writes vague comments, and never pins down assumptions.

If you want engineers who work well with AI assistants, test whether they can give both the machine and the team a clear brief. That skill matters now.

A simple hiring scenario

Imagine a startup needs an internal tool for its support team. The goal sounds simple: one screen to find a customer, another to see recent activity, and a few buttons for common actions. Agents are losing time hopping between admin panels, chat logs, and payment records, so the pressure to ship fast is real.

An AI assistant can draft most of that in one sitting. It can sketch the screens, write basic handlers, add a search box, and wire a database call. A candidate who only shows that part may look fast, but speed by itself tells you very little.

The difference appears when the draft meets real use. Support staff work under pressure. They click the wrong account, deal with angry users, and need answers in seconds. A strong candidate starts asking questions that protect the business before they write more code.

They ask who can issue refunds and who should only view data. They ask what the team must log for audits and dispute reviews, how the tool should behave if two agents edit the same case, how to roll it out without slowing support on day one, and what happens when search returns the wrong customer.

Those questions matter more than another polished screen. A weaker candidate treats the tool like a small CRUD task. A stronger one sees the system around it. They ask how many tickets the team handles per day, which actions create the biggest risk, and which mistakes cost real money. They may even suggest a view only first release, then add sensitive actions later once the logs look clean.

That is the signal worth noticing. Boilerplate is cheap now. Judgment is not. When a candidate thinks about permissions, logs, error recovery, and support load before they chase polish, you learn how they will behave once the code reaches production.

Mistakes that weaken the process

The weakest hiring loops still reward speed over reasoning. That was shaky before, and it is worse now. A candidate can produce working code quickly with an assistant, a decent prompt, and strong autocomplete. Speed still matters a little. It just tells you almost nothing unless you also ask why they chose that approach, what risks they accepted, and what they would change after feedback.

Trivia is another trap. If interviews are full of syntax questions, framework facts, or gotcha puzzles, you mostly test memory. Real work looks different. Engineers read unclear requirements, ask questions, make tradeoffs, and fix problems without breaking everything around them.

A small task based on real work gives a better signal than ten clever questions. Ask the candidate to review a short design, spot failure points, or explain how they would ship a change safely. That tells you much more than whether they remember an obscure command.

Teams also skip writing because code feels easier to score. That is a mistake. Engineers write design notes, pull request comments, incident updates, and handoff docs. If someone cannot explain a choice in plain language, they usually create confusion later.

A short written prompt often reveals more than another coding round. You can see whether the person names assumptions, separates facts from guesses, and notices missing information. Clear writing and clear thinking usually travel together.

One polished demo should not erase weak judgment. A candidate may show a slick side project and still miss obvious concerns like rollback, monitoring, cost, privacy, or support burden. Nice output can hide shallow thinking.

Watch for simple scoring mistakes: giving extra points because someone finished first, treating trivia as proof of seniority, ignoring weak writing because the code ran, letting charisma outweigh poor tradeoffs, or skipping follow up questions after a strong first answer.

A better process slows down at the right moments. Ask the candidate to explain tradeoffs, write a short plan, and respond to one change in requirements. That is where judgment shows up.

A short checklist before you decide

Final interview scores can hide the part that matters most: how a person thinks when the answer is not obvious. Before you decide, look at the candidate's last task, written note, or home assignment and ask four plain questions.

• Can they explain tradeoffs in plain English?
• Do they leave behind writing another person can use?
• Do they ask about failure modes and cost without being pushed?
• Can they revise the plan when new facts appear?

You do not need perfect answers in all four areas. You want a pattern. If someone writes clean code but cannot explain a simple tradeoff, they may struggle once AI handles more routine work. If another candidate offers a modest solution, then spots a hidden cost and adjusts the plan, that person is often the better hire.

A small example makes it clear. Candidate A gives a neat design and sticks to it. Candidate B starts simple, then changes course after you add a tight budget and a strict uptime target. Candidate B is usually safer.

That is the person other engineers can work with on a real team, under real pressure, when facts change halfway through the week.

What to change next

You do not need to rebuild your whole hiring process. You need better signals.

Start with the round that tells you the least. If you still use a generic coding task, replace part of it with a judgment exercise. Give the candidate a small but messy problem: an unclear product request, a production bug with missing facts, or a system that works but costs too much. Ask what they would do first, what they would ignore for now, and which tradeoffs they see. Forty minutes is enough if the prompt is realistic.

Near the end of the process, add a short written brief. Ask for a half page note after a system discussion or incident review. Strong engineers usually explain assumptions, risks, and next steps in plain language. That matters because much of the job now is making decisions clear to teammates, founders, and customers.

Interviewers need a better rubric too. Smooth talk, neat diagrams, and fast answers are easy to overrate. A stronger scorecard checks whether the candidate asked useful questions before choosing an answer, named tradeoffs instead of pretending there was one perfect option, explained reasoning clearly, and changed direction when new facts appeared.

This does not need a big internal program. One shared rubric and a calibration session can fix a lot. It also helps to review the process every few hires. Look for rounds where everyone gives similar feedback, or where comments focus on confidence and polish instead of decisions. Those rounds usually add noise.

For teams reworking hiring while AI takes over more routine engineering work, Oleg Sotnikov at oleg.is offers Fractional CTO and startup advisory support. A practical outside view can help when a hiring loop still rewards style over reasoning.

A good process should tell you how someone thinks when the path is unclear. That is the part AI still does not solve for you.