Hiring exercises for engineering judgment that show tradeoffs

Why clean coding tests miss the point

Good hiring exercises for engineering judgment should look like real work, not a puzzle book.

Most engineering teams don't spend their days solving neat problems with one correct answer. They deal with missing context, old code, deadlines, budget limits, and the risk that one rushed change breaks something customers use every day.

A clean coding test strips all of that away. It often rewards recall, speed, and practice with interview patterns. A candidate can do well because they've seen the same graph problem ten times, not because they make sound decisions when the facts are incomplete.

That gap matters. Real projects rarely start with perfect inputs. Engineers inherit half-written docs, vague requests, and systems with rough edges. If your exercise removes those constraints, you aren't testing judgment. You're testing how well someone plays a narrow interview game.

Strong candidates usually respond to a messy prompt by asking questions first. They want to know who uses the system, how risky the change is, how soon it must ship, what the team can support, and what happens if the plan fails. That's a good sign. It shows restraint.

A realistic exercise also reveals how people weigh speed, cost, and risk. Imagine a candidate has to fix a slow checkout flow in a legacy service before a sales event. They can add caching quickly, but stale data might cause pricing errors. They can rewrite part of the service, but that takes longer and raises delivery risk. There isn't a perfect answer, and that's exactly why the scenario works.

You learn more from how they reason through that choice than from whether they remember a textbook pattern. Do they name their assumptions? Do they notice what could fail? Do they choose the smallest safe change when time is tight, or explain why a deeper fix is worth the delay?

Trivia doesn't tell you much about how someone will work on your team. Reasoning does. Clean tests make candidates look tidy. Messy scenarios show whether they can make a sane call when the situation is anything but tidy.

What a messy scenario should include

A good exercise starts in the middle of something. That's closer to the job.

If you hand a candidate a blank page, you mostly learn how they like to build from scratch. Real work usually looks different. People inherit code, partial docs, old decisions, and a system that already has users.

That's why these exercises work better when they feel a little uneven. Give the candidate an existing service, a recent problem, and a few limits they can't ignore. One deadline is enough. One budget cap is enough. One reliability issue is enough. With just a few constraints, you can see whether the person reaches for the safest fix, the cheapest fix, or the fix they can still support six months later.

Leave a few facts unclear on purpose. Don't hide the whole setup, but do leave enough gaps to force questions. Maybe the error rate spikes only at peak hours. Maybe the team doesn't know whether the database or the queue is the bottleneck. Strong candidates don't guess too quickly. They ask what traffic looks like, who is on call, what failure costs the business, and what can wait.

The materials should look plain, like work people actually receive. A short note from a founder or manager, a simple architecture sketch, a few log lines or error samples, and one small table with traffic, cost, or uptime numbers are usually enough. Fancy interview portals often add noise. A rough diagram in a document usually works better because it keeps the focus on reasoning.

Keep the scope tight. One interview should cover one system, one problem, and a small set of decisions. For example, ask the candidate to improve an API that times out during a daily billing run while the team has two weeks to fix it and no budget for major new infrastructure. That's enough to surface judgment. You can watch how they sort facts, name risks, protect reliability, and choose what not to change yet.

If the candidate can understand the situation, ask smart questions, and make a clear tradeoff with limited time, the exercise is doing its job.

Choose tradeoffs that fit the role

A good exercise should feel like a real week at work, not a coding contest.

If every candidate gets the same kind of tradeoff, you miss what the role actually needs. The best scenarios match the decisions that person will face on the job.

Junior roles need narrower choices. Give them a bug in existing code, a little missing context, and a teammate who will inherit the fix. You want to see whether they pick the safest path, ask sensible questions, and leave notes another engineer can follow. A junior candidate doesn't need to redesign the system. They need to avoid making a small problem worse.

Mid-level roles should carry more tension. Add a deadline, a product request, and code that clearly needs cleanup. Then watch how they split the work. The strongest answers usually sound practical: ship the smaller fix now, add tests around the risky path, and name the cleanup that should happen next. That tells you more than a polished refactor with no sense of timing.

Senior roles need wider tradeoffs. Give them a scenario where one option lowers short-term risk but adds team load, and another saves time now but creates pain later. Strong senior candidates talk about rollback plans, monitoring, ownership, migration cost, and what happens when the system changes six months from now. They should also notice people limits. A design can look fine on paper and still fail if the team can't support it.

The scenario should also match your stack and the way your team works. If your team ships quickly with a small group and a lot of automation, test for that. If you work in a slower environment with approval steps and strict uptime needs, build those limits into the exercise. A backend candidate should face backend tradeoffs. A mobile candidate should deal with release risk, device limits, or offline behavior.

When the setup matches the job, candidates stop guessing what you want. They show how they think when the answer is messy, time is limited, and every option costs something.

Build the exercise step by step

A good exercise is light on paperwork and heavy on judgment. One page is often enough if it gives the candidate a real problem, a business goal, a few hard limits, and room to ask questions.

Start with the outcome, not the task. "Improve this system" is too loose. "Cut cloud spend by 30% without causing outages before the next launch" gives the candidate something real to optimize. Prompts like that work especially well for startups and small teams because cost, risk, and speed usually pull in different directions.

Write a brief that sounds like a real request from a manager or founder. Include the business goal, the current pain, and a little context. Keep it plain. If the candidate needs ten minutes just to decode the setup, the exercise has already gone off track.

State the constraints up front. Give them a budget cap, a time limit, a team size, or a rule they can't break. You might say the product serves paying users, the release date is fixed, and the team can't rewrite everything.

Hold back some details on purpose. Decide which facts you'll reveal only if the candidate asks. Good candidates ask about traffic, failure history, compliance needs, and who maintains the system after launch. If they never ask, that tells you something.

Make the scorecard before the first interview. Keep it simple. Score how they frame the problem, how they handle tradeoffs, what risks they spot, and whether their plan is easy for another engineer to maintain. Add notes for strong signals and red flags so different interviewers score the same way.

Then test the exercise with one teammate. Watch where they get confused, bored, or pushed into guessing. Trim anything that creates noise instead of signal. If two smart people misunderstand the same sentence, fix the prompt, not the candidates.

A small example helps. If you work with lean teams, give a scenario where a product is growing, the cloud bill hurts, and uptime still matters. That setup quickly shows who can make careful cuts and who jumps straight to a risky rebuild.

When this prep is done well, the interview feels calm. The candidate can think, ask, and choose. That's where judgment shows up.

A sample scenario you can adapt

A midsize online store wants a new returns flow before holiday traffic starts. Right now, customers contact support, one agent checks the order, and another person issues the refund. It works on quiet days, but it falls apart when volume climbs.

The real problem sits in the current order service. During busy hours, it starts throwing errors, timing out, or returning stale data. So a simple "build a returns feature" task isn't simple at all. Any plan that leans too hard on that service may fail when the store needs it most.

Add a few realistic constraints. Finance doesn't want higher cloud spend this quarter and doesn't want any new paid tools. Support wants fewer manual steps because each refund takes too long and errors create angry follow-up tickets. Give the candidate a small team and a short deadline, such as two engineers and three weeks.

Then ask for a first rollout plan, not a perfect final design. That's when judgment gets visible. The candidate has to decide what to fix now and what to leave for later.

You can ask them to cover four practical points: what they would ship first, how they would reduce manual work for support, how they would protect the system during peak traffic, and how they would roll out, measure, and roll back the change.

A thoughtful candidate will usually avoid a full rewrite. They may suggest a narrow first version: allow returns only for recent orders, use existing infrastructure, add a retry path for refunds, and keep manual review for edge cases. That answer shows restraint, which is often a better sign than ambition.

A weaker candidate often chases the cleanest architecture on paper. They propose new services, extra databases, and a queueing stack the team doesn't already run. That can sound smart, but it ignores the budget, the deadline, and the risk of adding more moving parts before the holiday rush.

One follow-up question works especially well: "What would you do in week one?" The answer tells you whether the candidate can separate urgent work from nice ideas. If they can name the first change, the biggest risk, and the metric they'd watch after release, you're seeing judgment instead of interview polish.

Questions that expose judgment during the exercise

A messy exercise only works if your follow-up questions force choices.

Good candidates don't try to fix everything at once. They pick an order, explain the tradeoffs, and admit what they'd leave unfinished.

Start with something simple: "You can ship one part this week. What goes out first?" Listen for a reason tied to user value, risk, or revenue. Weak answers sound like a task list. Strong answers sound like a plan: launch the narrow path that proves demand, keep the rest manual, and avoid building a lot of code before anyone uses it.

Then ask, "What risk would you cut before launch, even if it slowed you down?" This shows whether the candidate can separate annoying problems from dangerous ones. A startup checkout flow can survive a clumsy admin screen for a week. It shouldn't ship with weak access control, no backups, or no way to see failures.

Technical debt is where judgment gets real. Ask, "What would you do the fast way for now, and what line would you refuse to cross?" Good candidates accept some debt on purpose. They might postpone refactoring, write a manual support tool, or skip a nice abstraction. But they should also name the debt that gets expensive fast, such as tangled data models, missing tests around billing, or ad hoc permissions.

Metrics matter because they show whether the candidate thinks past launch. Ask what they would measure in the first week. Look for a short set of signals that match the scenario: error rate on the main workflow, time to complete the task users came for, drop-off at the fragile step, support tickets or manual fixes, and cost per transaction if infrastructure spend matters.

Last, ask what would make them change their plan. This is often the most revealing question. Strong candidates name triggers: higher traffic than expected, support pain, a security issue, poor conversion, or a teammate flagging a hidden dependency.

Don't score only the answer. Score the candidate's threshold for changing course. That's often where mature judgment shows up.

Mistakes that ruin the signal

A bad exercise can make a strong candidate look average, and an average candidate look polished.

The first mistake is turning the exercise into a guessing game. If the interviewer hides basic facts and waits for the candidate to ask the one "right" question, you aren't measuring judgment. You're measuring mind reading. Give enough detail to start, then let the candidate state assumptions and test them.

Speed can fool you too. Some people answer in 30 seconds and sound certain. Others take a few quiet minutes, then notice the rollback risk, the support burden, or the cost problem that everyone else missed. Good judgment often looks calm, not flashy.

Interviewers need the same scoring rules before the session starts. Without them, one interviewer rewards confidence, another rewards detail, and someone else rewards a design style they happen to like. A simple rubric keeps the discussion honest. Score how candidates weigh cost, risk, and maintainability, and how they adjust when new information appears.

Another common mistake is stuffing the exercise with architecture trivia. A messy scenario should feel like work, not a pub quiz for senior engineers. If the role is about shipping a stable product with a small team, ask about tradeoffs that small teams actually face: deadlines, on-call load, migration pain, and what happens when the first version goes wrong.

Polish can mislead you. A tidy diagram and smooth language can hide weak choices. A rough sketch with plain words can reveal better thinking. Pay more attention to whether the candidate protects users, limits blast radius, and leaves room to change course later.

Small details matter during the interview. If someone asks whether uptime matters more than speed of delivery, answer clearly or tell them to pick an assumption and defend it. If someone changes their plan after hearing a new constraint, treat that as a good sign, not a failure.

When teams miss these points, they often hire the person who performs best in the room, not the person who makes sound decisions under pressure. If two answers seem close, choose the candidate who sees the tradeoff, explains the downside, and picks a path the team can still live with six months later.

A simple scoring rubric

A good scorecard should reward clear judgment, not polished talking.

The strongest candidates often sound calm and practical. They don't chase the cleverest answer. They find the problem that matters most, then work from there.

Keep the rubric short. If you need a long checklist, the exercise is probably too fuzzy.

• Did they spot the real limit quickly?
• Did they separate today's fix from next month's cleanup?
• Did they ask for facts before picking a path?
• Did they explain their tradeoffs in plain language?
• Did they suggest a plan a normal team could actually run?

A small example makes the difference clear. Say the system fails during peak traffic, the logs are noisy, and the database schema is messy. One candidate proposes a rewrite in a new stack. Another says, "I would reduce load, add monitoring around the failing path, stop the data damage, and schedule schema cleanup after we stabilize." The second answer usually shows better judgment.

Score practicality, not confidence. Some candidates speak with total certainty and still miss the real risk. Others pause, ask two smart questions, and build a better plan.

One last filter helps: would the team trust this person during a rough week? If the answer is yes, the score should reflect it.

What to do after the interview

The exercise doesn't end when the candidate leaves. If your team waits until the next day to talk about it, you'll lose a lot of the signal.

Block 10 to 15 minutes right after the session. Ask each interviewer to write notes alone first, then compare them. That keeps the most confident person from rewriting everyone else's memory.

Write down what the candidate actually said. Skip labels like "smart," "senior," or "good communicator." A note like "kept the old queue for now because a rewrite could cause outages during billing week" tells you far more than a gut feeling.

That matters because these exercises are about reasoning, not style. You want proof that the candidate noticed tradeoffs, ranked risks, and picked a plan that fits the team you have, not the team they wish they had.

If several interviewers met the same person, compare patterns across the whole loop. One strong answer can happen by luck. Steady judgment shows up more than once. Maybe the candidate asks for missing facts in one round, then makes a careful scope choice in another. That kind of consistency is hard to fake.

A short debrief works well when it focuses on four questions: which tradeoffs the candidate noticed without prompting, how they weighed cost against risk and maintainability, whether their plan fit the role and team size, and whether they changed their mind for a solid reason when challenged.

Keep the exercise, notes, and scorecard in a shared hiring kit. Save the scenario, the follow-up questions, and a few anonymized examples of strong and weak answers. After a handful of interviews, review the kit. If interviewers score the same answer in wildly different ways, the problem is your interview design, not the candidates.

Small startup teams feel bad hires quickly. One person with weak judgment can create months of cleanup work. If you want outside help shaping hiring loops, scorecards, or practical engineering interviews, Oleg Sotnikov at oleg.is does this kind of Fractional CTO work in a hands-on, operator-focused way.