Measure repo friction before rolling out coding assistants

Why pilots stall early

Most coding assistant pilots do not fail because the model is weak. They fail because the repo asks too much before anyone can make and test one small change. A team installs tools, fixes missing env vars, looks for seed data, and spends half a day trying to run a single command. By then, the pilot already feels slow.

Orientation is the next problem. People open the codebase and cannot tell where tests start, which script still works, or which module owns the behavior they need to change. They guess, run the wrong command, wait through avoidable errors, and lose momentum. The assistant looks clumsy when the person using it keeps stopping to ask where anything lives.

Large files make this worse. A 2,000-line module with mixed concerns dumps too much messy context into one place. Instead of making one clean edit, the assistant might touch the wrong area, miss a local rule, or suggest a change that breaks something nearby. Teams often blame the tool, but the repo created the confusion first.

A simple example shows the problem. A new engineer joins a pilot to change one validation rule. Local setup takes three hours. Finding the right test command takes another twenty minutes. Understanding the module takes almost an hour because business logic, helpers, and API calls all live in one file. If the assistant then gives a mediocre answer, people say the pilot failed. In reality, the repo burned most of the time before the assistant had a real chance to help.

Measure repo friction before you judge the tool. A clean pilot repo tests the assistant. A messy repo mostly tests your setup pain.

What repo friction looks like

You usually feel repo friction before you can name it. A developer pulls the code, runs the usual setup, and still gets blocked by one missing tool, one private script, or one local secret that nobody wrote down. The repo may work fine for the person who set it up months ago. For everyone else, it feels brittle.

It also shows up in the path to one small task. A simple fix should take one editor and one clear command. In a high-friction repo, that same fix needs three terminals, a background worker, a local service, and a few steps passed around in chat. If the steps live in team memory instead of the repo, new people lose hours before they write a line of code.

Ownership problems create another kind of drag. You change one field in the product, then touch the API layer, a shared package, two config files, and a job runner in another folder. Nobody is sure who owns half of it. That is where assistants tend to slow down instead of help, because the repo gives weak signals about where a change should start and stop.

Tests expose the same issue. A healthy test failure points you toward the broken behavior. A high-friction repo throws a vague stack trace, fails in a helper far from the real bug, or needs special local data that only one person has. The assistant can read the error, but it still lacks the missing context.

A small startup example makes this obvious. A new engineer tries to rename one billing field. Setup takes ninety minutes, the app only boots after finding a hidden script, the change touches four folders, and the failing test says little more than "expected 200, got 500." That repo is not ready for a broad pilot.

If you want to measure repo friction, watch where people ask for tribal knowledge. That is usually where the repo fights back.

What to measure first

Start with a few plain numbers. You need facts a new teammate can hit on day one, not opinions from people who already know the codebase.

Time is the clearest signal. Track how many minutes it takes to go from clone to the first successful app run, then do the same for the first successful test run. If one person needs 12 minutes and another needs 55, the repo already told you something.

Use the same pass for discovery work. Count how many commands a new teammate has to find, guess, or ask about before they can run the app and run tests. Pilots usually work better when people are not stopping to ask, "What do I run next?"

Five checks are enough for a first score:

• Minutes from clone to a working local app
• Minutes from clone to a passing test run
• Number of commands a new person must discover
• Size of the largest files and busiest modules
• Number of flaky or skipped tests in the area you want to pilot

Module size matters because assistants do better when the job fits inside a readable chunk of code. A 300-line file with one clear purpose is easier to work with than a 4,000-line module full of side effects.

Flaky and skipped tests matter for a different reason. They make the pilot look worse than it is. If the assistant suggests a clean fix but the test suite fails for unrelated reasons, your team will blame the tool instead of the repo.

A simple rule works well: if checkout is fast, tests start with one command, and the target module is small, you have a fair place to try a pilot. If setup takes an hour and half the tests fail at random, fix that mess first.

Check local setup time

Local setup time is often the fastest signal. A repo can look tidy on a senior engineer's machine and still be painful for everyone else.

Use a clean laptop, a fresh virtual machine, or an empty workspace. Old packages, cached credentials, and leftover env files hide problems. You want the first-run experience, not the lucky one.

As you set the project up, write down every step. That includes package installs, SDK versions, secrets, shell commands, database seeds, and every env var you had to guess or ask for. Small missing details add up fast.

Keep a short log while you work:

• Time to clone and install dependencies
• Time to first successful app run
• Time to first passing test
• Every moment where you had to ask another person for help

The gaps between docs and reality matter more than teams think. If the README says "run one command" but you actually need to install two extra tools, edit a config file, and request access from ops, the docs are wrong. That mismatch is friction, and assistants will hit it too.

A short example makes this obvious. A developer opens the repo at 9:00. The app finally starts at 10:10, but tests do not run until 11:00 because one private package needs a token nobody documented. That repo is a weak place to start a pilot. The assistant may help with code, but it cannot fix missing access or team memory.

Teams that clean up setup first usually get quick wins. Removing extra local services, tightening setup docs, and making the first test easy to run can save 30 to 60 minutes per new engineer. That alone can change whether a pilot feels useful or annoying.

Find the test entry points

A coding assistant helps most when feedback is fast. If someone cannot find the smallest useful test command in a few minutes, the pilot starts with guesswork instead of progress.

Use a two-person check. Ask one person who knows the repo a little to run the unit tests with no coaching. Then ask another person, starting cold, to run the same tests from scratch. Watch where they stop, what files they open, and which commands they try first.

A messy repo often hides test commands in odd places. One command lives in a package file, another sits in a Make target, and a third only appears in old notes. That matters because an assistant cannot help much if people spend twenty minutes just finding the right test entry point.

Look for one short path after one small code change. If a developer edits a single helper, they should be able to run only the related tests and get a result quickly. That short loop is where a pilot has the best chance to show real value.

A quick check works well:

• Can both people find the same test command without help?
• Can they run tests for one file, package, or module?
• Does that path finish fast enough to use during normal editing?
• Do the docs match the command that actually works?
• Does every small change force the full suite?

Flag any area where only the full suite works. That usually means the repo has poor boundaries, weak test naming, or stale scripts. In practice, those spots create slow feedback, and slow feedback makes assistants look worse than they are.

If you find even one clean, short test path, start the pilot there. If you do not, fix that path first. A repo with a clear test entry point gives people a fair test of the assistant instead of a test of their patience.

Review module size

Large modules slow people down before any assistant touches the code. They also make bad pilot areas, because you cannot tell whether the tool helped or the code simply fought back.

Start with a plain standard for "too big." That number depends on your team, but most people feel the pain long before a file hits 1,000 lines. A 700-line file that mixes API calls, business rules, and UI logic is already hard to read, hard to test, and hard for an assistant to change safely.

A quick scan usually tells you enough:

• List files that feel past your normal comfort limit
• Mark modules that combine screen code, business logic, and data access in one place
• Check how many files a small bug fix usually touches
• Flag folders whose names say one thing but contain something else

That last point matters more than teams expect. A folder called "billing" sounds clear, but if it also holds auth checks, email templates, and PDF export code, people waste time guessing where a change belongs. An assistant will guess too, and sometimes with more confidence than accuracy.

Count file touches with a real example. If a simple discount bug needs edits in seven files across three folders, the boundary is weak. If the same kind of fix lives in two small files with clear names, that area is a much better candidate for a pilot.

This is one of the fastest ways to measure repo friction. Smaller files do not guarantee good results, but they make review easier, reduce side effects, and help you tell whether the assistant can follow the codebase.

Score one repo in one afternoon

Pick one real task that shows up often. A small change works best: update a label, adjust a validation rule, or tweak one business condition. If the task is too big, your score tells you more about planning than repo friction.

Run the task from a cold start if you can. Use a laptop that does not already have every tool, cache, and secret set up. That gives you a more honest baseline before a pilot starts.

Track four moments with a timer:

• Local setup finished
• App or service ran the first time
• Relevant test ran the first time
• First code edit landed and worked

Then score each step with simple colors. Green means the step felt clear and fast. Yellow means you got there, but only after guessing, waiting, or reading too much. Red means you got blocked, needed help, or still could not finish.

Write one short sentence every time something slows you down. "Missing env file name." "Three test commands, none obvious." "Build takes 14 minutes before one line change." These notes matter more than a polished spreadsheet because they tell you why the repo is hard to use.

A quick example helps. A team changes a pricing rule in an older service. Setup takes 25 minutes, the app starts after two retries, the right test command takes ten minutes to find, and the first edit touches a 1,800-line file. That repo is not a great first place for an assistant pilot, even if the code quality is decent.

Pick the pilot area from the score, not from gut feel. Start where setup is tolerable, tests are easy to reach, and modules are small enough that an assistant can stay on track. That usually gives you cleaner early wins and fewer false starts.

A simple pilot choice example

A small team wants to compare two repos before starting a pilot. Both matter to the business, but only one gives them a fair first test.

Repo A takes about 15 minutes to run on a new laptop. The setup notes are short, the app starts without guesswork, and the test commands are easy to find. Most files stay around 300 lines, so a developer can read a file, understand it, and check an assistant's suggestion without getting lost.

Repo B looks more important, so the team feels tempted to start there. That is usually a mistake. A fresh setup takes half a day, one senior engineer has to explain the order of commands, and several tests only make sense if you already know the codebase. Some files are close to 2,000 lines. An assistant can still help, but the team will spend more time fighting the repo than judging the pilot.

Repo A is the better first choice because it gives the team clean signals. If the assistant saves 20 minutes on a change, that gain is easier to see. If it writes a weak test, someone can spot the problem fast. If the workflow breaks, the team knows the issue came from the pilot process, not from a messy setup.

Repo B should wait. First shorten setup, write down the test entry points, and break oversized files into smaller parts. Then run the same pilot there and compare the results.

The boring repo often teaches more in week one. It shows where the assistant fits, where it slows people down, and what needs cleanup before a wider rollout.

Mistakes that skew the score

Most bad scores come from hidden familiarity. A repo can look easy when the person checking it already knows every script, every folder, and every odd little fix that never made it into the docs.

If the repo owner runs every check, you do not learn much about a real pilot. You learn how fast an expert can move inside their own habits. Ask someone who did not build the project to clone it, start it, run tests, and find the first safe place to make a change.

Setup time is only part of the picture. A project might install in 12 minutes and still confuse people for another hour because nobody can tell which test command matters, which package is active, or which module is safe to touch first. That confusion slows both humans and assistants.

A few mistakes show up again and again:

• Picking the most tangled area because it feels serious or high impact
• Testing on a laptop that already has old SDKs, cached secrets, or global tools
• Counting one clean run as the baseline even though the second run fails
• Ignoring the time spent finding the right test entry points
• Letting tribal knowledge fill gaps that a new contributor would hit

The cleanest check is a boring one. Use a machine with no hidden dependencies. Write down every missing step. Run the same setup and test flow twice. If the first attempt passes and the next one breaks, your baseline is unstable.

Module choice can skew the score too. Teams often grab the biggest or most complex part of the repo because it looks like the place with the highest payoff. Usually that is the wrong starting point. Pick an area with clear inputs, small files, and tests that people can actually run.

Quick checks before you start

Before you score anything, give the repo to someone who has not touched it in a few weeks. Ask them to do a tiny, normal task while you watch where they stop, guess, or open chat to ask for help. Those pauses tell you more than a long team debate.

You do not need perfect numbers yet. You need a clear sense of whether an assistant will help with real work or just mask basic repo problems.

• Can a new person get the app running from scratch with the current docs and commands?
• Can they run one focused test, not the whole suite, in a minute or two?
• Can they change a single file for a small task without a chain reaction across half the codebase?
• Do the setup notes still match what the repo actually expects today?
• Have you picked one module for the first trial, or are you still talking about the whole product at once?

A small example makes this real. A teammate clones the repo, follows the README, and hits three missing steps before the app boots. Then they try to run one test and discover the project only supports the full integration suite. The repo is telling you something: fix the entry points first, or your pilot will spend its time fighting setup noise.

This is a common pattern in growing products. The assistant is often not the first problem. The repo is. If two or more answers above are "no," pause the pilot for a day, clean up that path, and test again on the same module.

What to do next

If your scorecard shows several red items, stop shopping for more tools. Fix the first two problems that block normal work, usually local setup time and unclear test entry points. An assistant will not save much if a developer still spends 40 minutes getting the app running or guessing which command is safe.

Keep the pilot small. Pick one repo area, one workflow, and one kind of change. A good start is a small bug in a medium-size module or a cleanup task with clear tests. Skip migrations, shared auth code, and billing logic until the team sees steady results.

Use the same two trial tasks for every assistant you test:

• A small bug fix with an obvious expected result
• A small cleanup task, such as removing duplication or splitting a file that grew too large

Then compare more than the code diff. Measure the work around the diff too. Note how long setup took, how often the assistant chose the wrong file, whether it found the right tests, and how much review time the change needed. Ten minutes saved while coding can disappear in twenty minutes of cleanup.

Write the notes down after each run. A simple log with time saved, time lost, and the reason for both is enough. After five to ten runs, the pattern gets hard to ignore. You will know whether the repo is ready, which workflow fits, and where humans still move faster.

If you want an outside review before rolling a pilot out more widely, Oleg Sotnikov at oleg.is works as a fractional CTO and startup advisor and can help assess setup pain, test paths, and architecture issues that will distort the results. A short review is often cheaper than running a month-long pilot in the wrong repo.