Speed up Docker builds in CI with smarter layer order

Why slow CI builds hurt teams

A slow Docker build does more than annoy people. It breaks the rhythm of review, testing, and release. When every pull request waits on the same long image build, developers stop checking results right away and switch to other work. That context switch costs more than the build itself.

The problem gets worse in CI because the delay repeats on every branch, retry, and small fix. A build that takes 12 minutes instead of 4 may not sound dramatic once. Across a week, it turns into hours of runner time and a lot of dead waiting. For a team of six pushing several times a day, those extra minutes add up fast.

Slow feedback also changes behavior. People stop making small, safe commits and start batching changes together. Reviews get bigger. Failures get harder to untangle. A broken dependency or missed file copy can hide inside a much larger change set, which slows the team even more.

Fresh runners make this pain sharper. Many CI jobs start on clean machines with no local Docker layer cache, so the same system packages, tools, and app dependencies get rebuilt again and again. What feels acceptable on a laptop can feel expensive in a hosted pipeline.

The pattern is usually easy to spot. Deploys wait because one image rebuild blocks the rest of the pipeline. Engineers rerun jobs after tiny edits and still pay the full cost. Parallel branches burn cloud minutes with little to show for it. Testing moves later, when fixes are slower and riskier.

Build time is not just a technical metric. It changes how the team works, how often code ships, and how much each branch really costs.

Where your build time really goes

Most slow Docker builds come from a few repeated costs, not one bad command. The first delay is often the base image pull. If a fresh runner has to download a heavy image every time, you lose time before your own code even enters the picture.

The next cost is dependency installation. This is where many teams lose the most time. A tiny edit in one source file should not force npm install, pip install, or system package setup to run again, but poor layer order often does exactly that. If the Dockerfile copies the whole project too early, Docker treats a small change as a reason to rebuild expensive layers from scratch.

Build context size matters more than people expect. CI has to send files into the build before it can do useful work. If that context includes test reports, local caches, screenshots, or old artifacts, every job wastes time moving data the image never needed.

Asset compilation is another quiet time sink. Frontend bundles, generated code, and minified assets can take longer than the app compile itself. If those steps rerun after a README change or a small backend edit, the build is doing work with no payoff.

Then there is cache loss between jobs. This is common in hosted CI and also shows up in self-hosted runners that clean too aggressively. A team shipping a Next.js app through GitLab runners might see one fast build, then a much slower one an hour later because the runner forgot every cached layer.

Before changing anything, find which cost repeats most often. That usually shows where the easiest time savings are.

Put layers in a better order

Docker caches each build step in sequence. One noisy step near the top of the Dockerfile can force every step after it to run again. The fix is simple: put slow work that rarely changes near the top, and move fast-changing files closer to the end.

A sensible order starts with the base image and OS packages you rarely touch. Then copy dependency files such as package-lock.json, poetry.lock, or go.sum, and install dependencies before you copy the full project. App source and generated assets should come later, when cache misses cost less.

Teams often do the opposite. They copy the whole repository first and install packages after that. It works, but a tiny code edit can make Docker reinstall everything. In CI, that can add minutes for no good reason.

The same rule applies to system setup. If your image needs build tools, SSL libraries, or image processing packages, install them in an early layer. Those packages do not usually change with each commit, so Docker can reuse that layer across many builds.

App code changes all the time. Version files, templates, generated code, and frontend assets change even more. Keep them near the end so a cache miss affects seconds, not the whole dependency stack.

Multi-stage builds help when build-time tools and runtime files move at different speeds. Put compilers, package managers, and test tooling in the build stage. Then copy only the final binary or app bundle into the runtime stage. That keeps the runtime image smaller and avoids rebuilding extra layers when only the shipped artifact changes.

A good rule is easy to remember: if a file changes on almost every commit, it should not sit above a step that takes two minutes.

Send less data into the build

A slow build often starts before Docker runs a single command. CI first packs the build context and sends it to the builder. If that context includes test reports, local caches, screenshots, coverage files, or old logs, every build pays for that extra weight.

The fix is usually a stricter .dockerignore. Many teams ignore node_modules and .git, then forget about dist, build, .next, coverage output, temp folders, editor files, and local archives. Those files do nothing for the image, but they still cost time on every push.

Generated files need extra care. If your app creates compiled assets, docs, or codegen output during development, keep them out of the build context unless the image truly needs them. CI can generate them in a separate step and pass only the final result into the image build. That is often faster than shipping a huge folder into Docker again and again.

A common mistake is COPY . . from the repo root when the image only needs one service. In a monorepo, copy the service folder, its lockfile, and only the shared files that matter. If one frontend changed, the backend image should not drag the whole repository into the build.

Two checks catch this quickly. Look at the reported build context size in CI logs, then compare it before and after .dockerignore changes. Also check whether generated assets slipped into the context and whether large files should move to a separate artifact step.

If the context is hundreds of megabytes, start there before tuning caching or base images. Teams trying to get faster feedback often save more time by cutting context size than by tweaking Dockerfile commands.

Use remote cache carefully

Remote cache helps only when it saves more time than it costs to pull and push. In CI, that trade-off matters. A cache that takes 90 seconds to download before every build can erase most of the gain.

A practical setup restores cache from a shared location at the start of the job, then writes updated cache back only after a successful build. Failed builds often produce noisy layers you do not want other jobs to reuse. That one rule keeps the cache cleaner and more predictable.

Be selective about what you share across branches. Stable layers such as OS packages and language dependencies are good candidates for shared cache. Fast-changing app code is different. If every branch pushes its own app layers into the same cache, cache quality drops and builds spend more time checking entries they will never use.

It also helps to split cache by purpose. Keep dependency layers separate from app layers. Dependencies change less often, so they stay useful longer. App layers change all the time, so they need a shorter life and a narrower scope.

Do not judge remote cache by total build time alone. Measure where it actually helps: cache pull time at job start, time saved during dependency install steps, hit rate for stable layers, upload time after success, and rebuild time when dependencies change.

A small team can test this in a day. If one branch updates only UI text, the dependency layer should come from cache and the build should skip the expensive package install step. If that does not happen, the cache is too broad, too noisy, or saved at the wrong time.

Pick a base image that fits the job

A base image sets the pace for every build. If you pull a huge image for a small app, CI pays for it every time. Start with the smallest image that still has the tools your build needs.

Size is only part of the decision. Prefer images your runners pull often. If your CI already uses node:20-slim or golang:1.22, those layers may already be on the runner or in a nearby registry cache. In practice, a familiar image can beat a smaller one that CI still has to fetch from scratch.

Builder and runtime images

Two stages usually work better than one. Use a larger builder image with compilers, package managers, and other build tools. Then copy only the finished app into a slim runtime image. That keeps the final image small and cuts pull time in later jobs.

This matters most for teams that rebuild often. A small Go service, for example, may compile in a full Go image but run from a much smaller runtime image. The build stays simple, while deploys and test runs move faster.

Keep your base image stable. If you change it every few days, you reset cache and make build times jump around. Pick a version, pin it, and update it on a schedule. Do that during normal maintenance, not when the team is trying to ship a fix.

Treat base images like part of your build budget. Standardize on a short list, keep them current, and avoid random swaps.

Clean up in the right order

When you tune Docker builds, change one thing at a time and keep notes. If you tweak three settings at once, you end up guessing which one helped.

Start with a baseline. Run one build with no cache at all, then run the same commit again with cache available. That gives you two useful numbers: worst case and normal case. Save the logs too, because timing alone can hide where the slowdown actually sits.

Then fix layer order before anything else. Put the steps that change rarely near the top, and move fast-changing files lower in the Dockerfile. A common pattern is copying dependency files first, installing packages, and only then copying the rest of the app. Run the build again and compare it with the baseline.

Next, trim the build context. Large contexts waste time before Docker even starts useful work. Check what your build sends into CI, tighten .dockerignore, and rerun. You should see less data transferred and fewer cache misses caused by files that never needed to be there.

After that, add remote cache and test it on a fresh runner. This matters because local cache can make results look better than they really are. If a new runner can pull cache quickly and skip expensive layers, the setup will hold up across the whole team.

Leave the base image for later. Many people change it first because it feels like a big win, but it often is not. Once layer order, context size, and remote cache are under control, try a smaller or better-matched base image and measure again.

A simple order works well:

1. Measure a clean build and a cached build.
2. Reorder layers and test again.
3. Cut the build context.
4. Add remote cache on a fresh runner.
5. Change the base image last, and keep the change only if the numbers improve.

A simple example from a small team

A three-person team had a Node.js service that built on every commit. The app was not huge, but the CI job still felt slow. Their runners started fresh each time, so they lost local Docker cache between builds. On many commits, the pipeline spent more time rebuilding old layers than checking new code.

Most of the delay came from one Dockerfile choice. They copied the entire source tree before npm install. That sounds harmless, but it meant almost any code change invalidated the dependency layer. A small edit in one API file forced CI to download and install the same packages again.

They fixed it with a few small changes. They copied package.json and package-lock.json first, ran npm ci before copying the rest of the source, used a remote cache so fresh runners could pull earlier layers, and switched to a smaller runtime image for deploys.

After that, repeat builds got much shorter. A commit that changed only application code no longer triggered a full package install. Repeat build time dropped from roughly 7 minutes to a little over 2 minutes. That cut runner costs and made reviews less irritating because developers got results sooner.

The smaller runtime image helped in a different place. Deploy jobs finished faster because servers had less image data to pull. That did not matter as much as the build cache, but teams notice it when they deploy several times a day.

This kind of cleanup usually beats exotic tuning. Better layer order, a sensible remote cache, and a base image that fits the job can remove a lot of waste without changing the app itself.

Mistakes that keep builds slow

Many slow pipelines come from a few habits that look harmless in code review but cost time on every run.

One common mistake is changing version files on every build and copying them too early in the Dockerfile. A timestamp, build number, or generated commit file can force Docker caching to start over. If that file lands before dependency install steps, the job downloads and rebuilds the same packages again.

Teams also lose time when they keep build tools and runtime files in one stage. A single-stage image puts small app changes next to compilers, package managers, and temporary build output. Rebuilds get heavier, and the image often ends up larger than it needs to be.

Cache cleanup causes another self-inflicted delay. Some CI jobs wipe local and remote cache on every run because it feels safer. Usually it just turns normal builds into cold starts. Keep cache invalidation narrow and deliberate, or you pay the full install and compile cost every time.

Monorepos often add one more problem: a tiny change in one service triggers rebuilds for all services. That is hard to justify when a docs update or one small API edit wakes up half the pipeline. Build only what changed and what depends on it.

The last trap is chasing the smallest possible image even when it slows the path to a working build. Saving 40 MB in the final image is not a win if it adds minutes of package setup, makes debugging harder, or slows native builds.

Protect cacheable steps, split build and runtime cleanly, and avoid rebuilds no user will ever notice.

Quick checks before you keep tuning

Many slow builds come from one small mistake, not from Docker itself. Before you spend hours tuning, run a few repeatable checks.

If changing one app file causes Docker to run apt-get, npm install, or pip install again, your layer order is wrong. Put dependency files first, install once, then copy the rest of the code.

Check the build context size. If node_modules, dist, test output, or local caches go into the context, CI uploads junk on every run. A clean .dockerignore can save more time than a complicated cache setup.

Use a fresh runner and confirm that it restores cache before the build starts. Remote cache helps only if the runner can pull cache metadata early enough to reuse old layers.

Time the base image pull on its own. Some base images look convenient, but they add 30 to 60 seconds before the real work even starts. For a small service, a smaller base can beat a full image with extra tools.

Finally, record two separate numbers: a clean build and a repeat build. If you mix them together, you can fool yourself into thinking a change fixed the whole pipeline when it only helped warm builds.

It is not glamorous work, but it pays off. Once you know whether the delay comes from package installs, build context, cache restore, or image pull time, the next fix is usually obvious.

Next steps for a cheaper, faster CI loop

Do the first pass in one active repository this week. Change only three things: put stable layers earlier in the Dockerfile, send less into the build context, and turn on remote cache. That is enough to show where the real savings are.

Do not batch ten changes together. When everything changes at once, nobody knows which edit saved time and which one made the pipeline harder to trust.

Write the results in a small table and keep it with the repo notes:

This helps more than people expect. Teams often remember the worst build from last week, not the average build from the last 20 runs. A plain table makes the decision easier.

Keep the changes that save real time and stay boring under daily use. If a cache setup fails often, needs manual cleanup, or gives random misses, scale it back. A slightly slower pipeline is better than a flaky one that blocks merges.

After that, review the base image. If the build is still slow, the problem may be the starting point, not the Dockerfile details.

If your team wants an outside review, Oleg Sotnikov at oleg.is works with startups and small businesses as a Fractional CTO and can help audit Dockerfiles, CI cache flow, and infrastructure choices. Sometimes a short review is faster than spending another month guessing why builds still take too long.