Python packaging for internal services without chaos

Why internal Python repos get messy

Most internal Python code does not start as a product. It starts as a quick script to pull a report, fix a data issue, or call one API. That is fine for a day or two. The mess starts when the team keeps copying that script into new folders and changing just enough to make it work again.

After a few weeks, one file has turned into a service, but nobody stopped to decide what it is. Is it a package other code should import? Is it a runnable app? Is it just an ops script? When that choice stays fuzzy, people mix everything together and hope Python will sort it out.

Then imports start failing in weird ways. One person runs code from the repo root and it works. Another runs the same file from its own folder and gets a relative import error. A third person adds sys.path hacks to make the problem disappear for now. That fix usually makes the next problem harder.

A common example looks like this: a team has scripts/cleanup.py, then copies it into service_a/, then adds shared helpers in utils.py, then another service imports those helpers directly. Now a one-off script and production code depend on the same loose files, with no clear boundary. Small changes in one place break something far away.

That is why Python packaging for internal services goes wrong so often. The codebase grows by imitation, not by plan. Teams keep runnable code, reusable code, and throwaway experiments in the same tree. Once that happens, every new folder feels harmless, but each one adds another place where imports, tests, and deploys can drift apart.

Decide what is a service, package, or script

Most repo trouble starts before tools enter the picture. Teams call everything an "app", then imports leak across folders and a throwaway file slowly turns into production code.

Use plain rules. A service runs on a schedule or handles requests, has its own config, and someone on the team must keep it running. A package does not run by itself. Other code imports it. A script does one task, often by hand or for a short period, and nobody should mistake it for something you deploy and monitor.

That split should show up in the repo, not just in team chat.

• services/ for deployable units
• packages/ for reusable code
• scripts/ for manual or temporary jobs
• tests/ near the code they cover, or inside each unit
• one README per unit with its owner and purpose

Shared logic belongs in libraries, not inside service folders. If two services need the same parser, billing rule, or auth helper, move that code into a package as soon as the pattern is clear. Waiting too long is how Python packaging for internal services turns into copy-paste work.

A small example makes this easier. Say one team has an API that accepts orders, a nightly sync job, and a data fix tool an engineer runs once a month. The API and the nightly sync are services. The order validation code should live in a package if both use it. The data fix tool stays a script until it gets a regular schedule, alerts, and operational ownership.

Every package also needs a named owner. Not a committee. One person or one small team. Write down what the package is for, what it should never contain, and who approves changes to its import surface. That alone prevents a lot of mess.

Pick one repo layout and keep it boring

A boring layout saves time. Most import bugs start when one repo mixes reusable code, throwaway scripts, and real services in the same few folders.

Use a src layout. It makes imports behave the way they will behave after install, not the way they happen to work because someone ran code from the repo root. That catches bad imports early, which is the whole point.

repo/
  pyproject.toml
  src/
    invoice_worker/
      __init__.py
      main.py
    customer_sync/
      __init__.py
      main.py
    common/
      __init__.py
      auth.py
      db.py
      settings.py
  tests/
    invoice_worker/
      test_main.py
    customer_sync/
      test_main.py
    common/
      test_auth.py

Each deployable service gets its own directory and its own entry point, usually main.py or cli.py. Reusable code does not live inside a service folder just because one team wrote it first. Move shared code into one common package and keep it there.

That rule sounds strict, but it keeps small repos from turning into a pile of half-shared modules. If invoice_worker imports from customer_sync, you already have a smell. Both should import from common if the code is truly shared.

Tests should mirror the package structure. When a developer opens tests/customer_sync, they should find tests for src/customer_sync. No guessing, no hunting across random folders.

Names matter more than people think. Name folders after business tasks, like invoice_worker, customer_sync, or fraud_checks. Do not name them after machines, containers, or old deployment targets. server1_api gets silly fast, and six months later nobody knows what it does.

Oleg often pushes teams toward this kind of plain layout for internal services for one reason: it stays readable when the team grows. You can hand the repo to a new engineer, and they can tell what runs, what gets imported, and what should never ship as an app.

Choose tools with the fewest moving parts

Most teams do not get into trouble because Python packaging is hard. They get into trouble because every repo grows its own habits. One service uses setuptools, another uses Poetry, a third has a hand-written shell script, and nobody knows which command is the real one.

Start with pyproject.toml and treat it as the single place for package metadata, dependencies, and build settings. When people can open one file and see how the project works, they make fewer guesses and fewer local fixes.

One file, one tool

Pick one packaging tool for the whole repo and stick to it. For a fresh repo, Hatch or PDM keeps things clean and easy to explain. If your team already knows setuptools and the current setup works, keep it. Stability beats fashion.

Trouble starts when teams mix tools without a clear reason. A repo with setuptools config, stray Poetry lock files, and custom build scripts sends the wrong message: maybe all of them matter. Then people install packages three different ways, and import paths drift.

A simple rule helps: one repo, one build tool, one install command. If you need to change tools later, do it as a planned cleanup, not bit by bit.

Keep the workflow the same in dev and CI

Use one virtual environment workflow everywhere. If developers create an environment one way but CI builds another way, bugs slip through fast. The package installs locally, but the deploy job fails because it resolved dependencies differently.

Keep the routine boring. Create the environment the same way, install the package the same way, and run tests the same way in both places. Oleg often pushes teams toward this kind of plain setup for internal services because it cuts the small errors that waste hours. People should spend time on the service itself, not on guessing which packaging command applies today.

Keep runnable code separate from reusable code

A lot of import chaos starts when a useful script grows into a half-app. Someone adds startup logic, config loading, and a few helper functions in one file. A week later, another service imports that file because "it already works".

That is the moment to draw a hard line. Runnable code should stay thin. Reusable code should live in a normal package that any app, worker, or script can import without side effects.

A simple split works well:

src/
  billing_service/
    app.py
    api.py
    domain/
      invoices.py
      pricing.py
scripts/
  backfill_invoices.py

app.py starts the program. It reads settings, sets up logging, builds dependencies, and starts the CLI, worker, or web server. It should not hold billing rules. Put those rules in importable modules such as domain/invoices.py.

This keeps the direction clear. Apps call package code. Scripts call package code. Scripts should never become providers for services.

If a backfill job needs invoice logic, it should import a public function from the package, not reach into scripts/backfill_invoices.py or copy code from app.py. That one rule prevents a lot of hidden coupling.

PYTHONPATH hacks usually hide a layout problem. If people must tweak their shell to make imports work, the repo is telling you the package boundary is wrong. Install the package properly in development and run code from the package, not from random folders.

Expose a few public functions that other code can trust. Keep helper internals private. For example, create_invoice() is a better import target than _build_invoice_rows() buried three files deep.

Python packaging for internal services gets much calmer when every file has one job: start the program, or provide logic others can import safely.

Set versioning rules people can follow

Versioning only works when people can decide the number in under a minute. If the rule needs a debate every time, nobody will use it well. For Python packaging for internal services, keep the rule plain and strict.

Use patch versions for fixes that do not change how other code calls the package. If you fix a bug in a parser, tighten logging, or speed up a function, that is a patch change. Other services should upgrade without touching their imports or call sites.

Use a minor version when you add something safe. A new helper function, an extra optional setting, or a new module that does not break old code fits here. Old imports keep working, and existing function signatures stay the same.

Use a major version when other code must change. If you rename a module, move imports, remove a function, or change parameters in a way that breaks callers, bump major. Teams usually underuse major versions, and that is where import chaos starts.

This rule still matters inside one repo. A shared internal library is still a package with users. If billing_core changes from from billing_core.client import Client to from billing_core.api import Client, give it a new major version even if the code lives next to the service that uses it.

Write the rule down once, in a short team note, and keep it visible in the repo. It can be that simple: patch for fixes, minor for safe additions, major for breaking changes, every internal library gets its own version. If someone cannot tell which bump fits, they should ask before merge, not after release.

That sounds boring. Good. Boring versioning saves a lot of time.

Set up a new internal service step by step

Start before you write code. Pick one clear package name, write one sentence for its job, and name the team or person who owns it. If nobody owns the service, small messes turn into permanent messes.

For Python packaging for internal services, a boring starter shape beats a clever one every time.

1. Create pyproject.toml first. Add the project name, Python version, dependencies, and the build backend your team already uses.
2. Put importable code under src/your_package/. Keep a small __init__.py and add a main.py only if the service needs a command entry.
3. Add tests/ right away. Even two or three basic tests help catch bad imports and broken packaging before the service grows.
4. If the service needs shared code, install that shared package as a dependency. Do not copy a common folder into the repo. Copies drift fast, and nobody knows which one is correct after a month.
5. Wire CI before the first merge. Run lint checks, tests, and one clean install that proves imports work from the built package, not just from the repo root.

That last check matters more than people think. A service can look fine on a developer laptop while hidden relative imports keep it alive by accident. CI should build the package, install it in a fresh environment, and run the tests there.

Tag the first release before another team starts using it. 0.1.0 is enough. That tag gives people something stable to pin, and it tells everyone when a breaking change starts. If another team depends on an untagged branch, you are already back in import chaos.

A simple team example

One team has two Python programs. A billing worker runs on a schedule and applies invoice rules before charging customers. A report command runs once a month and builds finance reports from the same invoice data.

Trouble starts when both programs keep their own copy of the rules. Someone fixes tax rounding in the worker, but the report command still uses the old code. Two weeks later, finance sees different totals for the same invoice.

The clean fix is small. Put the invoice logic in one shared package, such as invoice_rules. That package should hold plain Python code: calculations, validation, date rules, and tests. It should not start workers, read CLI flags, or connect to queues by itself.

Then keep the runnable parts separate. The billing worker stays its own app, with its scheduler, retry logic, and logging. The report generator stays its own app too, even if it is just a command that someone runs from CI or a laptop. It is still an app because people execute it directly.

A repo like this stays easy to read:

• packages/invoice_rules for shared business logic
• services/billing_worker for the scheduled worker
• apps/report_generator for the command that creates reports

Now the team finds a bug in how credit notes reduce tax. They fix it once in invoice_rules, release version 0.4.2, and publish that package inside the company. After that, the worker updates its dependency to 0.4.2. The report generator does the same when the team is ready.

No one copies files between folders. No one imports code from a sibling app with path hacks. One fix reaches both tools through a normal package update, which is the whole point of Python packaging for internal services.

Mistakes that create import chaos

Import chaos usually starts with convenience. Someone runs a module from whatever folder they are in, adds a quick sys.path tweak, and moves on. It works on one laptop, then breaks in CI because imports depend on the current working directory instead of the package layout.

That same shortcut shows up when one service imports code from a sibling service. A billing app reaches into a notifications repo for one helper, then another team copies the pattern. Soon neither service can stand on its own. If code is truly shared, move it into a small internal package with its own name and version. If it belongs to one service, keep it there.

The utils.py habit is another quiet mess-maker. Every repo gets one. Then it gets split into utils.py, helpers.py, and common.py, which tells nobody what any of it does. A package with a plain name like date_parsing or logging_helpers is easier to import, test, and replace.

Renaming packages without a transition plan causes long, annoying breakage. Old imports sit in cron jobs, admin scripts, and forgotten automation. People only notice after a release fails. Keep the old import path alive for a while, warn people, and remove it on a set date.

A common trap in Python packaging for internal services is treating a script like a deployable app just because it already runs in production. A script that sends one report can stay a script. Once it has config files, retries, environment handling, logging, and a deployment job, it is not "just a script" anymore. Give it a real package structure before more code piles on.

One simple test helps: if a piece of code needs to be imported by other projects, version it and package it. If it only runs one task in one place, keep it isolated. Most chaos starts when teams blur that line.

Quick checks before merge and release

A merge should answer a few plain questions before anyone cuts a release. If the team cannot pass them in ten minutes, the repo still has hidden coupling. That is how Python packaging for internal services turns into late-night import fixes.

• Start on a clean laptop or in a clean container. After one install step, tests should run. If someone must remember extra commands or copy local files, the repo is not ready.
• Start each service from its own entry point. An API, worker, or scheduled job should have a clear way to run without relying on "open this file from the repo root."
• Run the code without touching PYTHONPATH. If imports fail until someone edits shell settings, the package layout is masking a structure problem.
• Declare every dependency and its version where your team expects it. Do not rely on a developer machine that already has the right package installed.
• Match the version bump to the change. Bug fixes get a patch bump, new behavior gets a minor bump, and breaking changes get a major bump.

These checks sound small, but they catch the mess that usually slips through. A repo can look tidy and still depend on local state, editor magic, or one person's shell profile.

Picture a team with two internal services and a shared package. One developer adds a helper, forgets to bump the package version, and tests still pass on their machine. Another developer pulls the branch on a new laptop, installs everything, and gets import errors because the service only works with a local path hack. That is not bad luck. It is a release process that skipped the last five minutes of discipline.

If a change fails any of these checks, do not merge it and promise to clean it up later. Fix the package boundary, entry point, or version now. It is faster than debugging a deployment that broke for reasons nobody can reproduce.

What to do next

Most teams do not need a big packaging reset. They need one written rule, one starter repo, and a few checks that stop bad habits before they spread.

Write your layout rule on a single page and keep it plain. Define where service code lives, where reusable package code lives, how runnable entry points work, and what is never allowed. That usually means no ad hoc PYTHONPATH edits, no imports from sibling folders, and no shared code hiding inside a service repo. Then apply the same rule to every repo. Boring is good.

This week, pick one shared module that two projects already use in a messy way. Move it into a real package with its own name, tests, and version. Publish it with the method your team already has, even if it feels basic. One clean package will change behavior faster than a long doc.

Before you split more services, add CI checks that catch drift early. Keep them small and strict:

• install the project the same way production does
• run tests against the installed package, not the raw source tree
• fail builds when scripts import from parent or sibling paths
• require a version bump when package code changes

If your team keeps slipping back into import hacks, copy-paste modules, or unclear version rules, an outside reset can help. Oleg Sotnikov works with startups and smaller companies as a Fractional CTO, helping them set package rules, versioning, and AI-assisted engineering workflows that people can follow without constant policing.

Start with one repo, one package, and one CI gate. If that sticks for two weeks, copy the pattern everywhere else.