Buf codegen pipeline for Go, TypeScript, and CI teams

Why protobuf workflows get messy fast

Protobuf looks tidy at first. You write a few .proto files, run generation, and both Go and TypeScript get what they need.

The mess starts when every team makes one small choice on its own. Backend engineers put files under api/proto, frontend engineers expect a different import shape, and someone copies an older setup from another repo. Each choice looks harmless, but together they turn a simple schema into a moving target.

Folder layout is usually the first problem. One repo keeps shared messages next to services, another splits them by language, and a third hides them under internal. After that, imports stop feeling obvious. A developer changes one path, generated code moves, import paths change, and review diffs fill up with noise.

Go and TypeScript pull in different directions. Go cares about package names and module paths. TypeScript tools usually care more about where files land and how imports resolve in the app build. If the repo does not make those rules clear, people start adding local fixes. Those fixes pile up fast.

Version drift causes a different kind of pain. One developer runs generation with one plugin version, CI uses another, and the output changes even though the schema did not. You end up with pull requests full of generated churn. Nobody wants to review that, and nobody feels sure which output is right.

Local commands and CI also drift apart in quiet ways. Someone generates code with a shell script on a laptop while the pipeline checks something slightly different. The branch works locally, then CI fails on a path rule, a lint rule, or a generated file that does not match. That kind of failure wastes time because the code itself may be fine.

Small teams feel this almost immediately. One person edits a service, another updates the web client, and a third tries to cut a release before lunch. If each machine produces different generated files, trust disappears.

That is why a plain Buf codegen pipeline beats a clever one. When paths, plugin versions, and commands stay fixed, protobuf stops surprising people and goes back to being part of the build.

Pick one repo layout and stick to it

Most protobuf pain starts with folders, not syntax. If one developer keeps .proto files next to Go code, another hides them under services/, and generated files land wherever a script writes them, the build starts to feel random.

Keep every source .proto file under one root folder. Call it proto/ or api/, but pick one and keep it boring. A Buf pipeline is much easier to read when every schema starts there, whether you generate Go stubs, TypeScript clients, or run checks in CI.

proto/
  billing/v1/*.proto
  users/v1/*.proto
gen/go/
gen/ts/

That split matters. Source files describe the API. Generated files are build output. Do not mix them. When someone opens the repo, they should know in five seconds which files they edit and which files generation replaces.

Package names need the same discipline. Pick protobuf package names and Go package paths once, then keep them stable. A rename looks harmless, but it can break imports, create noisy diffs, and change the full service names people use with grpcurl.

The same rule applies to output folders. Give Go one home, such as gen/go/, and give TypeScript one home, such as gen/ts/. Avoid half-measures like one script writing to pkg/, another to src/generated/, and a third to a temp folder in CI. That setup confuses people fast.

Once you lock the layout, write it down in a short README note and stop debating it. When a new service shows up, you add one folder under proto/, run the same generation command, and move on.

Set up Buf in a few steps

A Buf codegen pipeline stays readable when the repo owns the rules. Do not rely on copied shell commands or local plugin installs. Put the config in versioned files, and make every machine run the same steps.

Start with buf.yaml. This file sets module boundaries, lint rules, and breaking change checks. Keep it small at first.

version: v2
modules:
  - path: proto
lint:
  use:
    - STANDARD
breaking:
  use:
    - FILE

That gives the team one home for .proto files and one shared lint policy. If naming, package paths, or file structure drift, Buf catches it early instead of letting bad patterns spread into generated code.

Next, add buf.gen.yaml. This file should answer three plain questions: which plugins run, where files go, and which plugin version each developer uses. Pinning versions matters more than most teams expect. One person on a newer plugin can rewrite dozens of files and leave everyone else sorting through noise.

version: v2
plugins:
  - remote: buf.build/protocolbuffers/go:v1.35.2
    out: gen/go
    opt: paths=source_relative
  - remote: buf.build/grpc/go:v1.5.1
    out: gen/go
    opt: paths=source_relative
  - remote: buf.build/bufbuild/es:v2.2.3
    out: gen/ts

Use separate output folders for Go and TypeScript. That keeps generated files easy to find, easy to clean, and hard to mix with hand written code.

Run buf lint before buf generate. Lint is fast, and it stops bad schema changes before they turn into noisy diffs. Generation should come second.

Most teams need one repo command, not five. make proto, task proto, or a package script is enough if it always runs the same sequence:

• buf lint
• buf generate

Use that same command in CI. Do not create a special CI version with extra flags and different output paths. If a developer runs one command locally and CI runs the same one later, trust goes up quickly.

Generate Go and TypeScript without surprises

The same .proto files should produce the same Go and TypeScript output on every laptop and in CI. If Go comes from one command and TypeScript from another, drift starts early.

Use one source tree for contracts and one Buf config for generation. Keep package names close to the service boundary, not the org chart. billing.v1 and auth.v1 age well. Names like common.platform.shared.billingservice do not. Deep trees look tidy for a week, then every import turns into a hunt.

Keep paths short too. A file like proto/billing/v1/invoice.proto is easy to scan. Generate code into stable folders that mirror the package, such as gen/go/billing/v1 and gen/ts/billing/v1. When people can guess the path, they stop fighting the tool.

One config should do most of the work. Buf can run the Go and TypeScript plugins in one pass, so you rarely need shell scripts that rename files, patch imports, or copy outputs around. A small wrapper like make proto is fine. A pile of bash that "fixes" generation usually means the layout is wrong.

For a team with two services, a split like this stays readable:

• user.v1 for sign-in, profiles, and account settings
• billing.v1 for invoices, payments, and webhooks

That gives both languages the same map. A Go handler, a TypeScript client, and a CI job all point back to the same package names.

Decide early what to do with generated files. Teams lose time when half the repo expects committed output and the other half expects local generation.

• Commit generated code if other repos consume it, or if frontend developers should pull the repo and run it without extra setup.
• Skip committed output if CI always regenerates it and every environment pins the same plugin versions.

Pick one rule and keep it. Fewer moving parts mean fewer strange failures late in the day. When the folders, package names, and generation rules line up, protobuf code generation becomes ordinary build work instead of a recurring cleanup job.

Use grpcurl to check the real service

Generated code can look clean while the running service says something else. grpcurl gives you a direct view of the live endpoint, which makes it a good reality check before you spend time blaming Go code, TypeScript types, or the generator.

If server reflection is on, start by listing what the service actually exposes:

grpcurl -plaintext localhost:50051 list

That quick check catches a lot of boring but expensive mistakes. You may find the old service name, the wrong package, or a container that never picked up the latest proto change. In many cases, this is the first sign that code generation is fine and deployment is not.

Then inspect one method before you write client code:

grpcurl -plaintext localhost:50051 describe acme.user.v1.UserService.GetUser

Read the request and response shape closely. If a field moved, an enum changed, or a message name is different from what your client expects, you will see it right away.

Next, send a very small JSON request:

grpcurl -plaintext -d '{"id":"123"}' localhost:50051 acme.user.v1.UserService/GetUser

Keep the payload tiny. You are not testing every rule yet. You are checking that the JSON shape maps to the proto shape you think you generated.

If reflection is off, use a Buf image so grpcurl and your generators read the same contract:

buf build -o buf.binpb
grpcurl -plaintext -protoset buf.binpb -d '{"id":"123"}' localhost:50051 acme.user.v1.UserService/GetUser

Now compare the live response with the generated types. In Go, check the struct fields, enum values, and any timestamp or oneof handling. In TypeScript, check the generated interfaces or message classes and the JSON conversion code. If grpcurl returns displayName but your generated code expects name, stop there and fix the schema or the deployed service.

Teams that do this early avoid a lot of noisy CI failures later. One live method check can save a long round of guessing.

A small team example

A backend engineer on a small ecommerce team needs to expose one more detail in Order: a tracking_url field so the app can show shipment status. She first writes trackingURL out of habit. Buf lint fails in her branch before anyone reviews it, so she fixes the name to tracking_url and moves on in a minute instead of arguing about style in a pull request.

The team keeps one generation command, such as buf generate. That command updates both the Go server types and the TypeScript client code. Nobody edits generated files by hand, and nobody has to remember a second step for the frontend.

The pull request stays easy to read because reviewers focus on the schema and business logic, not on manual edits in two languages. If the diff looks odd, the engineer reruns the same command locally and gets the same output as everyone else.

The Go code reads the new field from the service layer, and the web app gets the updated type in the same commit. When another engineer pulls the branch, the repo already matches the schema.

CI still checks the boring stuff because people miss boring stuff. In this case, an old generated file from a previous package path is still sitting in the branch. The build job regenerates code, compares the working tree, and fails because that stale file should not exist anymore. It is a blunt check, but blunt checks work.

Before merge, someone calls the staging service with grpcurl and sends a sample GetOrder request. The response includes the new field, and the value matches what the app expects. That quick test matters because generated code can look fine even when the service forgets to fill the field.

Nothing fancy happened here. One schema change, one generation command, one CI guard, and one real request against staging. That is the sort of routine a small team can trust week after week.

Mistakes that break trust in the pipeline

A protobuf workflow stops feeling safe the moment two people can run the same command and get different results. That usually starts with small habits, not big failures. Someone runs generation from a nested folder, someone else runs it from the repo root, and now import paths or output folders shift for no good reason.

The fix is simple. In a Buf pipeline, every command should run from one place, usually the repository root, with the same config checked into git. If a new teammate has to guess where to stand before they type buf generate, the setup already has a crack in it.

Plugin drift causes the next round of damage. One laptop has an older Go plugin, another has a newer TypeScript plugin, and CI uses something else again. The generated code still compiles most days, which makes this worse. People stop trusting diffs because they cannot tell whether a change came from the schema or from somebody's machine.

Hand edits to generated files create a different kind of mess. It feels fast in the moment. A developer tweaks one import, fixes one method name, or adds a comment, then the next generation wipes it out. After that, teammates start protecting generated files with custom scripts and strange exceptions. That is how a simple pipeline turns into a weekly argument.

Package renames are even harsher. A team changes a protobuf package after clients already depend on it, and breakage spreads fast. Go imports move. TypeScript clients regenerate different names. CI goes red. If you need a new shape, add it in a way old clients can survive.

The last mistake is skipping a real service check. Schema diffs can look clean while the running server still behaves in an unexpected way. A quick grpcurl call against a real endpoint catches problems generation cannot catch on its own, like missing reflection, wrong field mapping, or a method that compiles but returns the wrong data.

One rule works well: generate in one place, with pinned versions, never edit the output, treat package names as public contracts, and run one real grpcurl check before merge.

Quick checks before you merge

A protobuf change should feel routine. If a small field rename makes people guess what to regenerate or whether the service still works, the pipeline is too fragile.

Run the same checks every time:

• Start with buf lint. It catches naming drift, package mistakes, and schema problems before generated files make the diff noisy.
• Regenerate code, then inspect git status. Expected changes in generated Go and TypeScript files are normal. Surprise diffs in unrelated folders usually mean your config or plugin setup needs attention.
• Build the Go code with the fresh output. This is where broken imports, package moves, and handler code mismatches show up.
• Compile the TypeScript side too. Frontend errors often appear after enum changes, renamed fields, or a method signature update.
• Hit a real running service with grpcurl. One live request can expose a bad route, missing reflection, or a server that still runs old code.

A small example makes this obvious. Say you add a status field to a response message. Generation may succeed right away. Then the Go build fails because a mapper forgot to fill the new field. Or the TypeScript app compiles most files but breaks in one view that assumed a smaller enum. A quick grpcurl call confirms whether the deployed service actually returns the shape you expect.

Local work and CI should use the exact same commands. If a developer runs one script on a laptop but CI runs a different chain of hidden steps, trust drops fast. Put the checks behind one script, one Make target, or one task file, and let both local runs and CI call that same entry point.

What to do next

A good Buf workflow feels boring in the best way. One command should lint your proto files and generate code, and the same command should run on every laptop and in CI. If your team has to remember three scripts, two plugin versions, and one local fix, the setup is already drifting.

Write down the command you want everyone to use and put it in the repo docs and your CI job. For many teams, this is enough:

buf lint && buf generate

If you prefer a wrapper like make proto, keep the wrapper thin and let Buf do the real work. Then remove one older script that does the same job in a different way. The usual culprit is a leftover protoc shell script that only one person trusts. Keeping both feels harmless until Go and TypeScript outputs stop matching.

Add one grpcurl smoke test to your release flow and point it at the real service after deploy. Keep it small. A health check is fine, and one safe read method is fine too. That single call can catch a stale image, wrong routing, or a service that starts but does not answer with the contract your team expects.

A short cleanup plan works better than a big rewrite:

• document one public command for lint and generation
• delete one duplicate script
• run one grpcurl check after release
• review any path, plugin, or CI rule that behaves differently on laptops and in the pipeline

If the workflow still feels brittle after that, an outside review can help. This is the kind of cleanup Oleg Sotnikov does through oleg.is in his fractional CTO and startup advisory work: pin versions, remove duplicate scripts, and make local runs match CI so the team can stop fighting code generation.