Which AI agent orchestration patterns fit your workload?

AI agent orchestration patterns are control systems, not personality charts for a group of bots. The useful choice is about who can assign work, who owns state, and who may decide that the job is finished. If those answers stay implicit, a polished demo turns into an expensive loop the first time a request crosses two specialties.

I choose the topology from the workload before I choose a framework. A predictable request with one clear destination needs a router. A job that must be decomposed, checked, and assembled needs a supervisor. A task whose next step depends on discoveries made during execution may justify a swarm. Many production systems use all three, but each added control loop must earn its latency and failure modes.

Choose by uncertainty, coupling, and risk

The right topology follows three workload properties: how well you can classify the task at the start, how tightly the subtasks depend on one another, and how much damage a wrong action can cause. Agent count is a poor proxy. Five specialists behind a deterministic router can be simpler than two agents that repeatedly negotiate ownership.

A router fits high classification confidence and low coupling. Think of an internal help desk that sends payroll questions to an HR workflow and access requests to an identity workflow. Once classified, each branch can finish without consulting the others. The router does not need to understand every branch result.

A supervisor fits moderate uncertainty or high coupling. A review of release readiness may need a code reviewer, a test analyst, and an operations reviewer, but one controller must reconcile conflicting findings and decide whether evidence is sufficient. Specialists contribute bounded results; they do not own the final decision.

A swarm fits high uncertainty when useful work changes what should happen next. Incident investigation is the respectable example. A log analyst can discover a database symptom and transfer control to a query specialist, which can discover a deployment correlation and involve a release agent. You cannot reliably list that sequence before the run.

Risk changes the answer. Dangerous actions should move control into code even when an LLM helps with diagnosis. An agent may propose a refund, database migration, or production rollback, but a policy function should validate limits and request human approval. OpenAI's Agents SDK documentation makes the same broad distinction between orchestration by an LLM and orchestration by code. I go further: use model judgment to interpret ambiguous evidence, and use code to enforce permissions, budgets, and irreversible transitions.

Do not confuse concurrency with collaboration. If four identical workers each summarize a separate document and a program concatenates the results, you have parallel execution, not a supervisor or swarm. That can still be the best design. Collaboration starts when one result changes another worker's task, an actor must resolve disagreement, or control moves because of evidence found during the run.

Also separate epistemic uncertainty from workflow uncertainty. Epistemic uncertainty means an agent does not know whether a claim is true; solve it with evidence, retrieval, tests, or review. Workflow uncertainty means the system does not yet know which action should run next; that is the problem topology addresses. Adding more agents to an evidence problem often creates several unsupported opinions instead of one.

Ask whether one agent with good tools can do the job first. Multiple agents add context boundaries, parallelism, and separate permissions. If you do not need one of those, a single agent with a deterministic workflow is usually cheaper to operate and easier to test.

A router should make one bounded decision

A router classifies an input and sends it to one specialist or one fixed workflow. It should not plan, synthesize, or keep calling branches until it likes an answer. Once it starts doing those things, you have built a supervisor with a misleading name.

Use a deterministic rule before an LLM whenever the signal already exists. An API path, customer tier, document type, language code, or explicit user selection is more reliable than asking a model to infer the same fact. Reserve model routing for semantic ambiguity, such as distinguishing a billing dispute from a cancellation request written in free text.

Make the routing output small and typed. This contract is enough for many systems:

{
  "route": "billing",
  "confidence": 0.91,
  "reason_code": "duplicate_charge",
  "needs_clarification": false
}

Do not accept an invented route. Validate route against an allowlist, set a confidence threshold from evaluation data, and send uncertain cases to clarification or human review. The model's prose explanation can help a reviewer, but it must not become an executable target.

The common failure is cascading classification. A support router picks billing, the billing agent decides it really belongs to retention, and retention sends it back. The request collects duplicated context and nobody owns the clock. Either let the first router make the only transfer, or record visited routes and cap transfers explicitly. I prefer one transfer because it produces a cleaner service boundary.

Routers can dispatch several branches when they are independent, but call that operation by its behavior. A document intake system might send the same file to extraction and policy screening in parallel, then join their typed outputs. The routing decision still happens once. Parallel dispatch does not turn it into a swarm.

Mixed requests need a declared policy. A message such as "cancel my plan and refund yesterday's duplicate charge" contains two intents that may have different permissions. The router can split it into typed tasks if the branches are independent, or send the whole request to a supervisor if cancellation changes refund eligibility. Do not let each branch act on a partial interpretation of the same account state.

Class imbalance also matters. A router trained or prompted on common requests may send rare security or legal cases into the nearest ordinary category with high confidence. Put sensitive routes behind explicit detectors and conservative fallback rules. The cost of an unnecessary human review is usually lower than the cost of letting an account support workflow interpret a data deletion demand.

Measure route accuracy per class, fallback rate, branch latency, and the cost of wrong routing. Overall accuracy can hide a rare route that fails constantly. In production, log the selected route, model or rule version, confidence, and final disposition. Without final disposition, you cannot tell whether a confident route was useful.

A supervisor owns the plan and the answer

A supervisor decomposes a goal, assigns bounded work, evaluates results, and decides what runs next. It remains responsible for the final answer. This pattern fits research synthesis, software changes across components, due diligence, and any job where specialist outputs conflict or depend on one another.

The OpenAI Agents SDK calls this manager orchestration when a central agent invokes specialists as tools and retains the conversation. That distinction matters. A specialist used as a tool returns to the manager; a handoff transfers control to another agent. Treating those operations as interchangeable causes missing conclusions and surprising permission changes.

Give every specialist a narrow input and a typed result. A research worker should receive a question, source policy, and budget, then return claims with evidence and uncertainty. Do not pass the entire supervisor transcript by default. That wastes tokens and lets irrelevant instructions leak across roles.

A useful supervisor loop has four explicit states:

1. Build or revise a task plan with dependencies.
2. Dispatch ready tasks, in parallel where they do not share mutable state.
3. Validate each result against its contract and acceptance test.
4. Stop, request approval, or schedule another bounded task.

The supervisor should not ask the same model to create a result and certify it with no independent evidence. A critic agent with identical context and incentives often produces confident agreement, not assurance. Prefer executable tests, schema checks, source requirements, or a different data path. Use a reviewer model for judgment that code cannot express, then record its rubric and verdict.

I have seen supervisors burn most of a run rewriting a plan after every minor observation. Freeze completed tasks, revise only affected dependencies, and limit replanning. If three workers return incompatible answers, the manager should identify the contested claim and commission a targeted check. Sending the whole assignment around again multiplies cost without improving the evidence.

The manager also needs a policy for partial success. Suppose a market analysis requires customer evidence, competitor evidence, and a pricing recommendation. If the competitor worker times out, the manager should not quietly write a complete recommendation. It should retry within budget, narrow the claim, or return a result that marks the missing evidence. A typed completeness field is more reliable than asking the final writer to remember every failed task.

Parallel work is safe only when outputs are independent or merges are deterministic. Two coding agents editing the same file do not create twice the throughput. They create a merge problem with probabilistic authors. Partition by component or let one agent implement while another reads tests and requirements.

A swarm trades predictability for adaptive handoffs

A swarm is a peer network in which the active agent can hand control to another agent based on the current state. There may be an entry agent and shared runtime, but no single model continuously owns the plan. This topology works when local specialists can recognize who should act next better than an upfront planner can.

Microsoft AutoGen's group chat documentation describes a related design where a manager selects the next speaker and participants publish into shared history. Its SelectorGroupChat uses a model for speaker selection and requires a termination condition. I would classify that implementation as a centrally selected swarm, not a pure peer swarm, because a manager still controls turns. The label matters less than identifying who selects the next actor and who can stop the run.

Use a swarm for exploration with branching clues, distributed operations where ownership follows the resource, or simulation where peer interaction is the subject. Do not use it for a known business process merely because the animation looks impressive. If the sequence is known, encode the sequence.

A handoff must carry a packet, not an unbounded transcript. The packet should contain the objective, relevant findings, artifacts, unresolved questions, permissions, remaining budget, and visited agents. The receiving agent should know why it was selected and what completion means.

Consider an incident run. The triage agent sees elevated checkout errors and asks the application specialist to inspect traces. That specialist finds connection timeouts confined to one release and hands the evidence to the deployment specialist. The deployment specialist proposes a rollback. The runtime blocks execution because rollback requires an operator approval token. After approval, a verification agent checks error rate and the run terminates. Each transfer follows new evidence; the approval boundary stays outside model discretion.

Now consider the failure version. Every agent sees the full chat, each can call every tool, and the stop condition is "until resolved." The application agent asks operations for help, operations asks application to confirm, and both repeat queries with slightly different wording. Token use grows, rate limits arrive, and the shared context fills with stale hypotheses. Limit repeated transfers, scope tools by agent, set an overall deadline, and define a terminal state that code can recognize.

Swarm membership should be small and legible. When every agent may transfer to every other agent, the number of possible paths grows quickly and role descriptions start doing the work of access control. Prefer an explicit adjacency list. If the database specialist never has a valid reason to call the customer communications agent, omit that edge.

Let a receiving agent reject a handoff for a structured reason such as missing_evidence, outside_scope, or budget_too_low. Rejection should return control to a known owner, not bounce the packet back into peer selection. This protects specialists from vague tasks and gives evaluation data about faulty transfer decisions.

Swarm traces need more than message history. Record from_agent, to_agent, reason_code, remaining budget, artifact references, and state version for every transfer. That turns a confusing conversation into a graph you can inspect.

Most useful systems are constrained hybrids

A hybrid uses deterministic routing for known boundaries, supervision for coupled work, and peer handoffs only inside the part that needs exploration. This is usually the production answer because workloads contain different kinds of uncertainty.

Take a startup engineering queue. Code ownership rules first route a request to product work, defect repair, security review, or operations. A supervisor handles a defect by asking one agent to reproduce it and another to inspect the relevant component. If the investigation crosses into an unfamiliar subsystem, the investigator may hand off once to that subsystem's specialist. Tests, approval policy, and deployment remain deterministic stages.

Write the topology as data so reviewers can see permissions and limits without reading prompts. This compact configuration does not depend on a framework:

entry: intake_router
agents:
  intake_router:
    can_call: [defect_supervisor, product_workflow, human_queue]
    max_handoffs: 1
  defect_supervisor:
    can_call: [reproducer, code_reader, test_runner]
    max_rounds: 4
  code_reader:
    can_handoff: [database_specialist]
    max_handoffs: 1
  database_specialist:
    tools: [read_schema, explain_query]
policies:
  write_production: human_approval
  total_model_calls: 18
  deadline_seconds: 240
terminal_states: [resolved, needs_human, budget_exhausted, failed]

This artifact prevents three failures. It keeps a specialist from discovering a production write tool it was never meant to use. It limits cycles even if prompts fail. It also gives the runtime a definition of done that does not rely on language. The exact numbers need evaluation; the presence of limits does not.

Avoid a universal "orchestrator agent" that receives every company request. It becomes a prompt full of exceptions, a large permission target, and a deployment bottleneck. Keep boundaries close to business domains. A finance workflow and an engineering workflow can share runtime components without sharing one omniscient supervisor.

Frameworks should implement your control graph, not define it. OpenAI Agents SDK agents as tools map naturally to a supervisor, and handoffs map to transfers. AutoGen group chats can implement collaboration in turns. Other graph runtimes can encode the same topology with nodes and edges. Choose on durability, tracing, state persistence, approval support, and operational fit after the control design is clear.

State contracts matter more than agent prompts

Orchestration fails at boundaries when agents disagree about state. Prompts cannot repair an undefined source of truth. Decide which data is immutable input, which artifacts only accept new entries, which fields one actor may update, and how concurrent updates are reconciled.

Separate conversation from working state. Conversation explains reasoning to people and models. Working state drives execution. A task record might contain status, owner, dependencies, artifact IDs, attempt count, deadline, and approval state. The supervisor should read those fields rather than infer completion from a sentence such as "looks good to me."

Use versioned schemas for handoffs and specialist results. When a worker returns invalid JSON, do not silently stuff the raw text into the next prompt. Record a contract failure, allow one repair attempt if the risk is low, then fail or escalate. Silent coercion hides the exact interface you need to improve.

Shared memory needs ownership rules. A common vector store can help agents find reference material, but retrieval is not truth and memory is not state. Store decisions in a durable record with provenance. Let agents propose memory additions; validate sensitive entries or those kept for a long time before other runs consume them.

Context should narrow as work moves outward. The router needs the fields required to classify. A specialist needs its task, relevant artifacts, and constraints. The supervisor needs result summaries plus evidence references. Passing every earlier thought to every later agent raises cost and creates instruction conflicts. It can also expose data to a role that did not need it.

Idempotency becomes important as soon as you add retries. Give every action that changes external state an operation ID and make the tool reject duplicate execution. If a payment agent times out after submitting a refund, the retry must check the existing operation instead of issuing a second refund. Model instructions that say "do not repeat" are not a transaction boundary.

Treat state migrations like API migrations. A new agent version may produce richer findings, but older active runs still carry the previous schema. Either support both versions at the boundary or drain old runs before deployment. Orchestration that spans hours or days turns prompt changes into changes across a distributed system.

Budgets, termination, and permissions belong in code

Every multi-agent run needs hard limits on model calls, tool calls, elapsed time, handoffs, and spend. A polite prompt request to be efficient is not a budget. The runtime must decrement counters and reject work after a limit.

Termination should be a set of outcomes that code can read. resolved, needs_human, budget_exhausted, and failed are more useful than waiting for an agent to say "TERMINATE." AutoGen's examples pair textual termination with a message limit, which is the right instinct: semantic completion needs a mechanical backstop.

Put permissions on tools and identities, not role descriptions. A prompt that says "you are an analyst who can only read" does not change a database credential. Give that agent a connection limited to reads, constrain network destinations, and validate tool arguments. When control transfers, do not transfer credentials automatically.

Approval should bind to a specific proposed action. The approver needs the command or transaction, target, expected effect, evidence, and expiry. If the plan changes after approval, ask again. A generic "continue" button can authorize more than the reviewer understood.

Failure policies should distinguish transient errors, contract errors, policy denials, and weak evidence. Retry a transient network error with backoff. Send invalid structured output through one bounded repair path. Do not retry a denied production write with different wording. If evidence remains weak, return needs_human with the unresolved claim.

Cost and latency grow differently across topologies. A router adds one classification call and one branch in the simple case. A supervisor adds planning, worker, validation, and synthesis calls, though independent workers can overlap in elapsed time. A swarm has the widest variance because the route length emerges during execution. Estimate the call graph, then set a budget from business value rather than accepting whatever the agents consume.

Security review should follow transitive capability. If a supervisor can call an agent that can call a shell tool, the supervisor effectively has a path to shell execution. Draw the capability graph, include handoffs and indirect tools, and remove paths that the workload does not require.

Evaluate decisions and completed work separately

An orchestration evaluation must score both control decisions and task outcomes. A system can produce a correct answer after wasteful routing, or route perfectly to a specialist that returns bad work. One aggregate success score cannot tell you what to repair.

Build a small evaluation set from real workload shapes, including ambiguous requests, requests with several intents, missing data, tool failures, conflicting evidence, and requests that should stop for approval. Synthetic cases help fill gaps, but keep actual redacted traces because users find boundary conditions that designers miss.

For routers, score the chosen route, clarification behavior, attempts to use forbidden routes, and downstream resolution. For supervisors, score task decomposition, dependency order, redundant work, evidence quality, merge correctness, and termination. For swarms, add transfer usefulness, cycles, repeated tool calls, path length, and whether the final owner was appropriate.

Trace one run as structured events:

00 route.selected      target=defect_supervisor confidence=0.94
01 task.dispatched     agent=reproducer task=T1 budget_calls=17
02 artifact.created    agent=reproducer artifact=A7 kind=test_case
03 task.completed      agent=reproducer task=T1 verdict=reproduced
04 approval.requested  action=deploy_patch target=staging
05 run.terminated      state=needs_human calls_used=9

This output shape lets you calculate latency and cost by phase, find repeated edges, and reproduce the state before a bad decision. A transcript alone makes those questions expensive.

Run topology comparisons on the same cases. Test a single agent, a router plus specialists, and the proposed multi-agent graph. The simpler version often wins on routine tasks. Keep the complex graph only where it improves completion, evidence, or safe parallelism enough to pay for its variance.

Do not evaluate only successful runs. Inject a worker timeout, invalid JSON, a denied tool call, a stale artifact, and a human who never approves. The run should land in a known terminal state without repeating side effects. Recovery behavior is part of product quality.

Move to multiple agents only after the boundary holds

Start production work with one complete path and explicit state, even if you expect several agents later. Instrument it, collect failures, and split a specialist out when you can name the boundary it needs: separate context, permission, parallel execution, or an independently testable skill.

The migration path is usually simple. First, turn a prompt with many tools into a deterministic workflow around one agent. Next, add a router if requests have distinct destinations. Add a supervisor when one destination contains coupled subtasks that benefit from separate contexts. Allow a peer handoff only when traces show that an upfront plan repeatedly chooses the wrong next specialist.

A Team & AI Audit applies the same discipline at company scale: map work, permissions, cost, and handoffs before buying more model calls or cutting roles. The useful output is a smaller operating design with measurable boundaries, not an org chart relabeled with agent names.

Assign one owner for the runtime even when no agent owns every run. Someone must maintain schemas, budgets, tool permissions, evaluations, and incident procedures. Without that ownership, each team adds a new agent and nobody controls the combined capability graph.

Use a short review before approving a topology:

• Can a deterministic rule make this decision?
• Which actor owns the final result and terminal state?
• What data crosses each boundary, and under which schema?
• What hard limit stops cycles and duplicate side effects?
• Which trace proves that this topology beats one agent?

If the team cannot answer those questions, the topology is not ready. Build the control plane first. Agent prompts will change every week; ownership, permissions, state, and evidence are what keep the system operable.