# A multi-agent architecture audit for failing pipelines > A multi-agent architecture audit separates tool limits from broken ownership, context drift, retry storms, and an overloaded human review queue. A failing agent pipeline rarely needs another control panel. When agents overwrite one another, reason from different facts, repeat expensive work, or bury reviewers, the pipeline has lost control of ownership and state. A new orchestration tool can make those failures easier to watch without removing their cause. I audit these systems by tracing one unit of work from request to accepted result. I want to know who may change each artifact, which context produced each decision, what a retry is allowed to repeat, and who can reject the final output. If the team cannot answer those questions from stored evidence, it has an architecture problem. Tool selection comes later. ## Classify the failure before changing the stack The first job is to classify the failure at the boundary where it becomes visible. Teams often call every bad run an orchestration failure, then shop for a scheduler. That label hides four different faults that demand different repairs. A merge conflict is an ownership fault when two agents have legitimate permission to change the same semantic unit. Context inconsistency is a provenance fault when an agent cannot prove which instructions, source snapshot, model settings, and predecessor outputs it used. A retry loop is a state fault when the system cannot distinguish an attempt from the work itself. A growing review queue is a capacity fault when machines create decisions faster than people can safely accept them. Start with a sample of failed and successful runs, not a diagram of the intended design. For each run, record: - the work item and its acceptance condition; - every agent attempt, including start time, input reference, and terminal state; - every artifact version and the agent that wrote it; - every retry reason and side effect; - the time spent waiting for human review. This evidence separates a rare implementation bug from a repeating architectural pattern. If failures cluster around one adapter, fix the adapter. If the same ambiguity appears across repositories, models, and tasks, audit the architecture. Buying software before making that distinction moves the uncertainty into a larger box. Keep the observation window long enough to include handoffs and recovery. A run that looks successful at generation time may fail when another agent consumes its artifact, when a reviewer asks for evidence, or when a delayed retry revives stale work. Link those later events to the original work item. Otherwise the dashboard records several local successes while the business request remains unfinished. Do not begin with model quality scores. A low score can describe a weak result, but it does not reveal whether the agent received the wrong source, violated an ownership boundary, or produced a valid answer that waited two days for review. Architecture decisions need causal evidence. Quality evaluation becomes useful after the pipeline can identify the attempt and inputs being evaluated. A useful incident statement names the violated invariant. "Agent B produced bad code" tells me almost nothing. "Two agents wrote different versions of the same migration after reading different schema snapshots" identifies ownership, context, and sequencing faults. The stronger statement also tells the team which evidence the pipeline must retain on the next run. ## Merge conflicts start before Git reports them Git reports a text collision, but the damaging conflict often starts earlier. Two agents may edit different lines and still make incompatible decisions about an API, a database field, or an error contract. A clean automatic merge can therefore be more dangerous than a visible conflict. I separate file ownership from decision ownership. File ownership answers who may write a path. Decision ownership answers who may define a contract that other work depends on. A pipeline needs both. If one agent changes a schema while another changes the consumer, either one agent owns the whole contract change or the schema change becomes an approved input before consumer work begins. Consider a billing change split between an API agent and a database agent. The API agent decides that `status` may be `pending`, `paid`, or `void`; the database agent uses `cancelled` instead of `void`. They touch separate files, pass their local tests, and merge cleanly. The failure appears later when a production event contains a value the consumer never handles. No merge algorithm could infer which vocabulary represented the intended business decision. The repair is to promote the status vocabulary into a versioned contract with one decision owner. Dependent agents receive that accepted contract rather than inventing parallel interpretations. Contract tests then verify both artifacts against the same version. This may reduce parallel execution for a few minutes, but it removes a class of rework that can consume hours of investigation and another full review cycle. The popular recommendation is to split work into smaller tasks and let more agents run in parallel. It is popular because small prompts look controllable and parallel graphs look fast. It is wrong when the tasks share a decision boundary. Ten narrow agents that can reinterpret the same contract create more coordination work than one agent with a larger, explicit responsibility. Define a write set before execution. It does not need perfect path prediction, but it must name semantic ownership. A change manifest can look like this: ```json { "work_id": "billing-142", "owner": "agent-api", "decisions": ["invoice-status-contract"], "write_scope": ["api/invoices", "schemas/invoice.json"], "depends_on": ["tax-rules@7f31c2"], "acceptance": ["contract-tests", "migration-check"] } ``` The orchestrator should reject overlapping decision ownership before it launches work. It can allow overlapping paths only when a deterministic merge policy exists and the policy can detect semantic disagreement. Ordering two writes is not conflict resolution. The later writer merely wins. When conflicts still occur, measure where they entered the graph. Did planning assign overlapping decisions? Did an agent exceed its declared write scope? Did a stale dependency change after the work began? Each answer has a different owner. A merge queue helps with orderly integration, but it cannot repair an ambiguous decomposition policy. ## Inconsistent context is a provenance failure Agents produce inconsistent results when the pipeline treats context as a bag of text instead of a versioned input. Repeating the same prompt does not repeat the same task if retrieved documents, system instructions, tool results, repository state, or predecessor artifacts changed. Every attempt should reference an immutable context manifest. Store content hashes or version identifiers for the parts, not only the final assembled prompt. The manifest should cover the system instruction, task instruction, source snapshot, retrieval results, tool definitions, model and settings, predecessor outputs, and policy version. Secrets can stay in a protected store; the manifest needs a stable reference and a record of which version the attempt accessed. This distinction matters: prompt observability shows what text reached the model, while context provenance shows where that text came from and whether another run can reconstruct it. Teams routinely blur the two. A long prompt captured in a log may still be impossible to explain if the system cannot connect a quoted requirement to its source version. Use one context identifier throughout the attempt: ```json { "context_id": "ctx_01J9M6", "work_id": "billing-142", "source_commit": "7f31c2", "instruction_rev": "review-v12", "retrieval_snapshot": "kb-2026-07-27T09:15Z", "toolset_rev": "tools-v8", "predecessors": ["plan:sha256:8c2a..."], "model_profile": "code-large-temp0" } ``` Pass the identifier to logs, artifact metadata, review screens, and retry records. Then an inconsistent answer becomes a comparison between two manifests. Without it, the team debates model behavior from memory. The manifest also needs assembly order and exclusion reasons when precedence matters. If a policy document overrides a project note, record that rule. If retrieval discards a source because it exceeds an age limit or fails an access check, record the omission. Two manifests with the same set of references can still produce different context when the assembler orders or trims them differently. Do not solve drift by freezing everything forever. Fresh context can be correct when a task explicitly permits it. The architecture must state which inputs remain fixed for the work item and which may refresh between attempts. For example, source code and acceptance criteria usually stay pinned, while a transient service status may refresh. Record the refresh as a new context version rather than silently changing the old one. A tool can capture traces, but it cannot decide your context contract. If nobody owns instruction versions or defines which retrieved sources are admissible, richer traces will document inconsistency in greater detail. ## Retry loops expose missing state boundaries A retry is safe only when the pipeline knows what completed, what changed outside the process, and what may run again. Retrying an entire agent graph after one late failure can duplicate comments, commits, tickets, deployments, or review requests even if the model response itself is harmless. The AWS Builders' Library article "Timeouts, retries, and backoff with jitter" calls retries selfish because a retry spends more server capacity to improve one request's chance of success. It also recommends retrying at one layer in a stack instead of multiplying attempts at every layer. Multi-agent pipelines need the same discipline, with an extra concern: an attempt may leave durable reasoning artifacts and external side effects. Give the work item a stable identity and every attempt a separate identity. Side effect handlers should accept an idempotency key derived from the work item, operation, and intended version. Before repeating an operation, the handler checks whether that exact effect already succeeded. This check belongs next to the side effect, not inside a model prompt that says "do not duplicate work." Model the attempt with explicit states such as `queued`, `running`, `waiting_review`, `accepted`, `rejected`, and `failed_retryable`. A terminal state must include a reason code. Free text can explain the incident, but machines need a bounded reason such as `context_unavailable`, `tool_timeout`, `contract_failed`, or `review_rejected` to choose the next action. Set a retry budget at the work item level. A sensible policy may allow one retry for a transient tool timeout, no automatic retry for a failed contract, and a fresh human decision after a rejected review. The exact numbers depend on cost and risk; the important choice is that nested agents do not each invent their own budget. Separate retry from compensation. A retry attempts the same intended operation again. Compensation performs a new operation that reduces or reverses an earlier effect, such as closing a duplicate ticket or rolling back a deployment. Calling compensation a retry obscures both histories and can cause the system to repeat the reversal. Store it as a linked work item with its own acceptance condition. Unknown outcomes need their own state. A timeout does not prove failure; the external service may have completed the request and lost the response. Mark the operation `outcome_unknown`, query the external system with the idempotency key, and resolve the state before retrying. Treating every timeout as `failed` is a common source of duplicate effects. Backoff reduces load, but it does not make a logically unsafe retry safe. A ten minute delay before duplicating a production change is still duplication. Likewise, a circuit breaker can stop a storm without telling the team which partial work remains valid. Preserve completed artifacts, invalidate only outputs downstream of the failed dependency, and resume from the narrowest safe boundary. Look for retry amplification in the event history. If an orchestrator makes three attempts and each agent wrapper makes three more, a single work item can trigger nine calls before tool specific retries even begin. The architecture audit should draw this multiplication explicitly. Teams usually underestimate it because each layer reports only its local count. ## Human review capacity limits the pipeline Human review is a queue with a finite service rate, not a ceremonial box near the end of a diagram. If agents submit more decisions than reviewers can evaluate, waiting time grows, context goes stale, and reviewers accept changes with less care. Adding agents makes this failure worse. Measure arrivals, completions, and age by review class. A low risk formatting change should not share a queue with a database migration or a financial policy decision. The reviewer pool, evidence required, expiration time, and escalation path can differ. One undifferentiated approval inbox hides the work that deserves attention. Little's Law connects the average number of items in a stable system to arrival rate and average time in the system. You do not need a simulation to use the lesson. If a team sends 24 reviews during an eight hour day and a reviewer can safely finish two per hour, one reviewer cannot keep up. The backlog will grow even when the orchestration graph runs perfectly. Time a review from ready to decision, then separate waiting time from active reading. Capture rework after rejection as new demand rather than pretending it was part of the first review. A pipeline with fast approvals and frequent reversals may have worse review quality than one with a modest queue. Set an age limit for each review class and define what happens when it expires. High risk work may require regeneration against fresh context, while a low risk change may only need its tests rerun. Automatic reminders are not an expiration policy. They move notifications while the evidence and assumptions continue to age. Sample completed reviews for decision quality, not just speed. Check whether reviewers saw the required evidence, whether accepted changes met their conditions, and whether later reversals trace back to missed information. This gives the team a feedback loop for the review packet. It also detects rubber stamping before an incident does. Review packets should contain the acceptance condition, changed decisions, relevant diff, test evidence, context identifier, side effects already performed, and rollback path. Do not make a person reconstruct those facts from several dashboards. The packet should also say what the reviewer is authorized to decide. An approval that means "I glanced at it" has no useful semantics. You can reduce load in four honest ways: submit fewer items, improve the evidence so each review takes less time, route low risk changes through preapproved policy, or add qualified reviewers. Hiding the queue or sending reminders changes none of those variables. Before adding an orchestration tool, calculate whether the current human capacity can absorb the tool's promised machine throughput. ## Four tests decide whether an audit comes first An architecture audit should come before a new orchestration tool when the team cannot state and verify its invariants. Four tests expose that condition quickly. First, ask who owns each decision and artifact. If the answer changes by run, the system lacks stable boundaries. Second, select any accepted output and reconstruct its context without asking the agent operator. If source versions are missing, the output lacks provenance. Third, choose a retried work item and account for every side effect. If the team cannot prove which effects occurred once, retry safety is unknown. Fourth, compare review arrival rate with completed reviews by risk class. If nobody has the data, human capacity is unmanaged. Failing one test does not require a grand redesign. It does require an architectural decision before a tool purchase. The decision might be a single writer rule, an immutable manifest, an idempotent adapter, or separate review queues. The audit finds the smallest boundary that restores control. A new tool is reasonable when the invariants already exist but the current implementation cannot enforce them. For example, the team has defined ownership scopes, but its homegrown scheduler cannot lock them atomically. Or the context manifest is sound, but trace collection loses events under load. Those are tool capability gaps with acceptance tests. Write the purchase test before evaluating vendors: "Given two tasks claiming the same decision, the system rejects the second before execution" is testable. "Better multi-agent collaboration" is not. Run candidate software against recorded incidents and a synthetic failure. A polished happy path says little about recovery. The audit also has a financial test. Estimate engineering time spent resolving duplicated work, reconstructing context, babysitting retries, and clearing reviews. Compare that cost with the cost of fixing boundaries and operating the proposed tool. License price is only one line. Migration, integration, training, and an extra control plane all consume attention. ## An architecture audit follows the evidence A useful audit produces decisions and executable checks, not a thick report. I use a trace based sequence because interviews alone describe the system people think they built. 1. Select representative work items: one success, one merge conflict, one context disagreement, one retry chain, and one review delay. 2. Rebuild each timeline from stored events and mark gaps as missing evidence, not assumptions. 3. Map decision owners, write scopes, context versions, state transitions, side effects, and review gates. 4. Define the invariants that each failure violated, then assign one owner to each invariant. 5. Test the proposed repair against the recorded runs and one deliberately injected failure. The event record needs enough structure to support that work. At minimum, every event should include `event_id`, `work_id`, `attempt_id`, `agent_id`, `context_id`, `artifact_refs`, `state_before`, `state_after`, `reason_code`, and a timestamp. Side effect events also need an idempotency key and the external system's result reference. Check continuity with a simple query or script. The exact storage system does not matter. The check should group events by attempt, order them by timestamp, and report any row where `state_before` differs from the previous row's `state_after`. Its output should name the work item, attempt, two event identifiers, expected state, and observed state. That turns "the run looked strange" into a reproducible defect. Validate the repair in shadow mode when production risk is high. The new policy can evaluate live events and record the action it would take without controlling the run. Compare those decisions with operator actions, investigate disagreement, then enable enforcement for one work class. Shadow evaluation is useful only when it sees the same inputs as the active path; a sanitized sample can hide the boundary that caused the incident. Track exceptions as architecture debt. Each temporary bypass should have an owner, an expiry condition, and the invariant it suspends. Review the list during pipeline changes. An exception without an expiry quietly becomes a second policy, and future agents will depend on behavior nobody intended to support. The deliverables should fit on working pages: a current state map, an invariant register, an incident replay, a prioritized repair list, and tool requirements tied to tests. Every repair needs an owner and a verification method. A diagram without operational checks will become outdated after the next pipeline change. In my Team & AI Audit, I apply this evidence first approach to the engineering system and identify at least $50,000 a year in savings within five business days, or the $5,000 audit is free. The promise does not change the technical standard: each saving must connect to observed work, a proposed change, and a way to verify the result. ## Add orchestration software only after boundaries hold Orchestration software earns its place when it enforces decisions the architecture has already made. It can schedule dependency graphs, persist state, coordinate locks, collect traces, apply retry policy, and route approvals. Those capabilities matter once the team can specify their required behavior. Evaluate a tool with failure drills, not a feature checklist. Kill a worker after it writes an artifact but before it reports success. Change a retrieved document between attempts. Submit two tasks with overlapping decision ownership. Let the review queue exceed its age limit. The tool should expose the condition and enforce your policy without silently creating a second source of truth. Pay close attention to escape hatches. Teams often adopt a strict workflow, then let operators bypass it through manual reruns, direct database edits, or untracked prompt changes. Emergency controls are necessary, but every override needs an actor, reason, scope, and resulting state transition. Otherwise the exception becomes the architecture. Avoid copying your entire pipeline into proprietary workflow syntax before you can describe the state machine in plain terms. Keep domain contracts, context manifests, and event records portable. The orchestrator may change; the meaning of accepted work should not. A small system can remain simple. A database table with conditional updates, an object store for immutable manifests, and a queue may be enough. Complexity becomes justified when coordination requirements exceed what the team can safely operate, not when a vendor diagram looks more complete than your own. ## Make every accepted result explainable A healthy pipeline can answer five questions about any accepted result: what request it satisfied, who or what made each decision, which exact context informed it, which side effects occurred, and which human or policy accepted the risk. If one answer depends on memory or chat history, the result is not yet operationally explainable. Turn those questions into release conditions. Do not accept an artifact without its work identifier and context manifest. Do not execute an external effect without an idempotency key. Do not close a review without the decision, reviewer identity or policy reference, and evidence viewed. Do not retry from a boundary whose completed effects are unknown. This discipline may reduce apparent throughput at first because the pipeline stops counting ambiguous activity as progress. That is healthy. A graph that launches fifty agents while people untangle its output is not faster than a graph that launches five well bounded tasks and completes them cleanly. The next time the pipeline fails, resist the reflex to add a smarter coordinator. Replay one work item and find the first invariant that became unverifiable. Fix that boundary, test the failure again, and buy software only if it can enforce the rule better than the system you can responsibly operate.