Multi-Agent Orchestration in Production: Taming the Combinatorial Complexity Problem
Adding AI agents multiplies the interactions you have to govern, not just the agents. Here are the coordination patterns, contracts, and safeguards that keep multi-agent systems stable in production.
A single AI agent in a pilot is easy to reason about: one input, one model, one output, one person checking it. Add a second agent and you add a conversation between them. Add a third and you have three possible pairs, each with its own failure modes. The agents themselves may be no harder to build. What grows quickly is everything between them.
This article explains why multi-agent systems get harder faster than teams expect, how to choose between central orchestration and choreography, which failures to design for, and the patterns that keep these systems stable once real work flows through them.
Your single-agent pilot does not predict production
A pilot proves that one agent can do one task. It rarely tests what happens when that agent's output becomes another agent's input, when two agents act on the same work order, or when a downstream system is slow. Those are the conditions production adds.
Here is an illustrative modeled estimate of why. With 5 agents there are 10 distinct pairs that can interact, and if each agent can be in one of 5 basic states, the system has 3,125 combined states. Model inputs: 5 agents, every pair able to interact (5 × 4 ÷ 2 = 10 pairs), and 5 states per agent (5 to the power of 5 = 3,125). Nobody tests 3,125 combinations by hand, which is why design discipline matters more than test volume.
| Agents | Possible pairs | Combined states at 5 states each |
|---|---|---|
| 2 | 1 | 25 |
| 3 | 3 | 125 |
| 5 | 10 | 3,125 |
| 8 | 28 | 390,625 |
Central orchestration or choreography
Microsoft's Azure Architecture Center describes the two basic coordination styles clearly. In the orchestrator pattern, requests flow through a central component that delegates work to each service; that component can consolidate the status of a transaction, but it can also become a performance bottleneck and a single point of failure, and adding or removing services means rewiring it [1]. In the choreography pattern, each service decides when and how to process a business operation, often through a message broker, instead of depending on a central orchestrator [1].
Choreography is not free. The same guidance warns that without a central orchestrator holding the full transaction state, no single component has a complete view of an in-flight operation, so you must consistently use distributed tracing and correlation identifiers, and that decentralized event topologies can produce feedback loops or event storms at scale [1].
| Choose | When | Watch for |
|---|---|---|
| Central orchestrator | The workflow is sequential, shared logic is significant, and you need one place holding transaction state | Bottlenecks and single point of failure |
| Choreography | Agents handle independent tasks, change often, and cross clear domain boundaries | Loss of end-to-end visibility, event storms, ordering problems |
| Hybrid | An orchestrator per domain, with events between domains | Clear ownership at each boundary |
The failures you will not see in staging
Three failure classes dominate multi-agent production incidents, and all three come from interactions rather than from any one agent:
- Feedback loops. Agent A's output triggers Agent B, whose output triggers Agent A again. The Azure guidance calls this emergent behaviour, where a minor event triggers a cascade of reactions, and recommends guardrails such as event filtering, concurrency limits, throttling, and explicit rules [1].
- Races and duplicates. Two agents act on the same asset or work order at once, or a retry delivers the same message twice. The Azure guidance notes that at-least-once delivery and retries can produce duplicates and out-of-order processing, and recommends idempotent consumers that track stable message identifiers [1].
- Cascading slowness. One slow dependency holds up every agent waiting on it until resources run out.
Interactions need owners too
Every agent has an owner. Most interactions between agents do not. Assign an owner to each agent-to-agent contract, with the authority to change or stop it, or the failures that live between agents will have nobody to fix them.
The agent manifest: contracts that prevent chaos
Write down, for every agent, the contract other agents and people can rely on:
| Manifest field | What it states |
|---|---|
| Purpose and scope | The task the agent performs and the assets or data it covers |
| Inputs and outputs | Schemas, with versioning rules |
| Permissions | What it may read and write, and in which systems |
| Decision rights | What it may do alone, and what needs human approval |
| Dependencies | The agents and services it calls, with time-outs |
| Failure behaviour | What it does when a dependency fails or input is invalid |
| Owner and escalation | Who maintains it and who is called when it misbehaves |
Treat schema changes like any other event contract change: the Azure guidance recommends a schema registry and backward-compatible evolution so one producer's change does not break every consumer [1].
Architectural patterns that survive production traffic
Pattern 1: circuit breakers for agent-to-agent calls
A circuit breaker temporarily blocks calls to a dependency after failures reach a threshold, instead of repeatedly retrying an operation likely to fail; it moves between closed, open, and half-open states, letting a limited number of trial requests through before resuming normal traffic [2]. Put one on every agent-to-agent and agent-to-system call so one failing agent cannot exhaust the rest.
Pattern 2: idempotent agent actions
Design every action so that doing it twice has the same effect as doing it once: check for an existing open work order on the same asset and failure mode before creating another, and key actions by a stable identifier. This is what makes retries safe [1].
Pattern 3: shared state with optimistic locking
When agents update shared records, use version checks so a stale update is rejected rather than silently overwriting a newer one. Combine that with an outbox pattern so a state change and the event announcing it are committed together [1].
Pattern 4: tracing across agents
OpenTelemetry defines a trace as the path of a request through your application, made of spans that each represent a unit of work, with context propagation as the core concept that enables distributed tracing [3]. Propagate one trace context from the triggering event through every agent call and every system write, so you can reconstruct any decision end to end.
Pattern 5: shadow-mode deployment
Run new agents, or new versions, alongside the existing process without letting them act. Compare their proposed actions with what people actually did, and promote them only when the record supports it.
Governance from pilot to production
The NIST AI Risk Management Framework is intended to help organizations incorporate trustworthiness considerations into the design, development, use, and evaluation of AI systems [4]. In multi-agent terms, that means documented decision rights per agent, measured accuracy for each, traceable decisions, and a clear way for people to stop an agent.
For agents that touch plant systems, add the operational technology view. The December 2025 joint guidance from CISA and partners on securely integrating AI in OT calls for assessing AI use in OT, establishing AI governance, and embedding safety and security, including maintaining oversight and integrating AI into incident response plans [5]. Keep agents read-only toward control systems unless a change has been designed and approved as part of the control system itself.
For practical examples of agents in maintenance, see our articles on agentic AI for root cause analysis and the trust scorecard for AI maintenance recommendations.
Your first 30 days
1. Inventory agents and interactions. List every agent and every agent-to-agent or agent-to-system call. The interaction list is usually longer than expected. 2. Write manifests for each agent, starting with permissions, decision rights, and failure behaviour. 3. Add tracing with a single propagated context across agents. 4. Put circuit breakers and idempotency checks on every call that creates or changes a record. 5. Move new agents through shadow mode before they act, and decide promotion from the comparison record.
Frequently asked questions
Why are multi-agent systems harder to run than single agents?
Because the number of interactions, and combined states, grows much faster than the number of agents. Most production failures live in those interactions.
Should we use an orchestrator or choreography?
Use an orchestrator for sequential workflows with significant shared logic, and choreography for independent tasks across domain boundaries. Each has documented tradeoffs [1].
What is the most important safeguard?
Traceability. If you can reconstruct any decision across every agent and system it touched, you can find and fix the other problems.
How do we move from pilot to production safely?
Write manifests, add tracing, circuit breakers, and idempotency, then run each agent in shadow mode with human approval until its record justifies more autonomy.
References
[1] Microsoft, Azure Architecture Center, "Choreography pattern." https://learn.microsoft.com/en-us/azure/architecture/patterns/choreography
[2] Microsoft, Azure Architecture Center, "Circuit Breaker pattern." https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker
[3] OpenTelemetry, "Traces." https://opentelemetry.io/docs/concepts/signals/traces/
[4] NIST, "AI Risk Management Framework." https://www.nist.gov/itl/ai-risk-management-framework
[5] CISA, "CISA, Australia, and Partners Author Joint Guidance on Securely Integrating Artificial Intelligence in Operational Technology," December 3, 2025. https://www.cisa.gov/news-events/alerts/2025/12/03/cisa-australia-and-partners-author-joint-guidance-securely-integrating-artificial-intelligence
Related articles
Ready to put this into practice?
See how Monitory helps manufacturing teams implement these strategies.