Multi-Agent LLM System on Azure

Overview

A multi-agent system breaks a complex task into roles handled by separate LLM-powered agents that plan, call tools, and hand work to each other. An orchestrator decides which agent acts next and when the task is done. Use this architecture when a single prompt cannot reliably solve the problem, such as multi-step research, code generation with review, or workflows that require calling several external systems.

Components

• Azure OpenAI: provides the reasoning models that power each agent.
• Azure Container Apps: runs the orchestrator and agent workers as independently scalable services.
• Azure Service Bus: passes tasks and results between agents as durable messages, decoupling them.
• Cosmos DB: stores shared state, conversation memory, and the task ledger.
• Azure AI Search: gives agents a retrieval tool over enterprise knowledge.
• API Management: exposes external tools to agents with rate limits and auth.
• Azure Monitor: traces each agent step, token use, and tool call.

Data Flow

A request reaches the orchestrator, which decomposes it into subtasks and dispatches them via Service Bus. Each worker agent reads its task, may call tools (search, APIs, code execution) through API Management, reasons with Azure OpenAI, and writes its result back. The orchestrator inspects results, decides whether to continue, retry, or finish, and assembles the final answer. All steps and state are recorded in Cosmos DB so a run can be resumed or audited.

Scaling and Resilience

Agents scale independently in Container Apps based on queue depth, so the system adds workers when tasks pile up. Service Bus provides at-least-once delivery and dead-letter queues for poisoned messages. Set hard limits on loop iterations and total tokens per task to prevent runaway cost and infinite agent loops. Use idempotent task handling so retries do not duplicate side effects. Time out stalled agents and reassign their work.

Security

Give each agent only the tools and data it needs, enforced through scoped API Management policies and least-privilege identities. Treat all agent output as untrusted: validate tool arguments and sandbox any code execution. Defend against prompt injection from retrieved content and from tool results that try to hijack the agent. Require human approval before agents take high-impact actions. Log every decision and tool call for accountability.

Trade-offs and Alternatives

Multi-agent systems can solve harder problems than a single prompt, but they cost more tokens, are slower, and are harder to debug because errors compound across steps. Start with a single agent plus tools and add agents only when measured quality demands it. Frameworks like Semantic Kernel or AutoGen speed development but add abstraction; a small custom orchestrator is often easier to reason about. Strong guardrails and step limits are mandatory, not optional, to control cost and risk.