What is AI orchestration?

What is AI orchestration?

AI orchestration sits above the model call. One chat.completions request is a function; orchestration is the workflow that strings 3+ of them together with retries, conditional branches, tool calls, and human-in-the-loop checkpoints. The hole most teams hit: glue code that works in a demo but loses state on every redeploy and silently retries deterministic 400s forever.

How it works

Two flavors dominate, and they solve the same problem differently.

1. Declarative graph. You describe the dependency graph as nodes and edges; the framework runs it. LangGraph is the canonical example for agent graphs. AWS Step Functions and Inngest steps cover the broader workflow case. Good when the control flow is mostly static.
2. Durable execution. You write straight-line code; the framework persists state between steps so a crash, deploy, or timeout resumes from the last completed step. Temporal, Restate, and Inngest ship this. Good when control flow is dynamic and you want code-not-config.

Both ship the same distributed-systems primitives (idempotent steps, exponential backoff with jitter, durable timers, signal-based human handoff), wrapped for LLM ergonomics.

When you'd encounter it

Any pipeline past 3 LLM calls. Specifically: research agents that fan out to N parallel sub-queries then synthesize, document processing where each page needs OCR + extract + validate, or any flow that pauses for human approval and resumes hours later. Multi-agent orchestration is the subset where multiple agents (not just multiple steps) coordinate.