Agent Architecture and Orchestration
Parallel Execution — Stack
Parallel execution › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
Concurrency primitives differ mainly in what they guarantee when something goes wrong, which is the only time the difference matters.
- Cancellation. Whether an outstanding branch can actually be stopped, how promptly, and whether the mechanism propagates through library code that was not written with it in mind.
- Scope discipline. Whether branches are guaranteed to complete or be cancelled before their parent returns, or can outlive it as orphans.
- Backpressure. Whether the mechanism limits outstanding work by itself, or whether unbounded fan-out is the default and limiting is your job.
- Per-branch observability. Whether each branch gets its own identifiable trace that can be read independently and related back to the parent.
- Deterministic combination. Whether results can be reassembled in a defined order regardless of completion order.
- Failure policy expressiveness. Whether fail-fast, wait-for-all, and best-effort-by-deadline are all expressible without hand-rolling.
- Cross-process reach. Whether the same construct works when branches run on different machines, or only within one process.
- Retry safety. Whether a branch can be made idempotent so a retry is safe while its siblings are still running.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Language-level structured concurrency (asyncio task groups, trio nurseries, Go errgroup, Java structured concurrency) | In-process fan-out over I/O-bound calls with guaranteed scope and cancellation | Cancellation only works if every layer cooperates; one blocking call in a thread defeats it |
| Thread and process pools | CPU-bound steps, or libraries with no asynchronous interface | Cancellation is usually impossible once a task has started; pool exhaustion deadlocks are easy to create |
| Native parallel tool calls from the model | Several independent lookups in one turn, with no extra machinery | The model decides the grouping; it will sometimes parallelise calls that had a dependency |
| Framework fan-out constructs (graph-runtime parallel branches and send primitives) | Agent graphs that need fan-out with checkpointing already handled | Merge semantics are framework-defined; non-trivial reconciliation ends up written around the primitive |
| Queue with a worker pool (Celery, SQS consumers, similar) | Bulk work, natural backpressure, and tolerance for unpredictable completion times | Not suitable while a user waits; per-branch tracing requires deliberate correlation |
| Workflow-engine fan-out (map states, child workflows) | Durable branches with per-branch retry, visibility, and survivable partial completion | Payload size and concurrency caps shape the design; engine overhead dominates for short branches |
| Actor runtimes (Ray, Dapr) | Compute-heavy branches needing placement control and distributed scheduling | Substantial operational surface for what is frequently an I/O-waiting problem |
| Provider batch interfaces | Large non-interactive workloads where completion within hours is acceptable | Entirely unsuitable for interactive latency; a different capability wearing similar clothing |