Parallel Execution

What it is

Having more than one piece of agent work in flight at the same time — several tool calls within a single turn, several independent subtasks, or several agents working on separate parts of one objective — and reconciling what comes back. It is a throughput decision about how many things are outstanding, not a decision about what the sequence of work is.

Why it matters

Agent latency is dominated by waiting: on model responses, on API calls, on retrieval. Much of that waiting is for things that have no bearing on each other, and overlapping it is usually the only way to turn a two-minute task into a twenty-second one without changing the model. The price is a step change in difficulty: concurrent writers, partial failure, non-deterministic completion order, and cancellation are all problems the sequential version simply did not have, and they produce intermittent bugs rather than reproducible ones.

Why this is not the control structure

Deciding that there will be another round of work, and deciding how many things may be outstanding at once, are separate choices. A single-threaded control structure that issues one call at a time and a heavily concurrent one that issues twenty have the same shape; only occupancy differs. Equally, running twenty things at once says nothing about the arrangement of participants or who is coordinating them — concurrency is a property you add to a design, not a design in itself.

Key decisions

• The unit: tool calls inside one turn, independent subtasks, or whole agents. Each has different isolation and different failure consequences.
• How independence is established — declared by whatever produced the units, derived from non-overlapping read and write sets, or assumed, which is where the intermittent bugs come from.
• The concurrency ceiling, and specifically which shared quota everything lands on: one provider's rate limit, one database's connection pool, one downstream service.
• Partial failure semantics: cancel siblings and fail fast, wait for everything and report, or return whatever finished by the deadline.
• Whether the combination step is order-independent, since an order-dependent one makes the whole run irreproducible and unevaluable.
• Whether in-flight work can genuinely be cancelled, and what happens to side effects it has already performed.

Failure modes

• Fan-out meets a provider rate limit and the entire batch fails at once, converting a latency optimisation into an availability incident that is worse than the serial version.
• Two branches write the same shared field and the later write silently discards the earlier one, with no error anywhere.
• A failed branch is retried while its siblings continue, so the retry observes state its original attempt never saw and produces a result inconsistent with the rest.
• Cancellation is never implemented, so a fail-fast design still pays in full for every branch it abandoned and holds their connections until they finish.
• Results are merged in completion order, so identical inputs produce different outputs on different runs and no evaluation of the system is reproducible.
• Tracing has no concept of a branch, so spans interleave into a single unreadable timeline and nobody can tell which branch caused the latency.
• The slowest branch sets the latency: a fan-out of twenty with one ninety-second outlier is a ninety-second task, and the mean is irrelevant to the user.
• Concurrency is applied to steps with an unnoticed ordering dependency, and the resulting bug reproduces roughly once a week.

Related capabilities

• Task Decomposition — produces the units and declares whether they are independent
• Execution Loop — the control structure this concurrency sits inside
• Shared State Management — where concurrent writers collide
• Multi-agent Orchestration — determines whether participants may act simultaneously
• Interrupt and Resume — stopping a run means stopping every branch of it

Applies: Production readiness · Design questions