Agent Lifecycle — Patterns

Transitions as recorded events

Problem. When state is a mutable field, the system can only say where an instance is now. The questions asked during an incident are about the past: when did it start waiting, what woke it, how long was it executing before it stalled.

Approach. Append every transition — previous state, new state, cause, actor, timestamp — to a durable log and derive the current state from the latest entry. The log is the record; the field, if it exists at all, is a cache of it.

Trade-off. A write per transition costs latency and storage, and the log grows for instances that churn between waiting and executing. It also tempts teams into replaying the log to reconstruct behaviour, which only works if transitions are the whole story and tool side effects are recorded alongside them.

Resources held by lease, not by reference

Problem. Cleanup written as a finally block runs when the process survives to execute it. Instances that are killed, evicted, or lost with their machine release nothing, and sandboxes, leases, and reserved budget accumulate.

Approach. Acquire every external resource with a bounded, renewable lease tied to the instance identifier. The instance renews while alive; expiry is what releases, with explicit release as an optimisation rather than the mechanism. A sweeper reclaims anything whose lease has lapsed.

Trade-off. Resources survive for up to one lease period after an instance is genuinely gone, so leases must be short enough to bound waste and long enough to survive a slow turn. Renewal traffic is constant and its failure modes become part of the lifecycle.

Every ending has a classified reason

Problem. A transcript that simply stops is indistinguishable from one that completed, one that exhausted its budget, and one whose worker was evicted. Success-rate metrics built on "produced a final message" quietly count all three as the same thing.

Approach. Make the terminal transition carry a reason drawn from a closed set, written before resources are released, with a last-resort writer that records an unknown reason when the instance is reaped by a sweeper rather than ending itself.

Trade-off. A closed set needs maintenance, and the first few months produce a large unknown bucket that has to be investigated rather than ignored. The alternative is a metric that cannot distinguish a working product from a broken one.

Admission control before creation

Problem. Instance creation is usually trivially cheap at the call site and expensive in aggregate. Without a gate, a retry storm or a loop in a calling service creates thousands of long-lived instances that then have to be found and killed.

Approach. Put a quota in front of creation — per tenant, per user, per agent — and make exceeding it a clear refusal at creation time rather than a degradation once everything is already running. Count live instances, not creation rate, because the problem is occupancy.

Trade-off. Legitimate bursts are refused, and the quota becomes a number nobody can justify that gets raised under pressure during incidents. It is still preferable to discovering the limit through resource exhaustion.