Agent Architecture and Orchestration
Shared State Management — Stack
Shared state management › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
The store has to survive concurrent, non-deterministic writers, so the criteria are mostly about what it refuses rather than what it accepts.
- Concurrency control primitives. Whether compare-and-set, transactions, or leases are available, or whether every write is a blind overwrite.
- Write validation. Whether the store can reject a structurally invalid write, or whether validation must happen in every caller and therefore eventually will not.
- Change notification. Whether participants can be told that something changed, or must poll, and what polling costs at the participant count you expect.
- Snapshot and restore. Whether a point-in-time copy can be taken cheaply, since resuming and debugging both need one.
- Size behaviour. What happens as the record grows, and whether a partial or projected read is possible rather than fetching everything.
- Provenance and history. Whether prior values and their writers are retained, or overwritten and lost.
- Scoping and deletion. Whether tenant and run isolation can be enforced by the store itself, and whether everything for a run can be deleted in one operation.
- Read latency. How much the record adds to each turn, given it may be read several times per iteration.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Relational database with transactions and row-level locking | Structured records needing real atomicity, uniqueness, and claim semantics | Schema migrations become part of agent changes; long-held transactions across model calls are a mistake |
| Key-value store with compare-and-set and expiry (Redis, etcd) | Fast claims, leases, and small coordination records | Durability guarantees vary by configuration; large values and history are not what it is for |
| Append-only event log (Kafka, or an events table) with a derived view | Full provenance, replay, and conflict visibility rather than conflict suppression | Readers need the projection to be correct and current; the projection is now a component that can be wrong |
| Workflow-engine execution state | Efforts already running inside a durable engine, where state and control flow stay together | State shape is constrained by the engine, and payload size limits arrive sooner than expected |
| Framework checkpointers over a database | Agent graphs needing resumability without new infrastructure | Contents are framework-shaped and opaque; concurrent writers across graph branches need checking |
| Conflict-free replicated data types (Automerge, Yjs) | Participants editing a shared document or structure with no coordinator | Convergence is guaranteed; the converged value is not necessarily the one anyone intended |
| Object storage for artefacts with pointers in the record | Large outputs — documents, datasets, images — that should never enter a context | Two stores to keep consistent, and orphaned objects when a run dies between write and reference |
| Vector index over accumulated findings | Long efforts where participants must retrieve relevant prior work rather than read all of it | Retrieval returns something plausible for queries with no genuine match; not a system of record |