State Checkpointing

What it is

Capturing the state of a run at a point it can later be restored from: where the agent is in its plan, which tool calls are outstanding, what has been accumulated, which memory items it had resolved. The capability is the capture-and-restore mechanism — what goes into a snapshot, which points are safe to take one at, how snapshots are versioned, and what restoring means once the world has moved on. Deciding to pause and later continue is a separate control-flow concern that uses this mechanism; it is not this.

Why it matters

Long agent runs cross deploys, preemptions, rate-limit waits, and human approvals. Without snapshots, every interruption costs the entire run — including the tokens already spent and the side effects already performed — and the only available retry is from the beginning, which performs those side effects again. Snapshots also make a failed run inspectable: a serialised state is something an engineer can load and examine, as opposed to a log of what a process was doing before it died.

Key decisions

• Where boundaries are placed: after every step, at explicit markers, or only before expensive or irreversible actions. Frequent snapshots cost storage and write latency on every step and shorten the worst-case replay.
• Whether the snapshot contains values or references. References keep snapshots small and make a restore non-deterministic, because the referenced store may have changed or expired.
• Whether a snapshot taken by one code version may be restored by another, and what the compatibility rule is. This is a schema migration problem that appears the first time a deploy lands during a long run.
• How side effects already performed are represented inside the snapshot, so that restoring does not repeat them.
• Retention: how long snapshots live, given that each one contains a full copy of whatever the run was handling, including anything that arrived from a tool.
• Whether restore is exact replay or re-execution from the captured state, which determines how much of the model's own non-determinism the design has to tolerate.

Failure modes

• The snapshot holds references to session-scoped memory that expired while the run was paused, and restore produces an agent with a plan and no context.
• A snapshot is taken after a tool call is issued but before its result is recorded, so resuming re-issues a payment.
• The snapshot schema changes with a deploy and in-flight runs cannot be restored, which is discovered when the first long run fails rather than at release time.
• Snapshots retain complete tool payloads and quietly become the largest store of personal data in the system, with no lifetime and no erasure path.
• The snapshot is written to the same store as the run's live state, so the failure that corrupts one corrupts both.
• Restore is only ever tested from the most recent snapshot, and the older ones that retention keeps have never been proven loadable.
• Snapshots are taken on a timer rather than at a defined boundary, so some capture a run mid-operation in a state that restore cannot make sense of.
• Nothing records which snapshot a resumed run came from, so a run that was resumed twice produces duplicate work that cannot be traced to its cause.

Related capabilities

• Interrupt and Resume — the control flow that calls on this mechanism
• Working Memory — the active set a snapshot has to capture faithfully
• Short-term Memory — the session state a snapshot references and may outlive
• Memory Expiration and TTL — why a referenced item may be gone by the time a restore happens
• Memory Deletion — snapshots as a copy that erasure has to reach

Applies: Production readiness · Design questions