State Checkpointing — Patterns

Snapshot only at declared safe points

Problem. A snapshot taken at an arbitrary moment — on a timer, on a signal, between two statements — can capture a run mid-operation: a tool call issued and unacknowledged, a partially applied state update, a lock held. Restoring from it produces behaviour that no code path was written to handle.

Approach. Define safe points explicitly as the boundaries between steps where the run's state is internally consistent and no external operation is outstanding, and permit snapshots only there. Everything between safe points is a unit that either completes or is redone from the last one. Because the points are declared, the worst-case amount of redone work is a known quantity rather than a function of when the process happened to die.

Trade-off. Long operations between safe points mean long replays, and the natural fix — adding more safe points — increases write cost and forces more of the run's state to be made serialisable. Operations that genuinely cannot be made atomic between two points need their own idempotency design regardless, so the pattern does not remove that work.

Side-effect ledger inside the snapshot

Problem. Restoring a run restores its intent, not the world. A snapshot taken before a tool call completes leaves the system unable to tell whether the call happened, and the two possible assumptions are "repeat the payment" and "skip it".

Approach. Record external operations in a ledger that is part of the checkpointed state: each entry has a deterministic key derived from the step and its arguments, a status, and the result once known. The ledger is written before the call is issued and updated after it returns. On restore, an entry that is issued-but-unknown is resolved by querying the external system or by replaying with the same idempotency key, rather than by guessing. Tools that cannot support a key are marked non-resumable and their steps fail closed.

Trade-off. The ledger requires every tool integration to cooperate, and the integrations that most need it are often the ones least able to provide an idempotency key. It also adds two writes around every external call, which is real latency on the common path to buy safety on the rare one.

Version the snapshot, not just the code

Problem. Snapshots outlive the build that wrote them. A deploy during a long run leaves state serialised by one version being loaded by another, and the failure is a deserialisation error at the worst possible moment, affecting exactly the runs that were long enough to matter.

Approach. Put an explicit schema version in every snapshot and keep a loader that can read every version still within the retention window, upgrading forward on load. Treat removing support for a version as a deliberate act gated on no snapshots of that version remaining, which requires being able to count them. Test the loader against stored fixtures from previous versions, not against state the current build just produced.

Trade-off. Maintaining upgrade paths is ongoing work that benefits nobody visibly, and the pressure to drop old versions is constant because the only cost of keeping them is maintenance. Forward-upgrading also means a restored run is not quite the run that was captured, which is occasionally the source of a subtle divergence.

Separate the narrative state from the bulk

Problem. A snapshot that embeds every tool result in full is large, slow to write, expensive to retain, and contains whatever those tools returned — which is frequently personal data that has now been copied into a store with no retention rule and no subject index.

Approach. Keep the snapshot to the run's control state and small values, with large payloads stored once by reference under a handle that carries its own lifetime and access rules. Restore resolves handles lazily, and a handle that no longer resolves produces an explicit, handled condition rather than a crash. Because the bulk is addressed separately, erasure can reach it by subject without having to parse every snapshot.

Trade-off. Restore now depends on a second store being available and on its retention being at least as long as the snapshot's, and a mismatch produces runs that are resumable in principle and not in practice. Lazy resolution also moves a class of failure from restore time to deep inside the resumed run, where it is harder to diagnose.