Agent Loop: Reliability & Verification · Part 4 of 7
The Crash Window: Recovering AI Agent Actions Without Blind Retries

Tool-using agents are often described as loops: ask the model, execute a tool, append the result, and continue. The difficult part starts when the tool has a side effect and the process dies at exactly the wrong moment.
Suppose a tool writes to an external database. The write commits. Before the agent can persist that the call completed, the process crashes. On restart, the journal still says running. Should the runtime execute the tool again?
No—at least not by default. The external effect may already exist.
I turned that boundary into a regression test in Agent Loop. The test is committed as 395e8e0 and uses two real subprocesses plus SQLite. It does not prove exactly-once execution. It proves a narrower and more useful invariant: after an unresolved call crosses a crash boundary, recovery records an unknown outcome and does not blindly redispatch it.
The execution-recovery work was subsequently hardened in commit cc577d2. That follow-up adds a second boundary: if compaction fails while a tool call is still unresolved, the runtime must not lose the execution record, mark the call complete, or dispatch it again after resume. It also adds session-integrity checks for journal/message consistency and durable replacement behavior.
The recovery work was then completed with two additional regression commits. b9f3325 verifies that truncated or structurally partial session files fail closed instead of becoming resumable. 798a589 adds real MCP stdio disconnect fixtures before and after an external side-effect commit; both cases are classified as unknown_outcome and non-retryable, while the external marker is checked independently of the MCP response. These commits extend the evidence from a local subprocess crash to transport loss and partial-session recovery. They add regression coverage; they do not claim that arbitrary external systems provide exactly-once semantics.
The current v0.1.0 reliability contract also makes the session-write boundary explicit: session data is validated, written to a same-directory temporary file, file-synced, atomically replaced, and followed by a directory fsync. A failure before replacement preserves the previous session; after replacement, a complete new session is observable. This is a bounded durability guarantee, not a claim about power loss on every filesystem, storage device, or hardware cache.
The broader ExecutionJournal failure-boundary audit is now complete. Its final report records 1,312 passed and covers pending-persist failure, persistence before dispatch, ambiguous external actions, crash after side effect, terminal-save failure, failed compaction with unresolved calls, restart recovery without redispatch, atomic persistence and fsync, partial or corrupt sessions, MCP timeout, and MCP connection loss both before and after an external side effect. The resulting invariant is broader than the original crash test: unresolved execution state remains separate from the summary and is never retried automatically.
The state machine
The runtime keeps a small ExecutionJournal for each tool call. A record moves through explicit states:
PENDING -> RUNNING -> COMPLETED
-> FAILED
-> UNKNOWN
Terminal states carry an outcome string. A completed call has a result; a failed call has a failure description; an unknown call has a recovery explanation.
The important state is UNKNOWN. It is not a synonym for failure. It means the runtime cannot establish whether the external operation took effect.
That distinction is easy to lose if recovery is designed around the assumption that the journal is authoritative. The journal is durable application state, but it is not a transaction spanning every external system the tool can touch.
The test boundary
Process A registers a tool call and moves it to RUNNING. The dispatched action inserts one row into an external SQLite database and commits. Immediately after the commit—and before the journal can record a terminal result—the process exits with a non-zero status.
The assertions after process A exits are deliberately concrete:
- the session file contains the tool call in
runningstate; - the external database contains exactly one effect;
- no terminal journal write has been assumed.
Process B loads the session. Its provider returns a normal stopping response, while the test replaces dispatch with a function that fails if called. Recovery then materializes the unresolved call as unknown and persists the explanation.
The second process must satisfy three conditions:
effect_count = 1
dispatch_count = 0
status = unknown
The test also checks that the recovered tool message says the action was not re-executed. This makes the no-replay behavior visible in both the execution journal and the conversation history.
Independent review and the compaction boundary
This recovery boundary was also raised during independent review. Tim Urista pointed out that unresolved tool calls must remain distinguishable from completed calls after older conversation messages disappear during compaction. His proposed invariant is deliberately narrower than “the agent remembers everything”: execution state should remain outside the prose summary, identified by the call ID and terminal outcome, so recovery can rebuild only the context needed for the next decision.
That concern connects directly to this test, but it is not the test's evidence. The subprocess regression checks a separate failure boundary: an external effect commits before the terminal journal record is written. On recovery, the call remains visible as UNKNOWN, the effect is not replayed, and the runtime does not invent a successful result. The same separation also means that compaction must not turn an unresolved action into either COMPLETED or an automatic retry.
Urista later asked for the execution-journal revision and suggested checking the external effect and journal together after recovery: no replay, no invented success, and the unresolved call still visible. The original external-effect test is 395e8e0; the subsequent recovery-hardening revision is cc577d2. The comments are independent review and test-design input, not additional experimental results.
References: compaction and unresolved tool calls, separating compaction from execution tracking, and the pinned revision and recovery test.
The updated compaction boundary
The new regression forces the summary provider to fail while a resumed session still contains a RUNNING tool call. The fallback path must preserve the execution record, recover it as UNKNOWN with an unknown_outcome category, append a tool message explaining that the action was not re-executed, and send no dispatch request.
This is a different failure boundary from the SQLite crash test. The crash test asks what happens when an external effect may already have committed. The compaction test asks what happens when the narrative summary cannot be produced. Both cases require the same conservative invariant: execution state lives outside the prose summary, and uncertainty must remain visible after recovery.
The current focused validation covers the execution-state, session-integrity, compaction, and MCP contract regression modules. The last directly reproduced subset passed 100 tests; the completed audit report records 1,312 passed across the full failure-boundary scope. These results describe the checked checkout and audit run; they are not a general reliability score for Agent Loop.
Why retrying is unsafe
A retry policy can be correct for a pure function. It is not automatically correct for an external action. A process crash removes the runtime's knowledge of where it stopped; it does not roll back a database commit, an email, a payment, a deployment, or a file upload.
The dangerous shortcut is:
if status == RUNNING after restart:
run the tool again
That converts uncertainty into a second side effect. If the first attempt succeeded, the retry duplicates it. If the first attempt failed before doing anything, the retry may be reasonable—but the runtime cannot safely know which case occurred without tool-specific idempotency or an external reconciliation check.
The conservative rule is therefore: recover the record, do not dispatch it, and require verification before relying on the result.
What the test does not guarantee
This is not a distributed transaction protocol. It does not provide exactly-once execution. It does not make arbitrary tools idempotent. It does not prove that a tool can never partially apply an effect. It does not replace reconciliation with the external system.
It proves one recovery behavior for one controlled failure boundary. A production tool still needs an idempotency key, a read-after-write verification path, or a domain-specific reconciliation procedure if the operation matters.
The word “unknown” is doing real engineering work here. It prevents the runtime from manufacturing certainty that it does not possess.
Why the subprocess matters
An in-process test could simulate an exception, but it would not exercise session durability across a process boundary. The regression starts a fresh interpreter for recovery, reloads the serialized state, and verifies the behavior against a second process.
The test also has explicit timeouts for both subprocesses. A recovery test that hangs can hide the very failure it is meant to catch. The focused validation passed 100 tests across execution-state recovery, session integrity, compaction, and MCP disconnect boundaries; the completed audit reported 1,312 passed across the full boundary matrix. git diff --check passed in the updated checkout before the later local working-tree change.
Those results are evidence about the tested code path—not a general reliability score for the whole agent runtime.
The design lesson
Durable state is valuable precisely because it makes uncertainty explicit. A good recovery path answers three separate questions:
- What did the runtime persist before the crash?
- What external effects can be observed now?
- What action is safe to take without duplicating an unknown effect?
The journal can answer the first. A reconciliation tool or external system must answer the second. The safest default for the third is often “do nothing automatically.”
That may sound less productive than retrying, but it is more truthful. An agent that pauses with an auditable UNKNOWN record is recoverable. An agent that silently duplicates a side effect has destroyed information about what happened.
Related: A Successful Search That Never Searched and Testing AI Agents With Independent Agents.