Harness Engineering for Agents

Durable Execution Replay Safety & Subagents

Your agent loop is alive—until it isn't. A network timeout, a process kill, a power outage, and that carefully constructed conversation disappears. Worse: the agent half-applied an edit, then crashed. On restart, it retries the last step and applies the edit again, corrupting the file. This is the core problem of durable execution: making the agent survive crashes without repeating or dropping side effects. You solve it by committing intent before action, checkpointing state machines through every effect, and classifying tools by whether they tolerate replay. And when you need parallel exploration—subagents forking off to research, edit, validate—the same durability principles extend to structured concurrency, where every child task is owned by a parent and dies when the parent does. By the end of this layer, your agent carries a crash-resistant journal, replays safely even after kill -9, and manages subagents with bounded lifetimes and clean cancellation.

Commit-First Architecture

The fundamental insight is simple: write down what you are about to do before you do it. Your agent's loop currently thinks, then acts. A commit-first architecture reverses the order for the durable parts: first append the intent to an append-only transcript, then execute the effect, then record the outcome. If the process dies between the first and second step, the recovery code sees a pending intent and can skip or re-construct that effect instead of blindly replaying an unknown state. The transcript becomes your source of truth—an append-only log of every message, tool call, and decision. Nothing gets rewritten or deleted. This means you can always reconstruct the exact sequence of events after a crash.

The second piece is atomic typed documents: small, strongly-typed records that represent durable state—the agent's conversation summary, the set of open files, the current task status. These documents use atomic compare-and-swap updates so that concurrent writers or recovery processes never see half-written state. You define each document's schema up front (defineDoc in typical implementations), and all updates go through the same commit-first path: log the update intent, apply the document change, log the result. Grouping multiple small writes into a single group commit batch improves throughput for low-latency streaming turns. During high-frequency tool output streams, you may defer the commit briefly—collecting a batch of log entries—then flush atomically. The storage backend itself is pluggable: SQLite with WAL mode for local durability, JSONL for debugging, IndexedDB for browser agents, or cloud Durable Objects for distributed setups. The loop does not care which one you plug in, as long as it guarantees append-only writes and atomic document updates.

spec: static diagram of a commit-first timeline. Three columns labelled "Time", "Transcript Log", "Document Store". Four rows, each row represents one step: step 1 shows "Append intent: tool_call(file_edit)" then a blank document cell; step 2 shows "Execute effect" with a gear icon over the tool call; step 3 shows "Append result: success/error" then a blank document cell; step 4 shows "Commit atomic update" with a lock icon over the document cell that now contains "file_edit applied". Arrows flow left-to-right across the columns showing the causal chain.

Checkpointed Task State Machines

A single long-running agent turn can involve dozens of tool calls, LLM decisions, and context transformations. If any of those steps crashes midway, you need to know exactly what state the task was in. The solution is to model each task as a state machine whose transitions are checkpointed. The states are something like: pending, awaiting_llm, awaiting_tool, completed, failed. Every transition between states writes a checkpoint—a snapshot of the current task's step number, pending effect, and accumulated conversation. When the process restarts, it reads the last checkpoint and knows precisely where to resume.

The critical pattern here is the effect sandwich: commit the intent to execute an effect (open one side of the sandwich), run the effect, commit the result (close the other side). If the process dies during the effect execution, the recovery code sees an unclosed intent and knows the effect may or may not have happened. It then either retries the effect (if idempotent) or marks it as interrupted and asks the LLM to recover. This pattern turns every crash into a deterministic state transition rather than a data-loss event.

Two safety mechanisms protect against repeated crashes. A restart limit circuit breaker counts how many times a task has restarted within a given window. After three or five restarts in quick succession, the circuit breaker trips and the task is permanently failed instead of retrying forever. This prevents crash loops from burning through retries or producing infinite recovery attempts. Separately, idempotent submission admission uses a requestId on every LLM call and tool execution. The first time a request arrives, the system records its ID. If the same request arrives again after a restart (because the previous attempt's commit failed before the result was recorded), the system recognises the duplicate requestId and returns the cached result instead of re-executing. This is the same principle as an idempotency key in payments APIs: the caller generates a unique key, and the system ensures exactly-once semantics.

spec: static diagram of a state machine with five states as labelled boxes: "pending", "awaiting_llm", "awaiting_tool", "completed", "failed". Arrows between them: pending -> awaiting_llm (commit intent); awaiting_llm -> awaiting_tool (effect sandwich: commit tool intent); awaiting_tool -> awaiting_llm (commit tool result); awaiting_llm -> completed (commit final result); any state -> failed (circuit breaker trips). The arrows are labelled with the type of checkpoint written.

Replay Safety and External Parking

A crash recovery replay must re-run the same LLM decision and tool calls that led to the crash. But not all tools can be safely re-run. Sending an email twice or charging a credit card twice is catastrophic. This is the distinction between safe (idempotent) and non-idempotent tools. A safe tool—read_file, list_directory, a pure computation—can be replayed freely because it produces the same result each time. A non-idempotent tool—send_email, charge_credit_card, create_stripe_invoice—must never be replayed. The harness marks each tool with a replay attribute: "safe" or "interrupted". On recovery, safe tool calls replay automatically. Non-idempotent tool calls that reached the execution phase but whose result was not committed raise an interrupted error, telling the LLM that the tool was attempted but the outcome is unknown. The LLM then decides how to proceed—usually by checking the external system to confirm whether the effect actually happened.

But decisions are also non-idempotent. An LLM might choose a different edit strategy if asked again. To make decisions deterministic across replays, you use deterministic memoization. When the LLM makes a decision—which file to edit, what arguments to pass—you record the prompt and the chosen response paired with a unique decisionId. On replay, if the same prompt appears with the same decisionId, you return the memoized response instead of calling the LLM again. This applies to any nondeterministic decision: random sampling, temperature-based generation, current time used for timestamps. The memoization layer uses api.memo or similar, keyed on the prompt hash plus a deterministic seed. Recovery becomes a pure replay of the logged past.

Finally, not all waits require computation. When an agent is waiting on a timer, a webhook callback, or an external human approval, you park the task at zero compute cost. The task's state machine enters a parked state and releases all memory and CPU resources. A signal from the timer or webhook wakes the task and resumes from the checkpoint. This is not polling—it is a true zero-compute wait, analogous to an interrupt handler. The harness uses typed signals (alarms) that carry the exact event payload, so the woken task knows exactly what happened and can proceed without re-interpreting raw data.

spec: static diagram of a replay safety decision tree. Top box: "Crash recovery starts". Two branches: left branch labelled "Tool replay: 'safe'" leads to "Re-execute tool call" and then "Commit result". Right branch labelled "Tool replay: 'interrupted'" leads to "Raise interrupted error to LLM" and then "LLM decides: check external state or abort". A separate parallel branch from the top box: "Decision replay: memoized" leads to "Return cached LLM response from decisionId lookup".

Structured Concurrency for Subagents

Subagents are parallel conversations that explore independent tasks—researching an API, editing a module, running a validation suite. Without structure, they leak resources, produce orphaned processes, and create data races when two subagents try to edit the same file. Structured concurrency gives them a parent-child hierarchy. Every subagent is a task-owned child conversation created by a parent task. The parent owns the child's lifetime: when the parent completes, fails, or is cancelled, all its children are automatically aborted. The cancellation propagates bottom-up: if a grandchild crashes, it notifies its parent, which may then fail and propagate upward, or spawn a replacement. You never have zombie subagents running after their parent has finished.

Three fan-out combinators control how parent tasks collect results from children. failFast aborts all siblings as soon as any one child fails—useful for parallel searches where one positive result is enough. allSettled waits for every child to finish regardless of failures—useful for parallel validation where you want all errors collected. any returns the first successful result and cancels the rest—useful for race conditions like "try three approaches, take the first one that works." Each combinator has bounded concurrency limits (you don't spawn 100 subagents at once) and propagates cancellation through the same structured hierarchy. Subagents that run in the background (fire-and-forget tasks like periodic log cleanup) are isolated: they have no parent that waits on them, but they still bound their own lifetime with a lease or timeout.

The trickiest part is zero-copy conversation forks. When a parent spawns a subagent that will read the current conversation, you don't want to copy the entire message history—that wastes memory and becomes inconsistent if the parent continues talking while the child reads. Instead, the child gets a shared read-only view of the parent's conversation up to the fork point. The child then appends its own messages to a private branch. If the parent needs the child's results, it merges the branch (or receives a summary) when the child completes. Multiplayer delta watching extends this to multiple subagents observing the same source of truth: each subagent subscribes to document changes and receives only the delta (the changed fields), not the full document. This is how you can have a research subagent and an edit subagent running in parallel without stepping on each other.

spec: static diagram of a tree of subagents. Root node labelled "Main Task". Two children: "Researcher" and "Editor". Researcher has a sub-child "API Explorer". Editor has a sub-child "Linter". Arrows show parent-child ownership. A lightning bolt icon on "API Explorer" shows a crash, with a dashed arrow going up to "Researcher" labelled "abort". "Linter" shows a success checkmark, with a solid arrow up to "Editor" labelled "merge result". The "Main Task" has a stopwatch icon that shows its lifetime bounds the entire tree.

Capstone: Journaling Dedup and Crash Resume

You now integrate all four patterns into a single agent that can survive a kill -9 at any point and resume without corruption. The capstone has three deliverables. First, replace the ephemeral in-memory conversation with a SQLite journal using WAL mode. The journal stores the append-only transcript and the atomic typed documents. Every LLM call and tool execution goes through the effect sandwich: write the intent, execute, write the result. The journal's write-ahead log ensures that a crash during a commit does not lose the last intact transaction.

Second, implement requestId deduplication on every LLM call. Generate a unique requestId for each call (based on the step number and a counter). Store the request and its response in the journal keyed by requestId. On recovery, before making any new LLM call, scan the journal for pending intents: for each pending LLM call, check if its requestId matches a completed entry; if so, return the cached response. For pending tool calls, check the tool's replay attribute—safe tools are re-executed, non-idempotent tools raise interrupted errors.

Third, add subagent structured concurrency with SQLite-backed state machines. Each subagent gets its own task document with parent-child links. When the parent crashes, the recovery process reads the task tree and aborts all children whose parent is missing. Use the same requestId dedup for subagent LLM calls, so even if a subagent was midway through a decision when the process died, the recovery can reconstruct its last stable checkpoint.

Then test it: write a script that sends kill -9 at random intervals during agent execution. After each restart, confirm that the transcript is consistent—no duplicate log entries, no missing entries. Verify that non-idempotent tools never execute twice by adding a mock send_email tool that records every call to a separate durable counter. After 100 random kills, that counter should equal the number of successful send operations, not the number of times the tool was attempted. If it does, your agent is truly crash-safe.

AI-generated study article from SophistAI. Sign up free to study it with an AI tutor.

Loading SophistAI