An agent does not recover from the model. It recovers from the place where the system records progress, confirms what already happened, and decides whether a step can be repeated. Without that layer, a restart turns completed work into lost context, while a partially applied external call becomes a duplication risk.
Durable execution is the contract that lets an agent pause, fail, wait for approval, and continue from confirmed state. A checkpoint, a queue, and a workflow can all help meet that contract, but they are not the same thing. The choice depends on the unit of work, the kind of external effect, and the recovery behavior the team needs to operate.
Quick decision
- Use an in-process loop only for short, reversible work that can restart without consequences.
- Add a checkpoint and a queue when each unit of work has an identifier, persisted state, and a resumable step.
- Use a state machine or workflow when the run has waits, signals, compensation, fan-out, replay, or execution history to audit.
- Treat every external write as potentially applied until the system confirms otherwise.
What must durable execution guarantee?
Durable execution means that a task's progress survives the process that started it. The Temporal Workflow documentation describes this principle through event history and replay: the workflow rebuilds its state from recorded events, while external calls run as Activities and are not performed again during replay.
For an agent, the contract must answer a few practical questions. What state has been confirmed? Which step is in progress? What can be repeated without duplicating an effect? What evidence allows the run to continue or ask for human help? The model conversation can be part of the state, but it should not be the only place where those answers exist.
A recoverable run is more than a session that keeps messages. It also records decisions, tool results, transitions, attempts, and external effects. If the process dies after sending a request, the system needs to know whether to query the state, wait for confirmation, or perform a compensating action.
This design changes the agent's role. The model suggests the next action and can explain a result. The runtime confirms the transition, applies retry limits, and decides whether the task is done, failed, or needs-human.
Checkpoint, queue, or workflow: which layer solves the problem?
The three layers address failure at different levels. A checkpoint persists progress. A queue delivers work to a consumer and allows the process to be replaced. A workflow organizes transitions, waits, signals, retries, and history for a run that may outlive a single process.
| Layer | What it persists | When it is often enough | What remains your responsibility |
|---|---|---|---|
| Checkpoint | State for a step or graph | An agent needs to continue after an interruption | State reads, leases, retries, and idempotency |
| Queue with worker | Task and processing attempt | Work can be split into independent units | Ordering, deduplication, timeouts, and coordination |
| State machine | Allowed states and transitions | The flow has clear rules and few valid paths | Persistence, effect execution, and observability |
| Durable workflow | Event history and signals | The flow waits, branches, calls external systems, or needs replay | Modeling Activities, versioning definitions, and cost control |
The mistake is choosing by the tool's name. Start with what needs to be proven. If a task only needs to remember that a read step finished, a checkpoint may be enough. If independent tasks arrive separately, a queue creates a clear operational boundary. If an agent must wait for approval and continue from the right point without losing earlier effects, the flow's state needs to be explicit.
When is a queue with a checkpoint enough?
A queue with a checkpoint works well for an agent that processes independent tasks with short steps and state that fits in one record. The worker receives a taskId, loads the last confirmed step, performs an action, records the result, and releases the task. Another worker can continue after a crash.
The minimum contract can be small and typed:
type RunState = "queued" | "running" | "waiting" | "done" | "failed" | "needs-human";
type AgentRun = {
id: string;
state: RunState;
step: string;
checkpoint: string | null;
lastEvidence: string | null;
idempotencyKey: string;
};
type RunStore = {
load(id: string): Promise<AgentRun>;
save(run: AgentRun): Promise<void>;
};The important part is not the type itself. The transition must be persistent, observable, and conditional on the previous state. A worker should not change running to done just because the model replied. It should save the evidence that confirms the expected result.
The queue lease also needs a policy. If the worker disappears, another worker may take the task after a timeout. If the first worker is still alive, both may attempt the same step. The store must therefore accept an update only while the expected lease and checkpoint remain valid.
This pattern fits queue consumers, evaluation jobs, file processing, and coding-agent tasks that produce an isolated artifact. It becomes harder to maintain when every step sends signals, waits for external responses, creates subagents, or needs compensation across multiple systems.
When does a state machine make an agent safer?
A state machine makes the actions available in each phase explicit. The model can classify an event or suggest an action, but code decides whether the transition is allowed. That prevents a plausible response from skipping authorization, marking a task complete before verification, or repeating a write that was already confirmed.
A simple flow can separate proposal, authorization, execution, confirmation, and verification:
const transitions: Record<RunState, RunState[]> = {
queued: ["running"],
running: ["waiting", "done", "failed", "needs-human"],
waiting: ["running", "failed", "needs-human"],
done: [],
failed: ["queued", "needs-human"],
"needs-human": ["queued", "failed"],
};
function canMove(from: RunState, to: RunState) {
return transitions[from].includes(to);
}The example does not execute the agent. It limits the runtime's decision space. The implementation still needs to persist the change, check concurrency, and record why the transition occurred. A state machine without durable storage still loses its place when the process dies.
This layer is especially useful for agents that access permissioned APIs, change data, wait for human approval, or coordinate subagents. The post on multi-agent orchestration with TypeScript explores supervisors, dependencies, and recovery. Here, the question is which contract keeps the entire flow recoverable.
What changes in a coding agent?
Imagine a coding agent that receives an issue, inspects the repository, changes files, and runs a patch check. If the worker dies after writing the diff, resume should not begin by reading the entire codebase. It should load the task state, reference commit, changed files, and last confirmed command.
If verification has not finished, the runtime can start that step again. If the commit already exists, the write step should not run again without checking repository state. If the test failed, the agent can receive the log and a correction hypothesis, but the system should keep the run incomplete until a new proof passes.
This design connects durable execution to the loop that sends CI failures back to the agent. The agent corrects one hypothesis at a time; the runtime keeps the history and limits the recovery path. An interrupted session therefore does not erase confirmed work or turn the conversation into the only source of truth.
When does a dedicated workflow pay off?
A dedicated workflow pays off when the system needs to treat a run as a long-lived entity. The task may wait for a signal, resume after a deployment, run parallel branches, apply compensation, or keep a history that another person can inspect.
AWS Step Functions models workflows as state machines with transitions, wait states, choices, maps, and parallel branches. Its error-handling documentation separates Retry from Catch, which makes recovery a declared property of the flow.
Temporal uses event history and replay to rebuild workflow state. Its Retry Policy documentation recommends putting failures from external operations in Activities, where retry policy can distinguish temporary failures from errors that should not be repeated.
LangGraph documents checkpoints as graph-state snapshots and preserves pending writes when one node fails while others complete. Its persistence documentation also warns that memory-only storage loses checkpoints after a restart and that an unbounded history can keep growing.
These tools do not remove domain decisions. A workflow may retry an Activity, but it cannot know by itself whether a charge was applied before a timeout. A checkpoint can restore graph state, but it does not authorize a tool. The durable layer stores what the system knows; business code decides what may happen next.
Retry is not resume
Retry means trying an operation again after it fails. Resume means continuing a run from the last confirmed state. Sometimes they overlap. Often they do not.
An idempotent read can usually be repeated with little risk. An external write needs an idempotency key, a confirmation query, or a compensating operation. Imagine that a timeout occurs after the provider receives the request. The local error does not prove that nothing was applied.
The flow should record the boundary between intent and effect. Before execution, record the intent and the key. After execution, record the response or observed state. On resume, query that evidence before calling the tool again. The agent can help interpret the response, but it should not invent confirmation.
The same principle applies to a model call. If a step generates a plan and fails while saving the result, repeating the call may produce a different output. Decide whether the plan is an artifact that needs deduplication, can be regenerated, or should go to human review.
How do you verify that the architecture really resumes?
The most useful test interrupts the system where a demo would usually hide the risk. Starting the worker and watching a final response is not enough. The proof must inspect the execution history.
- Start a task with an identifier and record its first checkpoint.
- Stop the process after a tool returns but before the next transition is saved.
- Start another worker and check which state it loads.
- Verify that the confirmed step was not executed again without an idempotency reason.
- Force a slow response, a permanent failure, and a timeout to confirm that the policies differ.
- Pause the task while it waits for approval and resume it later without keeping the original process alive.
- Inspect the final evidence, terminal state, and path that led to it.
For coding agents, add the commit, diff, tests, and produced artifact to the run record. An agent can say that it finished a change. The system should show which file changed, which verification passed, and which risk remains open. Agent observability in CI covers this evidence in the pipeline.
Is context part of recovery?
Resuming does not mean loading the entire old conversation into the prompt. The system should persist compact state, tool results, and references to artifacts. On resume, it retrieves only the context needed for the next transition.
During long runs, RemoteCode helps Claude Code and Codex continue agentic flows with less repeated context. The tool belongs to the author and appears here as an option for reducing context sent between sessions. It does not replace checkpoints, authorization, idempotency, or result verification.
This separation avoids two kinds of waste. The system does not pay again for the model to reread everything already confirmed, and the agent does not receive such a large history that it hides the current state. The checkpoint stores progress; the recovery mechanism builds the projection the next step needs.
Common mistakes and limits of the decision
Calling any persistence durable execution is a poor starting point. Saving messages in a database does not define transitions, detect an abandoned worker, or prevent a tool from running again. Storage must carry enough state to decide the next step.
It is also easy to put retries inside the agent loop and leave the runtime without visibility. An unlimited retry can consume context, repeat expensive calls, and hide a permanent error. The policy must sit outside the model's free-form decision.
Choosing a workflow because its diagram looks more complete creates another problem. A short read may be easier to operate with a simple queue or synchronous call. The workflow should pay for itself when the run waits, branches, compensates, or needs a reliable history.
Finally, a checkpoint is not long-term memory. A checkpoint keeps the state of one run. Persistent memory stores facts that cross runs. The layers are related, but their retention, authorization, and tests are different.
Frequently asked questions about durable execution for AI agents
Does every AI agent need a durable workflow?
No. A short agent without irreversible effects, running inside one request, can start with a simple loop. The need appears when work must survive a restart, wait for a person, consume a queue, repeat a step safely, or prove what happened. At that point, persist state before choosing a product.
Is a checkpoint the same as a retry?
No. A checkpoint records where the run is and which results are confirmed. A retry attempts an operation again after failure. To resume safely, the runtime must load the checkpoint and decide whether the step fails, continues, or first checks an external effect that may already have happened.
How do you stop a retry from duplicating an external call?
Use an idempotency key when the provider supports it. Also record the intent, query the operation state before repeating it, and treat a timeout as an unknown state until confirmed. The agent should not mark the action complete only because the call was sent.
Can a queue replace Temporal, Step Functions, or LangGraph?
Sometimes. A queue with persisted state can be enough for independent tasks and short steps. It does not automatically provide signals, replay, human waits, valid transitions, compensation, or workflow history. Compare the contract you need to operate, not the number of components in the architecture.
Where should agent state live?
Outside process memory. Use storage that records the task identifier, state, step, checkpoint, evidence, and idempotency key. The conversation and model results can be references in that record, but the application must be able to decide the next step after the original worker disappears.
