An agent should not resume by loading the conversation and asking the model to continue. The conversation records what was said. It does not prove that a tool finished, that an external system applied the effect, or that the response survived the process that made the call.

After a crash, load the last confirmed checkpoint, recover the intent and result for each step, and treat the first call without a receipt as a recovery decision. If the provider accepts an idempotency key, look up or repeat the operation with the same key. If there is no reliable way to learn what happened, mark the outcome as unknown and stop before producing another change.

This article is a spoke of durable execution for AI agents. The owner page explains when a queue, checkpoint, or workflow fits. This post focuses on the failure between an external effect and the local record, with a Node.js fixture that you can interrupt and restart.

Diagram shows an agent resuming from a checkpoint and checking an unknown effect before repeating a tool call.

Key Takeaways

  • Save the model's decision before running tools.
  • Save each tool result before starting the next step.
  • Generate one stable key per effect and reuse it on a repeat.
  • After a crash, check the external system before deciding whether to run the operation again.
  • If the outcome remains unknown and a repeat could cause harm, pause for human or operational reconciliation.

Why conversation history cannot recover the run

Imagine that an agent receives a request, asks create_ticket to run, and the process dies just after the ticket system accepts the operation. The history may contain the tool call, but it may not contain the ticket ID or a confirmation that survives the process.

If the restarted worker sends the entire conversation back to the model, the model can choose a different action. Even if it chooses the same tool, a new request can create a second ticket. Lost context is only part of the problem. The deeper mistake is treating a recorded intent as a confirmed effect.

The AWS guide to durable execution for agents describes the mechanism as step, checkpoint, and replay: each step's result lives outside the process and recovery continues from those records. The Cloudflare Agents task documentation adds an important limit: an interrupted step can run again, so an external effect needs a stable key or durable progress read.

That is why a trace is not a receipt. A trace helps investigate the path. A receipt, the provider's returned ID, or a query against the system that changed helps decide what can happen next.

What state should survive the process?

The recovery record does not need to keep the entire conversation in one document. It needs enough information for the runtime to choose the next step without relying on the worker's memory:

Record Purpose
runId and flow version Finds the run and avoids continuing with an incompatible definition
model turn and calls Replays the selected intent without asking the model to choose again
call ID and arguments Identifies the logical operation that may be repeated
idempotency key Connects attempts for the same operation in the external system
result or receipt Lets the runtime skip a completed step
unknown status Prevents a communication failure from being treated as proof of no effect
lease or fencing token Prevents two workers from advancing the same run at once

Write the record to storage that survives process death. An atomic file is enough for a local fixture. In production, use a database, durable volume, or workflow engine storage with concurrency control and a retention policy.

Microsoft Agent Framework checkpoint documentation describes snapshots that include executor state, pending messages, requests, and responses. The same page requires a rebuilt workflow to preserve its topology and executor identities. The idea applies beyond that framework: a checkpoint must remain compatible with the code that interprets it after a deployment.

What to do in each crash window

The failure point changes the recovery. Do not use one retry path for every situation.

Failure point What you know Next action
Before sending the tool call The effect was not requested Send it with the persisted operation key
After sending, without a response The effect may have happened Look up the key or read the target state
After the effect, before the checkpoint The effect exists, but the local record does not know Look it up or repeat with the same key, never a new one
After saving the receipt The effect and step are confirmed Reuse the result and advance
No reliable lookup and duplicate risk The state is still ambiguous Mark unknown, pause, and reconcile

A retry is another attempt at an operation. Resumption is continuing a run from the last confirmed state. A retry can be part of resumption, but only after the runtime classifies the effect.

How to implement a small journal in Node.js

The example below uses two files to make the boundaries visible. agent-run.json is the runtime journal. effects.json represents the external provider, which recognizes the same operation key. The model is replaced with a fixed plan so the test does not depend on network access, tokens, or a nondeterministic choice.

Save this as agent-recovery.mjs:

import { existsSync, readFileSync, renameSync, writeFileSync } from "node:fs";

const RUN_FILE = "agent-run.json";
const EFFECTS_FILE = "effects.json";
const runId = "ticket-77";

const plan = [
  [{ id: "c1", tool: "create_ticket", args: { title: "Login fails" } }],
  [{ id: "c2", tool: "send_email", args: { to: "dana@example.com" } }],
  [],
];

function load(file, fallback) {
  return existsSync(file) ? JSON.parse(readFileSync(file, "utf8")) : fallback;
}

function saveAtomically(file, value) {
  const temporary = `${file}.tmp`;
  writeFileSync(temporary, JSON.stringify(value, null, 2));
  renameSync(temporary, file);
}

function model(turns) {
  return plan[turns] ?? [];
}

function providerLookup(key) {
  const effects = load(EFFECTS_FILE, {});
  return effects[key] ?? null;
}

function providerExecute(tool, args, key) {
  const effects = load(EFFECTS_FILE, {});
  if (effects[key]) return effects[key];

  const result = {
    providerId: `${tool}-1`,
    tool,
    args,
    confirmedAt: "fixture",
  };

  // A real provider must persist this key with the effect.
  effects[key] = result;
  saveAtomically(EFFECTS_FILE, effects);
  return result;
}

async function run() {
  const state = load(RUN_FILE, { runId, turns: [] });

  for (let turnIndex = 0; ; turnIndex += 1) {
    if (!state.turns[turnIndex]) {
      state.turns[turnIndex] = {
        calls: model(turnIndex),
        results: {},
      };
      // Persist the model decision before any effect.
      saveAtomically(RUN_FILE, state);
    }

    const turn = state.turns[turnIndex];
    for (const call of turn.calls) {
      if (turn.results[call.id]) {
        console.log(`skip ${call.tool}: result already saved`);
        continue;
      }

      const key = `${runId}:${turnIndex}:${call.id}`;
      const known = providerLookup(key);
      const result = known ?? providerExecute(call.tool, call.args, key);

      if (process.env.CRASH_AFTER_EFFECT === "1" && !known) {
        console.log(`crash after effect: ${call.tool}`);
        process.kill(process.pid, "SIGKILL");
      }

      turn.results[call.id] = result;
      saveAtomically(RUN_FILE, state);
      console.log(`${call.tool}: ${known ? "receipt found" : "effect applied"}`);
    }

    if (turn.calls.length === 0) {
      console.log("run completed");
      return;
    }
  }
}

run().catch((error) => {
  console.error(error);
  process.exitCode = 1;
});

The example has two important properties. First, it saves the model output before the first tool. A restart does not call the model to rediscover the plan. Second, providerLookup checks the external system before providerExecute. When the process dies after providerExecute and before saving agent-run.json, the next run finds the receipt in effects.json and records it in the journal without creating a second effect.

effects.json is a simulation. It represents a guarantee your provider must offer, such as idempotency by key, a status query, or an operation table. A local file controlled by the same process does not prove atomicity in a distributed system.

How to crash and verify the resume

Run the first process with the injected failure:

rm -f agent-run.json effects.json
CRASH_AFTER_EFFECT=1 node agent-recovery.mjs || true

The first run writes the ticket effect and stops before writing the receipt to the journal. Restart the same run:

node agent-recovery.mjs
node agent-recovery.mjs

On the second run, create_ticket should print receipt found, and send_email should be applied once. On the third, both calls already have journal results. The final output should contain run completed, and effects.json should contain one entry for each key.

This test proves three harness decisions: the plan was not recalculated after the crash, the uncertain effect was reconciled with a stable key, and the recovered result was saved before continuing. It does not prove that a real provider treats the key idempotently, that two workers cannot run together, or that an external transaction is atomic.

When do you need a lease and a flow version?

The example assumes one process per run. In production, a deployment can start a new worker while the old one is still stuck in a network call. An idempotency key reduces the risk, but it does not replace ownership of the run.

Use an expiring lease and a fencing token when more than one worker can compete for the same run. The worker should renew the lease, validate the token before advancing the checkpoint, and lose the ability to write when the lease expires. The update should also check the expected state version so a delayed write is rejected.

Pin the flow version per run. A run started with a different tool description, argument format, or transition rule may not be compatible with an old journal. Microsoft Foundry's crash-resilient agent walkthrough shows the same principle: the server stores the response, checkpoints each stage, and skips the checkpointed stage after a restart. In that example, the resilience option must be enabled explicitly.

I use RemoteCode as my own tool when I need continuity between Codex and Claude Code sessions. It can reduce repeated context, but it does not replace the journal, lease, idempotency, or external-effect lookup.

When should recovery stop instead of retrying?

Stop when you cannot answer one of these questions:

  • Which logical operation is being recovered?
  • Does the external system offer an idempotency key or a reliable lookup?
  • Can you observe the effect without producing another change?
  • Has the old worker lost its lease and its ability to write?
  • Do the approval, budget, and flow version still apply?

If the answer is no and the operation can send a message, create a record, change a permission, or move money, unknown is better than failed. failed suggests that nothing happened. unknown keeps the obligation to reconcile before trying again.

This rule complements how to verify an AI agent tool call before retrying. That post focuses on checking one call. Here, the check is part of the full restart protocol, which begins at the checkpoint and ends with a recovered result or an explicit pause. For a task that depends on a broker and redelivery, see how to test a queue worker without duplicate side effects.

A workflow such as Temporal, Step Functions, LangGraph, or another durable runtime can take on part of the journal and replay. That reduces infrastructure code, but it does not make an external API exactly once. Check retry semantics, storage, interrupted-step behavior, and how to pass a key to the provider before relying on the product name.

How this article was produced

Samuel Fajreldines is responsible for this article. The research combined current AWS, Microsoft, and Cloudflare documentation, live-result inspection, recent public discussions, and comparison with the site's existing durable-execution cluster. The fixture ran locally with Node.js and uses only temporary files and a fake provider. It did not call a real payment, email, ticket, or external API.

I used AI assistance to discover sources, compare pages, organize a first draft, create the SVG illustration, and review consistency across languages. The assistance did not run a real agent, measure a production system, or replace verification of the guarantees offered by your provider.

Sources consulted