When an agent fails after calling an API, reading a database, or changing an external system, repeating the prompt rarely reproduces the failure. The model may answer differently, the API may have changed, and the execution may have observed different state.

Replay addresses a narrower problem: reconstructing a run from recorded events, replacing the model and tools with recorded responses, and stopping any attempt to reach live effects. You can inspect the first divergence without turning diagnosis into another production operation.

This article explains the mechanism, a minimum cassette format, and a runnable JavaScript fixture. To separate state, tools, and recovery, pair this approach with durable execution for AI agents. Replay diagnoses a known run; it does not resume an interrupted run.

Short answer: record the inputs and outputs at each relevant boundary, give replay only those events, make every live tool adapter fail explicitly, and compare the run until its first divergence.

Replay is not resuming a run

Three related problems need different contracts:

Mechanism Question it answers What happens to the outside world?
Observability What happened, and at which step? The original run may already have produced effects.
Replay How do I reproduce the observed behavior? External responses come from fixtures or cassettes.
Recovery How do I continue after an interruption? The system needs checkpoints, confirmed effects, and a retry policy.

A trace helps locate a step, but it is not necessarily enough to rerun it. OpenTelemetry traces describe spans and attributes for observing operations. A replay cassette needs more: the data required to answer the boundaries the agent would cross again.

That is also why an audit log should not automatically become a fixture. An AI agent audit trail may be minimized or redacted for accountability, while replay needs enough data to reproduce a decision. They can share events, but their retention and access controls serve different purposes.

What to record during the run

Start with boundaries that can change the next step. In a typical run, they include:

  • the normalized agent input and configuration that affects planning;
  • the model request and response, including model identity and relevant options;
  • each tool name, arguments, result, or error;
  • search results, retrieved documents, and state reads that change the decision;
  • clock, random values, and generated IDs only when the run depends on them.

Do not record everything by reflex. A raw tool payload may contain a token, personal data, or a secret. Create a policy that separates the value needed to reproduce a decision from the value that must be removed, masked, or replaced with a marker. OpenTelemetry GenAI semantic conventions can help name event parts, but they do not replace your privacy policy or make a trace a ready-made cassette.

Each event needs a stable identity and a verifiable order. A starting format might look like this:

{
  "runId": "run-2026-10-15-001",
  "events": [
    {
      "seq": 1,
      "kind": "model",
      "requestHash": "sha256:...",
      "response": { "tool": "lookupOrder", "args": { "id": "ord-7" } }
    },
    {
      "seq": 2,
      "kind": "tool",
      "name": "lookupOrder",
      "argsHash": "sha256:...",
      "response": { "status": "ready" }
    }
  ]
}

A hash helps reveal that replay received different input. It is not proof of integrity by itself: define the algorithm, normalize data before hashing, and protect the cassette from unauthorized changes.

How to block live tools during replay

The block belongs in the execution path, not only in a test naming convention. A production adapter that can send email, create an order, or delete a record must be unreachable when the mode is replay.

The fixture below uses Node's test module, whose official API lets you run it with node --test. The agent receives a real adapter, but replay never calls it. Instead, it consumes the recorded sequence and fails as soon as the expected step does not match the available event.

const assert = require("node:assert/strict");
const test = require("node:test");

const cassette = [
  { kind: "model", response: { tool: "lookupOrder", args: { id: "ord-7" } } },
  { kind: "tool", name: "lookupOrder", response: { status: "ready" } },
  { kind: "model", response: { final: "The order is ready." } },
];

function createReplay(events) {
  let cursor = 0;

  return {
    next(expectedKind) {
      const event = events[cursor];
      if (!event || event.kind !== expectedKind) {
        throw new Error(
          `first divergence at event ${cursor}: expected ${expectedKind}`,
        );
      }
      cursor += 1;
      return event;
    },
    done() {
      assert.equal(cursor, events.length, "replay left events unconsumed");
    },
  };
}

async function runInReplay(replay, realTool) {
  const plan = replay.next("model");
  const recordedTool = replay.next("tool");

  assert.equal(recordedTool.name, plan.response.tool);
  assert.deepEqual(recordedTool.response, { status: "ready" });

  // This argument makes an accidental live call detectable.
  void realTool;

  const final = replay.next("model");
  replay.done();
  return final.response.final;
}

test("replay uses the cassette and does not call the real tool", async () => {
  let realCalls = 0;
  const realTool = async () => {
    realCalls += 1;
    throw new Error("real tool reached during replay");
  };

  const result = await runInReplay(createReplay(cassette), realTool);

  assert.equal(result, "The order is ready.");
  assert.equal(realCalls, 0);
});

The example has two important properties. The first is negative: if the real tool is called, the test must fail. The second is diagnostic: the cursor points to the first event that stopped matching. A failure at the end of a run may only be the consequence of a divergence many steps earlier.

In a production agent, createReplay is usually a more complete dispatcher. It should verify operation identity, normalized arguments, schema version, and sequence. If a recorded response is missing, the safe mode is to fail with context, not silently fall back to the live API.

How to find the first divergence

Compare events in an order that preserves causality. A difference in final text is often not very informative. A difference in a tool argument, retrieval result, or state read that fed the decision usually is.

Record at least:

  1. the event position;
  2. the boundary type, such as model, tool, retrieval, or state;
  3. the operation identity;
  4. a redacted representation of expected and received input;
  5. the response source, such as cassette, fixture, or live, with live forbidden in replay.

Do not automatically compare every volatile field. A timestamp, trace ID, or random identifier may differ without changing behavior. On the other hand, ignoring a tool argument can hide the defect you are looking for. Define equivalence per event type and document which fields are semantic.

After changing the agent, replay important cases and group failures by their first divergence. Regression testing for agents without exact text matching helps turn that difference into a durable behavior assertion. Replay supplies the concrete path; the regression test prevents the fix from disappearing in the next change.

Retention, redaction, and safe forks

A cassette may contain more information than the log you normally keep. Before persisting it, define:

  • which fields are secret, personal, or regulated;
  • who can read, export, and invalidate the cassette;
  • how long it remains useful for diagnosis;
  • how prompt, schema, and tool versions are marked;
  • how to prevent an edited cassette from looking like the original run.

For an investigation, prefer an immutable fork. The original replay should remain the incident reference. Changes to the fork should appear as a new entry with the reason for the change and the observed result. This keeps “the run that happened” separate from “the behavior I wanted to test.”

If the cassette came from a run that produced real effects, replay does not undo those effects. It only avoids new ones. To understand that boundary, observe an agent run and its tool calls and keep the audit trail separate from the material used for simulation.

When replay should become a regression test

Not every cassette belongs in the suite. Promote one when it represents an important failure, a sensitive business decision, or a boundary that has already broken. Remove data that is not needed for the assertion and keep the expectation at the behavior level:

  • the correct tool was selected;
  • arguments respected the contract;
  • an error response led to the expected fallback;
  • no forbidden effect occurred;
  • the relevant final state was reached.

Do not freeze a model response as eternal truth. Replay can test how the system reacts to one specific response, but it does not prove that the model will keep producing that response in production. Keep dispatcher tests, tool contract tests, and model behavior evaluations separate when the question changes.

What this method does not prove

Replay is a controlled approximation, not a copy of the world. It does not prove that:

  • an external API will remain available or keep the same contract;
  • real database or service state will still match;
  • latency, concurrency, and rate limits will have the same effect;
  • the model will produce the same output outside the cassette;
  • a side-effecting tool is safe to execute again.

Recent research on deterministic replay for AI agent systems addresses the sources of nondeterminism between the model, external APIs, and the environment. Open-source projects such as agentrr also make the idea explicit by capturing model, tool, clock, and randomness boundaries for offline execution. These references help validate the mechanism, but they do not replace tests for your adapters or a data policy appropriate to your system.

Quick questions

Can I replay using only a trace?

Sometimes, but not by default. If the trace contains every input and output needed, it can be the cassette base. If it contains only span names, durations, and IDs, responses are still missing.

Should I call the live model during replay?

Not when the goal is to reproduce one specific run. Calling the model again turns replay into a new execution subject to another output. Run a separate test when you want to measure current model behavior against the same input.

How do I stop a live tool from escaping?

Make the production adapter fail by default in replay mode, inject cassette implementations, and keep a test that counts or forbids live calls. The protection should also exist in CI, not just on a developer's machine.

How this article was produced

I am Samuel Fajreldines, the author of this site. The executable part was designed as a local fixture with no model provider, database, or external API call. The test checks that the recorded sequence is consumed, the final response is preserved, and the real adapter is not called. This proves the example's contract, not the safety of a production runtime.

When a session needs to remain visible across machines and sessions, I use RemoteCode as the author's tool. It was not used to execute the fixture and is not evidence of the method's performance.

Sources consulted

Developer community discussions helped identify the language of the problem, especially the difference between caching tool responses and resuming a session. They were used as discourse signals, not technical proof.