A test that waits five seconds to find out whether a tool timed out is testing the machine's clock, not the agent's policy. The case gets riskier when the call reaches an external service, the effect happens, and the response is lost before the deadline.
To test this boundary, inject the delay into a fake executor, control timers with node:test, and observe the domain effect. The guide to checking whether a tool call had an effect before retrying covers reconciliation. This post shows how to turn those decisions into fast, repeatable tests.
The goal is not to prove that a production provider, queue, or API always behaves the same way. The fixture proves that your runtime distinguishes a timeout before the effect, a completion before the deadline, and a timeout after commit.
Short answer
- Do not use real
sleepto test a deadline. Advance the test runner's fake clock.- Inject the failure at the boundary that receives the
AbortSignal, while keeping the retry policy and real effect counter in the fixture.- Check external state, not only the agent's final message.
- If the effect happened before the timeout, mark the result unknown and reconcile before retrying.
What does a timeout test need to prove?
The test needs to answer two different questions: did the caller receive a response before the deadline, and did the external system receive an effect? The first question belongs to the runtime. The second belongs to the executor or to state it can query.
Microsoft's Dev Proxy guide to testing agent tool failures separates what a unit test of the wrapper can observe from what only a running agent reveals. It also lists paths that often escape the happy case: a tool can hang, the agent can repeat the call, or code can swallow the error.
A timeout has more than one useful position in the timeline:
| Case | What the test injects | Expected decision |
|---|---|---|
| Before the effect | The tool has not started the mutation | Timeout with no effect; policy may consider a retry |
| After the effect | The mutation happened, but the response did not arrive | Unknown state; query before retrying |
| After the response | The result arrived before the deadline | Success; no timeout should be recorded |
The article about checking a tool call's effect before retrying explains why unknown differs from failure. Here, unknown becomes an observable assertion: the test counts the mutation and prevents the runner from treating every timeout as proof of absence.
Make the deadline injectable
A test function should not hide its timer inside a wait that cannot be controlled. The wrapper below creates an AbortController, resolves the race when the deadline ends, and passes the signal to the tool. The result identifies a timeout without deciding whether an external operation was applied.
function runToolWithTimeout({ tool, timeoutMs }) {
const controller = new AbortController();
let timer;
const operation = Promise.resolve()
.then(() => tool(controller.signal))
.then(
(value) => ({ status: "completed", value }),
(error) => ({ status: "error", error }),
);
const deadline = new Promise((resolve) => {
timer = setTimeout(() => {
controller.abort(new Error("tool_timeout"));
resolve({ status: "timeout" });
}, timeoutMs);
});
return Promise.race([operation, deadline]).finally(() => {
clearTimeout(timer);
});
}
An AbortSignal interrupts the operation only if the tool observes it and passes it to libraries such as fetch. Node.js documents AbortSignal.timeout() as a native way to create a signal that expires, but this fixture uses an explicit timer so the runner can advance the clock and so domain policy stays separate from the chosen API.
This wrapper does not confirm remote state. If the tool writes before rejecting, the local result is still timeout. The next layer must query the postcondition, use the service's idempotency key, or leave the case pending.
Simulate the three moments without sleep
A useful fixture controls two instants: when the mutation happens and when the response would arrive. The signal cancels timers that have not started. If the effect timer has already fired, abort does not erase the effect. That is the case most happy-path suites miss.
function scriptedTool({ commitAtMs, responseAtMs, onEffect }) {
return (signal) => new Promise((resolve, reject) => {
let committed = false;
const effectTimer = setTimeout(() => {
committed = true;
onEffect();
}, commitAtMs);
const responseTimer = setTimeout(() => {
clearTimeout(effectTimer);
resolve({ receiptId: "receipt-1" });
}, responseAtMs);
signal.addEventListener("abort", () => {
if (!committed) clearTimeout(effectTimer);
clearTimeout(responseTimer);
if (signal.reason?.message === "tool_timeout") {
reject(signal.reason);
}
}, { once: true });
});
}
When the timeout happens before commitAtMs, the listener cancels the effect timer. When the commit has already happened, the listener only prevents the late response. The effect counter stays at one even though the agent saw only timeout.
Use node:test to advance the clock
Node's built-in runner provides mocked timers. The test runner documentation shows context.mock.timers.enable() and tick() for firing timers without waiting for the real duration. The file below runs with node --test tool-timeout.test.mjs after you include the two earlier functions.
import assert from "node:assert/strict";
import { test } from "node:test";
test("timeout before commit leaves no effect", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
let effects = 0;
const pending = runToolWithTimeout({
timeoutMs: 100,
tool: scriptedTool({
commitAtMs: 200,
responseAtMs: 400,
onEffect: () => { effects += 1; },
}),
});
await Promise.resolve();
t.mock.timers.tick(100);
assert.deepEqual(await pending, { status: "timeout" });
t.mock.timers.tick(300);
assert.equal(effects, 0);
});
test("timeout after commit preserves the ambiguous state", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
let effects = 0;
const pending = runToolWithTimeout({
timeoutMs: 100,
tool: scriptedTool({
commitAtMs: 50,
responseAtMs: 500,
onEffect: () => { effects += 1; },
}),
});
await Promise.resolve();
t.mock.timers.tick(50);
await Promise.resolve();
t.mock.timers.tick(50);
assert.deepEqual(await pending, { status: "timeout" });
assert.equal(effects, 1);
});
test("response before the deadline is not a timeout", async (t) => {
t.mock.timers.enable({ apis: ["setTimeout"] });
let effects = 0;
const pending = runToolWithTimeout({
timeoutMs: 100,
tool: scriptedTool({
commitAtMs: 200,
responseAtMs: 50,
onEffect: () => { effects += 1; },
}),
});
await Promise.resolve();
t.mock.timers.tick(50);
assert.deepEqual(await pending, {
status: "completed",
value: { receiptId: "receipt-1" },
});
assert.equal(effects, 0);
});
The three cases verify ordering, not the passage of time. The first proves that policy can end a slow read without an effect. The second proves that a timeout does not authorize a second mutation. The third prevents an old timer or a broad assertion from classifying a success as a failure.
How do you test the retry without duplicating the mutation?
Do not put the retry inside the same test that is meant to prove the timeout. First fix the observation: timeout with zero effects, or timeout with an effect already applied. Then test the recovery policy with a verification function that returns confirmed, not_found, or unknown.
The important path is:
- The executor accepts the operation.
- The fixture records the effect.
- The response is lost before reaching the agent.
- The runtime receives
timeout. - The policy queries external state.
- A
confirmedresult ends the action without a second call.
The API idempotency testing guide covers receiver-side protection. The timeout test remains necessary because an idempotency key can exist in the code without ever being exercised by an assertion about the effect.
Where do timeout tests usually lie?
Waiting in real time
A five-second setTimeout makes the suite slow and can still pass without exercising cancellation. Use mocked timers to prove the transition, then use a smaller integration test for the library or transport that actually accepts the AbortSignal.
Mocking the whole agent
If the mock already returns completed or timeout, the parser, dispatcher, effect counter, and policy are outside the path. Control the slow dependency and keep the decision boundary real.
Counting calls only
A call may have produced an effect before failing. The counter helps, but the main assertion should inspect fixture state, such as a message list, a database row, or a receipt linked to an actionId.
Rejecting and cleaning up everything
Cancelling a local promise does not undo a remote mutation. The article about cancelling an agent when a tool hangs explains the difference between interrupting a wait and erasing an effect. Keep that difference visible in the test.
Confusing a clock with a deadline
The guide to testing time-dependent JavaScript separates the current date from timers. Here, the tool deadline is an execution limit. If the agent also has a run deadline, test that limit in another layer or document which timer is being advanced.
CI checklist
- Can the timeout fire without real
sleep? - Does the test inject the failure at the boundary that receives the cancellation signal?
- Is there a case before the effect, one after commit, and one with a response within the deadline?
- Does the suite check external state rather than only the final message?
- Does a timeout after commit block automatic retry until reconciliation?
- Does the fixture use the same
actionIdor idempotency key across attempts? - Does the test state what it cannot prove about the real provider or service?
Frequently asked questions
Is AbortSignal.timeout() enough to test a tool?
Not by itself. It creates a signal that expires, but the test still needs to prove that the tool observes the signal and that policy classifies the result. An injectable timer wrapper can make explicit clock advancement easier to verify.
Does a timeout before the response mean the effect did not happen?
No. If the external operation was already accepted, the effect may exist even though the client did not receive the receipt. Model that case and query the postcondition before retrying.
Can I test an agent with a real model?
You can, but that is another layer. The deterministic test should protect the parser, timeout, trace, policy, and effect. A model-based eval can add choice variation, but it should not replace runtime assertions.
How this article was produced
Samuel Fajreldines is responsible for this article. The research compared Node.js documentation for AbortSignal and mocked timers, Microsoft's guidance on tool failures, and the OpenAI Agents SDK documentation for tool timeouts. Public discussions helped choose the timeout-after-commit case; they were not used as technical proof.
I ran the in-memory fixture with Node.js v25.5.0 on November 17, 2026. It covers a timeout before the effect, a timeout after commit, and a response before the deadline. That proves the fixture's declared behavior, not the semantics of an external provider, database, queue, or API.
I used RemoteCode as my tool for organizing context and reviewing the structure. It did not run the fixture or provide production evidence.
Sources consulted
- Node.js, "Global objects: AbortSignal.timeout()", retrieved November 17, 2026.
- Node.js, "Test runner", retrieved November 17, 2026.
- Microsoft Learn, "How to test what your AI agent does when its tools fail", retrieved November 17, 2026.
- OpenAI Agents SDK, "Tools", retrieved November 17, 2026.