How to Test an AI Agent Without Extra LLM Calls

An agent test can pass because a mock returned exactly the answer you wrote. The runner may never have processed the tool, checked state, or stopped at the right point.

To test the agent code without making another provider request, replace the model boundary with a local sequence. Keep the runner, dispatcher, and state fakes under test. Then check the trajectory and the effect instead of checking only the final sentence.

This is a different layer from testing an agent with a mock or real model. That post helps you choose the dependency. This one builds the fast test that needs no secret and shows exactly what it can prove.

Short answer

• Replace the model client with scripted replies, not the whole runner.
• Use observable fake tools with no network access.
• Check calls, arguments, state, and loop termination.
• Keep a real model for choices, prompts, and emergent behavior.

What does this test control, and what does it leave open?

The test controls the input that the runner receives from the model and the dependencies owned by your code. That lets you reproduce a tool call, a known error, an incomplete reply, or an extra turn. Anthropic's "Demystifying evals for AI agents" separates tasks, trials, graders, and traces as different parts of an evaluation.

Use this map before writing the fixture:

The last item is the important limit. A fake proves that your code handles a known decision. It does not predict how a model will interpret a new instruction. The current OpenAI guide to evaluating agent workflows also separates traces, graders, datasets, and evaluation runs. Do not merge those questions into one green check.

How do you connect a scripted model to the real runner?

Start with the interface the runner already needs. The fake should know only the input and output shapes, not the loop's internal rules. On each call, it returns the next scripted step. If the runner asks for a step that does not exist, the test fails instead of inventing a convenient answer.

The example below uses only Node.js modules. The Node.js test runner documentation supports asynchronous tests that return a Promise. This fixture imports no provider SDK and opens no network connection.

const test = require("node:test");
const assert = require("node:assert/strict");

function scriptedModel(...replies) {
  let position = 0;
  const calls = [];

  return {
    calls,
    async complete(input) {
      calls.push(input);
      const reply = replies[position++];
      if (!reply) throw new Error("unexpected model call");
      return reply;
    },
    assertConsumed() {
      assert.equal(position, replies.length, "the script was not fully consumed");
    },
  };
}

The assertConsumed method protects against a common false green. If the runner stops early and ignores an expected step, the fixture must report it. The same applies to an extra call: it should fail before reaching an external dependency.

How do you keep the loop and tools observable?

The runner remains the important object under test. It reads a model reply, dispatches the tool, sends the result into the next turn, and stops on a final reply. The fake tool should not be a hidden fixed return. It should record its argument and change a small state that the test can inspect.

async function runAgent({ model, tools, input }) {
  const messages = [{ role: "user", content: input }];
  const trace = [];

  for (let turn = 0; turn < 4; turn += 1) {
    const reply = await model.complete({ messages });

    if (reply.type === "final") {
      return { answer: reply.text, trace };
    }

    if (reply.type !== "tool") {
      throw new Error("invalid model reply");
    }

    const tool = tools[reply.name];
    if (!tool) throw new Error(`unknown tool: ${reply.name}`);

    trace.push({ name: reply.name, args: reply.args });
    const result = await tool(reply.args);
    messages.push({ role: "assistant", content: JSON.stringify(reply) });
    messages.push({ role: "tool", content: JSON.stringify(result) });
  }

  throw new Error("turn limit exceeded");
}

test("runs the real loop with a local scripted model", async () => {
  const orders = new Map([["ord-7", { status: "paid" }]]);
  const calls = [];
  const model = scriptedModel(
    { type: "tool", name: "getOrder", args: { id: "ord-7" } },
    { type: "final", text: "Order ord-7 is paid." },
  );

  const result = await runAgent({
    model,
    input: "What is the status of ord-7?",
    tools: {
      async getOrder({ id }) {
        calls.push({ name: "getOrder", id });
        return orders.get(id);
      },
    },
  });

  assert.equal(result.answer, "Order ord-7 is paid.");
  assert.deepEqual(result.trace, [
    { name: "getOrder", args: { id: "ord-7" } },
  ]);
  assert.deepEqual(calls, [{ name: "getOrder", id: "ord-7" }]);
  assert.equal(model.calls.length, 2);
  model.assertConsumed();
});

test("rejects a tool that is not in the test boundary", async () => {
  const model = scriptedModel({
    type: "tool",
    name: "deleteOrder",
    args: { id: "ord-7" },
  });

  await assert.rejects(
    () => runAgent({ model, input: "Delete ord-7", tools: {} }),
    /unknown tool: deleteOrder/,
  );
  assert.equal(model.calls.length, 1);
});

Vitest's function mocking documentation offers equivalent APIs for controlling returns and observing calls. The principle does not depend on Vitest: the model is replaceable, while the loop, tool, and state remain visible.

I ran this fixture with Node.js v25.5.0. It makes two calls to the local fake, no provider calls, and keeps the order in a Map. That proves the dispatcher and runner contract in this scenario. It does not prove authentication, SDK serialization, concurrency, a database, or real-model tool selection.

Which assertions prevent a false green?

Start with observable signals. The final answer is useful, but rarely enough for an agent that calls tools. The test should say which call happened, which arguments crossed the boundary, which state changed, and why the loop stopped.

Check at least:

1. Allowed tool: an unknown call fails before the effect.
2. Arguments: the dispatcher receives the expected identifier and defaults.
3. Consumed result: the tool reply reaches the next turn.
4. State: the record or event has the expected value.
5. Termination: the runner makes no call after the final reply.
6. Limit: a never-ending sequence reaches a turn limit.

These assertions should be smaller than the implementation. If the requirement is "look up the right order," do not freeze the entire response wording. If an exact legal sentence is the requirement, text equality can be part of the contract. The choice should come from the behavior you need to preserve.

When a tool has a mutable effect, use a fake that counts operations and exposes final state. For an email, record the recipient and body in memory. For an update, keep a synthetic record. If the test checks only "done", it can pass without executing anything or after executing twice.

How do you turn failures into repeatable cases?

A happy path is the beginning, not the suite. Add one fixture for each failure the runtime must handle. The case should state its input, initial state, controlled return, allowed effect, and expected result.

The failure test does not need to pretend it represents all production behavior. It proves one chosen combination. For broader coverage, pair it with the AI agent trajectory test and turn a reproduced defect into a regression case. For tool argument and shape checks at the boundary, use tool-call validation in TypeScript.

Avoid one trap: a fake tool can repeat the same assumption as the code under test. Keep fixtures simple, validate inputs, and inspect the produced effect. When behavior depends on the real service, add an integration test against an isolated environment. Do not turn the local fixture into an integration certificate.

How do you separate the offline test from a real eval in CI?

Run the deterministic layer in the pull request. It should work without a key, network, or shared state. Keep the real-model eval for a separate schedule, a prompt change, or a provider change. This division explains failures more clearly and keeps an external outage from hiding a runner regression.

The offline test can prove that a scripted reply travels through the expected path. A real eval must check whether the model chooses that path from natural language. One does not replace the other.

In CI, store the call sequence, normalized arguments, initial state, final state, and case version as artifacts. Do not publish tokens, secrets, or user data. A small trace is easier to review and enough to identify the broken boundary.

If the verifier itself changes, test it with positive and negative cases. A suite that checks only the happy path can accept correct state while ignoring a duplicate effect. The test must fail when relevant evidence disappears.

A checklist before trusting the suite

• Is the model local, scripted, and able to report unexpected calls?
• Is the real runner still executing?
• Do the scenario tools avoid the network and shared databases?
• Does the test check names, arguments, order when required, and result?
• Is external state represented by evidence the test can read?
• Are turn, retry, and effect limits explicit?
• Do error cases distinguish failure, unknown success, and invalid data?
• Is it written down which question requires a real model or integration?
• Can a person investigate the trace without opening a secret?

If you cannot answer the last question, the test may be green without helping you debug. The value of a deterministic layer is the explicit boundary: it makes your code repeatable and leaves the remaining uncertainty to the layer that can measure it.

Conclusion

To test an agent without extra LLM calls, script the model boundary and keep the rest of the runner observable. Use fake tools, small state, and assertions about calls, arguments, effects, and termination.

Then write the limit beside the test. The local fixture proves code behavior against a known sequence. A real-model eval proves something else. When both layers have their own names and criteria, a green test says more than "the agent replied."

How this article was produced

Samuel Fajreldines is responsible for this article. The research compared current Anthropic, OpenAI, Vitest, and Node.js documentation with the existing testing owners in this cluster. Public discussions were used only for language and failure modes, not technical proof. I ran the fixture locally with Node.js v25.5.0. It makes no provider call, measures no model quality, and does not represent a production environment.

I used RemoteCode as my tool for keeping context and evidence across sessions. It supported the writing workflow, but it did not run the fixture or prove the testing claims. AI assistance supported source discovery, comparison, drafting, image generation, localization, and consistency review. The code and limits were checked in the repository before publication.

Sources consulted

• Anthropic, "Demystifying evals for AI agents", retrieved 2026-10-27
• OpenAI, "Evaluate agent workflows", retrieved 2026-10-27
• Vitest, "Mocking Functions", retrieved 2026-10-27
• Node.js, "Test runner", retrieved 2026-10-27