Testing AI agents: mock the model or use the real API?

A green test can prove two very different things: that the agent code made the right decision, or that a mock returned exactly the response you wrote. Mixing those questions creates a fast suite that cannot reveal a change in the model, the prompt, or the tool choice.

To test AI agents with confidence, split the work into four layers. The first checks pure code. The second controls tools and state. The third runs a small set against the real model. The fourth reviews cases where success cannot fit in a deterministic assertion.

This article uses a small TypeScript interface and Vitest examples. The real provider stays behind an adapter. Fast tests need no key, network, or stable model response, while the behavior eval still has an explicit place to run.

It complements regression evals for coding agents: that post measures preservation in CI; this one decides which dependencies belong in each test.

Practical decision

• Mock the model when the question is about flow, state, schemas, or a known error.
• Use the real API when the question is about model or prompt behavior.
• Do not compare open-ended answers with exact equality without a strong reason.
• Keep the real eval separate from the fast gate that runs on every commit.

What does a model mock actually prove?

A mock proves how your code behaves against a known contract. Vitest's mocking documentation describes vi.fn() as a controlled function that records calls and lets you define returns or rejections. That is enough to test whether an agent validates a reply, retries a transient failure, stops its loop, or refuses a dangerous action.

The mock does not prove that a real model will choose that tool. It does not prove that a prompt remains clear after a change. It answers a narrower question: given this model event, does the system keep its contract?

Start with an interface that does not know the provider SDK:

export type ToolCall = {
  name: string;
  input: Record<string, unknown>;
};

export type ModelReply =
  | { kind: "tool"; call: ToolCall }
  | { kind: "final"; text: string };

export interface ModelClient {
  complete(input: {
    messages: Array<{ role: "user" | "tool"; content: string }>;
    tools: string[];
  }): Promise<ModelReply>;
}

The loop can now receive a fake without importing OpenAI, Anthropic, or another client. This fake returns a tool call and then a final answer. It is deliberately predictable. Predictability is the useful property of this test.

import { vi } from "vitest";

const model: ModelClient = {
  complete: vi
    .fn<ModelClient["complete"]>()
    .mockResolvedValueOnce({
      kind: "tool",
      call: { name: "get_order", input: { id: "ord-7" } },
    })
    .mockResolvedValueOnce({
      kind: "final",
      text: "Order ord-7 has shipped.",
    }),
};

The test can assert that get_order received an allowed identifier, that the tool result returned to the next turn, and that the loop stopped after the final reply. Do not compare the whole text if the business rule only requires the order to be shipped. Compare the decision, then validate language in a separate layer. For the tool schema and output boundary, see TypeScript tool-call validation.

When do you need the real API?

You need the real API when the behavior you want to measure comes from the model interaction. Anthropic defines an evaluation as a task with an input, success criteria, and grading logic. For agents, the evaluation also needs to observe trials, tool calls, changed state, and the trajectory, because the final answer can hide a bad path.

Use a small real set for questions like these:

• does the model choose the right tool when two look plausible?
• does it ask for clarification when data is missing?
• does it respect the permission boundary in the prompt?
• does it reach the right state after several calls?
• does it keep working when the prompt or model changes?

Each case must define what passing means. For an order lookup, that could be the correct ID in final state and no tool outside the allowlist. For an open-ended answer, it could be required facts, forbidden claims, and a grader score. Exact text is only one possible signal.

type EvalCase = {
  name: string;
  input: string;
  expected: {
    finalState: string;
    allowedTools: string[];
  };
};

const cases: EvalCase[] = [
  {
    name: "does not expose another customer's order",
    input: "Show the status of ord-7",
    expected: {
      finalState: "order:ord-7:visible",
      allowedTools: ["get_order"],
    },
  },
];

Run these cases with a real adapter in a separate command. Record the model, prompt, case set, and code version. Do not turn one run into a quality promise. Models vary, and an evaluation should show how scores changed between versions.

How should you split the test suite?

A practical suite puts the cheapest test first and the most model-dependent test last. That order shortens feedback without hiding the difficult part. The real model should not carry responsibility for schemas, permissions, state, or external side effects by itself.

This split also avoids a false choice. You do not need to choose between “mock everything” and “call the model everywhere.” Choose by assertion. If the assertion is about your code, use a mock. If it is about model behavior, run a real evaluation. If it is about a product or security decision, keep evidence for human review.

How do you test tool use and state without fixing the answer?

A tool-use test should observe the important boundary, not imitate every implementation detail. LangChain's tool-use benchmarks separate final output, expected tools, call order, and final state. The same split is useful without LangChain: each signal answers a different question.

When a tool crosses an MCP process, combine this layer with contract tests for TypeScript MCP servers, which check the published transport and schema.

When order can vary, check that every call is allowed and that final state is correct. Require order only when it is part of the contract, such as confirmation before a destructive action.

import { expect, test } from "vitest";

test("keeps the lookup inside the allowed tool", async () => {
  const result = await runAgent({
    model,
    input: "Show the status of ord-7",
    tools: { get_order: getOrderFromFixture },
  });

  expect(result.state).toBe("order:ord-7:visible");
  expect(result.toolCalls.every((call) => call.name === "get_order")).toBe(true);
});

Also simulate responses a provider or tool can return: incomplete JSON, a missing field, a timeout, status 429, and a 500 error. The aim is not to invent the real distribution of these events. It is to prove that your system does not silently treat a failure as success.

That is the difference between a useful mock and a comfortable snapshot. A useful mock represents a contract and includes relevant failures. A comfortable snapshot repeats the happy response the author wrote before the test existed.

How do you put the real model in CI without making the suite flaky?

Separate the fast command from the evaluation command. The first runs on every pull request without network access or secrets. The second runs on a schedule, after a prompt change, or before promoting a model. If it blocks a merge, the rule must tolerate variation and explain its threshold.

The pull-request gate remains a later layer. PR evals for code agents cover the evidence that reaches the reviewer; this post covers the dependency controlled by the test.

A simple setup can look like this:

{
  "scripts": {
    "test": "vitest run tests/unit tests/tools",
    "eval:agent": "tsx evals/run-agent.ts --dataset evals/cases.json"
  }
}

The eval should keep its output and trajectory as restricted artifacts. The report needs the case, model, prompt version, tools called, final state, score, and error. Do not put keys, sensitive prompts, or user transcripts in a public PR comment.

Anthropic's evaluation guidance calls each attempt at a task a trial and notes that multiple trials help account for model variation. Decide in advance how many runs form a comparison and how to treat a timeout. Without that rule, a team can choose the window that confirms the change.

For workflows that span sessions, I use RemoteCode to keep context and run evidence together for Codex and Claude Code. It is my tool, mentioned here because the separation between fast tests and real evals also needs operational continuity.

What can a mock never reveal?

A mock will not reveal a provider behavior change, a prompt the model misinterprets, or a tool choice that never appeared in your fixture. It also does not show cost, latency, context limits, or the exact way an SDK serializes a call.

That does not make the mock bad. It defines the boundary of its proof. Add a small integration set instead of turning every test into a real call. The real set should cover representative and difficult cases, while the local suite covers most error combinations.

The reverse is also true. A real evaluation can pass while an authorization rule is broken if the case never requests the protected action. Keep deterministic tests for permissions, schemas, state, and effects. Use the real model for the part that actually depends on it.

Which combination should you choose?

Use mocks by default when the agent is still code under development, the tool has expensive side effects, or the failure must be reproduced every time. Add a real eval when the prompt, routing, or response quality is part of the change. Ask for human review when the success rule is still vague or the agent can affect data, money, or production.

Before publishing a test, answer these questions:

1. What claim does this test prove?
2. Which dependency is controlled?
3. What failure cannot the mock represent?
4. What evidence will the real eval keep?
5. Which decision blocks the merge, and who can override it?

If the answer is “the model responded well,” the test still lacks a criterion. If it is “final state was correct, tools were allowed, and no external call was duplicated,” you have a contract that can be checked.

Frequently asked questions

Should I mock the model in every unit test?

No. Mock the model when the test checks your code, tools, state, or a controlled failure. Vitest's documentation recommends mocks to replace dependencies, control returns, and observe calls. Keep a separate real-model set for behavior, prompts, and trajectories.

Does a real-model eval replace traditional tests?

No. An eval can show that an agent case passed, but it does not cover every branch, schema, permission, or code effect. OpenAI's documentation separates equality, similarity, and score graders. Use deterministic assertions where they are sufficient, and a grader only for the semantic part.

Can I compare the agent response with a snapshot?

Yes, when the text is a deliberate and stable contract. For open-ended answers, check required facts, final state, allowed tools, and forbidden rules instead. Text equality makes tests sensitive to wording changes and can hide a regression that reaches the same text through the wrong path.

Sources consulted

• Anthropic, "Demystifying evals for AI agents", retrieved 2026-08-04, https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
• OpenAI, "Graders API Reference", retrieved 2026-08-04, https://platform.openai.com/docs/api-reference/graders?api-mode=chat
• LangChain, "LangChain Benchmarks: Tool Usage", retrieved 2026-08-04, https://langchain-ai.github.io/langchain-benchmarks/notebooks/tool_usage/intro.html
• Vitest, "Mocking Functions", retrieved 2026-08-04, https://vitest.dev/guide/mocking/functions
• Vitest, "Mock Functions", retrieved 2026-08-04, https://main.vitest.dev/guide/learn/mock-functions