How to Test an AI Agent When a Tool Returns Invalid JSON

A tool can return HTTP 200 and still deliver a result the agent cannot use. The body may be malformed JSON, have the wrong type, or omit the field that authorizes the next step. To test this case, inject the defective response at the tool boundary and check the whole path, not just the agent's final sentence.

The test needs to prove four things: the parser rejected the value, the trace recorded why, the recovery policy made a known decision, and no tool with an external effect was called. The guide to validating tool calls in TypeScript covers the input contract. This article asks what to test after a tool returns output that does not satisfy its contract.

This is also narrower than a full eval. Testing an agent trajectory looks at the general path. For the persistence boundary after a proposal, see how to validate AI agent output before saving it. This test fixes one tool boundary failure and makes its consequence observable: the next send operation must not happen.

Short answer

• Inject the invalid payload into the tool adapter without depending on a real API or model.
• Separate malformed JSON from valid JSON with the wrong shape.
• Record the error in the trace and verify the stop or recovery policy.
• Count calls to the next tool and confirm that the effect stayed at zero.

What does the test need to prove?

Do not start with the text the agent showed the user. Start with the boundary that turns a tool response into internal data. The test should answer different questions:

1. Can the text be parsed as JSON?
2. Does the parsed value have the shape the runtime requires?
3. Should the agent stop, ask for a correction, retry, or hand off?
4. Did any later tool receive permission to produce an effect?

JSON.parse() turns a string into a JavaScript value and fails when the string is not valid JSON (MDN, JSON.parse()). That answers only the first question. {"accountId":"acct-1"} is valid JSON, but it is still insufficient if the contract also requires status.

An agent eval commonly separates task, trial, grader, trace, and the outcome in the environment. Anthropic describes these parts in its guide to agent evals, including the difference between what appears in the transcript and the final state outside it. Here, the final state is simple: an invalid response must not reach the send effect.

Inject the failure at the right boundary

A mock at the model level does not test the tool boundary. It can return the result the code expects directly and leave the parser, trace, or dispatcher outside the path. The injection point should be the adapter the runner actually calls.

In this example, the agent looks up an account. If the account needs review, it calls sendEmail. The test controls lookupAccount, but keeps the parser, decision, and effect counter real inside the fixture.

async function runAgent({ lookupAccount, sendEmail }) {
  const trace = [];
  trace.push({ kind: "tool_call", name: "lookup_account" });
  const raw = await lookupAccount({ accountId: "acct-1" });

  try {
    const account = parseAccount(raw);
    trace.push({ kind: "tool_result", name: "lookup_account", status: "valid" });

    if (account.status === "needs_review") {
      await sendEmail({ accountId: account.accountId });
    }

    return { status: "completed", trace };
  } catch (error) {
    trace.push({ kind: "tool_error", name: "lookup_account", code: error.code });
    return { status: "blocked", trace };
  }
}

The fixture's trace does not try to reproduce a provider's format. It records only the events the local contract needs to verify. In production, use the observability format already present in your runtime. OpenAI's guide to evaluating agent workflows uses traces for model calls, tool calls, guardrails, and handoffs, and recommends structured graders for finding workflow failures.

The important detail is that sendEmail is not hidden inside the lookupAccount mock. If the test controls both functions so that no path between them can exist, it cannot prove that an invalid response blocks the effect.

Make the case executable with node:test

Node's built-in runner executes this case without another test framework. The Node.js node:test documentation covers the runner, assertions, and mocking features. The fixture below uses only node:test and node:assert/strict.

import { test } from "node:test";
import assert from "node:assert/strict";

class ToolOutputError extends Error {
  constructor(code) {
    super(code);
    this.code = code;
  }
}

function parseAccount(raw) {
  let value;
  try {
    value = JSON.parse(raw);
  } catch {
    throw new ToolOutputError("invalid_json");
  }

  if (
    !value ||
    typeof value !== "object" ||
    typeof value.accountId !== "string" ||
    typeof value.status !== "string"
  ) {
    throw new ToolOutputError("invalid_shape");
  }

  return value;
}

// Include runAgent from the previous example in the same file.
test("malformed JSON stops before the effect", async () => {
  let sent = 0;
  const result = await runAgent({
    lookupAccount: async () => '{"accountId":"acct-1"',
    sendEmail: async () => { sent += 1; },
  });

  assert.equal(result.status, "blocked");
  assert.equal(sent, 0);
  assert.deepEqual(result.trace.at(-1), {
    kind: "tool_error",
    name: "lookup_account",
    code: "invalid_json",
  });
});

test("valid JSON with the wrong shape is also rejected", async () => {
  let sent = 0;
  const result = await runAgent({
    lookupAccount: async () => JSON.stringify({ accountId: "acct-1" }),
    sendEmail: async () => { sent += 1; },
  });

  assert.equal(result.status, "blocked");
  assert.equal(sent, 0);
  assert.equal(result.trace.at(-1).code, "invalid_shape");
});

test("valid output reaches the effect only with the precondition", async () => {
  let sent = 0;
  const result = await runAgent({
    lookupAccount: async () => JSON.stringify({
      accountId: "acct-1",
      status: "needs_review",
    }),
    sendEmail: async () => { sent += 1; },
  });

  assert.equal(result.status, "completed");
  assert.equal(sent, 1);
});

Save the snippet in an .mjs file that also contains runAgent, then run:

node --test invalid-tool-output.test.mjs

In my local run on November 10, 2026, with Node.js v25.5.0, all three cases passed. That is evidence for the in-memory fixture: it reaches the three decisions the code declares. It is not a test of a model provider, an external API, or email delivery.

Assert the error, path, and effect

A weak test checks only that an exception was thrown. That can leave the runtime without a usable outcome. Another weak test checks only status: "blocked" but does not confirm whether a write already happened before the block.

Use layered assertions:

• Boundary error: invalid_json and invalid_shape are different codes. The operator needs to know whether to investigate transport, parsing, or the contract.
• Trace: the trace contains the original call and the error event. This distinguishes a failed tool from a tool that was never recorded.
• Policy: execution ends as blocked, is handed off, or enters a bounded retry according to the contract. Do not use a final message as a substitute for that decision.
• External effect: the sendEmail counter, command list, or database fixture stays unchanged when the precondition was not proven.

This format also avoids requiring one rigid sequence when more than one path is safe. Fix the invariant, such as “do not send without a valid account,” rather than every sentence the model might write. When model choice is part of the risk, add a model eval with a separate grader. The local test remains responsible for the deterministic boundary.

Expand the matrix without mixing layers

After the three basic cases, add failures that answer different questions:

A unit test can use in-memory functions. A contract test should check the format the adapter actually receives. An integration test can connect to a test API and inspect the recording system. A real-model eval should observe the trace and state, because a model can respond plausibly without fixing the problem.

Do not turn the table into a permanent list of mocks. Keep each case close to the boundary it protects. The post on testing agent regressions without exact text matching helps when the problem stops being one payload and becomes behavior that must remain stable.

Failures that leave the test green

Validate only the parser

assert.throws(() => JSON.parse(raw)) proves that the parser recognizes bad input. It does not prove that the runner records the error, ends the turn, or blocks the next tool. Keep the parser call inside the path the agent uses.

Mock the whole runner

If the mock directly returns { status: "blocked" }, the test does not exercise the decision. Replace the model or external adapter, but leave the dispatcher, parser, trace, and effect assertion real.

Check only the final response

A sentence such as “I could not complete this” does not say whether the agent already sent the email before writing it. Count the operation, inspect the external fixture, and record the event sequence.

Retry every error the same way

Invalid output may be recoverable, but the retry needs a budget and a newly verified input. If every attempt returns the same malformed response, the test should end with an explicit limit rather than a loop.

Use a write tool in the unit case

To test policy, a fake sendEmail with a counter is enough. Leave SMTP, a database, or a queue to the test that needs to prove integration. A local case should not trigger real effects by accident.

CI checklist

• Is the failure injected at the boundary the runner actually calls?
• Does the suite separate malformed JSON, wrong shape, invalid value, and transport error?
• Does the trace record the tool, reason, and runId without storing secrets or sensitive payloads?
• Does the test verify the stop, bounded retry, or handoff policy?
• Is there an assertion about the effecting tool and external state?
• Does the case remain deterministic without network, clock, or model calls?
• Does a contract test cover the adapter's real format?
• Is there at least one separate eval for model-dependent decisions?
• Does the case fail when code calls the effecting tool before validating the response?

An invalid-output test is small, but it protects an important boundary. It does not need to know whether the model is good. It needs to show that the system does not turn an unproven response into permission to change the world.

How this article was produced

Samuel Fajreldines is the accountable author. The research compared current OpenAI and Anthropic documentation on agent evaluation with the node:test documentation and recent public discussions. Discussions helped choose failure language; they were not used as technical proof.

I ran an in-memory fixture locally with Node.js v25.5.0. It injects malformed JSON, JSON with an incomplete shape, and a valid response. The observed result was three passing tests. The fixture did not call a model, external provider, account, or email service.

I used RemoteCode as the author's tool to organize context and review the structure. It did not run the tests or provide production evidence.

Sources consulted

• Anthropic, “Demystifying evals for AI agents”, consulted November 10, 2026
• OpenAI, “Evaluate agent workflows”, consulted November 10, 2026
• Node.js, “Test runner”, consulted November 10, 2026
• MDN, “JSON.parse()”, consulted November 10, 2026