A correct final answer can hide a bad path. The agent may have called the wrong tool, sent an invalid identifier, repeated a lookup, or needed a human intervention that never appeared in the report.

That is why evaluating an agent means looking at its trajectory, not only its last sentence. Here, trajectory means the observable sequence of actions between the user's input and the final answer. The focus is what the system did: tools, arguments, results, state, and stopping decisions.

This article complements the guide to choosing between a mock and a real model when testing agents. That post helps you separate test layers. This one asks a different question: what should each run prove when the agent uses tools?

Diagram comparing an agent's expected and observed trajectory to its final answer.

Practical decision

  • Record the trajectory alongside the final answer.
  • Require exact ordering only when order is part of the contract.
  • Check tools, arguments, forbidden actions, and final state deterministically.
  • Use a semantic evaluator only for what an observable rule cannot decide.

What does an AI agent trajectory measure?

A trajectory measures the path an agent took to complete a task. The Google ADK evaluation guide separates tool-use and trajectory evaluation from final-response evaluation. That split matters because an answer can look good even when the process took unnecessary or wrong steps.

Imagine an agent that must look up an order status. The final result should contain the right status, but the system may also require it to use get_order, pass the received order ID, and never call cancel_order. Those are three different assertions: result, argument, and action boundary.

The trajectory record does not need to expose the model's private reasoning. It needs enough operational events to check the contract. Usually that includes the input, tool calls, validated arguments, summarized results, state changes, attempts, interventions, and output.

This also connects to validating tool arguments and tool responses in TypeScript. Validation protects each boundary. Trajectory evaluation checks whether the full run crossed those boundaries correctly.

How do you record a trajectory without keeping too much data?

A useful record is small, stable, and sufficient for the assertion you want to make. Do not copy the entire prompt or every infrastructure log by default. Use a format for events that can survive a provider change.

The example below is illustrative. It shows one possible representation, but it was not executed in this repository and does not depend on a specific SDK.

type ToolEvent = {
  kind: "tool_call";
  name: string;
  args: Record<string, unknown>;
  result: "ok" | "error";
};

type AgentTrajectory = {
  caseId: string;
  events: ToolEvent[];
  finalState: string;
  humanInterventions: number;
};

caseId connects the trace to an evaluation case without putting personal data in the report. name and args let you check choice and contract. result separates a successful call from an attempt that ended in an error. finalState avoids depending on the wording of the answer. humanInterventions shows when the result appeared only after a human decision.

Adapt the schema to the risk of the application. For an agent that changes records, also store the normalized target, policy version, and an idempotency identifier. For a coding agent, the equivalent may be files touched, commands run, tests triggered, and artifacts produced. Do not include secrets, tokens, or complete payloads when a verifiable summary is enough.

Which assertions should a trajectory evaluation contain?

Start with observable assertions. They are easier to explain, repeat, and run in CI. The expected trajectory does not need to copy the transcript. It can declare only the boundaries that matter for the task.

Assertion Question Example failure
Allowed tool Did the agent use an action available for this case? It called delete_order during a lookup.
Arguments Did the tool receive the expected fields and values? It used another order's ID.
Order Did one step need to happen before another? It published before approval.
Forbidden action Did the agent avoid actions outside the case? It made an unnecessary external lookup.
State Did the observable effect end in the expected state? It said “cancelled,” but the order stayed active.

The Google Cloud guide to agent evaluation describes trajectory metrics for matching, precision, recall, and single-tool use. Names differ between frameworks, but the decision is the same: choose the metric that matches the contract instead of defaulting to the strictest one.

A state check is often more important than an identical tool list. If two independent reads lead to the same state and no forbidden action occurred, requiring one particular order can reject a valid run. In contrast, a payment confirmation, approval, or destructive migration usually has an order that is part of its safety model.

When is exact ordering a good rule?

Use exact matching when the sequence is a business or security precondition. A simple case is authorize_user before read_balance. Another is validate_payload, then request_approval, and only then publish_change.

Do not require exact order just because the first trace you saw had that order. The model may choose two independent reads in a different sequence without changing the result or increasing risk. In those cases, prefer set, subsequence, or final-state assertions.

The LangChain AgentEvals project separates strict matching, unordered matching, subset matching, and argument comparison. Those options are not universal answers. They help you express the contract you should already have defined.

A practical policy could look like this:

const contract = {
  requiredTools: ["get_order"],
  forbiddenTools: ["cancel_order", "delete_order"],
  requiredArgs: { get_order: { id: "ord-7" } },
  order: "any",
  finalState: "order:ord-7:visible",
};

The order value is a contract decision. If it is any, the evaluation checks presence, arguments, forbidden actions, and state. If it is exact, it compares the full sequence. The format is illustrative and must be adapted to the real executor.

How should you split deterministic checks from semantic judgment?

Semantic evaluation helps when there is no single answer or simple rule. It can check whether an explanation is understandable, whether a response is grounded in a tool result, or whether an open-ended conversation reached its goal. It should not replace a rule that code can check directly.

Use deterministic checks for tool names, arguments, required order, forbidden actions, final state, schemas, and attempt limits. Reserve a semantic judge for clarity, completeness, or meaning equivalence. Even then, write a short rubric and keep the evidence the judge used.

The ADK evaluation documentation recommends separate trajectory and response criteria. For CI and regression, it points to tool-trajectory and response-comparison metrics as fast, predictable options. Rubric-based criteria fit cases without a trusted reference response.

This split avoids two errors. The first is accepting a persuasive answer that used the wrong tool. The second is rejecting a correct answer because it does not match a reference sentence word for word. The evaluation should measure the risk you actually want to control.

How do you bring trajectory evaluation into CI?

Separate the fast test from the eval that depends on a network, model, or judge. The first should run on every change and inspect controlled traces. The second can run on a schedule, after a prompt change, or when comparing model versions. The article on regression evals for coding agents in CI comes next when you need to preserve behavior that already worked.

A minimal flow has these steps:

  1. Create cases with input, available tools, initial state, and expected result.
  2. Run the agent and save a sanitized trajectory alongside the answer.
  3. Apply checks for tools, arguments, forbidden actions, and final state.
  4. Run a semantic rubric only on cases that pass the basic rules.
  5. Publish a summary with failures, attempts, and human intervention.
  6. Turn each important failure into a versioned regression case.

A job can call a custom script, a provider evaluation tool, or a library such as AgentEvals. What matters is that the contract is visible in the repository and that the report distinguishes final answer from trajectory.

{
  "case": "show-order-status",
  "required_tools": ["get_order"],
  "forbidden_tools": ["cancel_order"],
  "expected_state": "order:ord-7:visible",
  "checks": ["tool", "args", "forbidden", "state"]
}

This file is only a format idea. ord-7 is an example value, not production data. In a real suite, create controlled fixtures and avoid reusing identifiers between cases that can change the same state.

For loops that cross sessions and accumulate working context, the author uses RemoteCode as his tool. This describes the author's production context, not a benchmark or a quality promise.

Which failures can the trajectory reveal?

The first signal is the wrong tool with a correct answer. A secondary lookup may have returned the same fixture text, but the agent still violated the boundary you wanted to test. The second is an almost-correct argument: the tool name is right, but the ID, tenant, or filter belongs to another case.

The third is silent intervention. A person fixes the payload, approves an action, or repeats a step, while the report records only “success.” Count that decision as part of the trajectory. An agent that finishes only after rescue is not equivalent to one that completed the case alone.

The fourth is wasted path. The agent reaches the right state after repeated calls, ineffective reads, or attempts that a rule should have blocked. This does not mean every long trajectory is wrong. It means efficiency and intervention can be separate signals from the result.

How do you verify that the suite measures the right risk?

Review the suite with a passing case, an invalid argument, a forbidden tool, an error response, and a run that needs intervention. Do not add examples only to increase apparent coverage. Each case should represent a decision someone would make while operating the agent.

Also check what the report does not show. If it stores only the final text, you cannot distinguish a safe trajectory from one that reached the same place by accident. If it stores the full transcript without redaction, it may leak data that does not help the evaluation.

A useful review asks:

  • Do cases define initial and final state?
  • Are forbidden tools explicit?
  • Are arguments compared at the level that matters, without requiring irrelevant fields?
  • Is exact ordering tied to a business or security rule?
  • Does the report record retries, failures, and human intervention?
  • Does the semantic judge receive only the context it needs?
  • Does a new failure become a reproducible case?

If any answer is “no,” the problem is still the evaluation contract. Changing the model or increasing the score does not fix a poorly defined question.

Frequently asked questions

Is evaluating the final answer enough?

Not when the trajectory carries risk. An agent can answer correctly after using a forbidden source, sending the wrong arguments, or receiving human help. Evaluate the answer for the visible result and the trajectory for the actions that produced it.

Does every trajectory need an expected order?

No. Require order when one step depends on another, such as authorization before a protected read. For independent reads, check allowed tools, arguments, forbidden actions, and final state without rejecting an equivalent order.

Does an AI judge replace test assertions?

No. Use code to check fields, tools, state, and safety rules. A judge can help with clarity, completeness, or semantic equivalence, but its result needs a rubric and reviewable evidence.

Can I evaluate without an agent framework?

Yes. The concept is a contract over observable events, not an SDK brand. You can record calls in your own loop and apply assertions with the test runner you already use. Libraries and services help compare trajectories, but they cannot decide which actions your domain allows.

Conclusion

An agent does not pass just because its last sentence looks correct. It passes when the run respects a contract you can explain: it chose allowed tools, sent valid arguments, followed required order, avoided forbidden actions, and ended in the expected state.

Start with a small, sanitized trajectory. Add deterministic assertions before a semantic judge. Then take failed cases into CI and turn incidents into regressions. Your suite will measure what the agent did, not only the story it told at the end.

Sources consulted