An LLM provider can accept a request and fail before your application receives the response. If the runtime sends the same turn to another provider without checking state, the backup response can produce a second tool call. For an external effect, that can mean a repeated charge, duplicate message, or state change applied twice.

Safe failover separates three things: the attempt to obtain a model decision, the application’s action intent, and execution of the external effect. This separation is the practical complement to durable execution for AI agents. Each model attempt can have its own identifier, but the action keeps a key created by the application. The runtime then classifies the error, respects a time and cost budget, switches only when capability is compatible, and confirms state before repeating.

Diagram showing LLM failover classifying failures, controlling budget, switching providers, and deduplicating an external action.

Short answer

  • A retry repeats a model attempt; failover changes provider; stopping records that there is not enough evidence to continue.
  • A timeout after sending a request is an unknown state, not proof that the action did not happen.
  • The action key belongs to your runtime and should reach the tool executor when the effect might be repeated.
  • The backup provider must satisfy the capability, format, and authorization contract before taking over.

What does failover need to decide?

Failover is not just a list of providers in order. It is a policy for four states: an error before the model processes the request, a transient error with no known external effect, an invalid response, and an ambiguous result after the application may have started an action. Each state needs a different outcome.

If the failure is transient and the operation is still only a model attempt, the runtime can retry within its budget. If the provider is unavailable, it can switch to another provider that satisfies the same contract. If the response is invalid, the runtime should validate or stop. If a tool call may have run, it should query state or wait for confirmation before repeating.

That boundary between decision and effect connects failover to the durable layer. Checkpoints preserve what the execution knows; they do not automatically make an external operation idempotent.

When does an error allow retry or failover?

Start with the provider contract, not a catch block that treats everything as retryable. OpenAI’s error-code guidance separates rate limits and temporary failures from authentication, configuration, billing, and quota problems. Gemini troubleshooting guidance also recommends bounded backoff for transient errors and advises against automatically retrying request or permission errors.

A starting classification looks like this:

Observed state Preferred action What still needs confirmation
Rate limit or outage before a tool call Bounded retry or failover Budget, deadline, and backup capability
Authentication, permission, schema, or configuration error Stop and fix configuration Whether a different credential is authorized outside the loop
Timeout while only waiting for the model response Follow policy and retry within limits Whether the request was accepted and whether the attempt is billable
Timeout after requesting an external effect Do not blindly repeat Action state, idempotency key, or operation query
Backup response lacks the required tool or format Stop or request a new decision Actual capability, not a similar model name

Status codes are not a universal policy. SDKs may apply their own retries, proxies can hide responses, and providers can change details. Record the original category and the decision so a silent hop cannot look like success.

How do you separate the model call from the tool call?

A model response is a proposal. The application should turn that proposal into an action intent only after validating name, arguments, authorization, and task state. Anthropic’s tool-use documentation exposes an identifier for a tool_use block, but that identifier describes the provider message. It does not replace the action identifier in your domain.

Keep two identifiers:

  1. modelAttemptId identifies an attempt with a provider and model.
  2. actionId identifies the logical intent, such as send-order-847, regardless of how many responses were obtained.
  3. The executor records actionId before applying the effect and returns the known result when the same action arrives again.
  4. A timeout keeps the state as unknown until a query, callback, or reconciliation changes the evidence.

The snippet below is illustrative. It shows the decision boundary, not a production-ready router, and it was not run against a live provider.

type ActionState = "not_started" | "running" | "succeeded" | "failed" | "unknown";

type ProviderResult = {
  kind: "answer" | "tool_call" | "error";
  retryable?: boolean;
  actionState?: ActionState;
};

async function resolveTurn(actionId: string): Promise<ProviderResult> {
  const existing = await actionStore.read(actionId);

  if (existing?.state === "succeeded") {
    return { kind: "answer", actionState: "succeeded" };
  }

  const attempt = await askProviderWithBudget();

  if (attempt.kind === "error" && attempt.retryable && !attempt.actionState) {
    return failoverOnceWithTheSameAction(actionId);
  }

  if (attempt.kind === "tool_call") {
    return executeWithIdempotencyKey(actionId, attempt);
  }

  return attempt;
}

The important part is not the function names. It is preventing failoverOnceWithTheSameAction from repeating a tool that may already have been accepted. The tool must know actionId, or the runtime must query a service that knows that key.

For the schema and error boundary before execution, see how to validate tool calls in TypeScript. Shape validation does not prove that an action has not already run, but it reduces the chance of sending an incompatible response to the executor.

How do you bound time, cost, and attempts?

A failover policy without a budget can turn an outage into a sequence of expensive, slow calls. Set a deadline for the turn, a maximum number of attempts, and a spend ceiling that the runtime checks before calling another provider. The limit must be shared by the turn, not reset at every hop.

Backoff should follow the provider’s guidance and the remaining deadline. A retry whose delay exceeds the deadline is not resilience; it is a later error. Also record provider, model, attempt, error class, and duration. This trail shows whether failover prevented an error or only accumulated cost.

If a tool call hangs, cancellation has its own boundary. Canceling an AI agent when a tool hangs can release the worker, but it cannot erase an effect that already reached the external system. The next step should be reconciliation, not automatic repetition.

How do you validate the backup provider?

Two models can return a similar overall shape and still offer different capabilities. Before switching, compare allowed tools, argument schemas, context limits, streaming behavior, safety policies, and error representation. If the backup cannot execute the required tool, returning text is not a successful fallback.

Validate in two layers. First, confirm that the response meets the syntax contract. Then confirm that the decision can advance the business rule. The tool-call validation guide explains why a TypeScript type or valid JSON does not prove that a result is safe for the next step.

Failover must also preserve relevant context and drop material that increases risk without helping the decision. Do not send integration secrets to a provider that is not authorized to operate that tool. When capability is not equivalent, the honest outcome is to stop or request approval, not silently adapt the action.

How do you record a provider hop without hiding it?

Use one identity for the logical run and a list of attempts inside it. Each attempt should record provider, model, exit reason, duration, token usage when available, tool state, and next decision. Avoid storing prompts or sensitive arguments in logs merely to prove that failover happened.

One success metric can mislead. Track how many turns finished, how many needed a hop, how many remained unknown, and how many effects required reconciliation. The coding-agent observability guide uses the same idea of separating the logical run from the events that compose it.

For long-running flows, I use RemoteCode as my tool for working with agent sessions and keeping this kind of operational context visible. That describes my tool use, not a benchmark in this article or an endorsement of a specific provider.

How do you test the failure path?

Do not start by testing only whether the backup provider answered. Test whether the runtime preserves the action identity and stops when it cannot prove state. A fake provider and fake executor let you exercise the decisions without sending real messages or charges.

Scenario Expected result
Primary returns a rate limit before a decision One bounded retry or a recorded hop
Primary fails and backup lacks the required tool Explicit incompatibility stop
Timeout before the model response, with no action started Bounded retry within the deadline
Timeout after sending an action with actionId State query or unknown, never blind duplication
Backup response has invalid arguments Rejection before the executor
Every attempt exceeds the deadline Observable failure with original reason and action state

The most valuable test is ambiguity: let the executor accept the action, discard the response, and simulate a timeout. A second attempt with the same actionId should return the stored result or require reconciliation. If it creates a new effect, failover is still coupled to message retry.

Frequently asked questions

Should failover always switch providers?

No. A configuration, permission, schema, or business-rule error does not become correct by changing vendors. Switch only when the error matches policy, the deadline allows it, and the backup provider satisfies the turn’s contract.

Can I repeat a tool call after a timeout?

Not without additional evidence. Treat the effect as unknown, query its state, or use an idempotency key that the external system actually honors. A local timeout reports what your client saw, not what the server completed.

Does an idempotency key solve everything?

No. AWS guidance on idempotent APIs explains that the service must record intent and operation consistently to recognize a repeat. Well-Architected guidance also treats the key as part of the service contract, not a decorative field sent by the client.

Conclusion

Reliable failover starts with a simple question: what exactly might have happened before the switch? If the answer is only a model attempt, bounded retry or failover may be enough. If a tool call is unknown, preserve the action, query the effect, and then decide the next step.

This design does not depend on one provider. It requires explicit policy for errors, budget, capability, validation, idempotency, and observability. Without those boundaries, failover can improve availability on a dashboard while weakening system integrity.

Sources consulted