How to Limit Concurrent Tool Calls in an AI Agent

A model response can contain several tool calls, but that does not mean your service can run them all at once. An external API may have limited connections, a query may compete for the same resource, or a background agent may consume the workers needed by an interactive request.

Put the limit in the runtime, before the tool handler starts. The guide to deciding when to allow or block parallel tool calls answers the previous question: are these calls independent? This article answers the next one: after the batch is considered safe, how many executions may consume capacity now?

The code below uses a TypeScript semaphore for global execution limits and per-tool pools. It is a local, illustrative implementation. It does not coordinate different replicas, replace rate limiting, or make a write idempotent.

Short answer

• Separate the permission for the model to emit multiple calls from the execution limit in your process.
• Reserve a slot before calling the tool and release it in finally, including on errors and cancellation.
• Use a global limit for worker capacity and smaller pools for sensitive dependencies.
• Make the wait observe an AbortSignal; a canceled agent should not leave a waiter stuck in the queue.
• Test the maximum number of active executions and the cancellation path. Counting only final results does not prove that the limit worked.

Concurrency is not the model's parallelism setting

parallel_tool_calls and similar options say whether a provider may return several calls in one response. The OpenAI Agents SDK for JavaScript also documents toolExecution.maxFunctionToolConcurrency as a local execution limit per turn. These are different layers: one controls model emission, while the other controls the SDK dispatcher.

The OpenAI Agents SDK for Python makes the distinction explicit: parallel_tool_calls controls what the model may emit, while max_function_tool_concurrency limits how many local function tools run at once. Even so, an SDK setting may be insufficient when several runs share an HTTP client, queue, API, or database outside that instance.

Think in terms of four different limits:

Disabling parallel emission can be valid for tools with shared state, but it does not limit two separate runs. A total budget also does not stop five calls from starting in the same millisecond. Use the control that matches the resource that can saturate.

Reserve capacity before the handler

The semaphore below keeps three invariants: the counter never exceeds the limit, every acquisition has a matching release, and a canceled waiter leaves the queue without consuming a slot.

type Waiter = {
  resolve: (release: () => void) => void;
  reject: (error: unknown) => void;
  onAbort?: () => void;
};

export class Semaphore {
  private active = 0;
  private readonly waiters: Waiter[] = [];

  constructor(private readonly limit: number) {
    if (!Number.isInteger(limit) || limit < 1) {
      throw new Error("limit must be a positive integer");
    }
  }

  acquire(signal?: AbortSignal): Promise<() => void> {
    if (signal?.aborted) {
      return Promise.reject(signal.reason ?? new Error("aborted"));
    }

    if (this.active < this.limit) {
      this.active += 1;
      return Promise.resolve(() => this.release());
    }

    return new Promise((resolve, reject) => {
      const waiter: Waiter = {
        resolve: (release) => {
          signal?.removeEventListener("abort", waiter.onAbort!);
          this.active += 1;
          resolve(release);
        },
        reject,
      };

      waiter.onAbort = () => {
        const index = this.waiters.indexOf(waiter);
        if (index === -1) return;
        this.waiters.splice(index, 1);
        reject(signal?.reason ?? new Error("aborted"));
      };

      signal?.addEventListener("abort", waiter.onAbort, { once: true });
      this.waiters.push(waiter);
    });
  }

  private release(): void {
    const next = this.waiters.shift();
    if (next) {
      next.resolve(() => this.release());
      return;
    }
    this.active -= 1;
  }
}

The counter increases when the acquisition is handed over, not when the Promise enters the queue. When a waiter is canceled, the queue loses the reference and the counter does not change. The caller's finally should be the only place that releases the slot. Releasing again from an error callback creates negative counts or phantom capacity.

Combine a global limit with per-tool pools

A global limit protects process capacity. A per-tool pool stops one dependency from consuming every slot. The trpc-agent-go documentation describes global and per-tool groups and also warns that per-invocation limits do not protect capacity shared across runs or replicas.

type ToolCall<T> = {
  name: string;
  run: (signal: AbortSignal) => Promise<T>;
};

const globalPool = new Semaphore(4);
const toolPools = new Map([
  ["search", new Semaphore(1)],
  ["read_customer", new Semaphore(2)],
]);

export async function executeTool<T>(
  call: ToolCall<T>,
  signal: AbortSignal,
): Promise<T> {
  const releaseGlobal = await globalPool.acquire(signal);
  const specificPool = toolPools.get(call.name);
  let releaseSpecific: (() => void) | undefined;

  try {
    releaseSpecific = specificPool
      ? await specificPool.acquire(signal)
      : undefined;
    return await call.run(signal);
  } finally {
    releaseSpecific?.();
    releaseGlobal();
  }
}

This order avoids one class of deadlock: every execution takes the global limit first and the specific pool second. The cost is that a global slot may remain reserved while a call waits for its tool pool. If that matters, use a scheduler that admits the pair of resources atomically or define a group limit instead of stacking semaphores without measuring the queue.

Do not let the tool name be the only configuration. The registry should also say whether a call reads or writes, which resource it shares, and whether it requires ordering. The independence classification remains the decision in the article about parallel tool calls; the pool controls capacity after that decision.

Choose between waiting and failing fast

When no slot is available, the runtime can queue the call or return a structured error to the agent. The right choice depends on the resource:

The Microsoft Agent Framework explains that call-count and duration limits are best effort and may be checked after a model-requested batch. That differs from rejecting an execution before the handler. If the protection must be exact, enforce admission in your own runtime or in the component that owns the capacity.

Do not turn a rejection into an automatic retry. A concurrency_limit error can return to the model as a tool result, end the turn, or enter a durable queue. The runtime should record runId, tool name, policy, wait time, and rejection reason without copying sensitive arguments. Whether to try again belongs to the run policy and the effect's idempotency.

Test capacity, not only the response

A useful test controls tool duration so it can observe how many executions are active. The example below can run with tsx or be adapted to JavaScript. It calls neither a model nor an external API.

import assert from "node:assert/strict";

const gate = new Semaphore(2);
let active = 0;
let maximum = 0;

async function fakeTool(signal: AbortSignal): Promise<void> {
  const release = await gate.acquire(signal);
  active += 1;
  maximum = Math.max(maximum, active);

  try {
    await new Promise((resolve) => setTimeout(resolve, 20));
  } finally {
    active -= 1;
    release();
  }
}

await Promise.all(
  Array.from({ length: 6 }, () => fakeTool(new AbortController().signal)),
);

assert.equal(maximum, 2);

Add a second case with an AbortController: occupy the only slot, queue another call, abort the signal, and confirm that the second tool never entered the handler. Then make a call fail and verify that the slot becomes available again. Finally, run two runs at the same time. A semaphore created inside the run tests only that run; a pool created outside it tests shared process capacity.

The test still does not prove a distributed policy. With two replicas, each process may accept four calls while the external service sees eight. For that boundary, move admission to a gateway, queue, or the shared resource itself. The article on testing a queue worker without duplicate side effects helps with behavior after execution leaves the local process.

Failures the limit does not fix

Confusing concurrency with rate limiting. Four slots can start four requests at one instant and still violate a quota of two per second. Use a token bucket or equivalent policy when the limit is temporal.

Applying the limit only to the model. The provider may emit one call, a retry may emit another, or a worker may start another later. The boundary must wrap the executor that calls the dependency.

Using a counter without reservation. The pattern "read active, compare, start" lets several Promises pass the same check before any of them increments the counter. Semaphore acquisition must be one indivisible operation inside the event loop or a shared coordinator.

Holding slots in a tool that ignores cancellation. The AbortSignal must reach the operation waiting on I/O. A caller timeout can release the local slot while the upstream keeps working, so classify that effect as unknown when it includes a write.

Creating a pool per run and calling it global protection. That limits one execution, not all executions. Document each pool's scope and test the scenario that actually competes for the resource.

Checklist before enabling concurrent tools

• Was independence classified before the scheduler ran?
• Is there a global limit for worker capacity?
• Does any dependency need a smaller pool or resource-level serialization?
• Does acquisition happen before the handler, with release in finally?
• Does canceling a waiter remove it from the queue without consuming capacity?
• Does the wait have a deadline and a rejection policy?
• Are rate limiting, total budget, and idempotency handled separately?
• Does the test measure maximum active executions, failure, cancellation, and two concurrent runs?
• Does the trace identify admission, rejection, waiting, and effect without leaking arguments?

Conclusion

The model may request several tools, but the runtime should decide how many start. First classify dependencies and shared state. Then put admission in the executor, combine a global limit with specific pools, propagate cancellation, and test the maximum number of active executions.

This limit is a capacity control. It does not replace rate limiting, approval, idempotency, timeouts, or coordination across replicas. Treat each protection as a separate question and let the test show which boundary is actually covered.

For keeping context across long development sessions, I use RemoteCode as the author's tool. It does not coordinate production slots, replace the scheduler, or provide evidence of throughput.

How this article was produced

Samuel Fajreldines is the accountable author for this article. The research compared current references from the OpenAI Agents SDK, Microsoft Agent Framework, and trpc-agent-go with recent public discussions and the site's existing cluster. The semaphore and test are a local reference implementation. The code was checked in Node.js v25.5.0 without calling a model provider, external API, or multi-replica environment. AI assistance supported discovery, organization, drafting, localization, and consistency review; it did not supply production experience or measure latency, cost, or throughput.

Sources consulted

• OpenAI Agents SDK for JavaScript, “Running agents”, retrieved November 13, 2026.
• OpenAI Agents SDK for Python, “Running agents”, retrieved November 13, 2026.
• Microsoft Agent Framework, “Function tools”, retrieved November 13, 2026.
• trpc-agent-go, “Tool documentation”, retrieved November 13, 2026.