Every step of a tool-using agent usually resends the same tool definitions, the same system prompt, and the history that already existed. Without prefix caching, the provider processes that block as fresh input. With caching, it reuses the identical stretch and mostly bills (and processes) what changed: the tool result, the assistant turn, and the next user message.
Prompt caching is the reuse of an identical prompt prefix across API requests. In an agent loop, that is not only a billing detail. It is a layout contract: what stays the same goes first; what changes each step goes last; and the harness records whether the hit actually happened.
If you are still cutting harness tokens, start with the hidden context cost in coding agents. Budget decides what enters the model. Caching decides how often you pay again for the stable block on each tool step.
Practical outcome
- Stable prefix: tools + system + fixed examples first.
- Dynamic tail: tool results, timestamps, and the current ask last.
- Measurement: log cache-read and cache-write tokens on every step.
- Common miss: timestamps, reordered tool lists, or a mutable system block in the prefix.
What does prompt caching reuse in an agent loop?
On current providers, the cache works on an exact prefix. OpenAI describes identical prefix matching at the start of the prompt, automatic caching for eligible prompts from about 1,024 tokens, and notes that tools, messages, and structured outputs can contribute to that prefix (OpenAI, "Prompt caching", accessed 2026-08-06).
Anthropic documents the same idea with a fixed build order for the prefix: tools, then system, then messages, up to the block marked with cache_control. The docs list agentic tool use with many tool calls as a scenario where caching cuts cost and latency, because each step is typically a new API call (Anthropic, "Prompt caching", accessed 2026-08-06).
In system terms, the usual loop looks like this:
- Send tools + system + task.
- Receive
tool_use. - Run the tool in your runtime.
- Resend the history with the
tool_result. - Repeat until a final answer or a step limit.
Step 4 is where caching matters. If tools and system are byte-for-byte equal to the previous step, the prefix can be read from cache. If any stable part changes, the hit disappears and you pay for the full prefix again.
This sits next to context engineering for coding agents: that post reduces what enters; this one reuses what must stay.
How should you structure tools, system, and history for hits?
Treat the prompt as two zones.
Stable zone (prefix): tool names and schemas, system prompt, policies, fixed examples, reference documents that do not change per step. Put this zone first. Do not interpolate dates, request ids, turn counters, or JSON with random key order into it.
Dynamic zone (tail): the user message, tool_results, task state, and any telemetry that changes. Keep this zone last.
OpenAI recommends static content at the beginning and variable content at the end, and notes that tools and images must also be identical across requests for the prefix to match (OpenAI, "Prompt caching", accessed 2026-08-06). Anthropic stresses the same layout and warns that changing tool definitions invalidates the tools, system, and messages caches (Anthropic, "Prompt caching", accessed 2026-08-06).
// Illustrative: stable prefix + dynamic tail layout.
// Not a full production client.
type ToolDef = {
name: string;
description: string;
input_schema: Record<string, unknown>;
};
type Message =
| { role: "user" | "assistant"; content: string }
| {
role: "user";
content: Array<{
type: "tool_result";
tool_use_id: string;
content: string;
}>;
};
const STABLE_TOOLS: ToolDef[] = [
{
name: "search_orders",
description: "Look up orders by customerId.",
input_schema: {
type: "object",
properties: {
customerId: { type: "string" },
},
required: ["customerId"],
},
},
];
const STABLE_SYSTEM = [
"You are a support agent with tools.",
"Call at most one tool per step.",
"Do not invent a customerId.",
].join("\n");
function buildRequest(history: Message[]) {
return {
model: "claude-sonnet-4-5",
max_tokens: 1024,
// Anthropic: mark the end of the reusable block.
system: [
{
type: "text",
text: STABLE_SYSTEM,
cache_control: { type: "ephemeral" },
},
],
tools: STABLE_TOOLS,
messages: history,
};
}Two implementation details matter more than the model name:
- Stable serialization. In some languages, key order in tool JSON changes across processes. Anthropic lists unstable key order in
tool_useblocks as a miss cause (Anthropic, "Prompt caching", accessed 2026-08-06). - Fixed tool surface during the loop. If the harness adds or removes tools mid-task, the prefix changes and the tools cache drops. Prefer a fixed set per task type and leave tool choice to the model, with runtime validation of tool calls.
When I run long loops in Claude Code, Codex, or a custom harness, I use RemoteCode as my layer to push the work with less context waste. It is the author's tool: it does not replace prefix layout, cache-hit measurement, or human review.
How do you measure cache hits on every tool step?
Without metrics, "we enabled caching" is hope. Both providers expose counters on the response usage object.
On Anthropic, the useful fields are:
cache_read_input_tokens: tokens read from cachecache_creation_input_tokens: tokens written to cache on this responseinput_tokens: tokens after the last breakpoint (the non-cacheable tail)
The docs define total input as the sum of those three fields (Anthropic, "Prompt caching", accessed 2026-08-06).
On OpenAI, the matching detail field is cached_tokens. Newer model families also document cache_write_tokens and a prompt_cache_key parameter to improve routing for requests that share a long prefix (OpenAI, "Prompt caching", accessed 2026-08-06).
// Illustrative: record hit ratio per loop step.
type UsageLike = {
input_tokens?: number;
cache_read_input_tokens?: number;
cache_creation_input_tokens?: number;
prompt_tokens_details?: {
cached_tokens?: number;
cache_write_tokens?: number;
};
};
function summarizeCache(step: number, usage: UsageLike) {
const read =
usage.cache_read_input_tokens ??
usage.prompt_tokens_details?.cached_tokens ??
0;
const written =
usage.cache_creation_input_tokens ??
usage.prompt_tokens_details?.cache_write_tokens ??
0;
const uncached = usage.input_tokens ?? 0;
return {
step,
read,
written,
uncached,
// Prefix hit: this step read from cache.
hit: read > 0,
};
}Write these numbers next to tool name, latency, and cost. Coding-agent observability in CI is a natural place for the event: without it, a cache miss only looks like "the API got expensive."
Prices change by model and retention policy. As of 2026-08-06, Anthropic's table shows cache reads at 0.1× base input and 5-minute cache writes at 1.25× base input, with a 1-hour TTL option at a higher write rate (Anthropic, "Prompt caching"). Use the provider's current table when you project savings; do not hard-code a percentage.
What invalidates the cache mid-loop?
The most expensive agent misses are usually self-inflicted.
1. Timestamp or id in the system block. An Now: 2026-08-06T12:01:03Z at the top of system rewrites the prefix every second. Move clock and request id into the tail, or omit them if the agent does not need them.
2. Tools rebuilt every step. Building the tools array from an unordered Map, or injecting session data into descriptions, breaks the match. Materialize the list once per task type.
3. System prompt assembled from task state. Current customer: ACME in system feels handy and destroys reuse across tenants and steps. Prefer a user message or a block after the breakpoint.
4. Expired TTL. Anthropic's ephemeral cache defaults to a 5-minute lifetime, refreshed on each hit; a 1-hour TTL exists at a higher write cost (Anthropic, "Prompt caching", accessed 2026-08-06). If the agent waits 20 minutes for human approval and you depend on the 5-minute TTL, the next step rewrites the prefix.
5. Expecting cache to change the answer. Both providers state that caching does not change output token generation; it reuses processing of the prefix (OpenAI, "Prompt caching"; Anthropic, "Prompt caching", accessed 2026-08-06). If the output changed, look at sampling, tools, history, or the model, not "the cache hallucinated."
Verification checklist for the harness
Run this sequence before calling it done:
- Run two consecutive steps of the same agent with the same tool set.
- On the second step, confirm
cache_read_input_tokensorcached_tokens> 0. - Intentionally add a timestamp to system and confirm the miss.
- Remove the timestamp and confirm the hit returns.
- Keep the tool set stable during the loop; validate arguments at the boundary.
- Record read/write/uncached per step in a CI artifact or structured log.
- For multi-tenant OpenAI traffic, use a stable
prompt_cache_keyper shared prefix, per the current docs.
Limits of this article
This post does not promise a fixed savings percentage. That depends on prefix size, step count, TTL, model, and invalidation rate. It also does not replace context reduction. A 40k-token prefix with a 90% hit rate is still a large prefix; budget still matters.
It does not cover semantic response caching (similarity-based output stores). Prompt caching is provider-side prefix reuse, not a Redis of answers. It does not cover your organization's data-retention policy: read the provider data guide when ZDR or residency matters.
Finally, the TypeScript sample is illustrative. Model names and usage fields change; treat official docs as the source of truth on the day you deploy.
FAQ: prompt caching for agents
Does prompt caching change the model response?
No. OpenAI and Anthropic describe the feature as reuse of prefix processing. Output token generation is still a new computation from that prefix. Do not treat cache as memoization of the final answer.
Do I need cache_control on every provider?
No. OpenAI documents automatic caching for eligible requests and, on newer families, explicit breakpoints and prompt_cache_key. Anthropic offers top-level automatic caching and per-block explicit breakpoints. Stable layout matters on both; the marking API differs.
What is the minimum prefix size?
It depends on the model and platform. OpenAI cites about 1,024 tokens as a reference for cacheable prefixes. Anthropic publishes per-model minimums (for example 1,024 tokens on several recent Sonnet/Opus models, with larger minimums on some others). If read and creation both come back zero, the prefix is likely under the minimum or the breakpoint sits on the wrong block.
Is caching worth it for a single-step agent?
Only if the same prefix repeats across users or nearby jobs. In a multi-tool loop, the benefit usually appears from the second step. For a one-shot with a short system prompt, the gain can be zero while cache writes still cost on plans that bill writes.
