A retry can save a failed task. It can also repeat a write that happened before the process died. If the pipeline creates a file, inserts a record, or charges for an operation, “try again” is not an idempotency policy.
With Cloud Run Jobs, the protection belongs in your code and durable storage. Give the task a stable identity, persist progress outside the container, and make the output accept the same key only once. The retry can then repeat the execution without turning a partial failure into two results.
This article uses illustrative TypeScript and a batch Job. To choose between a Service, Job, and worker pool, see the Cloud Run resource comparison for long-running agents. The narrower problem here is how a restarted task recovers without duplicating its effect.
Practical rule
- Retry decides how many times a task may try.
- A checkpoint decides where an attempt resumes.
- Idempotency decides whether repeating an input creates another effect.
- Verification proves that the expected output exists once, not merely that the process exited.
Why does a retry not make a task idempotent?
In 2026, the Cloud Run Jobs retries and checkpoints documentation says that each task can fail, restart, and try again. The documented default is up to 3 retries. That improves the chance of recovering from transient failures, but it does not stop two attempts from reaching the same external operation.
Imagine a task that computes a result, writes result.json, and dies before it
marks the state complete. The next attempt does not know whether to recompute,
replace the file, or skip the stage. If the output is an insert without a unique
key, it can create a second row. If it is an email or charge, lowering
max-retries does not undo the effect.
The first separation is simple: retry is a platform recovery policy; idempotency is a property of the operation. You can set zero retries and still duplicate a result by running the Job manually. You can set three retries and avoid duplicates if every effect uses the same key and a conditional write.
What must survive a restart?
In 2026, Google Cloud’s Jobs retries and checkpoints guide recommends making Jobs idempotent and using persistent checkpoints so a restarted task does not redo all its work. Process memory is not a checkpoint. It disappears when the instance stops.
For each unit of work, persist four things:
- Identity: Job name, task index, and a business key that stays the same across attempts.
- State:
queued,running,checkpointed,done, orfailed. - Progress: the last completed stage and the location of its partial artifact.
- Output: a record or object addressed by the same idempotency key.
The durable execution guide for agents and pipelines covers the broader boundary between checkpoints, queues, and workflows. For a Cloud Run Job, you do not need to start with an orchestrator. You need a new attempt to read durable state and distinguish “the stage has not started” from “the output has already been confirmed.”
Do not save only one checkpoint after the whole pipeline. Save each stage that produces a useful artifact. A rerun can then skip decoding that already finished and continue with transformation. The checkpoint reduces repeated work. The idempotency key protects the effect when the process dies between the write and the acknowledgement.
How do you write an idempotent TypeScript stage?
Cloud Run exposes the task index through CLOUD_RUN_TASK_INDEX, but the output
identity remains an application decision. The Create jobs documentation describes that index and the total task count. Use it when the batch partition is part of the contract. For a business entity, include a stable item ID as well.
The following example is illustrative. putIfAbsent represents an atomic
operation in your database or object store. Without that condition, two
concurrent attempts can both observe “missing” and write two outputs.
type Progress = {
key: string;
status: "running" | "checkpointed" | "done";
artifactKey?: string;
leaseUntil: number;
};
interface ProgressStore {
get(key: string): Promise<Progress | null>;
claim(key: string, leaseUntil: number): Promise<"claimed" | "busy" | "done">;
put(value: Progress): Promise<void>;
putIfAbsent(key: string, value: { artifactKey: string }): Promise<boolean>;
}
function outputKey(jobName: string, taskIndex: string, itemId: string) {
return `jobs/${jobName}/tasks/${taskIndex}/items/${itemId}/result.json`;
}
async function runItem(input: {
jobName: string;
taskIndex: string;
itemId: string;
store: ProgressStore;
buildArtifact: () => Promise<string>;
}) {
const key = outputKey(input.jobName, input.taskIndex, input.itemId);
const existing = await input.store.get(key);
if (existing?.status === "done") return existing.artifactKey;
const claim = await input.store.claim(key, Date.now() + 5 * 60_000);
if (claim === "done") return existing?.artifactKey;
if (claim === "busy") {
throw new Error("item is already leased");
}
const artifactKey = existing?.artifactKey ?? await input.buildArtifact();
await input.store.put({ key, status: "checkpointed", artifactKey, leaseUntil: 0 });
await input.store.putIfAbsent(key, { artifactKey });
await input.store.put({ key, status: "done", artifactKey, leaseUntil: 0 });
return artifactKey;
}The contract has two decisions that cannot remain implicit. claim must be
atomic, and the lease must expire because a process can die without releasing
running. The final write
must be conditional or use a unique key. In production, make state transitions
atomic too. The example does not choose Firestore, Cloud Storage, or PostgreSQL;
it shows the guarantees any adapter must provide.
For external effects, such as a tool call that creates a resource, send the same idempotency key to the destination when it supports one. If it does not, put the operation behind an intent table or approval step. TypeScript tool-call validation helps control the schema, but it does not prevent an already-valid effect from being repeated.
How should you configure retries and timeout?
In 2026, the Create jobs documentation lists a default task timeout of 10 minutes and a configurable retry count. The task timeout documentation allows a timeout up to 168 hours for tasks without GPUs. Those limits apply to each attempt. They are not a promise that one execution will run without interruption.
An initial command might look like this:
gcloud run jobs create process-items \
--image=REGION-docker.pkg.dev/PROJECT/REPOSITORY/IMAGE:TAG \
--tasks=1 \
--max-retries=3 \
--task-timeout=20m \
--region=REGIONThis is a configuration example, not a ready-to-run deployment. Confirm the region, image, service account, and project limits. Choose enough retries for transient failures, but do not treat the number as a fix for a non-idempotent effect.
If the batch can be divided into independent tasks, use the task identity to create repeatable partitions. The code must map the same item to the same partition on every attempt. If the input list changes during execution, record the version or snapshot used. Otherwise, the same task can process different items on a retry and make the audit trail confusing.
How do you separate retryable and terminal failures?
Google Cloud documents retries for failures that may be transient, but the application still needs to decide when another attempt helps. A 429 response, network timeout, or 5xx error may deserve a retry. A missing file, invalid schema, or denied permission usually needs a fix or quarantine.
Use an explicit classification:
- Record the item, attempt, stage, and reason before throwing the error.
- Throw only failures that can change without changing the input or configuration.
- Mark terminal errors as
failedwith enough context to repair them. - Send exhausted retries to a dead-letter queue or report.
- Let a watchdog find expired leases and states stuck in
running.
Do not confuse “the function returned an error” with “the effect did not happen.” The process can write to a database and die before the final log line. Recovery must consult durable state before running again, rather than looking only for the last log entry.
How do you prove a rerun did not duplicate output?
After configuration, test a failure at several points: before the checkpoint, after the artifact, and before the final confirmation. A Job is safe to rerun only when the second attempt finds the same identity and produces the same observable result.
A minimum verification can look like this:
it("reuses output when the same task runs twice", async () => {
const store = makeInMemoryStore();
let builds = 0;
const input = {
jobName: "process-items",
taskIndex: "0",
itemId: "item-7",
store,
buildArtifact: async () => {
builds += 1;
return "artifacts/item-7.json";
},
};
await runItem(input);
await runItem(input);
expect(builds).toBe(1);
expect(await store.countOutputs("item-7")).toBe(1);
});The test is a behavior specification. makeInMemoryStore and
countOutputs must exist in the real suite. Add one test that fails after
buildArtifact and another that runs two workers against the same lease. The
important result is not an identical string. It is output cardinality, the
persisted key, and the final state.
The Jobs documentation says that when a task exceeds its retry limit, the Job execution finishes as failed after Cloud Run has tried the tasks. Inspect the execution and logs, identify the failed task, and compare output count with the input snapshot. A green Job without that check can still have processed the wrong input.
Common mistakes and limits
The first mistake is using --max-retries=0 as a substitute for idempotency. It
turns off automatic retries. It does not stop manual executions, repeated input,
or two workers from competing for one key.
The second is saving only the final file. If one expensive stage finishes and the next fails, a rerun cannot tell whether the file is valid, which version produced it, or which input it represents. Include input version, stage name, and timestamp in the progress record.
The third is making state durable but not the effect. A done record written
before the external call can hide a failure. Record the intent, execute with an
idempotency key when possible, and confirm state only after checking the result.
The fourth is putting every stage into one long task. The guide to testing AI agents with a mock or the real API helps verify that a rerun calls the external dependency at the right time. For a multi-stage pipeline, addressable checkpoints let you repeat only what is needed. They are not a distributed transaction, though. A crash can still happen between two systems, so recovery must be safe in that interval.
Frequently asked questions about Cloud Run Job retries
How many retries does a Cloud Run Job make by default?
The Create jobs documentation says the default is up to 3 retries per task, and the value can be configured from 0 to 10. A retry belongs to the task. It is not a guarantee that an external effect happens once. Use the setting for transient failures and implement idempotency separately.
Does a checkpoint remove the need for idempotency?
No. A checkpoint reduces repeated work and shows where to resume, but the process can die after the write and before the checkpoint. The retries guide recommends both practices. Use those two controls: persistent progress to recover the stage and a unique key or conditional write to protect its output.
What is the default task timeout?
In 2026, the documented default task timeout is 10 minutes. It can be shortened or increased to 168 hours for tasks without GPUs, while GPU tasks have a 1-hour limit. The timeout applies to each attempt. Split work or save checkpoints when a stage can exceed it.
Does max-retries=0 prevent all duplicates?
No. With 0 automatic retries, it prevents new attempts after a failure, but not manual runs, repeated input, or two workers competing for one key. The defense is a deterministic identity, an expiring lease, a conditional write, and a check that counts the confirmed output.
