A lease can expire while the worker that received it is paused. When another worker takes over the task, the old process can still wake up and try to write the result it calculated before ownership changed. If the resource accepts that write because the worker checked the lease a moment earlier, the protection still has a race.

Fencing tokens close that race at the right boundary: the one that accepts the mutation. Each acquisition receives a higher generation. The worker carries that generation with the write, and the database or write proxy rejects any token lower than the highest token it has already accepted. The guide to resuming an agent after a crash covers the broader recovery problem. This article focuses on stopping an old owner from writing after recovery.

This article uses a small Node.js fixture to reproduce the event order. It proves the resource rule in memory. It is not a distributed lock, does not measure latency, and does not prove the behavior of a specific broker or database.

Diagram shows an old worker with token 1 rejected after a new worker writes with token 2.

Short answer

  • The lease decides who should work now, but it cannot stop a process that is already running.
  • The ownership authority should issue a higher token for every new acquisition.
  • The protected resource must compare the token and the mutation atomically.
  • A stale-token rejection should not become an automatic retry with the same token.

Why is checking the lease before writing not enough?

Suppose worker A receives the lease for job-7 and starts calculating. It pauses for long enough to miss renewal, and the lease expires. Worker B takes the same task and writes a new result. A then resumes and checks a local variable that still says it owns the job. Between that check and the write, nothing stops the old result from replacing the new one.

The problem is not limited to garbage collection. A process can be suspended, lose CPU, lose network access, or send a request that arrives late. Martin Kleppmann's analysis of distributed locks explains why an expiry check before the write does not close the race and why the service receiving the write must reject older tokens.

The important sequence looks like this:

Moment Worker Token Result
The task starts A 1 May work while its ownership is valid
The pause outlasts the lease A 1 The process may continue without knowing it lost ownership
The task is reclaimed B 2 B becomes the newer owner
B writes B 2 The resource accepts the mutation
A wakes and writes A 1 The resource rejects the old mutation

That last step separates "the worker should stop" from "the resource will not accept an old write." The first is a liveness policy. The second is the safety guard that remains in force when a process fails to stop.

Separate the lease, heartbeat, and fencing token

These mechanisms often appear together, but they do different jobs:

Mechanism Question it answers Limit
Lease Who may try to work during this interval? A paused process may not notice that it lost ownership.
Heartbeat Can the worker still prove that it is alive? A failed renewal does not undo a write already sent.
Fencing token Can this ownership generation still change the resource? It protects only the boundary that compares the token.

The Microsoft global distributed lock sample for Azure Cosmos DB combines atomic acquisition, ETag-based renewal, TTL cleanup for an abandoned lock, and a monotonically increasing token that a downstream system can use to reject an old owner. The combination is useful because TTL helps recover the task while the token protects later writes.

Do not treat a heartbeat as permanent authorization. It helps a worker discover ownership loss early and stop calls that cooperate with cancellation. The resource still needs to check the generation on a mutation that an old worker must not overwrite.

The resource must enforce the token rule

The token is useless if it appears only in the worker's logs or context. The write destination must store the highest accepted generation and compare the received value in the same operation that changes state.

An illustrative SQL update could look like this:

UPDATE job_results
SET result = $1,
    accepted_fence = $2,
    updated_at = now()
WHERE job_id = $3
  AND accepted_fence < $2;

If the update affects zero rows, the executor should treat it as stale_fencing_token, not as a transient error that deserves a retry of the same write. The column needs a defined initial value, and every path that mutates the result needs the same condition. One administrative script that skips the comparison can reopen the bug.

When a PostgreSQL sequence creates the token, nextval returns distinct values atomically across concurrent sessions, according to the sequence function documentation. The sequence is not gapless, and that does not matter for fencing. What matters is that the generation never moves backward.

For an external service that does not understand the token, put the check in a proxy or in a store that records the highest generation before calling the service. If the external call already happened, fencing does not undo the effect. You still need idempotency, a state lookup, or reconciliation.

Node.js fixture: the stale worker is rejected

The fixture below represents two acquisitions of the same task. The second acquisition stands for lease recovery. The store accepts the newer token and rejects the old token even though worker A is still executing code.

import assert from "node:assert/strict";

class FencingAuthority {
  #next = 0;

  acquire(resource) {
    this.#next += 1;
    return { resource, fence: this.#next };
  }
}

class FencedStore {
  #highestFence = new Map();
  #values = new Map();

  write(resource, value, fence) {
    const highest = this.#highestFence.get(resource) ?? 0;
    if (fence < highest) {
      throw new Error("stale_fencing_token");
    }

    this.#highestFence.set(resource, fence);
    this.#values.set(resource, value);
  }

  read(resource) {
    return this.#values.get(resource);
  }
}

const authority = new FencingAuthority();
const store = new FencedStore();

const workerA = authority.acquire("job-7");
const workerB = authority.acquire("job-7");

store.write(workerB.resource, "result-from-B", workerB.fence);

assert.throws(
  () => store.write(workerA.resource, "late-result-from-A", workerA.fence),
  /stale_fencing_token/,
);
assert.equal(store.read("job-7"), "result-from-B");

console.log("stale worker write rejected");

Save it as fencing-token-fixture.mjs and run it with Node.js:

node fencing-token-fixture.mjs

The test checks the local invariant: once the resource has accepted token 2, a write with token 1 cannot replace the value. It does not check database atomicity, token issuance by a real coordinator, lease renewal, or any effect outside this store. Those boundaries need separate integration tests.

Where does a PostgreSQL queue fit?

If tasks already live in PostgreSQL, a transaction can select an available task, record ownership, and return the generation to the worker. PostgreSQL's SELECT documentation describes SKIP LOCKED as an option that can reduce contention in queue-like tables, while warning that it produces an inconsistent view and is not suitable for general queries.

That solves the competition to claim a queue item. It does not by itself solve a worker that calculates for a long time and returns after losing ownership. The final result still needs to carry the generation and use a conditional update. If multiple processes can compete for the same run, the article on durable execution with queues and workflows helps separate queues, checkpoints, and workflows.

Order the claim explicitly when priority or fairness matters. SKIP LOCKED does not turn a table into a fair scheduler, and a row lock is not permission for the application to ignore the task version.

How should the runtime react to stale_fencing_token?

A resource rejection is ownership information, not an invitation to repeat the same operation. The executor can follow this path:

  1. Stop the current step and do not retry the write with the old token.
  2. Record runId, jobId, the presented token, the accepted token, and the boundary that rejected the mutation.
  3. If an external call happened before the rejection, look up the effect by a stable key or mark the result unknown.
  4. Let the new worker reread state and decide whether to continue, reconcile, or stop.
  5. Emit an operational signal that distinguishes ownership loss from a dependency failure.

The queue-worker test after a lost acknowledgment covers another window: the effect was applied, but delivery confirmation failed. Here the window is task ownership. Both cases need to observe the actual effect, not only the number of calls.

Mistakes that make fencing decorative

Checking and writing as separate operations

An if (token === currentToken) followed by another command still allows ownership to change between the two operations. The comparison and mutation need to share a transaction or the conditional operation the resource actually guarantees.

Keeping the token only in memory

If the process protecting the resource restarts and forgets the highest accepted generation, an old worker may look valid again. Persist the high-water mark or use a mechanism whose ordering survives a restart.

Resetting the token sequence

A token is an ownership generation, not an attempt counter that can return to zero after cleanup. If the order must be reset, treat it as a protocol change and preserve a barrier that prevents old tokens from becoming valid again.

Confusing cancellation with fencing

An AbortSignal can interrupt an operation that cooperates with it. It does not cancel a request that already reached the server and does not stop a delayed response from reaching its destination. Fencing is still needed at the state-accepting boundary.

Promising exactly once across several systems

Fencing can prevent an old generation from updating a protected resource. It does not make transactions between a database, broker, and external API atomic. For those boundaries, combine idempotency, an outbox, compensation, or reconciliation according to the effect you need to protect.

Release checklist

  • [ ] Each acquisition receives a new generation that never moves backward.
  • [ ] The token travels with the mutation to the resource that can accept or reject the write.
  • [ ] Token comparison and state change are atomic at the protected boundary.
  • [ ] The stale-token path ends without an automatic retry using the same generation.
  • [ ] Heartbeats and cancellation help liveness, but are not the only safety defense.
  • [ ] The system records ownership loss and exposes the winning generation for investigation.
  • [ ] External effects have an idempotency key, a state lookup, or an explicit reconciliation path.
  • [ ] The fixture forces an ownership change and confirms rejection of the old write.

Method and limits

Samuel Fajreldines is responsible for this article. The research combined Martin Kleppmann's distributed-lock analysis, Microsoft's current distributed-lock example, and PostgreSQL documentation on row locks, SKIP LOCKED, and sequences. Public discussions were used only to recognize practitioner terms and failure modes. I wrote the fixture for this article and ran it locally with Node.js v25.5.0. It uses in-memory state, so it does not measure distributed recovery, test a broker, or prove exactly-once behavior in production.

I used AI assistance to organize sources, review the prose, and produce the localized versions. It did not run a production worker or provide operational experience. When I need continuity across long Codex and Claude Code sessions, I use RemoteCode, a tool I built. That does not replace a journal, lease, fencing, or reconciliation.

Conclusion

Leases and heartbeats help recover a task and move a stalled worker out of the way. They do not guarantee that the old process stopped. The decisive protection belongs at the resource: accept the mutation with an ownership token and reject any generation lower than the last accepted one.

This design does not promise exactly once for the whole system. It answers a narrower and more useful question: can a worker that lost ownership still overwrite protected state? If the answer is yes, the lease is only a convention. If the answer is no, the write boundary is enforcing fencing.

Sources consulted