Engineering note · 1 of 6

A stale runner cannot win.

When a job is reclaimed, the next attempt gets a higher generation. Every later heartbeat, status report, and terminal event from the old attempt is rejected — so a network blip cannot rewrite the winner’s result.

The race that breaks queues

Runner A takes a job. Heartbeats stop (pause, partition, OOM). After ~6s the reclaim loop bumps generation and requeues. Runner B finishes successfully. Then A wakes up and tries to report success.

t=0   A dequeues  generation=1
t=6   reclaim → B dequeues  generation=2  · B reports success
t=?   A recovers · ReportStatus(gen=1)

without fencing:  A’s late Ack overwrites B  →  corrupt final status
with fencing:     Ack(gen=1) rejected · Heartbeat returns superseded=true
        

What generation actually gates

• Heartbeat — control plane answers with superseded: true; the runner cancels local work.
• ReportStatus / Ack — non-current generation is ignored; the thread stays on B’s terminal state.
• Terminal StreamEvents — completion events from a superseded generation are dropped before publish.

Progress tokens from a dying runner can still slip through until the next heartbeat — only the final status is iron-fenced. That trade is deliberate and documented.

Same idea, three transports

Redis bumps a hand-rolled counter on reclaim. NATS reuses JetStream NumDelivered. Kafka stores generation in the compacted state topic and bumps it on every re-produce. The Runner Protocol surface is identical: carry generation on every bound RPC.