An agent run is a long-lived, stateful, side-effecting computation that calls flaky external services and must not be lost when a process dies. That sentence is the entire specification for a durable execution engine — and it is why a web framework or a plain task queue is the wrong foundation. This chapter explains durable execution, why it fits agents better than anything else, and how the leading engines differ.

4.1 The durability problem, precisely

Consider an agent that spends ninety seconds and forty thousand tokens reasoning across eight tool calls, then the pod is rescheduled during a routine deploy. With an ordinary service, the in-memory state vanishes: the run is gone, the tokens are wasted, and any side effects already performed (a payment, an email, a database write) may be repeated when the client retries. Multiply by thousands of concurrent runs and a single deploy becomes a mass-extinction event. The platform needs runs that survive process death and resume exactly where they left off, without repeating completed work or side effects.

4.2 How durable execution works: deterministic replay

The dominant technique, pioneered at scale by Temporal (and its Cadence lineage), is event sourcing with deterministic replay. Your workflow is ordinary code, but each interaction with the outside world — scheduling a step, receiving its result, starting a timer, receiving a signal — is appended to an immutable event history in durable storage before control proceeds. If the worker crashes, a new worker re-executes the workflow code from the top, but instead of re-running the side-effecting steps, it feeds the recorded results back from the history. Execution fast-forwards deterministically to the exact point of failure and continues. The completed payment is never made twice because its result is already in the log.

Other engines reach the same guarantee by different routes. DBOS checkpoints each step's output to Postgres and, on recovery, skips steps whose output is already recorded — durability as a thin library over your existing database. Inngest memoizes each step.run() so re-invocations replay completed steps from its event store. Restate journals invocations and state to an embedded log. The mechanism varies; the contract — completed work is never repeated, and runs survive failure — does not.

4.3 Why this fits agents better than a task queue

Teams often reach first for a task queue (Celery, Sidekiq, BullMQ, SQS). Queues give at-least-once delivery of a message, which is necessary but far from sufficient. They do not give you persisted intermediate state, exactly-once side effects, automatic resume-from-failure, durable timers, or the ability to pause for days awaiting a human — all of which you would have to build yourself, badly. Durable execution provides them as primitives. Four agent-specific needs map directly onto durable-execution features:

• Long runs & resumability — the agent loop is the workflow; each turn is a step; a crash resumes mid-loop.
• Human-in-the-loop — durable timers and signals let a run block on human approval for hours or days at essentially zero cost, then continue (Ch. 5).
• Fan-out / fan-in — child workflows and parallel steps let an orchestrator spawn many sub-agents or parallel tool calls and await them (Ch. 12, 21).
• Idempotent side effects — recorded step results make retries safe, so a flaky tool call can be retried without double-charging a customer.

4.4 Choosing an engine

The engines cluster by weight and by where state lives. Heavier engines (Temporal) offer the most power, polyglot SDKs, and battle-tested scale at the cost of operational complexity and a determinism discipline your team must learn. Postgres-native engines (Hatchet, DBOS) trade some power for radically simpler operations — your durable state lives in a database you already run and can query. Event-first engines (Inngest) optimize developer experience and serverless deployment.