Chapter 21 gave the patterns that keep a platform standing when its dependencies fail. This chapter is the discipline around them: defining what "reliable" even means for a system whose outputs are probabilistic, committing to it as a number, sizing capacity from first principles, and surviving the disasters that the resiliency patterns alone do not address. It is the operational-maturity layer that turns a working architecture into a service you can put an SLA on.

29.1 What "reliable" means when the output is probabilistic

Classical reliability assumes a correct answer exists and the system either returns it or fails. Agents break that assumption: there is often no single correct output, and quality is a distribution. So you must separate two kinds of reliability. The mechanical kind — availability, latency, and successful completion — is measured exactly as in any distributed system. The quality kind — was the answer good? — is itself probabilistic and can only be measured statistically, by sampling production and scoring it with the online evaluation of Chapter 19. The crucial consequence: you can promise availability and latency, but you cannot promise a correct answer — only a measured quality floor.

29.2 SLOs and error budgets

An SLI is a measurement; an SLO is a target on it (for example, 99.5% availability and a quality score above a set threshold, per month), and the gap between the target and 100% is the error budget — the amount of unreliability you are permitted to spend. The error budget is the link between reliability and velocity (Ch. 21): while the budget is healthy, ship quickly; when it is being burned, freeze feature work and stabilize. Burn-rate alerts warn when you are consuming the budget too fast to last the period. SLOs are set per tier and per tenant (Ch. 26), because a premium tenant's promise differs from best-effort.

29.3 Capacity planning, quantitatively

The capacity floor of Chapter 21 must be sized, and the arithmetic is approachable. The sustained inference demand is, to first order:

Worked: 200 concurrent runs, ~8 model calls each, ~1,500 tokens per call, completing in ~30 s, implies on the order of 80,000 tokens/second of sustained demand — which you translate into provisioned-throughput units or a GPU count for your serving stack (Ch. 21), then add headroom. Little's law gives the queuing intuition: average concurrency equals arrival rate times average run duration, so long agent runs inflate concurrency fast. Size the floor to the mean, absorb peaks in the queue, and let per-tenant fairness (Ch. 26) decide who waits when the floor is saturated.

29.4 Disaster recovery and business continuity

Resiliency patterns handle a dependency failing; disaster recovery handles you failing — a region outage, a corrupted store, a bad deploy that poisons data. Two numbers frame it: RPO (recovery point objective — how much data you can afford to lose) and RTO (recovery time objective — how long restoration may take). The non-obvious, critical point for this architecture: the durable execution store is your system of record (Ch. 4) — it holds every in-flight run — so its backup, point-in-time recovery, and replication govern whether a storage failure is a blip or a mass loss of work. Back up the event history and the data stores of Chapter 27 to their RPO; decide active-passive versus active-active multi-region against your RTO; and recall the control-plane/data-plane split of Chapter 3 — they fail over differently, the control plane prizing consistency and the data plane prizing availability. A provider outage is itself a DR scenario the gateway's fallback (Ch. 6) already addresses. Write the runbooks, and rehearse them with game-days, before the incident — restore procedures that have never been run do not work.