Scaling, Reliability & Deployment
Everything so far assumed one agent running once. Production means thousands of concurrent runs, each fanning out to many model and tool calls, against dependencies that rate-limit, slow down, and fail. Scaling an agentic platform is unlike scaling a stateless web service: the bottleneck is rarely your own CPU and almost always your model-inference capacity, and the runs are long-lived and stateful. This chapter is about making the platform fast, reliable, and affordable under real concurrency.
21.1 The three-layer capacity model
The single most useful mental model for capacity is to separate three concerns that teams routinely conflate. (1) The capacity floor — guaranteed inference throughput. Shared model APIs are subject to rate limits and noisy-neighbor variance; for predictable performance at scale you secure dedicated capacity (provisioned throughput on managed model platforms, or your own GPU fleet running an inference server like vLLM). This is the bedrock that determines how much the platform can think per second. (2) The LLM gateway — the smart layer (Ch. 6) that meters out that finite capacity: routing, queuing, load-balancing across providers and deployments, caching to reduce demand, and failing over when a source is exhausted. (3) The durable workflow engine — the orchestration layer (Ch. 4) that runs the long-lived agent loops, fans work out and back, survives failure, and absorbs bursts by queuing runs rather than dropping them. Each layer scales on a different axis; conflating them is why platforms fall over.
21.2 Concurrency, queuing, and GPU scheduling
Because inference is the scarce resource, the platform's concurrency strategy is really a queuing strategy: admit runs, and when capacity is saturated, queue rather than fail (the durable engine makes a queued run free to hold — Ch. 5). For self-hosted inference, GPUs are scheduled on Kubernetes with awareness that they are expensive and indivisible; serving frameworks batch requests to maximize utilization. The choice between serverless inference (elastic, pay-per-use, cold starts) and dedicated capacity (predictable latency, fixed cost, no noisy neighbors) is the same provisioned-throughput trade-off as §21.1, now at the deployment layer.
21.3 Resiliency patterns for a probabilistic, dependency-heavy system
Agentic platforms depend on flaky, rate-limited external services (model providers, tools, APIs) and run non-deterministic work. The classic distributed-systems resiliency patterns are therefore not optional; they are how the platform stays up when its dependencies don't.
| Pattern | What it does | Where it matters most |
|---|---|---|
| Timeouts | Bound how long any call may hang | Every model and tool call — never wait forever |
| Retries + backoff | Re-attempt transient failures, spaced out | Provider 429/5xx; safe only with idempotency |
| Circuit breakers | Stop calling a failing dependency, fail fast | A down provider or tool — prevents cascading stalls |
| Bulkheads | Isolate resource pools so one failure can't drain all | Per-tenant / per-provider isolation |
| Fallback | Degrade to an alternate path | Secondary model/provider via the gateway (Ch. 6) |
| Idempotency | Make repeated execution safe | Every side-effecting tool, so retries don't double-act (Ch. 4, 8) |
21.4 Deployment and the CI/CD gate
Shipping changes to a non-deterministic system demands a release pipeline that treats prompts, models, and policies as versioned artifacts and refuses to ship a regression. A robust gate (built with a CI system, e.g. Tekton) runs the evaluation suite (Ch. 19) on every change, checks policy with OPA (Ch. 17), and only then promotes — ideally behind canary or staged rollout so a bad change reaches a fraction of traffic first. Combined with the gateway's ability to swap models by config, this lets you adopt improvements quickly and roll back instantly.
Guaranteed inference throughput, intelligent allocation of it, and durable orchestration of workloads are three different problems that scale on three different axes. Design and scale them independently. Most "the platform melted under load" failures are really one layer absorbing pressure that belonged to another.
Aggressive retries against a struggling dependency without idempotency cause double-execution (double charges, duplicate messages) and can turn a brief outage into a self-inflicted thundering herd that prevents recovery. Pair every retry with backoff, a circuit breaker, and idempotent side effects — the durable engine of Chapter 4 provides exactly these guarantees.
21.5 The platform as code
Everything in this chapter — the capacity floor, the autoscaling policy, the CI/CD gates — is infrastructure, and infrastructure clicked together by hand is infrastructure you cannot reproduce, review, or recover. The discipline is infrastructure as code: the platform's cloud resources, clusters, and configuration are declared in version-controlled definitions and applied by tooling, so the environment is reproducible, diffable, and auditable. OpenTofu (the open-source Terraform fork) and Crossplane (Kubernetes-native) are representative; the choice matters less than the principle that there is a single declarative source of truth for what the platform is.
For an agentic platform this pays off in four places. Ephemeral sandbox fleets and microVM pools (Ch. 9, 24) are provisioned and torn down declaratively rather than by hand. It overlaps with policy as code — the same GitOps discipline that governs infrastructure governs the OPA policies of Ch. 17, the prompt and model-registry artifacts of Ch. 28, and per-tenant configuration (Ch. 26). Multi-region disaster recovery (Ch. 29) is only credible if the standby region can be stood up from the same definitions. And drift — the silent divergence of running infrastructure from its declared state — becomes detectable rather than a debugging mystery discovered mid-incident. The goal is a platform whose entire definition, from GPU floor to guardrail policy, lives in version control and ships through the same gated pipeline as application code.