Every agent action that involves thinking is a call to a model. If those calls scatter across the codebase — each service holding its own API key, its own retry logic, its own provider SDK — you lose the three things a platform exists to provide: control, observability, and the freedom to change your mind. The model gateway is the single chokepoint through which all inference flows, and it is one of the highest-leverage components you will build or adopt.

6.1 Why a single chokepoint

From first principles: the model is the platform's most volatile dependency. Prices change monthly, new and better models ship constantly, providers have outages, and rate limits bite without warning. Code that calls a provider SDK directly is welded to that provider and blind to its own behavior. Routing every call through one internal service decouples your application from any single model and gives you one place to measure cost, enforce policy, cache, and experiment. The application asks for inference; the gateway decides how it is served.

6.2 What the gateway must do

• Unified interface — one request schema across providers, so swapping a model is a config change, not a code change.
• Routing & fallback — direct each request to a model by task, cost, or latency target; on error or timeout, fail over to an alternate automatically.
• Load-balancing & rate-limit handling — spread load across keys/regions/deployments and back off gracefully when a provider throttles.
• Caching — exact-match for identical requests, semantic for near-duplicate prompts, and provider prompt caching for stable long prefixes (system prompts, tool schemas). Caching is the single largest cost lever for many workloads.
• Structured outputs — enforce JSON-schema / typed responses centrally so tool arguments and downstream parsing are reliable.
• Observability & cost attribution — record tokens, latency, and dollar cost for every call, tagged by run, tenant, and feature (feeds Ch. 18 and 20).
• Experimentation — A/B or bandit-route between models and prompts, and capture outcome feedback to learn which performs best.

6.3 Build, adopt, or buy

A gateway is plumbing with sharp edges. Most teams should adopt rather than write one. The candidates cluster by ambition: thin universal adapters that normalize providers; gateways that add a control plane of caching, routing, and analytics; and gateways that close the loop with built-in experimentation and feedback-driven optimization.

The advanced posture is to treat the gateway as the seam where the platform learns. TensorZero illustrates the pattern: every inference is logged with its inputs and, crucially, the downstream feedback (was the tool call valid? did the user accept the answer?). That dataset powers dynamic in-context learning (DICL) — automatically retrieving high-performing past examples into the prompt — bandit-based routing that shifts traffic toward whatever model and prompt is winning on your real metrics, and curated datasets for later fine-tuning. The gateway stops being a pipe and becomes a flywheel; this ties directly to evaluation (Ch. 19) and prompt optimization (Ch. 7).

A closing observation: several platforms now fold the gateway together with observability (Ch. 18) and evaluation (Ch. 19) into one control plane — LangWatch, Portkey, and Helicone among them. Because all three planes sit on the same model call, consolidating them removes integration glue; it also concentrates a critical dependency and invites lock-in. Adopting an integrated plane versus composing best-of-breed per layer is the same build-versus-buy judgment as anywhere else (Ch. 30) — what is fixed is the gateway's responsibilities, not which vendor bundles them.