Reasoning & Context Engineering
The gateway decides which model answers; this chapter concerns how well it reasons. Two levers dominate. The first is the reasoning strategy — how much structure and deliberation we impose on the model's thinking. The second, and more underrated, is context engineering: the window is a small, fixed, expensive resource, and what you put in it determines the quality of everything that comes out. Most "the model isn't smart enough" problems are really "the context was assembled poorly" problems.
7.1 Reasoning strategies, from cheap to deliberate
Reasoning quality can be bought with inference-time compute: spending more tokens and more model calls to think harder before acting. The strategies form a ladder of cost against capability. The platform's job is to let each task sit on the lowest rung that solves it — not to apply tree search to a date lookup.
| Strategy | Mechanism | Relative cost | Strongest for |
|---|---|---|---|
| Chain-of-Thought | Model reasons step-by-step in one pass | 1× | Arithmetic, logic, anything needing intermediate steps |
| ReAct | Interleave reasoning with tool calls (the core loop) | N× turns | Tasks requiring external information or actions |
| Plan-and-Execute | Plan the whole approach first, then execute steps | Plan + N | Multi-step tasks where a global plan reduces wandering |
| Reflexion / self-critique | Act, evaluate own output, retry with the critique | 2–3× per try | Code, writing — anywhere a draft can be checked and improved |
| Tree-of-Thoughts / search | Explore multiple branches, score, backtrack | High (k× branches) | Hard search/planning with a verifiable objective; rarely worth it online |
A practical default for agents is ReAct with a lightweight planning preamble, escalating to reflection only when an output can be cheaply verified (tests pass, schema validates). Reserve search-based methods for offline or high-value problems; their token cost compounds against the multipliers of Chapter 2.
Inference-time compute trades tokens for accuracy. That trade is excellent on hard, verifiable problems and wasteful on easy ones. Match the reasoning strategy to task difficulty — ideally let a cheap router classify difficulty and pick the rung.
7.2 The context window is the scarce resource
However large the context window grows, it is finite, it is filled anew on every single turn, and you pay for every token in it on every call. Worse, models attend unevenly: relevant facts buried mid-context are often missed (the "lost in the middle" effect), and overstuffed windows degrade reasoning and raise both cost and latency. The context window is therefore an actively managed working set, not a bucket to fill. Treating context assembly as a deliberate engineering discipline — deciding what earns a place in the window each turn — is what the term context engineering names.
7.3 Techniques for living within the budget
- Selective retrieval — pull only the top-k reranked chunks actually relevant to the current step (Ch. 11), never the whole knowledge base.
- Compaction & summarization — as a run grows long, summarize older turns into a compact running state and drop the verbatim history; persist the full transcript out-of-band for audit.
- Stable prefixes for caching — keep the system prompt and tool schemas byte-identical across turns so provider prompt caching (Ch. 6) discounts them.
- Structured over prose — a typed state summary conveys more per token than narrative recap.
- Offload to memory and tools — what the model can fetch on demand need not sit permanently in context (Ch. 8, 10).
7.4 Prompts as optimized assets, not handwritten strings
The instructions, examples, and formatting that make up a prompt are usually hand-tuned by trial and error — brittle, undocumented, and silently broken by a model upgrade. The maturing alternative is to treat prompting as a program that is compiled and optimized against data. DSPy embodies this: you declare the task as typed input→output signatures and compose modules; an optimizer then searches for the instructions and few-shot examples that maximize a metric on your dataset. Optimizers such as MIPROv2 (joint instruction + demonstration search) and GEPA (reflective, evolutionary prompt search) can lift quality materially without you editing a single prompt string by hand. The output is a prompt artifact you version and re-optimize whenever the model or task changes — closing the loop with the gateway's feedback data (Ch. 6) and your evals (Ch. 19).
Two slow failures haunt this layer. Context rot: long-running agents accumulate stale, contradictory, or redundant context until reasoning degrades — fix with disciplined compaction. Prompt drift: a prompt tuned for one model silently underperforms after an upgrade — fix by re-optimizing against an eval set rather than trusting a frozen string. Both are invisible without the evaluation discipline of Chapter 19.