Model Lifecycle & Customization
The gateway (Ch. 6) decides which model serves a request and context engineering (Ch. 7) shapes what it sees. Neither manages the models themselves — choosing them, customizing them to your domain, versioning them, and retiring them safely. Models are the platform's most capable and most volatile dependency; treating them as static givens is how teams end up locked to a deprecated model or burning training budget on a problem prompting would have solved. This chapter is the lifecycle of the model as a managed asset.
28.1 The customization decision
When a base model is not good enough at your task, there is a ladder of interventions ordered by cost and iteration speed, and the discipline is to climb only as high as you must. Prompt and context engineering (Ch. 7) is first — cheapest, instant to iterate, no training infrastructure. Retrieval (Ch. 11) is next when the gap is knowledge: facts the model lacks belong in context, not in weights. Only when the gap is behaviour — a consistent format, tone, structured output, or a latency/cost target at volume — does fine-tuning earn its cost. The guiding heuristic: fine-tune for form, retrieve for facts. Pre-training or continued pre-training is a last resort for deep domain shift, rarely justified for an application platform.
28.2 Fine-tuning and distillation
When you do train, the techniques form their own ladder. Supervised fine-tuning (often parameter-efficient via LoRA) teaches format and behaviour from input→output examples. Preference tuning (DPO and kin) aligns the model to chosen-over-rejected judgements. Distillation is the highest-leverage move for a platform: use a strong, expensive model plus your evaluation set to generate training data, then fine-tune a small, fast, cheap model to match it on your specific tasks — collapsing the cost and latency of routine sub-tasks (Ch. 6 routing, Ch. 20 cost) without sacrificing quality where it matters. The training data, in every case, comes from assets you already have: the gateway's logged inputs and feedback (Ch. 6) and your curated eval cases (Ch. 19). The train→evaluate→deploy loop is the evaluation flywheel of Chapter 19 with a training step inserted.
| Technique | Changes | Cost / iteration | Reach for it when |
|---|---|---|---|
| Prompt / context | What the model sees | Lowest · instant | Always first |
| RAG | Facts in context | Low | The gap is knowledge, not behaviour |
| SFT (LoRA) | Behaviour / format in weights | Moderate | Consistent form, tone, or structured output at volume |
| Preference (DPO) | Alignment to judgements | Moderate | You have chosen-vs-rejected feedback to learn from |
| Distillation | Big-model behaviour → small model | Moderate | Cut cost/latency of routine sub-tasks at quality |
| Continued pre-training | Deep domain knowledge in weights | Highest | Rarely — severe domain shift only |
28.3 Selection and adoption
Better models ship constantly, and the platform's gateway architecture (Ch. 6) exists precisely so you can adopt them without touching application code. The adoption discipline is empirical: evaluate a candidate against your eval set (Ch. 19), not a public leaderboard, scoring it on capability, latency, cost, safety behaviour, licensing, and data-residency terms; then roll it in gradually via the gateway's A/B or bandit routing, shifting traffic as it proves itself on your real metrics. Open-weights models you can self-host (vLLM, Ch. 21) trade operational burden for control, residency, and cost predictability; managed APIs trade control for zero operations. The choice is per task, and the gateway lets it differ per task.
28.4 Versioning, deprecation, and drift
A model is a versioned dependency, and providers deprecate versions on their schedule, not yours — a forced migration that can silently change behaviour. Manage it like any dependency: pin explicit model versions rather than floating aliases; re-run the eval set on every version change to catch the prompt drift of Chapter 7; keep a known-good previous version as an instant rollback target (the gateway makes this a config change); and maintain a model registry — the approved models and versions permitted per task, mirroring the MCP registry of Chapter 22 and governed by the same policy discipline as Chapter 17. A model in production that no one chose, evaluated, and pinned is an outage waiting for the provider's next deprecation notice.
Reach for prompting, then retrieval, then training — in that order — and stop at the first that clears the bar. Whatever you run, pin it, evaluate it, register it, and keep it swappable behind the gateway, so adopting a better model or surviving a deprecation is a measured config change, not a crisis.
Training is the most expensive, slowest-to-iterate intervention, and it cannot fix a knowledge gap — that is what retrieval is for. Fine-tuning on un-curated or contaminated data bakes errors into weights you then have to re-train to remove, and an un-evaluated custom model can regress in ways a prompt change never could. Always gate a custom model on the eval set, and never let it be the only model you can serve.