One workflow,from spec to self-healing.
A field manual for running the AI Development Lifecycle (AIDLC) end to end: a twelve-stage pipeline where humans steer with specs and gates, fleets of agents do the work in isolated sandboxes, and incidents feed fixes back to implementation. The workflow is the spine of this document.
Harness engineering and loop engineering are how you make each stage reliable — the deterministic scaffolding and the bounded feedback loops underneath the pipeline. They appear throughout as the "how," never as the headline.
Security and compliance run as a rail beneath all of it — the same gates that make agents trustworthy double as audit evidence, mapped to the SOC 2 Trust Services Criteria in §08 and packaged as a detachable auditor's runbook in §16.
The AIDLC, end to end
This is the canonical pipeline. A human steering layer sets intent and holds the gates; an orchestration layer fans work out to parallel agents in isolated sandboxes; automated review and security gates stand between a green PR and production; and an agentic-SRE loop closes back onto implementation. Everything else in this manual hangs off these twelve stages.
Rendered as a control plane, work flows top-to-bottom; the violet bands are where a human must act; the green/red dashed bands are gates; the amber block is the orchestration engine; and the dotted arc is the self-healing loop.
Specify
Spec Kit / Kiro / Tessl → versioned in repo. docs/ = system of record.
Plan & decompose
Exec-plans as repo artifacts; non-overlapping tasks.
Design
ADRs + mechanically-enforced architecture (layer linters, structural tests).
Write the code
Claude Code (Opus 4.8) / Codex (GPT-5.5) / Devin / Cursor bg. Tools via curated MCP subset.
Contained run
E2B / Daytona / Modal / devbox — QA-isolated, no prod, no egress.
Prove it
Agent-gen unit/integration/e2e; CDP/Playwright UI; coverage lints.
Iterate
Agent reviews own diff; loop until agent reviewers pass.
Green or escalate
Pre-push hooks (<1s) → selective tests → hard 2-round cap → else escalate.
Release
Progressive delivery (Argo Rollouts/flags); auto-rollback on SLO breach.
Watch
OTel + Grafana/Datadog; telemetry made legible to agents (LogQL/PromQL).
Agentic SRE
incident.io/Rootly/Resolve: triage → RCA → bounded remediation → fix PR.
Doc-gardening
An agent keeps docs/ fresh; freshness is CI-validated.
Sections 04–08 walk the workflow in order, with the real configs each stage needs. Sections 09–10 are the two enablers — harness engineering and loop engineering — that make stages 4–7 trustworthy. Section 11 is the orchestration layer (the fleet manager). The workflow is the destination; the enablers are the road.
Humans steer. Agents execute. Gates decide.
The pipeline only works because authority is split cleanly. Humans own intent and accountability; agents own throughput; deterministic gates own the go/no-go. The bottleneck has moved from writing code to reviewing it — so the leverage is in specs, decomposition, and the gates.
Intent & judgement
- The spec and acceptance criteria (stage 1)
- Plan approval (gate after stage 2)
- Architecture invariants (stage 3)
- PR review & merge accountability (gate 6b)
- Promotion to prod for high-risk (gate 8)
Throughput
- Implementation in isolated sandboxes (stage 4)
- Test generation (stage 5)
- Self-review iteration (stage 6)
- Fixing their own CI & remediation PRs (stages 7, 10)
Go / no-go
- CI with a hard 2-round cap (stage 7)
- AI review + SAST (gate 6b)
- SLO-breach auto-rollback (stage 8)
- Security/compliance throughout (stage 12)
Three load-bearing gates, condensed from about eight. Read end to end, the pipeline has roughly eight points where a human can act (the gate column in §03 tags them at stages 1, 2, 3, 6b, 8, 9, 10 and 12). In practice they concentrate into three high-value gates — approve the plan (after stage 2), review the PR (gate 6b), and promote to production (stage 8). This mirrors the prevailing industry framing that an agentic SDLC reduces roughly eight human interruptions to three checkpoints (Augment Code), raising reviewer leverage while preserving governance.
How many gates bind is risk- and autonomy-dependent. At the autonomous end they collapse toward roughly one — exception-based escalation (OpenAI permits but doesn't require human PR review; humans set acceptance criteria and step in only when judgment is required). At the strict / regulated end the count climbs to five or more — spec-driven tools alone gate requirements, design and tasks separately before a single line is written. Stripe sits in between, with mandatory human review on every PR.
Sensitive code is an intensifier, not a separate gate. Auth, crypto, payments and other high-blast-radius changes don't add a stage — they tighten the gates already present, forcing a mandatory human reviewer at 6b and a human promotion decision at 8 even where those gates would otherwise be optional or automated. Set the thresholds explicitly and encode them — don't leave it to habit.
Every stage: what the agent does, the gate, the enabler
One table to hold the whole workflow. The final column is the point of this manual: each stage is made reliable by a specific harness or loop mechanism (detailed in §09–10). The gate tags read HUMAN AGENT HARD.
| # | Stage | Agent does → gate | Representative tooling | What makes it reliable (the enabler) |
|---|---|---|---|---|
| 1 | Specify | NL → structured, testable spec (EARS) HUMAN | Spec Kit · Kiro · Tessl | Harness: repo as system of record; spec versioned in docs/ |
| 2 | Plan & decompose | Spec → non-overlapping tasks HUMAN: approve plan | exec-plans in-repo; plan modes | Harness: non-overlapping decomposition (the scarce skill, §11) |
| 3 | Design | ADRs + boundaries HUMAN: own invariants | ADRs; layer linters; structural tests | Harness: machine-enforced architecture (§09) |
| 4 | Implement | Writes all code in a sandbox AGENT | Claude Code (Opus 4.8) · Codex (GPT-5.5) · Devin · Cursor | Harness: disposable sandbox + curated MCP; Loop: inner loop |
| 5 | Test | Generates unit/integration/e2e AGENT | agent-gen suites · Playwright/CDP · coverage lints | Loop: tests are the iteration contract (computational sensors) |
| 6 | Self-review | Reviews own diff, iterates AGENT | evaluator subagent · fresh context | Loop: bounded self-review; default-FAIL contract (§10) |
| 7 | CI | Fixes own build, capped HARD 2-ROUND CAP | GitHub Actions · GitLab CI · pre-push hooks | Loop: hard iteration cap → escalate (§10) |
| 6b | AI review + SAST | Automated review & scan AGENT → HUMAN | CodeRabbit · Greptile · Diamond · Snyk · Semgrep · CodeQL | Loop: inferential sensors as a merge gate (§03,§10) |
| 8 | Deploy | Progressive delivery HUMAN: high-risk promo | Argo Rollouts · Flagger · feature flags | Harness: auto-rollback on SLO breach bounds blast radius |
| 9 | Observe | Emits agent-legible telemetry HUMAN: define SLOs | OpenTelemetry + Grafana · Datadog | Harness: telemetry made legible to agents (LogQL/PromQL) |
| 10 | Agentic SRE | Triage → RCA → fix PR HUMAN: escalation thresholds | incident.io · Rootly · Resolve AI | Loop: bounded remediation; closes [10]→[4] (§07) |
| 11 | Docs | Keeps docs/ fresh AGENT | doc-gardening agent · freshness linters | Harness: scheduled drift/GC agent; CI-validated |
| 12 | Security | Runs throughout HARD: auth/crypto/payments | SAST · dep CVEs · MCP gateway · audit | Harness: gateway as single policy-enforcement plane (§08) |
The shape of the job. Read the gate column top to bottom: there are about eight points where a human can act, but the load-bearing ones concentrate into three — approve the plan, review the PR, and promote to production — with spec and design sign-off folded into planning, and sensitive-code review tightening the rest. That is the entire post-transition role: engineers move up the stack from authoring lines to authoring intent and judging output.
The front of the pipeline is where humans win or lose
Underspecified intent multiplies errors across a whole fleet, so stages 1–3 carry the heaviest human gates. Three tool categories matter here — and they are not interchangeable: a portable process scaffold, an agentic IDE, and a spec-as-source framework.
Spec Kit vs Kiro vs Tessl — three different roles
| Dimension | GitHub Spec Kit | AWS Kiro | Tessl |
|---|---|---|---|
| What it is | Open-source CLI + commands/templates that scaffold spec-driven development inside your existing agent | Agentic IDE (VS Code fork) + CLI where specs are the unit of work | Spec-as-source framework and a registry of 10,000+ OSS "usage specs" |
| Artifacts | constitution.md, spec.md, plan.md, tasks.md | requirements.md, design.md, tasks.md + steering/ + hooks/ | .spec.md (frontmatter + capabilities + [@test]) + usage specs |
| Flow | /speckit.constitution → specify → clarify → plan → tasks → implement | Create Spec → requirements → design → tasks → execute; hooks on save | Agent writes specs → you approve → implement → verify; edits flow spec-first |
| Role in the pipeline | Portable, agent-agnostic process for stages 1–2 | Enterprise spec-first IDE (esp. AWS) for stages 1–3 | Specs as durable source of truth + reliable OSS API usage |
| NOT | Not an IDE, not a registry | Not a registry, not just templates | Not a scaffold, not an IDE — a registry + spec-as-source framework |
Tessl treats the specification as the primary artifact and lets code follow. Its Framework keeps specs (or AI-generated "vibe-specs") in the codebase as long-term memory, flows edits spec→code, and detects/reconciles drift; its Spec Registry ships 10,000+ usage specs for OSS libraries, installed like dependencies and optimized for agent readability (Tessl reports up to 3.3× improvement in correct API usage). It runs as a CLI for humans or an MCP server for agents — a different job from Spec Kit (a command scaffold) or Kiro (an IDE).
# bootstrap uvx --from git+https://github.com/\ github/spec-kit.git specify init APP # inside your agent: /speckit.constitution /speckit.specify <feature> /speckit.clarify /speckit.plan <tech> /speckit.tasks /speckit.implement
.kiro/
specs/<feature>/
requirements.md # EARS
design.md
tasks.md
steering/ # persistent
# project knowledge
hooks/ # on save:
# lint · test · scan
--- name: User Auth targets: [../src/auth/*.py] --- Users log in with email + password. def login(email, pw) -> Session [@test] ../tests/test_login.py # run: tessl eval run .
CONFIG EARS — the structured-spec grammar (stage 1)
EARS keeps acceptance criteria testable. It is the connective tissue between the human's intent and the agent's contract, and is the native requirements format in Kiro.
# Ubiquitous The system shall encrypt all data at rest. # Event-driven WHEN a user submits invalid credentials, the system shall return 401. # State-driven WHILE a session is active, the system shall refresh the token every 15m. # Unwanted IF a payment webhook signature is invalid, THEN the system shall reject it. # Optional WHERE multi-factor auth is enabled, the system shall require a TOTP code.
Stage 3 · architecture the agent cannot violate
Design is enforced mechanically, not by hoping the agent remembers. Encode a forward-only dependency direction and a single seam for cross-cutting concerns as linters set to error; the rule is then a hard gate for every agent on every run.
# MyProject — what this service does, in one line ## Commands - build: `make build` # lint rules are ERROR-level, not warn - test: `make test` ## Where things live (point outward; don't inline detail) - Architecture & layer rules .... docs/architecture.md - ADRs (why X over Y) .......... docs/adr/ - Testing strategy ............. docs/testing.md ## Boundaries - Dependency direction: Types → Config → Repo → Service → Runtime → UI - Cross-cutting concerns enter ONLY via the Providers interface - Never edit: .env, infra/prod/**, .git/**
A monolithic instruction file rots; keep AGENTS.md ~100 lines and point into a structured docs/ tree (one frontier agent-first repo composes 88 of them across subcomponents). And follow the field's highest-leverage habit: "if you can articulate what you don't like about the code, write that down" — as a lint, a structural test, or a bespoke reviewer, set to error.
Inside the box: implement → sandbox → test → self-review → CI
This is where loop engineering earns its keep. Each agent runs a bounded inner loop in an isolated sandbox: generate → run sensors → self-review → fix, exiting on pass and escalating when capped. Nothing here is open-ended.
CONFIG The sandbox (stage 4) — disposable & isolated
Isolation is what lets an agent run with full permissions safely; pre-install dependencies at build time so a new agent starts ready to code (the "warm devbox" / cached-dev-container idea). Two real options:
{
"name": "my-app",
"image": "mcr.microsoft.com/devcontainers/python:3.12",
// run once at build → agents start in seconds
"onCreateCommand": "pip install -r requirements.txt",
"forwardPorts": [3000, 8000]
}
# pip install e2b-code-interpreter from e2b_code_interpreter import Sandbox with Sandbox() as sbx: sbx.run_code("x = 1") r = sbx.run_code("x += 1; x") print(r.text) # 2
CONFIG Hooks (stages 4–7) — turn the design into enforcement
Claude Code hooks make "the agent can't make that mistake again" literal: auto-format/lint every edit, deny destructive commands, inject branch context, and block stopping while tests fail. exit 2 is a blocking error fed back to the agent.
{
"hooks": {
"PostToolUse": [
{ "matcher": "Write|Edit|MultiEdit",
"hooks": [{ "type": "command", "command": "npx prettier --write \"$CLAUDE_TOOL_INPUT_FILE_PATH\" && npx eslint --fix \"$CLAUDE_TOOL_INPUT_FILE_PATH\"" }] }
],
"PreToolUse": [
{ "matcher": "Bash",
"hooks": [{ "type": "command", "command": "echo \"$CLAUDE_TOOL_INPUT\" | grep -qE 'rm -rf|DROP TABLE' && exit 2 || exit 0" }] }
],
"Stop": [
{ "hooks": [{ "type": "command", "command": "npm test || exit 2" }] }
]
}
}
CONFIG The CI cap (stage 7) — green in two rounds, or escalate
The hard iteration cap is the single most important loop bound: it prevents infinite token-burning loops and routes genuinely hard problems to a human. (Documented at the strict end as a two-round cap.)
# the agent gets at most 2 attempts to make CI green, then a human is paged MAX=2; n=0 until npm run ci; do n=$((n+1)) [ "$n" -ge "$MAX" ] && { gh pr comment --body "CI red after $MAX rounds — escalating to a human."; exit 1; } claude -p "CI failed. Read the logs, fix the cause, do not disable tests." done echo "green in $n round(s)"
Every inner loop in stages 4–7 carries four bounds: an iteration cap (e.g. --max-iterations or /goal), a budget cap, a kill switch (a watched AGENT_STOP file), and the CI round cap. Configure all four before you let a stage run unattended. Full mechanics in §10.
Between a green PR and production
A green CI means the code runs, not that it's right. Two gates stand between the PR and prod: an automated gate (AI review + SAST + dependency CVEs) and a human gate whose strictness you set by blast radius. These are inferential sensors used as gates — rich but non-deterministic, so they gate rather than decide alone.
Template-compliant, green CI
The inner loop (§05) has exited; the change arrives with tests and a description.
AI review + SAST
CodeRabbit / Greptile / Diamond for review; Snyk / Semgrep / CodeQL for security; dependency CVEs.
Merge accountability
Strict (every PR) → autonomous (optional). Always for auth/crypto/payments/high-blast-radius.
CONFIG AI review gate (6b)
reviews: profile: "assertive" request_changes_workflow: true # block merge until review threads resolve path_instructions: - path: "src/payments/**" instructions: "High blast radius. Require a human approver; flag any auth or money-movement change." tools: semgrep: { enabled: true } github-checks: { enabled: true }
The design reason the 6b gate is load-bearing is that AI-authored code carries more correctness and security defects per unit than human code — which is exactly why layered review and SAST sit here. An earlier draft of this manual cited a specific December-2025 study for the magnitude; that figure predates the 60-day recency window and has been dropped (see §15). No in-window first-party replacement was found, so the manual states the principle without a stale statistic. Keep the gate regardless — it is cheap insurance against the failure mode.
Ship safely, watch closely, heal automatically
Past the gates, the workflow keeps containing risk: progressive delivery bounds the blast radius, telemetry is made legible to agents, an agentic-SRE loop turns incidents into fix PRs that re-enter implementation, and a doc-gardening agent keeps the system of record fresh.
Progressive delivery
Canary or flagged rollout with automatic rollback the moment an SLO is breached. The human gate here is reserved for high-risk service promotions.
Agent-legible telemetry
OpenTelemetry into Grafana/Datadog, with logs and metrics queryable by agents (LogQL/PromQL) so the SRE loop can reason over them. Humans define the SLOs.
CONFIG Deploy with auto-rollback (stage 8)
apiVersion: argoproj.io/v1alpha1 kind: Rollout spec: strategy: canary: steps: - setWeight: 10 - pause: { duration: 5m } - analysis: # auto-rollback if the SLO query fails templates: [{ templateName: error-rate-slo }] - setWeight: 50 - pause: { duration: 10m }
Stage 10 · the agentic-SRE loop that closes back onto [4]
This is the loop that makes the pipeline self-healing. On alert, an SRE agent triages, runs root-cause analysis over the agent-legible telemetry, performs a bounded remediation, and opens a fix PR — which re-enters the workflow at stage 4 and passes the same gates. A human stays on the loop via escalation thresholds.
Bounded, reversible, approved
The agentic-SRE tools in the field converge on one rule: actions must be bounded, reversible, and require human approval at defined confidence/severity thresholds. Augment the on-call; don't hand over the pager.
Doc-gardening agent
A scheduled agent keeps docs/ — the system of record — fresh, with freshness validated in CI. This is harness maintenance: fighting entropy on a cadence so the next agent reads accurate context.
Security, compliance & audit, throughout
Stage 12 is not a step — it's a rail spanning the whole pipeline. In an agentic SDLC the same gates that make agents trustworthy double as your control evidence: the change-management trail an auditor samples is produced automatically, on every PR. The job is to make those gates emit immutable, attributable, replayable records by construction — so a SOC 2 audit becomes a query against artifacts you already have.
The two questions a SOC 2 audit asks
Are the controls designed correctly?
Show the gates exist and are sound: branch protection requires review + SAST + green CI; the MCP gateway enforces an allowlist; sandboxes are isolated from prod; deploys roll back on SLO breach.
Evidence = configuration. Largely your settings and policy-as-code, captured at one date.
Did they operate effectively?
Show the gates actually fired on every change across the window: each merge had a reviewer who wasn't the author, SAST ran on 100% of PRs, no agent bypassed the gateway, the audit log is complete and tamper-evident.
Evidence = operating records over time. This is where most orgs struggle — and where an agentic pipeline can excel.
When humans hand-merge, change-management evidence is patchy and reconstructed at audit time. When every change is an agent-authored PR through a fixed gate, the evidence is uniform and machine-generated — provided you wire the gates to record who/what authored it, who reviewed it, which scans ran, and which scoped identity acted. The discipline that makes agents reliable is the same discipline that makes you audit-ready.
Mapping the pipeline to the Trust Services Criteria
Every AIDLC mechanism already maps to a SOC 2 Common Criterion. The columns that matter for the audit are the last two: what proves the control was designed (Type 1) and what proves it operated (Type 2).
| AIDLC mechanism (stage) | SOC 2 criterion | Type 1 — design evidence | Type 2 — operating evidence |
|---|---|---|---|
| Plan approval gate (stage 2) | CC3, CC8.1 | Documented plan-approval workflow | Approval record linked to each change over the period |
| PR review + human gate (6b) | CC8.1 | Branch protection requires a review | 100% of merges had a reviewer ≠ author |
| AI review + SAST + dep CVEs (6b, 7) | CC7.1, CC3.2, CC4.1 | Scanners set as required status checks | Scan results retained per PR; findings triaged & closed |
| MCP gateway: RBAC, allowlist, OAuth (4) | CC6.1, CC6.2, CC6.3 | Gateway policy + per-agent scoped identities | Access logs; zero shared tokens; periodic access review |
| Disposable sandbox, no prod/egress (4) | CC6.6, CC6.7 | Sandbox isolation & egress config | Evidence agents never touched prod data in the window |
| Progressive delivery + auto-rollback (8) | CC8.1, A1.2 | Rollout policy with SLO analysis | Deploy & rollback records per release |
| Observability + agentic SRE (9–10) | CC7.2, CC7.3, CC7.4 | Monitoring + incident-response runbook | Incident timeline, RCA, remediation PRs |
| Immutable audit log (12) | CC2.1, CC4.1, CC7.2 | Append-only SIEM pipeline | Complete, tamper-evident log across the period |
| Docs as system of record (1, 11) | CC1.x, CC2.1 | Structured docs/ + ownership | Doc-freshness CI logs; change history |
References follow the AICPA 2017 Trust Services Criteria (rev. 2022); Security (CC-series) is mandatory. Confidentiality (C1) and Privacy (P-series) attach the same way when in scope — map the data-handling stages (sandbox isolation, audit redaction) to them.
Agents are non-human identities, and SOC 2's access criteria (CC6.1–6.3) demand least privilege and attribution for every actor. The failure mode is a shared service token several agents reuse — it collapses attribution and over-grants access, failing CC6 on both design and operation. Issue per-agent, per-task scoped credentials, route every tool call through the gateway as the policy-enforcement point, and keep each call attributable to one identity. That is also what makes the periodic access review CC6.2/6.3 expect even possible.
CONFIG Change management as code (CC8.1)
The gate that satisfies CC8.1 should be enforced by the platform, not by convention. A branch ruleset makes review, status checks, and linear history mandatory — and is itself the Type 1 design artifact.
{
"target": "branch", "enforcement": "active",
"conditions": { "ref_name": { "include": ["refs/heads/main"] } },
"rules": [
{ "type": "pull_request",
"parameters": { "required_approving_review_count": 1,
"require_code_owner_review": true,
"dismiss_stale_reviews_on_push": true } },
{ "type": "required_status_checks",
"parameters": { "required_checks": [
{ "context": "sast/semgrep" }, { "context": "deps/cve-scan" },
{ "context": "ci/tests" }, { "context": "compliance/evidence-gate" } ] } },
{ "type": "required_linear_history" },
{ "type": "non_fast_forward" }
]
}
CONFIG Make compliance a blocking check, not a hope
A small policy-as-code gate (the compliance/evidence-gate check above) refuses the merge unless the evidence a Type 2 auditor will sample actually exists. Encoding it means the control can't be silently skipped under deadline pressure.
package merge.evidence default allow := false allow if { input.review.approvals >= 1 input.review.approver != input.pr.author # CC8.1 — reviewer ≠ author input.scans.sast == "passed" # CC7.1 input.scans.cve_high == 0 # CC7.1 input.actor.identity_scope == "per-task" # CC6 — no shared tokens input.plan.ref != "" # CC3/CC8 — change traces to an approved plan }
CONFIG The audit record — your Type 2 evidence unit
Type 2 lives or dies on the operating log. Emit one append-only record per gate event, attributable to a single identity, hash-chained, shipped to an immutable store. This is the artifact the auditor samples.
{
"ts": "2026-06-15T14:03:11Z",
"actor": { "identity": "agent://impl-7f3a", "scope": "task-4821", "human_owner": "u/eng-114" },
"event": "pr.merged", "pr": 9182, "change_ref": "plan/exec-4821",
"controls": { "reviewer": "u/eng-220", "sast": "passed", "cve_high": 0, "ci": "green" },
"tool_calls": 37, "prev_hash": "e3b0c4…", "hash": "a17fde…" // hash-chained = tamper-evident
}
CONFIG Least-privilege MCP for stage 4 (CC6)
{
"servers": {
"github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"],
"env": { "GITHUB_TOKEN": "${scoped_repo_token}" } },
"prod-db": { "command": "uvx", "args": ["db-mcp@1.4.2"],
"disabled": true, "comment": "never enabled in the agent sandbox" }
}
}
The MCP spec enforces neither authentication nor authorization, so a gateway is the single policy-enforcement plane for CC6: a per-consumer virtual key with an explicit tool allowlist, deny-by-default at both advertisement and execution, per-user OAuth 2.1, and an immutable per-call audit log to your SIEM. Documented gateway capabilities are vendor-reported; start in log-only mode, then enforce. Curate tightly — too many tools degrade the agent and slow the harness start.
For any change sampled from the audit window: it traces to an approved plan; it was reviewed by someone other than its author; SAST and CVE scans ran and passed; the acting identity was task-scoped, not shared; and the whole chain sits in a complete, tamper-evident log. Hold those five for 100% of changes across 3–12 months and the agentic pipeline isn't a compliance risk — it's the strongest change-management evidence you've ever had.
Harness engineering — the "how" under stages 1–4 & 8–12
A coding agent is a model wrapped in a system: Agent = Model + Harness, where the harness is everything except the model. It is where almost all of your engineering leverage lives, and it is what makes the front and back of the pipeline reliable. The operating heuristic of the field:
"Every time the agent makes a mistake, don't just hope it does better next time. Engineer the environment so it can't make that specific mistake the same way again."— Mitchell Hashimoto, on harness engineering (Feb 2026)
Pictured as an exploded view, the agent is a small model core wrapped in concentric, mostly-deterministic rings — and every ring is something you author and version. Each one also maps onto the pipeline: context & tools feed stages 1–4, the sandbox is stage 4, the sensors are stages 5–6b, and orchestration & hooks span the whole engine.
The seven harness components — and the stage each one serves
docs/ + lean AGENTS.md → stages 1, 3, 11Feedforward guides vs feedback sensors
The clearest current framing (Thoughtworks, Apr–May 2026) splits the harness into what you give the agent before it acts and what you measure after. The "after" signals divide into two kinds with very different economics — and they map directly onto pipeline stages.
Loop engineering — the "how" under stages 4–7 & 10
Loop engineering is designing the agent's iteration cycles so they converge on something verified and cannot run forever. The inner loop (§05) is one instance; the most reliable inferential sensor in the field is a fresh-context evaluator paired with a default-FAIL contract, so an agent can't claim a success it hasn't observed.
Under-specify on purpose
Sets goals + acceptance criteria, leaves implementation open — early over-specification cascades downstream.
Build against the contract
Writes code + tests; gathers evidence (screenshots, logs, DB state) by driving the app.
PASS / NEEDS_WORK
No Write/Edit tools; judges the diff + evidence it never built. On NEEDS_WORK its findings become the next session's prompt.
CONFIG Fresh-context evaluator + default-FAIL (stage 6)
--- name: evaluator description: Skeptical reviewer. Reads the diff and the builder's evidence, returns PASS or NEEDS_WORK with specific findings. No Write/Edit tools. tools: Read, Glob, Grep, Bash --- You did NOT build this change. Begin your reply with the bare word PASS or NEEDS_WORK on its own line. Judge ONLY against the spec and the evidence that was actually Read.
// every acceptance criterion starts FALSE { "feature-1": { "passes": false }, "feature-2": { "passes": false } } # PreToolUse verify-gate.sh blocks writing "passes": true until the agent # has Read real evidence (a screenshot / console log) this session: { "decision": "block", "reason": "No evidence Read this session. Open the evidence file first, then retry." }
CONFIG Repeat-until-green, with a hard cap (stages 4–7)
# first-party plugin: loop until a completion promise or the cap /plugin install ralph-wiggum@claude-plugins-official /ralph-loop "<prompt>" --max-iterations N --completion-promise "DONE" # built-in & version-stable — prefer this for "keep working until correct": /goal all tests in test/auth pass and lint is clean
The discipline that ages well: re-audit the harness every model release
A harness encodes assumptions about what the model can't do; those assumptions go stale as models improve. On every major upgrade, ask "which components exist only to compensate for now-fixed limitations?" and delete them — the best harnesses get simpler over time.
When stage 4 becomes many agents at once
The orchestration layer in the pipeline is a multi-agent workspace manager: it isolates each agent (worktree or container), tracks work that survives restarts, runs watchdogs, and merges through a verifying queue. It is the control plane for stages 4–7 at scale.
| Tool | Type | Isolation | Coordination model | License | Best for |
|---|---|---|---|---|---|
| GitHub Agent HQ | control plane | cloud + worktree | "mission control" to assign/steer/track many agents in GitHub/VS Code | commercial | Governed multi-agent inside GitHub (rolling out through 2026) |
| Conductor | local orchestrator | worktree per task | human-in-loop dashboard; multi-model compare | macOS · free (BYOK) | Mac devs wanting a clean dashboard beside the editor |
| Composio Agent Orchestrator | autonomous | worktree/branch/PR | each agent fixes its own CI + review comments | commercial | The "orchestrator-shaped" autonomous step |
| Gastown | unattended town | worktree (git-backed) | coordinator (Mayor) + watchdogs + bisecting merge queue | OSS (MIT) | Self-hosted autonomous fleets, 20–50 agents |
| Factory | enterprise | cloud "Droid Computers" | coordinator → specialized droids; Missions | commercial | Enterprise, ticket-to-PR, multi-day Missions |
| Devin (Cognition) | enterprise | cloud + local | agent-first; Agent Command Center | commercial | Enterprise autonomy with an agent control center |
| Sculptor (Imbue) | containers | Docker per agent | parallel agents + Pairing Mode (IDE sync) | free (BYO LLM) | When isolation & safety matter most |
Isolation buys parallelism, not coordination. The moment two agents must modify overlapping code, worktree/container isolation doesn't save you — you get several drift-parallel solutions and merge review becomes the bottleneck. The genuinely scarce skill is non-overlapping task decomposition (stage 2). No tool removes that responsibility; merge queues and coordinator agents only manage its consequences.
CONFIG Gastown — agent presets + a concurrency cap
# define agent aliases (any CLI runtime) + a default gt config agent set codex-low "codex --thinking low" gt config agent set claude-haiku "claude --model haiku --dangerously-skip-permissions" gt config default-agent codex-low # cap concurrent workers so you don't exhaust API rate limits gt config set scheduler.max_polecats 5 # >0 enables deferred dispatch gt sling gt-abc gastown # auto-defers when a cap is set
Read the open ones as a blueprint. Even if you never run Gastown, its architecture is the clearest open catalogue of what an unattended orchestrator needs: persistent work state, a coordinator, isolated worker storage, a verifying (bisecting) merge queue, health watchdogs, severity-routed escalation, capacity governance, and OpenTelemetry. Evaluate every commercial platform against that checklist.
A concrete stack for the pipeline
One defensible pick and a strong alternative for each stage. Tool names are representative of the field, not endorsements — choose against your own constraints and the recency caveats in §15.
| Stage | Primary pick | Strong alternative |
|---|---|---|
| 1 Specify | GitHub Spec Kit | Kiro (IDE) · Tessl (spec-as-source) |
| 2 Plan | Spec Kit /tasks + exec-plans in repo | Claude Code plan mode |
| 3 Design | ADRs + layer linters + structural tests | Sourcegraph / CodeScene context |
| 4 Implement | Claude Code (Opus 4.8) | Codex (GPT-5.5) · Cursor · Devin · Droid |
| 4 Sandbox | Warm devbox pool / Sculptor | E2B · Daytona · Modal |
| 4 Tools | MCP via governed gateway | curated per-task tool subsets |
| 5 Test | Agent-gen suites + coverage lints | Playwright/CDP · property tests |
| 6b Review | CodeRabbit (breadth) + Greptile (depth) | Graphite Diamond · Claude Code Review |
| 7 CI | GitHub Actions + pre-push hooks | GitLab CI · Depot |
| 8 Deploy | Argo Rollouts + feature flags | Flagger · LaunchDarkly |
| 9 Observe | OpenTelemetry + Grafana | Datadog · New Relic |
| 10 SRE | incident.io AI SRE | Rootly · Resolve AI |
| 11 Fleet | Gastown (OSS) / Factory (enterprise) | Agent HQ · Conductor · Sculptor |
| 12 Security | Snyk + Semgrep + CodeQL + MCP gateway | SonarQube · Checkmarx |
Stage-4 models — current lineup (vendor-reported, harness-sensitive)
| Model | Released | Vendor headline | Price /M (in/out) |
|---|---|---|---|
| Claude Opus 4.7 | 2026-04-16 | 87.6% SWE-bench Verified | $5 / $25 |
| Claude Opus 4.8 | 2026-05-28 | 88.6% SWE-bench Verified | $5 / $25 |
| Claude Fable 5 (GA) | 2026-06-09 | 95.0% SWE-bench Verified | $10 / $50 |
| OpenAI GPT-5.5 | 2026-04-23 | 82.7% Terminal-Bench 2.0 · powers Codex | $5 / $30 † |
| Cursor Composer 2.5 | 2026-05-18 | 79.8% SWE-bench Multilingual | $0.50 / $2.50 |
| Gemini 3.5 Flash | 2026-05-19 | powers Antigravity 2.0 agent harness | n/a |
† GPT-5.5 price via secondary source — unconfirmed. SWE-bench Verified is near saturation and harness-sensitive: the same model scores differently under different harnesses, which is the whole reason §09–10 matter more than this table. Don't select a model on a leaderboard — evaluate it inside your harness on your own task set.
Instrument the workflow, not the hype
Measure the pipeline's effect on outcomes you already track, plus the agentic signals that tell you the fleet is healthy. Resist vanity counters.
Outcome metrics
- Change failure rate — the most meaningful signal for AI code
- MTTR — the most distortion-resistant
- Deploy frequency & lead time (distort when much code is AI-gen)
Is the AI code good?
- AI code share & adoption
- Code durability — does it survive or get reverted?
- Time-in-review (quantifies the new bottleneck)
Is the fleet healthy?
- Task success rate & review-intervention rate
- CI pass-on-first-try; retries-to-merge; escalation rate
- Cost-per-task; SRE autonomous-resolution rate
Are we shipping risk?
- AI-introduced vulnerability / logic-error rate
- SAST findings per PR
- % of incidents traceable to AI-authored changes
"Tokenmaxxing" — treating raw token spend or lines generated as success — is a vanity metric flagged by DORA in 2026. It tells you how much AI is used, never whether the org is more effective. Tie every stage's metric back to an outcome.
Stand the workflow up in stages, with hard gates
Don't wire the whole pipeline at once. Build the harness floor before the fleet; orchestration on a weak harness amplifies dysfunction. Each step has an exit criterion mapped to the workflow.
Baseline
- Instrument cost/merged PR, time-to-merge, review velocity, defect-escape, compute/dev
- The numbers that gate everything after
Harness floor
- Repo as system of record; lean AGENTS.md (stages 1, 3)
- Top-3 failures → error-level lints (stage 3)
- Hooks + disposable sandboxes (stage 4)
Bound the loops
- Inner loop with hard caps (stages 4–7)
- Fresh-context evaluator + default-FAIL (stage 6)
- AI review + SAST as merge gates (6b, 12)
Close the loop & scale
- Spec tool by need (stage 1); progressive delivery (stage 8)
- Agentic SRE feeding [10]→[4]; doc-gardening (stages 10–11)
- Add an orchestrator (stage 11); audit the harness each model release
What's in window, what's quarantined
This manual applies a hard 60-day cut (≥ 2026-04-01) for any claim, statistic, benchmark, version, or "what a company does today," and uses primary engineering sources only. Foundational harness/loop posts that predate the window are used for stable architecture, never for time-sensitive metrics.
| Source | Date | Window | Used for |
|---|---|---|---|
| Anthropic — Scaling Managed Agents | 2026-04-08 | IN | Harness-decoupling thesis |
| anthropics/cwc-long-running-agents (repo) | 2026 · CwC | IN | Evaluator, verify-gate, default-FAIL configs (§10) |
| anthropics/claude-code — ralph-wiggum / /goal | 2026 | IN | Loop primitives (§10) |
| Thoughtworks/Fowler — harness eng. + sensors | 2026-04-02 / 05-13 | IN | Guides/sensors taxonomy (§09) |
| Anthropic — Opus 4.7 / 4.8 launches + system card | 2026-04-16 / 05-28 | IN | Model facts (vendor) |
| OpenAI — GPT-5.5 launch · Codex docs | 2026-04-23+ | IN | Model + Codex facts (vendor) |
| Cursor / Google — Composer 2.5 · Antigravity 2.0 | 2026-05-18 / 19 | IN | Model + product facts (vendor) |
| github/spec-kit · kiro.dev · tessl.io | 2026 active | IN | Spec Kit / Kiro / Tessl roles + configs (§04) |
| gastownhall/gastown · gastown.dev | 2026 active | IN | Orchestrator config (§11) |
| imbue.com (Sculptor) · e2b.dev · Argo Rollouts | 2026 active | IN | Sandbox / deploy configs (§05,§07) |
| Claude Code hooks · code.claude.com | stable schema | IN | Hook config (§05) |
| OpenAI — "Harness engineering" | 2026-02-11 | QUARANTINED | Architecture only (AGENTS.md, layering) — NOT live metrics |
| Anthropic — "Harness Design for Long-Running Dev" | 2026-03-24 | QUARANTINED | Planner/generator/evaluator shape only |
| Stripe — "Minions" | 2026-02 | QUARANTINED | Blueprints, 2-CI-cap, devboxes, Toolshed — NOT current PR/week |
| arXiv — "Securing the MCP" 2511.20920 | 2025-11 | QUARANTINED | Stable controls taxonomy only (§08) |
| MCP-gateway vendor pages · sandbox comparisons | 2026 | IN | Vendor-reported — capability claims flagged |
The earlier draft's December-2025 AI-code-defect study was dropped (pre-window); §06 states the principle without it. The headline throughput numbers — OpenAI's "1M lines / 3.5 PRs per engineer per day" and Stripe's "1,300+ PRs/week" — are pre-window and are deliberately not presented as current; only Stripe's stable architecture (two-CI cap, disposable devboxes, curated Toolshed) is used. Listicle/aggregator/vendor-comparison sites were not cited for any claim — only used to locate primaries, and flagged vendor-reported where no primary existed. Fast-moving repo field names (Gastown especially) should be verified against main before copy-paste.
The SOC 2 evidence runbook
A detachable one-pager that operationalizes §08. For each technical control the pipeline enforces, it names the criterion, the exact evidence artifact, where it's stored, who owns it, and how often it's collected. Type 1 wants the design artifacts (configuration); Type 2 wants the operating records across the audit window.
| Control area | TSC | Evidence artifact (what you hand over) | System of record (where it lives) | Owner · cadence |
|---|---|---|---|---|
| Change authorization | CC8.1 | Branch ruleset (required review + checks); PR approval with reviewer ≠ author | GitHub rulesets + PR / audit log | Platform · every merge |
| Plan / intent approval | CC3, CC8.1 | Approved exec-plan linked to the change | docs/exec-plans/ | Eng lead · per change |
| Secure-dev scanning | CC7.1, CC4.1 | SAST + dependency-CVE results and triage, per PR | CI logs + Semgrep / Snyk | AppSec · per PR |
| Logical access — agent identity | CC6.1–6.3 | Gateway RBAC/allowlist policy; per-agent identity registry; access-review sign-off | IdP + gateway config + SIEM | Security · logs continuous, review quarterly |
| Boundary protection | CC6.6–6.7 | Sandbox isolation & egress (network policy) | IaC repo + cloud config | Platform · config + runtime logs |
| Supply-chain integrity | CC6.8, CC9 | Dependency pins, private registry, SBOM per build | Registry + artifact store | AppSec · per build |
| Deployment control | CC8.1, A1.2 | Progressive-rollout policy; deploy + rollback records | Argo / CD logs + release records | SRE · per release |
| Monitoring & detection | CC7.2–7.3 | SLO/alert config; alert + triage records | Observability platform | SRE · continuous |
| Incident response | CC7.4–7.5 | IR runbook; incident timeline, RCA, remediation PR | incident.io / Rootly + repo | On-call · per incident |
| Audit logging | CC2.1, CC4.1 | Append-only, hash-chained audit-event stream | SIEM (WORM / immutable) | Security · continuous |
| System docs | CC1, CC2.1 | Structured docs/ + freshness CI | repo docs/ | Eng · continuous |
One sampled change, one query
The point of wiring evidence into the gates: an auditor picks any merged change from the period, and a single query reconstructs its entire control chain from the hash-chained audit record plus the artifacts it points to.
# 1) one query returns the gate record for the sampled change $ aud get --event pr.merged --pr 9182 actor agent://impl-7f3a (scope task-4821 · owner u/eng-114) reviewer u/eng-220 sast passed cve_high 0 ci green change_ref plan/exec-4821 hash a17fde… (chain verified) # 2) cross-check the artifacts the record points to GitHub PR 9182 ......... approver ≠ author ✓ (CC8.1) CI run for the merge ... required checks present ✓ (CC7.1) docs/exec-plans/4821 ... approved before work began ✓ (CC3) SIEM chain 0…9182 ...... unbroken, append-only ✓ (CC4.1)
One record plus the artifacts it references closes the loop for that change. Repeat across the auditor's sample; if all five Type 2 conditions from §08 hold for every sampled change, the operating-effectiveness test passes.
This runbook covers the technical, automated controls the pipeline enforces — CC6 (access), CC7 (operations & monitoring), CC8 (change management), CC9 (risk mitigation), and A1 (availability) where in scope. Org-level controls live elsewhere: CC1 governance, CC2 role communication, HR onboarding/offboarding, vendor management, the formal risk-assessment cadence, and physical security belong in your GRC platform, not here. Criteria references follow the AICPA 2017 Trust Services Criteria (rev. 2022). This is an engineering implementation guide, not legal or audit advice — your auditor sets the final scope and sampling.