Harness Engineering

Presented at the Brisbane AI Paper Reading Club (Oct 2026) on the paper:
Harness Engineering: Anatomy, Architecture and Evolution of Coding Agents, https://arxiv.org/abs/2609.00006

What do modern coding agents actually look like under the hood? Join the AI Paper Reading Club this Tuesday where I'll present a source-code study of 11 leading coding agents, covering nearly 4 million lines of code across systems such as Claude Code, Codex, Gemini CLI and OpenHands. We'll explore the seven core subsystems that make up a coding-agent harness, examine how these systems implement loops, tools, memory, safety and orchestration, and discuss 13 surprising observations from the study. Along the way we'll see why many popular assumptions about agent frameworks and RAG don't appear in production systems, why file editing may be more important than prompting, and how coding-agent harnesses are evolving into full-fledged platforms.

Transcript

1. HARNESS ENGINEERING LOOP TOOLS MODEL What production source code tells us about coding agents SAFETY Mark Perry – Principal AI Engineer Based on “Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents” CONTEXT
2. 02 MOTIVATION The Missing Layer in Model-Centric Discussion • Model capability is only part of the coding-agent experience. • The runtime controls how actions become repository changes. • Similar models can operate through very different harnesses. H ARNES S ENG INEERING 2
3. 03 MOTIVATION Task Completion Versus Production Readiness Task success: did the issue get resolved? H ARNES S ENG INEERING Operational trust: were actions safe, reviewable, and recoverable? Adoption also depends on interaction, integration, and cost. 3
4. 04 MOTIVATION Research Question and Platform Thesis 01 02 Question: "What architectural patterns recur across codingagent harnesses? H ARNES S ENG INEERING Output: anatomy, observations and recommendations. 4
5. 05 METHOD Provider-Native Systems Claude Code Codex Anthropic OpenAI Gemini CLI Mistral Vibe Google Mistral AI H ARNES S ENG INEERING 5
6. 06 METHOD Open-Source Diversity and Contrast Points Coding-focused: OpenHands, Aider, Mini-SWE-Agent, Pi, OpenCode. Non-SWE corpus contrast: OpenClaw. Hybrid coding/personal assistant: Hermes. Separate meta-layer contrast: Omnigent. H ARNES S ENG INEERING 6
7. 07 METHOD Harness Size, Largest to Smallest Scale Approx. lines of code Harnesses Very large ~600K+ Codex (~1.1M), Gemini CLI, Hermes Large ~500K+ Claude Code, OpenClaw, OpenCode Medium ~80–110K OpenHands, Pi Small ~20–40K Aider, Mistral Vibe Tiny ~5K Mini-SWE-Agent Qualitative source-tree scale. Harnesses share size tiers. Counts vary by language and counting method. H ARNES S ENG INEERING 7
8. 08 METHOD How the Source Audit Works Approximately four million lines across Python, TypeScript, and Rust. Inspect manifests, imports, control paths, and subsystem implementations. Compare retained April and July snapshots for eight systems. Snapshot -> subsystem analysis -> cross-system comparison -> source diff. H ARNES S ENG INEERING 8
9. 09 METHOD What This Method Can and Cannot Establish 01 Can reveal contracts, control flow, persistence, and policy surfaces. H ARNES S ENG INEERING 02 03 Cannot establish runtime speed or matched-model success rates. Source visibility, selection, and interpretation constrain conclusions. 9
10. 10 F R A ME W OR K A Runtime, Not a Linear Pipeline Executes permitted actions and returns observations. Repeats until a stop condition is satisfied. Calls the model with selected context. H ARNES S ENG INEERING 10
11. 11 F R A ME W OR K Four Terminology Boundaries Scaffold: structural code; closely related to harness. Evaluation harness: wraps an agent to assess it. Framework: library used to build an agent. Meta-harness: coordinates whole harnesses. H ARNES S ENG INEERING 11
12. 12 F R A ME W OR K The Seven-Subsystem Map Agent loop, LLM integration, tools and actions. Memory and context, safety and permissions. Orchestration, extensibility. Attributed redraw of Figure 1 with the loop at the center. H ARNES S ENG INEERING 12
13. 13 F R A ME W OR K Trace a Bug Fix, Request to Edit 01 02 03 Load the goal and repository instructions. Search and inspect the relevant implementation. Propose an edit and evaluate permissions. H ARNES S ENG INEERING 13
14. 14 F R A ME W OR K Trace a Bug Fix, Verification to Completion Feed failures back or establish completion evidence. H ARNES S ENG INEERING Preserve useful state and report the result. Execute the edit and run a focused check. 14
15. 15 AGENT LOOPS The Minimal Action-Observation Cycle Build context -> call model -> execute -> observe. Mini-SWE-Agent demonstrates the small-loop floor. Stop when the model finishes or a limit is reached. H ARNES S ENG INEERING 15
16. 16 AGENT LOOPS Reflection and Delegation Are Added Shapes Aider: apply edits -> lint/test > corrective reflection. Coordinator-worker: delegate -> collect -> synthesize. These shapes can compose with iterative actionobservation. Iterative, reflection, and coordinator-worker diagrams. H ARNES S ENG INEERING 16
17. 17 AGENT LOOPS Event Logs Versus Middleware Policies 01 OpenHands: eventsourced conversations and replayable state. H ARNES S ENG INEERING 02 03 Mistral Vibe: composable turnlevel middleware. Both retain the underlying inference/action cycle. 17
18. 18 AGENT LOOPS Stopping Requires a Contract Verification during execution or at turn exit. H ARNES S ENG INEERING Model finish, resource cap, or repetition signal. Hermes can veto an unverified code-mutating completion. 18
19. 19 LLM INTEGRATION Provider Integration Strategies Tight coupling: Claude Code, Codex, Gemini CLI. Shared abstraction: LiteLLMbased systems. Provider-first fallback: Mistral Vibe. Owned transports/transforms: Hermes, Pi, and OpenCode's SDK-based integration. H ARNES S ENG INEERING 19
20. 20 LLM INTEGRATION Optimizations and the Update Loop Multi-provider systems can expose caching and reasoning controls. Provider-specific metadata pays the compatibility cost centrally. Vendor-controlled catalogs support coordinated model/harness updates. Provider catalog updates versus maintained compatibility metadata. H ARNES S ENG INEERING 20
21. 21 LLM INTEGRATION Prompt Guidance Is Not Enforcement 01 Recurring prose: be concise, read first, avoid unrelated changes. H ARNES S ENG INEERING 02 03 Runtime gates enforce what prose merely requests. Removing a rule does not prove the model internalized it. 21
22. 22 TOOLS & EDITING Design the Agent-Computer Interface Contracts include schemas, errors, permissions, and output limits. H ARNES S ENG INEERING Common actions: shell, search, read, write, edit. Tool count is not a quality ranking. 22
23. 23 TOOLS & EDITING Deterministic Repository Retrieval ripgrep, glob, file traversal, and context files. No vector retrieval over repository code in the inspected corpus. Aider adds ranked tree-sitter symbol context. H ARNES S ENG INEERING 23
24. 24 TOOLS & EDITING What the RAG Absence Does Not Mean Retrieved code still augments model input. H ARNES S ENG INEERING Conversation-memory retrieval is a different substrate. Corpus absence is not universal evidence against semantic search. 24
25. 25 TOOLS & EDITING Exact Replacements and Patch Contracts 01 02 03 Exact replacement requires a unique match. Patch DSLs require valid structure and matching context. Ambiguity should produce an actionable failure. H ARNES S ENG INEERING 25
26. 26 TOOLS & EDITING Handling Drift and Model Differences Hermes and OpenCode use fuzzy matcher cascades. H ARNES S ENG INEERING Aider and OpenCode select editing behavior by model. Gemini CLI adds an LLMassisted edit repair step. 26
27. 27 TOOLS & EDITING Deferred Tools and Concurrent Actions Load large schemas only when needed. Parallelize independent reads; serialize conflicting writes. Search deferred catalogs rather than exposing everything. H ARNES S ENG INEERING 27
28. 28 MEMORY & CONTEXT Four Context-Management Strategies Linear history: MiniSWE-Agent. H ARNES S ENG INEERING Recursive summaries: Aider. Pluggable condensation: OpenHands. Threshold compaction: common in production harnesses. 28
29. 29 MEMORY & CONTEXT What Compaction Must Preserve 01 02 03 Original goal, decisions, unresolved work. Recent verbatim context and valid tool-call/result groupings. Incremental summary updates and overflow recovery. H ARNES S ENG INEERING 29
30. 30 MEMORY & CONTEXT Session History Need Not Be Destroyed Pi: append-only session tree and movable active branch. H ARNES S ENG INEERING Hermes: compaction rotates to a parent-linked session. OpenCode: anchored incremental summaries. 30
31. 31 MEMORY & CONTEXT Who Writes and Reviews Durable Memory? Codex: agent-maintained extraction/consolidation. Hermes: bounded, sessionfrozen snapshots. Gemini CLI: human-reviewed patch inbox. OpenClaw: pre-reply agentic recall. H ARNES S ENG INEERING 31
32. 32 SAFETY Approval Is Not Isolation Permissions decide whether an action may run. Sandboxes constrain reachable resources. Audit trails record actions and decisions. Permission gate before execution; sandbox around execution; audit alongside. H ARNES S ENG INEERING 32
33. 33 SAFETY Native Isolation and Layered Review 01 02 03 Codex: policy, hooks, Guardian review, native sandbox. Gemini CLI: approval modes and crossplatform sandbox policies. Claude Code: layered permissions and opt-in sandbox. H ARNES S ENG INEERING 33
34. 34 SAFETY Other Safety Investments and Their Limits OpenCode: syntax-aware command permissioning. Large codebase does not imply native sandboxing. H ARNES S ENG INEERING Hermes: content scanning and a policy floor under bypass mode. Pi: minimal core with extension-provided controls. 34
35. 35 O R C H E S TR A T I O N Task Delegation Shapes Sequential delegation: Mistral Vibe. Thread trees/recursive composition: Codex, Claude Code. Concurrent child sessions: OpenHands, OpenCode. Process/extension variants: Hermes swarms, Pi. H ARNES S ENG INEERING 35
36. 36 O R C H E S TR A T I O N When Delegation Pays Off Good candidate: independent breadth-first investigation. H ARNES S ENG INEERING Harder case: simultaneous mutation of shared files. Costs: tokens, coordination, cancellation, and authority propagation. 36
37. 37 O R C H E S TR A T I O N Protocol Placement Matters 01 02 03 ACP: editor integration and external harness hosting. A2A: separate crossvendor communication protocol. Own children: usually in-process or bespoke process communication. H ARNES S ENG INEERING 37
38. 38 E X T E N S I B I LI TY Four Extension Mechanisms MCP: external tools, resources, and prompts. Skills: procedural instructions and optional assets/scripts. Hooks: lifecycle interception and policy integration. Plugins: packaged runtime extensions. H ARNES S ENG INEERING 38
39. 39 E X T E N S I B I LI TY Capability Packages Acquire a Supply Chain Skills 9/11; MCP 8/11, within this selected corpus. Registries, provenance, trust tiers, quarantine, and agent authors emerge. Eight of nine skill adopters defer bodies until needed. H ARNES S ENG INEERING 39
40. 40 SYNTHESIS Observations 1-4, Capability and Contracts Loop sophistication does not establish benchmark superiority. H ARNES S ENG INEERING Multi-provider systems can implement providerspecific optimizations. Prompt rhetoric converges and evolves by model and snapshot. Editing contracts are consequential and increasingly modelaware. 40
41. 41 SYNTHESIS Observations 5-8, Operational Choices 01 02 03 04 Durablememory governance is an emerging frontier. Native sandboxing is expensive, not dictated by codebase size. Hierarchical delegation recurs with different containment choices. Skills lead MCP adoption in the selected corpus. H ARNES S ENG INEERING 41
42. 42 SYNTHESIS Observations 9-11, Absences and Boundaries Designs align with public engineering guidance; causation is unresolved. H ARNES S ENG INEERING No general frameworks in inspected runtime paths; no vector code retrieval. Protocols now also support hosting whole harnesses. 42
43. 43 SYNTHESIS Observations 12-13, Platforms and the Minimal Floor SDKs, marketplaces, governance, and metaharnesses support a platform-turn thesis. H ARNES S ENG INEERING A small illustrative harness captures patterns, not proven equivalent performance. 43
44. 44 R E C O MMEN DA T I O N S A Practical Build and Configuration Order Start with a measurable task and small runtime. H ARNES S ENG INEERING Make editing and error feedback precise. Budget context and preserve useful state. Match safety to deployment; add delegation/extension s for observed needs. 44
45. 45 R E C O MMEN DA T I O N S What Still Needs Evidence? 01 Small scaffold: illustrative, not a production-ready demo. H ARNES S ENG INEERING 02 03 Needed: matchedmodel comparisons, ablations, and safety evaluation. Test framework/retrieval additions against a simple held-out baseline. 45
46. 46 C O N C LUS I O N Conclusion Editing, context, permissions, and verification determine practical usability. H ARNES S ENG INEERING A coding agent is a model plus an engineered runtime. Platform surfaces create interoperability opportunities and governance trade-offs. 46