Harness Engineering or How to Keep Agents on Track

by Iury Souza

Embed

Start on current slide

Slide 1

Slide 1 text

@iurysza iurysouza.dev Harness Engineering How to Keep Agents On Track or

Slide 2

Slide 2 text

About me •Google Dev Expert •Co-host @ Fragmented-AI •Platform Engineering @ SumUp @iurysza iurysouza.dev

Slide 3

Slide 3 text

Non deterministic Coding

Slide 4

Slide 4 text

Non deterministic Coding “They’ve done studies, you know? 60% of the time, it works every time” - Brian Fantana (Anchorman)

Slide 5

Slide 5 text

Non deterministic Coding •Agents are writing more and more code

Slide 6

Slide 6 text

Non deterministic Coding •Agents are writing more and more code •They write solid code, until they don’t

Slide 7

Slide 7 text

Non deterministic Coding •Agents are writing more and more code •They write solid code, until they don’t •New bottlenecks • Keeping code quality • Steering • Containing blast radius

Slide 8

Slide 8 text

Non deterministic Coding •Platform Teams attempts to keep quality standards for developers

Slide 9

Slide 9 text

Non deterministic Coding •Platform Teams attempts to keep quality standards for developers •Harness Engineering does that for agents

Slide 10

Slide 10 text

Non deterministic Coding •Prompt Engineering •Context Engineering •Harness Engineering

Slide 11

Slide 11 text

Non deterministic Coding Made up word

Slide 12

Slide 12 text

Non deterministic Coding The Agentic Coding Ladder

Slide 13

Slide 13 text

Non deterministic Coding The Agentic Coding Ladder

Slide 14

Slide 14 text

How we got Here

Slide 15

Slide 15 text

How we got Here The Anthropic Moment

Slide 16

Slide 16 text

The Anthropic Moment •Claude Code preview: Feb 25 •Claude 3.7 Sonnet launch •CLI-first agentic coding Feb 2025: Claude Code

Slide 17

Slide 17 text

The Anthropic Moment •Claude Code preview: Feb 25 Feb 2025: Claude Code

Slide 18

Slide 18 text

The Anthropic Moment •Claude Code preview: Feb 25 •Claude 3.7 Sonnet launch Feb 2025: Claude Code

Slide 19

Slide 19 text

The Anthropic Moment •Claude Code preview: Feb 25 •Claude 3.7 Sonnet launch •TUI / CLI based agent Feb 2025: Claude Code

Slide 20

Slide 20 text

•Opus 4.5 Sep 2025: Opus The Anthropic Moment

Slide 21

Slide 21 text

•Opus 4.5 •Step change in quality Sep 2025: Opus The Anthropic Moment

Slide 22

Slide 22 text

•Opus 4.5 •Step change in quality •Model can prompt itself Sep 2025: Opus The Anthropic Moment

Slide 23

Slide 23 text

•Opus 4.5 •Step change in quality •Model can prompt itself •Improved planning mode Sep 2025: Opus The Anthropic Moment

Slide 24

Slide 24 text

•The IDE starts losing ground Holidays: Claude Code + Opus The Anthropic Moment

Slide 25

Slide 25 text

•The IDE starts losing ground •The harness was the main improvement Holidays: Claude Code + Opus The Anthropic Moment

Slide 26

Slide 26 text

•The IDE starts losing ground •The harness was the main improvement •Competition shows up • Codex • Opencode • Amp • Pi Holidays: Claude Code + Opus The Anthropic Moment

Slide 27

Slide 27 text

•The IDE starts losing ground •The harness was the main improvement •Competition shows up • Codex • Opencode • Amp • Pi •It’s not magic Holidays: Claude Code + Opus The Anthropic Moment

Slide 28

Slide 28 text

•The IDE starts losing ground •The harness was the main improvement •Competition shows up • Codex • Opencode • Amp • Pi •It’s not magic •The harness matters more than the model Holidays: Claude Code + Opus The Anthropic Moment

Slide 29

Slide 29 text

What is a harness anyway?

Slide 30

Slide 30 text

What is a harness anyway? It’s all about steering

Slide 31

Slide 31 text

What is a harness (…) a set of straps and fittings used to control an animal (…) •Control

Slide 32

Slide 32 text

What is a harness (…) a set of straps and fittings used to control an animal (…) •Control •Direction

Slide 33

Slide 33 text

What is a harness (…) a set of straps and fittings used to control an animal (…) •Control •Direction •Constraints

Slide 34

Slide 34 text

What is a harness (…) a set of straps and fittings used to control an animal (…) •Control •Direction •Constraints A horse Harness

Slide 35

Slide 35 text

What is an agent harness It's what the LLM uses to navigate the world and complete tasks. An Agent Harness

Slide 36

Slide 36 text

What is an agent harness It is what the LLM uses to navigate the world and complete tasks. •Makes a model useful for real work •From text replies to action An Agent Harness

Slide 37

Slide 37 text

Should I build a Harness? Short answer: no •You can customize it •You can extend it •You can build infrastructure for it

Slide 38

Slide 38 text

A minimal Agent Agent: an LLM running inside a harness •Instruction Layering A typical agent definition

Slide 39

Slide 39 text

A minimal Agent Agent: an LLM running inside a harness •Instruction Layering •Tools A typical agent definition

Slide 40

Slide 40 text

A minimal Agent Agent: an LLM running inside a harness •Instruction Layering •Tools •Permissions A typical agent definition

Slide 41

Slide 41 text

A minimal Agent Agent: an LLM running inside a harness •Instruction Layering •Tools •Permissions •Context Management A typical agent definition

Slide 42

Slide 42 text

A minimal Agent Agent: an LLM running inside a harness •Instruction Layering •Tools •Permissions •Context Management •Loop control A typical agent definition

Slide 43

Slide 43 text

Make it your own Adapt it to your workflows •AGENTS.md •Skills •Commands •MCPs •Event Hooks

Slide 44

Slide 44 text

Make it your own Adapt it to your workflows Pi Agent Extensions API •AGENTS.md •Skills •Commands •MCPs •Event Hooks

Slide 45

Slide 45 text

Stacking Loops What agents are made of

Slide 46

Slide 46 text

What agents are made of The ReAct Architecture Stacking Loops

Slide 47

Slide 47 text

The Inner Loop Next Token Generation Stacking Loops

Slide 48

Slide 48 text

The agentic loop Stacking Loops The Outer Loop

Slide 49

Slide 49 text

Stacking Loops How agents get things done

Slide 50

Slide 50 text

Stacking Loops Why this works? •Non deterministic -> deterministic •Code is verifiable •Feedback loop

Slide 51

Slide 51 text

Stacking Loops Why this works? Tool Calling Feedback Loop

Slide 52

Slide 52 text

Stacking Loops Why this works? E.G.: Opencode’s AskUserQuestion Tool Error

Slide 53

Slide 53 text

Closing the Loop How to leverage the harness to make the agent smarter

Slide 54

Slide 54 text

Closing the Loop Give the agent senses •Observability enables self correction

Slide 55

Slide 55 text

Closing the Loop Give the agent senses •Observability enables self correction •Design agent friendly tools, CLIs/MCPs

Slide 56

Slide 56 text

Closing the Loop Give the agent senses •Observability enables self correction •Design agent friendly tools, CLIs/MCPs •Favor them over raw text instructions

Slide 57

Slide 57 text

Closing the Loop Give the agent senses •Machine Readable Output

Slide 58

Slide 58 text

Closing the Loop Give the agent senses •Machine Readable Output •Self describing schemas

Slide 59

Slide 59 text

Closing the Loop Give the agent senses •Machine Readable Output •Self describing schemas •Safety rails against hallucinations

Slide 60

Slide 60 text

Closing the Loop Give the agent senses •Machine Readable Output •Self describing schemas •Safety rails against hallucinations •Instrument the repository for agents

Slide 61

Slide 61 text

Closing the Loop Give the agent senses •Token Efficiency

Slide 62

Slide 62 text

Closing the Loop Give the agent senses •Token Efficiency •Reliability

Slide 63

Slide 63 text

Closing the Loop Give the agent senses •Token Efficiency •Reliability •Reproducibility

Slide 64

Slide 64 text

Closing the Loop Give the agent senses •Token Efficiency •Reliability •Reproducibility •Auditable

Slide 65

Slide 65 text

Give the agent senses Closing the Loop

Slide 66

Slide 66 text

Context Management Give the agent a memory

Slide 67

Slide 67 text

Context Management Give the agent a memory •Context is a scarce resource

Slide 68

Slide 68 text

Context Management Give the agent a memory •Context is a scarce resource •A well crafted AGENTS.md + docs goes a long way

Slide 69

Slide 69 text

Context Management Give the agent a memory •Context is a scarce resource •A well crafted AGENTS.md + docs goes a long way •Progressive disclosure

Slide 70

Slide 70 text

Context Management Give the agent a memory •Context is a scarce resource •A well crafted AGENTS.md + docs goes a long way •Progressive disclosure •Agents ❤ file system + find + grep

Slide 71

Slide 71 text

Context Management Give the agent a memory •Context is a scarce resource •A well crafted AGENTS.md + docs goes a long way •Progressive disclosure •Agents ❤ file system + find + grep

Slide 72

Slide 72 text

Context Management Give the agent a memory

Slide 73

Slide 73 text

Context Management Give the agent a memory

Slide 74

Slide 74 text

Platform Engineering And the tragedy of the commons

Slide 75

Slide 75 text

Platform Engineering The tragedy of the commons An economic and ecological concept describing how individuals, acting strictly in their own self-interest, deplete or spoil a shared resource, ultimately ruining it for everyone

Slide 76

Slide 76 text

Platform Engineering Enables Speed and Cohesion •Company Scaling Implications • Shared standards • Tooling • Guardrails

Slide 77

Slide 77 text

Platform Engineering Dedicated Agent Infra •Fast moving environment •Agent Evals and benchmarking •Skills versioning and distribution •Safely expose company systems for agents •Research and Development

Slide 78

Slide 78 text

Platform Engineering Dedicated Agent Infra •Fast moving environment •Agent Evals and benchmarking

Slide 79

Slide 79 text

Platform Engineering Dedicated Agent Infra •Fast moving environment •Agent Evals and benchmarking •Skills versioning and distribution

Slide 80

Slide 80 text

Platform Engineering Dedicated Agent Infra •Fast moving environment •Agent Evals and benchmarking •Skills versioning and distribution •Safely expose company systems for agents

Slide 81

Slide 81 text

Slide 82

Slide 82 text

Platform Engineering Dedicated Agent Infra Bounded Work Agent

Slide 83

Slide 83 text

Conclusion Experiment & Iterate

Slide 84

Slide 84 text

•Mastering Harness engineering • Enables shipping faster • Fewer regressions • Lower Organizational drift •No one-size-fits-all •Start experimenting and find out what works best for your team/org Conclusion Experiment & Iterate