Harness Engineering in AI

In this Knowledge Ketchup session, Raksha explored the concept of Harness Engineering in AI and its growing role in building reliable, scalable, and production-ready AI systems. The presentation highlighted how engineering practices help organizations effectively develop, manage, and optimize AI applications beyond the model itself.

Through practical examples and real-world use cases, she demonstrated how harnessing the right engineering frameworks, workflows, and tools can improve AI performance, accelerate deployment, and ensure long-term maintainability in modern software development.

Transcript

1. Harness Engineering in AI Raksha Khadka Associate Software Engineer
2. The task has changed • Early transformers ran stateless queries. Example: Movie review in, sentiment out. • Now we hand a model a whole repo: here's a bug, here are the tests, go fix it. • Act over response to Prompts
4. What a harness is “Model + Harness = Agent”
5. What the model can actually do by itself Nothing. It produces text. Everything else is the harness acting on its behalf.
6. Model + Harness = Agent
7. The loop the harness exists to support loop until the tests pass, not until the model stops talking
8. Opus 5, two harnesses 95% 42% Claude Code HF smolagents Same model. The harness is the only variable.
9. Harnesses in LLM • Model is the ceiling and harness is the ladder • Outer harness (Skills, MCP, claude.md) and inner harness (loop, tool calling, subagents, sandbox) • The harness writes the standing instructions, and they keep growing, not shrinking.
10. Five levers Model, Retrieval, Context and memory, Loops and tools, Orchestration
11. Retrieval, from cheapest to smartest Nothing. It produces text. Everything else is the harness acting on its behalf.
12. Long context degrades • 1M-token windows are common. Our appetite for data outruns them anyway. • Models drop information as you fill the window. • Past half the window, expect to start losing information. • Memory solves this issue.
13. Retrieval, from cheapest to smartest context is rented, files are owned, skills compound
14. Skills • Instructions plus code plus reference material. Externalized experience. • Cursor replaced 15,000 lines of orchestration code with a 200-line skill. • Reduces performance when skills not required are loaded
15. Open harness versus closed harness • Closed harness: company owns, neither can be modified nor seen internally Example: Fix bug • Open harness: inspect code, modify or build it Example: search task
16. Loops: from brute force to hypothesis • Giving the model time to think • Cognitive discipline • Karpathy's auto-research added a hypothesis and a verification step. • Feedback taken for the next approach
17. Safety and graded friction • Your laptop has API keys and secrets sitting within reach of the agent. • Run code in an isolated sandbox. • Use of guardrails.
18. Single agent beats multi-agent • The case for splitting context across agents is sensible. The coordination is not. • Across many popular benchmarks, single agents won on average. • Factory found serial orchestrator, workers, validation beat a parallel swarm. • Constrained forms do work: parallel subtasks, or one orchestrator with workers. • Splitting work off is a memory decision first: it keeps a big search out of the main window.
19. Four shapes, and what they cost A harness splits work off mainly to keep a big search out of the main window. It is a memory decision first.
20. What lasts
21. What lasts, and what does not • Skills last. They capture expertise from the whole org, not one person. • Memory is unsolved. What to keep and what to throw away is still open.
22. Thank you Any queries?