Measure whether the harness works (Compare with tasks, trials, and graders) | Learn Harness Engineering with Kiro 10

This Level 300 deck, the 10th of Learn Harness Engineering with Kiro, shows how to check whether the harness you added to Kiro (steering, a hook, a skill) works: make tasks from real misses, keep the grading tests out of the workspace, run the same tasks before and after the harness a few times each, and compare scores, transcripts, and cost. On a small booking app built for the series, Kiro CLI headless mode ran 3 tasks 3 times each before and after adding the harness (the 2 feature tasks passed 2 of 6 runs before and 6 of 6 after, at about 2.1 times the credits), and real screens show what that result can and cannot say.

Transcript

1. Learn Harness Engineering with Kiro Run the loop Level 300 Measure whether the harness works Compare with tasks, trials, and graders 10 As of November 2026 November 2026 Riku Inada Amazon Web Services Japan + Kiro + Harness Engineering © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 1
2. Introduction About me Riku Inada Amazon Web Services Japan G.K. Solutions Architect Works mainly with manufacturing customers Recent work Promoting the AI-Driven Development Life Cycle (AI-DLC) Writing and speaking about Kiro and AWS Amplify © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. X @inada_riku Speaker Deck speakerdeck.com/inariku Zenn zenn.dev/inariku GitHub github.com/inariku 2
3. Introduction Where this deck fits You are here: 10 Measure whether the harness works. Read first: 08 Keep improving the harness The idea Intro to harness 01 engineering 01 Guide 02–03 Check Rules and 02 context Hand over 03 intent as specs 04–05 Contain Computational 04 checks Contain what 06 the agent can do 06 Run the loop 07–10 07 Run it in CI Read first 05 AI review improving 08 Keep the harness long work 09 Connect across sessions whether 10 Measure the harness works © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 3
4. Introduction What you'll learn After you add a harness, run a fixed task set before and after it a few times, and compare 1 Compare before and after with tasks, trials, graders, and transcripts 2 Make tasks from real misses, and don't show Kiro the grading tests 3 Results of 3 tasks run 3 times each on the booking app, and what they can say Read first: 08 Keep improving the harness (01 and 07 too) + We try it on a small meeting-room booking app made for these decks (Kiro CLI 3.1.0) Measuring an agent you built yourself: Learn Strands Agents with Kiro 07 In the pictures, the ghost is Kiro. Colors: purple is the point, blue is Kiro, green passed © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 4
5. Part 1 The idea © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 5
6. The idea Without measuring, you can't tell Only by running and comparing can you tell whether the harness worked Added harness Passed once Tasks not run yet Without evals, you fix a miss after it shows up, and the fix breaks something else, again and again If a sensor never fires, is that high quality, or a weak way to detect misses? © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 6
7. The idea One success is not enough A task that passes 75% of the time passes all 3 runs only about 42% of the time A task that passes 75% of the time, run k times At least 1 of k passes All k pass 98% 94% 75% 75% For work that should 56% 42% behave the same each time, look at "all k pass" This deck runs each 3 times 1 run 2 runs 3 runs pass@k: at least one of k runs passes. pass^k: all k runs pass Use pass@k when one hit is enough. Use pass^k when you count on every run © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 7
8. The idea Task, trial, grader, transcript A trial solves the task once; the grader checks the final workspace and the transcript Trial (one run) Task input, pass rule Kiro solves it in Outcome: final workspace a clean workspace Transcript (events) The grader checks both Kiro saying "done" is not the outcome. Tasks together make the task set Each trial starts from a new workspace. Keep no git history or files from earlier runs © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 8
9. The idea Compare before and after the harness Run the same task set the same number of times, before and after Before 3 runs Today's harness The same task set Compare scores, the same model transcripts, cost After 3 runs Rules, hook, skill added Run both the same number of times. Averaging over different sets favors the stronger model 05 covers reviewers, 07 CI gates, 08 how to fix. This deck compares the whole harness, before and after © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 9
10. Part 2 Making tasks and graders © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 10
11. Making tasks and graders Make tasks from real misses Misses seen in 01 and 02 (Japanese runs) became 3 booking app tasks Misses in 01 and 02 (Japanese runs) Task (request) What grading checks 02: all 3 runs adding "extend" kept Cancel a booking Held-out test, tests no test and left the spec as is (DELETE) kept, spec fixed 01: in 1 of 2 runs, the Property 11 Fix the flaky Test not cut, passes numRuns was cut from 100 to 10 Property 11 3 times in a row 02: in all 3 runs, the check Filter a room's list Held-out test, was not kept as a test by date tests kept Property 11: a test that checks, with 100 inputs, that the DB returns the same list after reopening 20 to 50 tasks from real misses is a good start. Hold some back to check the harness, not to build it © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 11
12. Making tasks and graders Don't show Kiro the grading tests Write the pass rule as tests first, and put them in the workspace only to grade The workspace while Kiro solves At grading time only The task and the code no grading tests Kiro cannot read the pass rule Kiro stops 1 Copy the held-out tests in 2 Run them and score 3 Delete them after grading Kiro sees only the request and the code; the tests come in after Kiro stops A model that noticed it was being evaluated looked for the answer key and decrypted it This trial copied its 2 held-out tests in only to grade, and deleted them after grading © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 12
13. Making tasks and graders Check the tasks themselves Grade the untouched app once: a task that passes unchanged can't show a gain Task Graded on the untouched app Cancel a booking Flaky Property 11 Filter by date Failed (no DELETE) How to read it Fails as it should. Usable to compare Passed (3 of 3 at Passes untouched. It can this machine load) measure only "not cut" Failed (no date filter) Fails as it should. Usable to compare A task nobody passes and a task that passes untouched can't show a difference. Also check a sample answer Near 0% over many runs usually means a broken task or grader. An audit found about 30% broken © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 13
14. Making tasks and graders Three kinds of graders Use code where code can check. Calibrate a model grader against people first Grader Checks Strong at Weak at Code Tests, file diffs Fast, the same each time Misses valid alternatives Model Reads only, answers in a set form Can judge wording Can drift from people Person Reads transcript and outcome Most accurate Slow, does not scale Give a model grader a way out: let it answer "unknown" This trial graded with code only. A model grader works read-only, in its own session, one aspect at a time © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 14
15. Part 3 Comparing on the booking app © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 15
16. Comparing on the booking app The harness we added We added always-on steering with 3 rules (below), an Agent Stop hook, and a how-to skill .kiro/steering/working-rules.md in the after workspace Cut from the screen: the grep command, the heading, blank lines inclusion: always loads the file for every request © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 16
17. Comparing on the booking app How we ran it Kiro CLI in headless mode, each run from a new workspace steering_inclusion: read the steering. Load skill: opened the skill. hook_update: the hook ran runFinished is the run stopping, not a pass. One run outside the comparison (the 18 runs: A2) --trust-all-tools uses tools without asking: only in a throwaway place (a container or VM; 06) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 17
18. Comparing on the booking app Grade the result Check the final workspace, not the steps. For canceling, the spec fix made the difference Before adding, 2 of 3 runs kept a test and 0 of 3 fixed the spec's Out of scope (the request does not ask for it) We don't grade the steps, such as which file Kiro read first © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 18
19. Comparing on the booking app Read the transcript Before adding, Kiro reported a fix, and also cut the Property 11 runs from 100 to 25 The report from Kiro at the end of run 2, in the transcript. The grade alone said only "numRuns 25" Run 3 cut them to 20 the same way. After adding, no run set numRuns below 100 © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 19
20. Comparing on the booking app The totals The 2 feature tasks: 2/6 before, 6/6 after. Property 11: 1/3 and 3/3 Property 11 measures only whether the test stays as strict as before (slide 13) Every is yes when all 3 runs passed (pass^3, slide 7). Read 3 runs as a trend; a rerun can change it © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 20
21. Comparing on the booking app Which checks made the difference The checks match the added rules, so they show whether Kiro kept the rules Task Check Cancel a booking Before (3 runs) After (3 runs) Spec: Out of scope fixed 0/3 3/3 Tests added and kept 2/3 3/3 Flaky Property 11 Runs and checks not cut 1/3 3/3 Filter by date Tests added and kept 2/3 3/3 Out of scope left as it was 3/3 3/3 Last row: the spec rule should not act here (a date filter is not Out of scope). Not graded © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 21
22. Comparing on the booking app Measure the cost too After adding, a run took about 2.1 times the credits and 2.1 times the time, on average Credits per run (mean). On the right, minutes per run (mean) Cancel a booking Flaky Property 11 Filter by date 1.51 before 1.1 min 3.44 after 2.4 min 1.65 before 2.6 min 3.42 after 6.0 min 1.98 before 1.6 min 4.01 after 2.8 min Measure credits and time along with the score. Another study saw about 2 times the tokens from the harness alone Credits are the transcript's session_usage. Minutes cover the whole run, the hook's tests included © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 22
23. Comparing on the booking app What we can say, and what we can't Say only what you tried. The input can stay the same, but the result can change each run What we can say (within this trial) What we can't say The 2 feature tasks: 2/6 before, Whether the gap is chance (3 tasks, 6/6 after. Property 11: 1/3 and 3/3 3 runs each, no error bars) After adding, every run kept the rule that Whether one model ran them all its task checks (spec 3/3, tests 6/6, runs 3/3) (auto may differ from run to run) After adding, runs took more credits Which of steering, hook, and skill and more time worked (we added them together) Whether it works with the rule in the request, or on tasks not used to build it © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 23
24. Part 4 Keep it running © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 24
25. Keep it running Same setup, many runs Compute allocation moves scores, and so may the time of day. Run often, same setup Timed out Idle machine Busy machine Allocation alone made a 6-point gap. Until setups are written down and matched, doubt gaps under 3 points This trial logged the machine load before each run and ran before and after side by side (Appendix A2) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 25
26. Keep it running Run evals in CI too Run the same task set when the harness changes and when the model changes Capability evals The harness changed Run the same task set What to improve; add hard tasks in CI (07) The model changed Regression evals Tasks that keep passing move here A set at 100% watches for regressions but shows no gain How to run it: 07 (Kiro CLI headless mode and GitHub Actions) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 26
27. Keep it running Measure the checker too Measure the deck 05 reviewer with a rule-breaking, a safe, and an unrelated change Change given to the reviewer Wanted result Questions A change that breaks A finding that the rule names the rule Coverage Flags the violation? A safe counterexample No finding on Restraint Quiet on clean changes? (similar, but allowed) the rule A change unrelated to the rule No extra findings Retention Still finds plain bugs? Actionability Rule, place, priority Reviews given the rules raised 98% of the needed findings; without them, 58.3% (not measured on Kiro) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 27
28. Summary Start on your own project Make tasks and grading, check them on a dry run, then run before and after equally 1 Collect misses 2 Write tasks 3 Grade a dry run Log request, miss, count. A request, plus grading Fails untouched, passes Hold some back to check tests kept outside with a sample answer 4 Set up before/after 5 Run headless 6 Sum up and read Two commits that differ Fix the model. 3 runs Passes, every time or not, only in the added parts each, fresh worktrees credits, transcripts The booking app's code and this trial's scripts are not public. Appendix A2 shows the 18 runs' command © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 28
29. Summary Summary Run the same tasks before and after a few times; read the scores, transcripts, cost Make tasks from real misses, hide the grading tests, and say only what you tried Three moves for tomorrow 1 2 3 Make 3 tasks from misses, Run before and after, Read transcripts and grade once untouched 3 times each, headless credits with the score When the tasks grow, run them in CI when the harness or the model changes (07) To learn which part worked, remove one at a time and compare on the same set (08) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 29
30. Summary Read next To run it in CI, 07. To fix misses, 08. To measure your own agent, Strands 07 To run evals in CI To fix the misses you see 07 To measure your own agent 08 07 Run it in CI Keep improving Learn Strands Agents (Kiro CLI headless mode) the harness with Kiro 07 (evals) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 30
31. Appendix A1 Glossary Term Meaning Term Meaning Task Set input and pass rule Task set Tasks put together Trial One run of a task Grader Checks the result Transcript What happened in a trial Outcome The workspace at the end pass@k At least 1 of k passes pass^k All k pass Held-out test Placed only to grade Capability eval A set for what to improve Regression eval Watches what keeps passing Headless mode Asking Kiro with no UI stream-json One JSON line per event Credit The unit of usage in Kiro Agent Stop hook Runs as a response ends Always-on steering Rules read every time Miss Not as asked or as the rules say Request Work you ask Kiro to do Skill Steps opened when needed git worktree A second workspace © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 31
32. Appendix A2 Trial record Task Harness Passed Min (mean) Credits (mean) Hook ran Load before Cancel a booking before 0/3 1.1 1.51 0/3 16.0-32.3 after 3/3 2.4 3.44 3/3 8.8-32.9 before 1/3 2.6 1.65 0/3 11.6-23.0 after 3/3 6.0 3.42 3/3 11.0-14.8 before 2/3 1.6 1.98 0/3 7.9-15.6 after 3/3 2.8 4.01 3/3 11.7-19.7 Flaky Property 11 Filter by date F="--v3 --trust-workspace --trust-all-tools --output-format stream-json" kiro-cli chat $F "<request>" The 18 runs' command. Unlike slide 17: --trust-workspace, no --no-interactive, stdin not a terminal Load: the load average before the run. Model: auto. Before and after ran in two lanes side by side No errors logged (no HTTP 500 or 429). The skill opened in all 6 after runs of the feature tasks © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 32
33. Appendix A3 Sources Anthropic, January 2026, Demystifying evals for AI agents (slides 4, 6, 7, 8, 9, 11, 12, 13, 14, 17, 18, 19, 23, 26, 28, 29, 31) https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents Anthropic, February 2026, noise from the infrastructure in coding evals (slides 23, 25) https://www.anthropic.com/engineering/infrastructure-noise Anthropic, March 2026, a model that noticed it was evaluated and decrypted the answer key (slide 12) https://www.anthropic.com/engineering/eval-awareness-browsecomp OpenAI, January 2026, Testing Agent Skills Systematically with Evals (slides 11, 14, 19, 22, 26, 27, 28, 29) https://developers.openai.com/blog/eval-skills OpenAI, July 2026, Separating signal from noise in coding evaluations (slide 13) https://openai.com/index/separating-signal-from-noise-coding-evaluations/ © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 33
34. Appendix A4 Sources OpenAI, February 2026, why it no longer reports a well-known benchmark's score (slides 12, 13) https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ OpenAI, July 2026, an article on writing review rules in the instruction file and evaluating AI review (slide 27) OpenAI, May 2026, a tax agent improved by turning practitioners' fixes into evals (slide 11) martinfowler.com (Birgitta Böckeler of Thoughtworks), April 2026, Harness engineering for coding agent users (slide 6) https://martinfowler.com/articles/harness-engineering.html Strands Agents, October 2026, We ran 94,000 trials to measure the cost of harness choice (slides 9, 22, 29) https://strandsagents.com/blog/we-ran-94000-trials-to-measure-the-cost-of-harness-choice/ © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 34
35. Appendix A5 Sources Anthropic, November 2024, A statistical approach to model evaluations (slide 23) https://www.anthropic.com/research/statistical-approach-to-model-evals Anthropic, September 2025, Writing effective tools for agents (check on tasks you did not tune on) (slide 11) https://www.anthropic.com/engineering/writing-tools-for-agents Kiro, docs as read on November 10, 2026 (Headless mode, Agent harness, Hook types, and more) (slides 16, 17, 28, 31) https://kiro.dev/docs/ Kiro, November 2026, the Kiro CLI changelog (3.0.0 and 3.1.0) (slide 17) https://kiro.dev/changelog/cli/ © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 35