Keep improving the harness (So the same miss doesn't happen twice) | Learn Harness Engineering with Kiro 08

A Level 200 deck on keeping a harness improving with Kiro: log each miss in one line, turn a repeated or costly miss into steering, a hook, or a skill, check whether it worked, watch for slow drift on a schedule, and take away mechanisms that stop helping, based on public articles from OpenAI, Anthropic, martinfowler.com, Mitchell Hashimoto, and the kiro.dev blog. On a small meeting-room booking app made for this series, real Kiro CLI 3.1 screens show Kiro proposing mechanisms from the miss log, asking before it writes a file, keeping a property-based test's counterexample as a regular test, writing a skill's common mistakes, and saving a memory you can open and read. This is deck 08 (Run the loop pillar) of the "Learn Harness Engineering with Kiro" series.

Transcript

1. Learn Harness Engineering with Kiro Run the loop Level 200 Keep improving the harness So the same miss doesn't happen twice 08 As of November 2026 November 2026 Riku Inada Amazon Web Services Japan + Kiro + Harness Engineering © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 1
2. Introduction About me Riku Inada Amazon Web Services Japan G.K. Solutions Architect Works mainly with manufacturing customers Recent work Promoting the AI-Driven Development Life Cycle (AI-DLC) Writing and speaking about Kiro and AWS Amplify © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. X @inada_riku Speaker Deck speakerdeck.com/inariku Zenn zenn.dev/inariku GitHub github.com/inariku 2
3. Introduction Where this deck fits You are here: 08 Keep improving the harness. Next: 09, 10 01 Guide The idea Read first to harness 01 Intro engineering 02–03 Check Rules and 04 checks Hand over 05 AI review 02 context 03 intent as specs 04–05 Contain Computational Contain what 06 the agent can do 06 Run the loop 07–10 07 Run it in CI Keep improving 08 the harness Read next long work 09 Connect across sessions Read next whether 10 Measure the harness works © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 3
4. Introduction What you'll learn Fix the harness from each miss, and take away what you no longer need 1 Record what Kiro missed, and turn it into a mechanism so the same miss doesn't happen twice 2 See whether each addition worked, and watch on a schedule for drift that builds up slowly 3 Take away mechanisms that stopped working, one at a time, and watch for misses Read first: 01 Intro to harness engineering (guide, check, contain, and run the loop) The screens use a small meeting-room booking app made for these decks (the booking app; its code is not public), with requests to Kiro CLI 3.1.0 A miss is when Kiro does not do what you asked. In the pictures, the ghost is Kiro Colors in the figures: purple is the point and where a person decides, blue is Kiro at work © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 4
5. Part 1 The idea © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 5
6. The idea Agents copy bad patterns too Agents copy the repo's patterns, bad ones too. Hand cleanup can't keep up One bad pattern Copies spread Hand cleanup One team first spent every Friday (20% of the week) cleaning up, and could not keep up So turn misses into mechanisms and clean up a little on a schedule. A harness is not built once and done © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 6
7. The idea Don't let the same miss happen twice When Kiro misses, add a rule or a tool so the same miss doesn't come back Rule The first miss Caught next time Check Each time an agent makes a mistake, build something so it never makes that mistake again This deck's suggestion: log each miss; add a mechanism the second time, or when the damage is large © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 7
8. Part 2 Collect the misses © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 8
9. Collect the misses The miss log One line per miss: the request, what happened, how often, which way to fix # Request What happened Times Fix direction 1 Extend, time display Returned with an unrelated test failing 5 of 8 Check: a Stop hook 2 Fix a failing test Cut the test runs from 100 to 10 1 of 2 Guide: don't weaken tests 3 Extend a booking Didn't keep the tests it checked with 3 of 3 Guide: steering, skill 4 Extend a booking Didn't settle "Out of scope" first 3 of 3 Guide: scope first 5 Extend a booking Used another task's DB to check the page 1 time Not yet (maybe contain) 6 Review a PR Proposed "one README line" as the fix 1 time Guide: not a doc alone The booking app's docs/failure-log.md. Times: how often it happened in the Japanese editions of 01-05 A person writes the fix direction. When you add a mechanism, write the date too (this deck's suggestion) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 9
10. Collect the misses Feed back what people find, too Review comments, fix PRs, and users' bugs also go back as a rule or a tool If not enough Review comments Fix PRs Bugs users found Make it a rule Make it code (steering, docs) (lint, tests) Kiro reads that text Put the fix in the error text and fixes it Misses people find go into the same log (previous slide), one line each © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 10
11. Part 3 Make it a mechanism © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 11
12. Make it a mechanism Where to add it Bad if it happens: contain. A machine can tell: check. Otherwise: guide Bad if it happens? Yes Contain Permissions, sandbox #5 (candidate) No A miss Can a machine tell? Yes Check Tests, hooks Guide Steering, skills #1 No #2-#4, #6 Guiding alone does not tell you it worked. Pair it with a check Apply them from the top (this deck's suggestion). #1-#6 are the rows of the log on slide 9 © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 12
13. Make it a mechanism Have Kiro propose the mechanisms Let Kiro read the log and propose; a person reviews before Kiro writes # Mechanism Location 1 Hook that runs npm test when the agent stops (Stop trigger) .kiro/hooks/test-on-stop.json 2 Steering: "Don't cut what a check checks (run counts, assertions, skips). Fix the cause of a slow test." .kiro/steering/testing.md (always) The answer table from Kiro, first two rows, copied from the terminal (#3-#6 left out; requests in appendix A3) A Stop hook that exits 1, as a failing npm test does, starts another agent turn. Not tried in this trial (A3) So first decide what Kiro does when tests still fail, e.g. report the failing test and stop (slide 26) Kiro turns each fix direction into a file. You check the order (slide 12) and the choices only you make © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 13
14. Make it a mechanism A person decides whether to contain Kiro proposes ways to contain #5, with their costs; a person decides Option Mechanism Trade-off A. Fence Agent permissions that deny shell commands matching :3001 or meeting-rooms.db Applies only when that agent is used, and regexes are easy to get around B. Hook PreToolUse hook on the shell tool. It exits 2 on commands that touch :3001 or meeting-rooms.db Works for any agent, but it's still pattern matching C. Isolate An npm run server:check script (PORT=3101 DB_PATH=:memory:) and a Vite proxy target read from env. A skill step says "check pages only on your own server" Removes the reason to touch the shared server, but changes app config and is only a guide The options from Kiro for #5, copied from the terminal (location column left out). Kiro recommended C + B Only C was chosen (not built here), so #5 is a guide for now. B would also block your own 3001 server © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 14
15. Make it a mechanism Read it before you approve the write Kiro shows the file before it asks. Read rules before they go in, like code ## Keep the tests you used to check your work (failure log #3) - Any test you write to check a change goes into the repository as a `src/**/*.test.ts` file … - A change is done only when `npm test` passes with those tests included. Above: lines 13, 15 and 16 of the testing.md Kiro writes, copied from the file (line 15's end cut) Below: the approval before the write (default settings). The #3 in the heading is the log's number Kiro wrote the files; the author committed them to the trial branch © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 15
16. Make it a mechanism Keep a counterexample as a regular test Keep a property test's counterexample as a regular test, shrunk input as is × Property 2: times on 30-minute slots are accepted in any time zone, … 7ms Counterexample: [1794236400000,1,0,-1] Shrunk 12 time(s) it("Property 2 counterexample: an end time written with -00:01 is accepted …" Above: the failing property, copied from the terminal (the end of its name cut). Below: the test Kiro wrote Run on a copy with a planted sign bug. Inputs change each run, so the counterexample may differ © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 16
17. Make it a mechanism Turn steps that worked into a skill Put steps that worked into a skill, with the common mistakes added Log # Mistake How to avoid it 1 Reported done while an unrelated test was failing Run the full npm test and treat any failure as not done 3 Did not keep the tests used to check the work Save them as src/**/*.test.ts and keep them in the repository 4 Built first and wrote the spec's out of scope at the end Settle out of scope in the spec before step 2 .kiro/skills/add-feature/SKILL.md, the list under "## Common mistakes", shown as a table Its steps: 1 settle out of scope · 2 implement · 3 keep the tests · 4 npm test, typecheck · 5 report Check that the next request calls the skill. If not, make its description say when to use it © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 17
18. Make it a mechanism Memory is a guess A memory is a guess at what's needed. Check it against the conversation Read every time Keep it small Check notes against Memory notes Recalled when needed; can be wrong or stale the conversation, fix what is wrong The conversation What actually happened What must always hold goes in steering, not memory Checking notes against the original conversation fixes wrong ones. Keep the notes read every time small Memory kept in git can be read as a diff. You can open and read the Memory of Kiro (next slide) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 18
19. Make it a mechanism Open and read the Kiro memory You can open the Memory of Kiro. Put what must always hold in steering db-test-property-11-flaky-timeout Scope: repo:github/example/meeting-rooms · Type: Semantic · Updated: Nov 11, 2026 at 3:18 AM Description: Read when db.test.ts Property 11 fails or times out, or when judging whether a test failure was caused by a recent change. Content: - `src/server/db.test.ts` "Property 11" (close the file DB and open it again, same list returned) sometimes fails at vitest's 5-second timeout when the machine is busy. (3 more items) One memory from the /memories list in Kiro CLI (text copied from the terminal) Kiro IDE opens the same memories in its Memories view. Kiro CLI needs memory.mode read_write (A3) It could not save without a git remote, so a placeholder remote was added (the Scope above shows it) What must always hold goes in steering, misses in the log. Don't limit what Kiro remembers too strictly © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 19
20. Part 4 Watch © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 20
21. Watch See whether it worked After you add one, count again: the same miss, reruns, and people's fixes 1 2 3 4 Write the date Before vs. after Try another request Read the exchange On the miss log row, Same miss, reruns, Not only the request Not only the numbers: the date it was added people's fixes used for the fix read what Kiro said Measuring with tasks run many times and scored: deck 10 In the booking app, write the date on the miss log row (slide 9) and count before and after it © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 21
22. Watch Write principles, clean up a little at a time Write principles; a scheduled task finds drift and sends small fixes By hand By a mechanism Bad patterns pile up Write principles A big weekly cleanup Scheduled scan Small fix PRs Falls behind A person reads Write principles a machine can scan for (e.g. shared utilities over hand-rolled helpers) Keep the scheduled task's fixes small: most should take under a minute to read © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 22
23. Watch Watch for drift Watch dead code, old dependencies and docs, and test strength on a schedule What to watch Tools A bad sign Dead code Dead-code detection It grows: clean up Old dependencies, docs Dependency scans, doc sweeps Stale ones stay How well tests check Mutation score (StrykerJS) The score drops Run them on a schedule, separately from PR checks (next slide) Mutation score: the share of planted bugs that the tests catch. Watching it move is this deck's suggestion © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 23
24. Watch Put the watch on a schedule Run the watch at a set time each week with Schedule in Kiro Crew Message The prompt or task sent to the agent when this job fires In the meeting-room booking app, check dead code, duplicates, old dependencies, and the tests' mutation score, and report only what got worse than last week. Don't fix anything. The Message on the screen is in Japanese; above is its English translation This job only reports; slide 22's small fix PRs would be a separate job. Last week's numbers: A2 Setup only: no job was created. How to use Kiro Crew: Kiro: Zero to Hero 11 © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 24
25. Part 5 Take away what doesn't work © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 25
26. Take away what doesn't work Look back If you keep adding to the same place, choose: stop, revert, or simplify Fixes keep landing in one place (10-line fix → subsystem) People look back (read the history) Stop Stop adding Revert Go back to before Simplify A smaller fix Usually, start by taking away what is not needed The Kiro Crew team (the always-on Kiro agent) added retrospectives that read the history of fixes The Stop hook of slide 13 can loop while a test fails (e.g. property 11): decide first what Kiro does then © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 26
27. Take away what doesn't work Signs a mechanism doesn't work Rewrite a rule that didn't help; plant a bug where a check never fails The sign What to do A rule that didn't reduce the miss Rewrite or remove it (see the miss log) A check that never fails Plant a bug, see the check fail Containment that blocks normal work Fix only what users can hit, smallest change The more rules Kiro reads every time, the less it follows each. Add one, take one away A check that never fails can't tell high quality from problems it misses Planting a bug to see the check fail is the idea of mutation testing (this deck's suggestion) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 27
28. Take away what doesn't work On a new model, remove one part at a time Each part bets on what the model can't do. Take one away at a time and compare 1 2 3 4 Remove one part Compare fairly Read the result Put back or drop Instructions first, Same requests, Did misses rise? Worse: put it back. hooks later same number of runs Read transcripts No change: leave out Cutting a lot at once hides which part was working With a new model, an instruction like "run the tests" may no longer be needed A new model may take instructions too seriously and stop. The order of removal is this deck's suggestion © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 28
29. Summary Summary Record misses, make mechanisms, see if they work, take away what doesn't 1 2 3 4 Record Make a mechanism Watch Take away One line a miss Guide, check Did it work? Remove what does (failure-log.md) or contain Any drift? not work, one by one Back to step 1 From tomorrow: write each miss by Kiro as one line in one file, such as docs/failure-log.md On a second miss, add one mechanism and write the date. Once a month, review the ones that don't help © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 29
30. Summary Read next Long work is 09; measuring whether it worked is 10 Read next To measure it 09 For scheduled jobs 10 11 Connect long work Measure whether Kiro: Zero to Hero 11 across sessions the harness works Intro to Kiro Crew Where rules go: 02. Property-based tests: 03. Hooks: 04 (this series) The series' decks are on speakerdeck.com/inariku © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 30
31. Appendix A1 Glossary Term Meaning Term Meaning miss what Kiro didn't do as asked miss log a file, one line per miss guide tell it before (steering) check look after it runs (hooks) contain limit actions (permissions) take away fix or remove what doesn't work Stop hook a hook as the agent stops (04) inclusion: always steering read every time (02) property-based test tests what always holds (03) counterexample an input that breaks a property Memory what Kiro keeps across sessions look back read the fixes' history, then choose mutation score share of planted bugs caught watch a check run on a schedule Kiro Crew The always-on Kiro agent copy the booking app copy in the trial © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 31
32. Appendix A2 What to know about the watch job Item What to know Last week's numbers Write in the request: "Add one line to docs/drift-history.md and compare it with the line before (write only this file)" (this deck's suggestion) Where results go Dashboard notifications (and a Slack DM if set). For a team, put a Slack channel in the Channel ID field of New Job Runs The Logs in Schedule show OK, Error, Timeout, or Cancelled. On Error and Timeout, Ask the agent opens a chat to look into it Gateway The Kiro Crew server. A run due while it is stopped is missed and not replayed Approvals A scheduled run never gets "auto-approve this session". If writing drift-history.md asks, allow only that file in the permission rules (06; not tried) Session Use Fresh (a new session each run) for a watch. The default, Persistent, carries over the last run's context © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 32
33. Appendix A3 What the trial did (1/3) What What this deck's trial did (November 11, 2026, Kiro CLI 3.1.0, V3) Personal settings HOME is an empty folder, so the author's steering, skills, and memories are not read (it holds only a link to the sign-in data) Use memory In that HOME only, memory.mode is read_write in the settings file of Kiro CLI (automatic updates off). In the app, the switch is under /memories → config Request 1 (slides 13, 14) I want to turn misses 1-6 in docs/failure-log.md into mechanisms, so the same miss doesn't happen twice. For each one, give the mechanism to add and where its file goes, as a table (three columns: #, mechanism, location. Keep each row short). #5 has no fix decided yet, so suggest options. Don't write any files yet. Request 2 (slide 15) Thanks. Write the hook for #1 (the CLI JSON only) and the steering for #2 and #3 (.kiro/steering/testing.md). The skill and #4-#6 come later. For #5, I'll take C only and leave B out for now (don't block port 3001). Stop when you've written them. Don't commit. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 33
34. Appendix A3 What the trial did (2/3) What The full requests (sent in order, in one conversation) Request 3 (slide 17) Next, write the skill for #3. In .kiro/skills/add-feature/SKILL.md, put the steps to add a feature and a "Common mistakes" section. Under common mistakes, list the misses from the miss log that happened while adding a feature (#1, #3, #4), one line each, with how to avoid it. Keep it short. Stop when you've written it. Don't commit. Request 4a (slide 19) One more thing. Property 11 in db.test.ts (close the file DB and open it again) sometimes fails at the 5-second limit when the machine is busy. It's an old, separate problem, not caused by the other fixes. Remember it. Request 4b (slide 19) [Note: Kiro could not save it, so a placeholder git remote was added first] I added a git remote, so try putting it in memory again. Don't write it in steering yet. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 34
35. Appendix A3 What the trial did (3/3) What The full requests (sent in order, in one conversation) Request 5 (slide 16) [Note: sent on a copy with a sign bug put into the time zone offset] npm test fails property 2 in reservation.test.ts. The counterexample is [1794236400000,1,0,-1] (when the end time is written with -00:01). Fix the cause. Keep this counterexample as a regular test, with the shrunk input as it is. Leave the property test as it is. Stop when you've written it. Don't commit. Differences From the Japanese trial: this trial gave #5 as three options (A permissions, B a PreToolUse hook, C an isolated check server), so request 2 uses those letters. The counterexample was the same; it shrank 12 times (14 in the Japanese trial). The Japanese trial's Kiro asked first whether B could block port 3001. Here, Kiro said the Stop hook could not make it keep working; the docs say exit code 1 starts another turn (not tried) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 35
36. Appendix A4 Sources (1/2) OpenAI, February 2026, harness engineering (a product built with no hand-written code) (slides 4, 6, 10, 13, 22, 23) https://openai.com/index/harness-engineering/ Mitchell Hashimoto's blog, February 2026, my AI adoption journey (slides 4, 7, 9, 29) https://mitchellh.com/writing/my-ai-adoption-journey martinfowler.com (Birgitta Böckeler, Thoughtworks), April 2026, harness engineering for coding agent users (slides 4, 6, 7, 10, 12, 13, 23, 27, 29, 31) https://martinfowler.com/articles/harness-engineering.html kiro.dev blog, October 2026, keeping a software factory stable at 1,000 PRs a week (slides 4, 14, 22, 23, 26, 27, 31) https://kiro.dev/blog/software-factory-stability/ Anthropic, March 2026, harness design for long-running application development (slide 28) https://www.anthropic.com/engineering/harness-design-long-running-apps The numbers are the publishers' own. The articles were read on November 9 and again on November 10, 2026 © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 36
37. Appendix A5 Sources (2/2) OpenAI, September 2026, an article on reviewing skills and instruction files for a new model (slide 28) Anthropic, November 2026, the components of agent memory (slides 18, 19, 27) https://www.anthropic.com/engineering/components-of-agent-memory OpenAI, June 2026, a guide to long-running work (PDF) (slides 17, 18) Anthropic, October 2025, Agent Skills (procedures in folders, read step by step) (slide 17) https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills Anthropic, September 2025, writing effective tools for agents (slide 21) https://www.anthropic.com/engineering/writing-tools-for-agents Kiro, the docs as read on November 10, 2026 (the page names are in each slide's notes) (slides 13, 15, 16, 17, 19, 24, 26, 31, 32, 35) https://kiro.dev/docs/ Kiro, November 2026, changelog (CLI 3.0.0 and 3.1.0, IDE 1.4.121) (slide 19) https://kiro.dev/changelog/ © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 37