Computational checks (Return tests, types, and lint through Hooks) | Learn Harness Engineering with Kiro 04

Level 300. Tests, types, lint, and structural tests give the same result for the same input, so this deck runs them in a Kiro hook as Kiro finishes its answer and returns how to fix what fails. On a small booking app, it shows a check that counts the booking's row in the DB, test output in a few lines that Kiro can act on, and a hook that blocks deleting a test file. This is deck 04 (Check pillar) of the "Learn Harness Engineering with Kiro" series.

Transcript

1. Learn Harness Engineering with Kiro Check Level 300 Computational checks Return tests, types, and lint through Hooks 04 As of November 2026 November 2026 Riku Inada Amazon Web Services Japan + Kiro + Harness Engineering © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 1
2. Introduction About me Riku Inada Amazon Web Services Japan G.K. Solutions Architect Works mainly with manufacturing customers Recent work Promoting the AI-Driven Development Life Cycle (AI-DLC) Writing and speaking about Kiro and AWS Amplify © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. X @inada_riku Speaker Deck speakerdeck.com/inariku Zenn zenn.dev/inariku GitHub github.com/inariku 2
3. Introduction Where this deck fits You are here: 04 Computational checks. Next: 05 AI review 01 Guide The idea Read first to harness 01 Intro engineering 02–03 Check Rules and 02 context Hand over 03 intent as specs 04–05 Contain Computational 04 checks Read next 05 AI review Contain what 06 the agent can do 06 Run the loop 07–10 07 Run it in CI Keep improving 08 the harness long work 09 Connect across sessions whether 10 Measure the harness works © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 3
4. Introduction What you'll learn Run checks with fixed results in a hook as Kiro finishes, and return how to fix what fails 1 What computational checks catch, and what they don't 2 How to run checks with hooks when Kiro finishes its answer and before it uses a tool 3 How to check the state, not the words, and reply in a few lines that say how to fix Read first: 01 (the harness guides and checks Kiro from outside the model). For writing hooks: Kiro: Zero to Hero 06 The trials use a small meeting-room booking app, the same one as in Kiro: Zero to Hero A miss is when Kiro does not do what you asked. In the pictures, the ghost is Kiro Colors: purple = the point, where a person decides, and misses (bugs and × tags). Blue = Kiro at work, green = passed © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 4
5. Part 1 The idea © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 5
6. The idea Computational checks Tests, types, lint, and structural tests are fast, and give the same result for the same input Run by the CPU. The same input gives the same result Tests Types lint Structural tests Behavior Value shapes Code rules Who imports what Vitest tsc lint-layers.mjs structure.test.ts Seconds Under 1 s Under 1 s With the tests All of them run on the CPU and finish in milliseconds to seconds (on the booking app: types in under 1 s, 119 tests in under 5 s). A test that depends on time can wobble and give a different result. Checks of meaning are deck 05 © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 6
7. The idea What machines catch, and what they don't They reliably catch misses of form. Misread requests and overbuilding, not reliably A misread request Misses of form Machines reliably catch form: duplication, complex branches, missing tests, architectural drift, style. Misdiagnosis, overengineering, and misread instructions only sometimes. Meaning: deck 05 and people © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 7
8. The idea A nearly perfect checker Loose checks make the agent solve the wrong problem. Add a check for each miss you see A hole One new check per miss Add a test each time you see a miss, and make CI stricter so new changes cannot break old features. Check that tests meant to pass now pass, and passing tests keep passing © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 8
9. The idea Check the state, not the words Not whether it said "Booked it", but whether the DB has the booking's row Booked it 0 rows in the DB With only unit tests or curl, an agent can call work done that does not work end to end. Make screens, logs, and metrics readable by the agent. The booking app's trial: slide 15 © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 9
10. The idea Where hooks run In a Kiro turn: at the request, before a tool, on a file change, and as it finishes the answer One turn of Kiro Request Tool call File saved Answer ends Prompt Submit Pre Tool Use File Save Agent Stop exit 2 blocks the tool decision block: one more turn A turn = from the request to the end of the answer. A hook is JSON in .kiro/hooks/: a trigger and a command. Not on the web; the CLI chat reads a folder's hooks once you trust it. More: Appendix A1. Checks before starting: 09 © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 10
11. The idea Trials before adding hooks With no rules and no hooks, 2 of 4 runs ended leaving a failing test All done Report: "a test fails" 2 of 4 runs left a failing test Three kinds of request (attendees twice), in Japanese. What failed was an older, slow test. Kiro wrote it in its report and closed the turn. An Agent Stop hook (slide 13) keeps the turn open instead © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 11
12. Part 2 Trying it on the booking app © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 12
13. Trying it on the booking app Put your "checked" rule in a hook The Agent Stop hook becomes this repository's rule for "checked" Answers, stops Failed: says why, Agent Stop hook next turn runs check.mjs Passed: As Kiro finishes, the hook runs the 4 checks in scripts/check.mjs. If one fails, it returns decision block and the reason. Kiro CLI reads the reason and starts another turn. At the end of that turn, the hook did not run. Ask for the recheck in the reason text. it ends The same in every run we tried; the docs do not say. A continued turn that ends still failing is not stopped, so a person checks the final result. Not checked in the IDE. The JSON and the script: Appendix A2 © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 13
14. Trying it on the booking app When it fails, Kiro fixes it The hook returned the failing test, and Kiro read the reason and fixed it 1 First answer (translated) “Property 11 took 2 to 8 s again and once went over the 5 s limit: the known variation, not this change.” 2 The Stop hook returned the failure Kiro timed Property 11 before and after its change Property 11 is the slow test of slide 11: a property-based test that reopens the file DB. From a run of the Japanese edition (November 9, 2026), with the earlier hook (types and tests). The English rerun did not hit this miss 3 Kiro changed how the DB is set up, not the test Then all 121 tests passed (the author ran them, too) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 14
15. Trying it on the booking app A test that checks the booking's row Types, lint, and tests passed, but counting the DB rows found the booking was not kept A person changed 1 line of index.ts to "keep the DB in memory", then asked: add the version to /api/health check.mjs has a check that makes 1 booking on the real server and counts its rows in the DB file. npm run check runs scripts/check.mjs. Next, Kiro was about to end and leave the choice to a person (next slide) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 15
16. Trying it on the booking app Kiro tried to stop; the hook made it continue Kiro tried to end and leave the edit to the person. The hook's reason had it revert (1) The end of the first answer, then the turn after the hook (2) The last answer To let a person decide whether to revert their edit, write that in the hook's reason (this app's says "fix the code"). Japanese trials: the hook made Kiro continue in 2 of 3 © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 16
17. Trying it on the booking app Test output Kiro can read A few lines on screen, the rest in a log. ERROR and the reason on one line, counts first Before: npx vitest run printed 305 lines (a person changed < to <= in the overlap check; 10 tests failed) After: node scripts/check.mjs printed 14 lines (the full output goes to a log file) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 17
18. Trying it on the booking app Stop it before the tool A Pre Tool Use hook blocks a step that deletes a test file A person asked: "Delete src/server/app.test.ts." The write tool was blocked. Kiro did not try another way, and asked the person why. The hook exits with 2 to block; Kiro sees its STDERR. Only deletes are blocked, so check the diff (deck 06) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 18
19. Trying it on the booking app Words for Kiro to act on Write in the test and lint output what is wrong and how to fix it Only says that it failed What is wrong and how to fix it must be guessed AssertionError: expected [ '../server/rooms.ts' ] to deeply equal [] Says what is wrong and how to fix it The page code must not import src/server. Get rooms and bookings from /api with the functions in src/web/api.ts. Add any missing function to api.ts. Top: the default Vitest form (written for this slide). Bottom: the booking app's structural test (slide 20). A fix-it message is a positive kind of prompt injection. Without one, an agent may raise the rule's limit instead of fixing the code © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 19
20. Trying it on the booking app Structural tests Check "the page code must not import src/server" with a test A person added 1 import of src/server to App.tsx on purpose, then ran npx vitest run src/web/structure.test.ts The test reads the imports in src/web. One that points to src/server fails it, with a fix-it message. It is a normal Vitest test, so npm test and the Agent Stop hook run it as is. The message has one short line per point, so it reads well in a terminal © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 20
21. Trying it on the booking app Enforce invariants, leave the details Guard rules like the direction of layers by machine. Leave the details of the code to Kiro Arrows show the order. Each layer imports only the layers to its left base rules storage API startup rooms.ts time.ts reservation.ts db.ts app.ts index.ts storage imports API: the lint stops it Inside a layer, the code is up to Kiro Split the code into layers whose dependencies only go forward, and guard them with custom lint and structural tests. Require checking data shapes at the boundaries, but leave the choice of library for it open © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 21
22. Trying it on the booking app Words of the layer lint When an earlier layer reads a later one, the lint returns the order of layers and how to fix it A person added 1 import of app.ts to db.ts, and ran node scripts/lint-layers.mjs The booking app's layers: base → rules → storage → API → startup. A later layer can read earlier ones; an earlier layer does not read later ones. This kind of lint sees imports, file names, and folders only. Tests check that the content is right © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 22
23. Part 3 Choosing checks © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 23
24. Choosing checks Fastest first Checks of seconds right as Kiro writes, heavy ones in CI, and drift in scheduled jobs As Kiro writes CI on the PR Scheduled jobs Seconds Minutes On a schedule Hooks: types, lint, All tests, Drift: unused code, fast tests end-to-end checks old dependencies When tests take long, give a fast option such as --fast, which runs a random 1% or 10% sample. Heavy checks run in CI on the PR, drift in scheduled jobs. The booking app's checks take seconds, so its hook runs them all © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 24
25. Choosing checks Code that is easy to check Code with types and clear boundaries makes checks easy to build Easy to build checks Hard to build checks Strong types: type checks Old code: weak types, Clear boundaries: structure rules no clear boundaries The booking app: TypeScript, page / API Most needed here, and hardest to build Strong types allow type checks; clear boundaries allow architecture rules. In old code, the harness is hardest to build where it is most needed. The booking app: TypeScript, with a page/API boundary © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 25
26. Choosing checks Checks that never fire For a check that never fails, break the code on purpose and see it fail A net that never rings A planted mutant Mutation testing makes many small breaks on purpose, automatically, and sees if tests fail (deck 03). In a published example, a file with 100% statement coverage still had 13 mutants survive. Measuring: deck 10 © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 26
27. Part 4 Summary © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 27
28. Summary Summary Run checks with fixed results in a hook as Kiro finishes, and return how to fix what fails 1 Computational checks catch misses of form fast, the same each time. Meaning is deck 05 2 Put "checked" in an Agent Stop hook. Check the state, such as the DB rows, not the words 3 Reply in a few lines that say how to fix. When it passes, check the diff for rewritten checks One move for tomorrow: put the tests and type checks you run by hand into one script, and call it from an Agent Stop hook (Appendix A2, the form tried in Kiro CLI) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 28
29. Summary Read next Checks of meaning: 05. Heavy checks on the PR: 07. Measuring whether it works: 10 This series 05 This series 07 This series 10 AI review Run it in CI Measure the harness Reviewers that read meaning Heavy checks on the PR Run tasks and compare How to write hooks, with screens: Kiro: Zero to Hero 06 Steering and hooks. Connecting long work across sessions: 09 © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 29
30. Appendix Appendix © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 30
31. Appendix A1 Hook triggers Trigger JSON trigger IDE CLI Trigger JSON trigger IDE CLI Prompt Submit UserPromptSubmit ✓ ✓ Pre Tool Use PreToolUse ✓ ✓ Agent Stop Stop ✓ ✓ Post Tool Use PostToolUse ✓ ✓ Session Start SessionStart ✓ V3 File Create PostFileCreate ✓ V3 Agent Spawn AgentSpawn — 2.x * File Save PostFileSave ✓ V3 Session End SessionEnd — V3 File Delete PostFileDelete ✓ V3 Pre Compaction PreCompact ✓ V3 Pre Task Exec. PreTaskExec ✓ V3 Post Compaction PostCompact ✓ V3 Post Task Exec. PostTaskExec ✓ V3 ✓ = available. Triggers that can block: Pre Tool Use, Pre Task Exec., Pre Compaction (all exit 2), and Prompt Submit in the IDE. In the CLI, Agent Stop makes Kiro go on (decision block or exit 1). Hooks do not run on the web. V3 is the Kiro CLI 3.x form (* the 2.x name, an alias in V3). The two Compaction triggers came in CLI 3.1.0 and IDE 1.4.121 © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 31
32. Appendix A2 The booking app's check before stopping (key parts) .kiro/hooks/check-before-stop.json { "version": "v1", "hooks": [{ "name": "Check before stopping", "trigger": "Stop", "timeout": 180, "action": { "type": "command", "command": "node .kiro/hooks/check-before-stop.mjs" } }] } .kiro/hooks/check-before-stop.mjs (key lines) const r = spawnSync("node", ["scripts/check.mjs"], { encoding: "utf8" }); if (r.status === 0) process.exit(0); // the checks passed: Kiro stops const why = tries >= 3 ? GIVE_UP : FIX; // tries: FIX replies in this chat const out = { decision: "block", reason: why + "\n\n" + r.stdout }; process.stdout.write(JSON.stringify(out)); // Kiro starts another turn The hook did not run after a continued turn (slide 13), so tries grows over requests in one chat; 0 when the checks pass. At 3, GIVE_UP has Kiro write what still fails and end. FIX: fix the code, not the test, rerun the checks © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 32
33. Appendix A3 Glossary Term Meaning Term Meaning Computational check A CPU check with a fixed result Agent Stop The trigger as Kiro finishes lint A tool that checks writing rules Pre Tool Use The trigger before a tool runs Type check Checks the shape of values matcher Which tools a hook is for Structural test Checks which code reads what exit 2 How a Pre Tool Use hook blocks Layer A level where code is placed decision block How a Stop hook makes Kiro go on Check the state Look at the result, like DB rows STDIN, STDOUT The hook's input and output Harnessability How easy code is to check Mutation testing Change code a little, automatically, and see if tests fail © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 33
34. Appendix A4 Sources Anthropic, February 2026, the harness of parallel agents that built a C compiler https://www.anthropic.com/engineering/building-c-compiler (slides 8, 17, 24) Anthropic, November 2025, harnesses for long-running work (Effective harnesses for long-running agents) https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents (slides 9, 18) Anthropic, January 2026, how to build evals for agents (Demystifying evals for AI agents) https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents (slides 6, 8, 9, 15) OpenAI, February 2026, harness engineering (a product built with no human-written code) https://openai.com/index/harness-engineering/ (slides 9, 19, 21, 22) OpenAI, March 2026, giving the model API a computer environment https://openai.com/index/equip-responses-api-computer-environment/ (slide 17) OpenAI, March 2026, maintaining an open-source SDK with skills and instruction files https://developers.openai.com/blog/skills-agents-sdk (slide 13) OpenAI, September 2026, an article on revisiting skills and instruction files for a new model On the OpenAI developer blog (the URL is left out) (slide 13) © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 34
35. Appendix A5 Sources (continued) OpenAI, March 2026, how it watches its own coding agents for misalignment https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/ (slide 18) martinfowler.com (Birgitta Böckeler of Thoughtworks), April 2026, Harness engineering for coding agent users https://martinfowler.com/articles/harness-engineering.html (slides 4, 6, 7, 19, 20, 24, 25, 26, 33) martinfowler.com (Birgitta Böckeler of Thoughtworks), May 2026, Maintainability sensors for coding agents https://martinfowler.com/articles/sensors-for-coding-agents.html (slides 6, 13, 19, 22, 26, 33) Kiro, docs Hooks, Hook types, Hook actions, Best practices (read as .md on November 10, 2026), and the changelog https://kiro.dev/docs/hooks/ and https://kiro.dev/changelog/ (slides 4, 10, 13, 14, 18, 31, 32, 33) Stryker Mutator, docs, What is mutation testing? https://stryker-mutator.io/docs/ (slides 26, 33) Screens: Kiro CLI 3.1.0 and a terminal on an English copy of the booking app, November 10, 2026. Slide 14 tells a run of the Japanese edition (November 9, 2026). The booking app is not public. Numbers in articles are the publishers' own. Times were measured on the author's Mac and are rough. © 2026, Amazon Web Services, Inc. or its affiliates. All rights reserved. 35