Harness Engineering on Rails

Agents have re-shaped how AI can work with code yet we still use them like ChatGPT and wonder why they disappoint. The quest to one-shot our code with the perfect prompt is a trap. We get much better results by hooking into signals from RSpec, Rubocop, or even a self-review. This requires a paradigm shift: from chatbot to reactive system. When your reaction to mediocre code is to fix the harness rather than the mistake, agents can become valuable assistants that write code you are proud of.

Originally given at RailsWorld 2026

Transcript

1. Harness Engineering on Rails Joël Quenneville
8. What went wrong?
10. Raising the floor on review Human LLM Before After
11. How to Raise the Floor? Better Prompting AGENTS.md like the pros!
13. II What is an Agent?
14. FizzBuzz WRITTEN BY AN LLM
15. Raw API call curl https://api.anthropic.com/v1/messages \ -d "$(jq -n --rawfile tests fizz_buzz_test.rb '{ messages: [{ role: "user", content: ( "Implement fizz_buzz.rb so that all the tests pass.\n\n" + $tests ) }] }')"
16. Writing a Ruby Wrapper prompt = <<~PROMPT Generate an implementation such that all the tests pass. #{File.read("fizz_buzz_test.rb")} PROMPT response = model.ask(prompt) File.write("fizz_buzz.rb", response)
17. We just created an agent harness!
18. Mostly Correct Code def fizz_buzz(number) if number % 3 == 0 "Fizz" elsif number % 5 == 0 "Buzz" elsif number % 15 == 0 # will never trigger "FizzBuzz" else number.to_s end end
19. Use a better prompt? response = <<~PROMPT Implement fizz_buzz.rb so that all the tests pass. #{File.read("fizz_buzz_test.rb")} Remember to handle numbers divisible by both 3 and 5. PROMPT
20. Use a better prompt? prompt = <<~PROMPT Implement fizz_buzz.rb so that all the tests pass. #{File.read("fizz_buzz_test.rb")} Remember to handle numbers divisible by both 3 and 5. MAKE NO MISTAKES! PROMPT
21. Prompt and Pray Prompt Model Code
22. How would you solve this by hand?
23. Human plumbing 1 1 You: Send test failures to LLM You have to copy-paste this 4 2 LLM: generates code 3 You: Save the LLM's new version 2 You have to copy-paste this 3 4 RSpec: runs the tests
24. Adding a loop loop do test_result = run_tests("fizz_buzz_test.rb") break if test_result.success? prompt = <<~PROMPT Write an implementation so that all of the tests pass. Current implementation: #{File.read("fizz_buzz.rb")} Test failures: #{test_result.output} PROMPT response = model.ask(prompt) File.write("fizz_buzz.rb", response) end
25. Agentic Loop Model Prompt Harness Acts Observes Environment
26. The loop allows an agent to respond to failure
27. Commercial Harnesses Cursor Claude Code Codex
28. III Raising the Floor on Rails
29. Sloppy Ruby def dashboard_path_for(user) if user.moderator? == true and user.suspended? == false return moderation_queue_path else dashboard_path end end
30. AGENTS.md? ## Ruby style - Use `&&` and `||` instead of `and` and `or`. - Do not compare predicates with `true` or `false`. - Avoid explicit `return` at the end of a method.
31. AGENTS.md is not the place for your styleguide
32. Prompt and Pray Prompt Model Code
33. How would you solve this by hand?
35. Human plumbing 1 1 You: Send lint failures to LLM You have to copy-paste this 4 2 LLM: generates code 3 You: Save the LLM's new version 2 You have to copy-paste this 3 4 Rubocop: Run the lint rules
36. AGENTS.md directive After changing Ruby files, run `bundle exec rubocop -a`.
37. Claude hook "Stop": [ { "hooks": [ { "type": "command", "command": "grep -q '\"stop_hook_active\": *true' && exit 0; { git diff --nameonly --diff-filter=d HEAD 2>/dev/null; git ls-files -o --exclude-standard; } | grep \"\\.rb$\" | xargs -r bundle exec rubocop --force-exclusion >&2 || exit 2", } ] } ]
38. Be ambitious!
39. Multi-tenant bugs # BAD def show @document = Document.find(params[:id]) end # GOOD def show @document = current_account.documents.find(params[:id]) end
40. Custom Cop def_node_matcher :constant_lookup, <<~PATTERN (send (const {nil? cbase} $_) $_ ...) PATTERN def on_send(node) constant_lookup(node) do |model, method| association = tenant_owned_models[model.to_s] next unless association next unless restricted_methods.include?(method.to_s) receiver = node.receiver add_offense(receiver, message: message) do |corrector| corrector.replace(receiver, "#{tenant_scope}.#{association}") end end end
41. Config Tenancy/ScopedLookup: Enabled: true Include: - app/controllers/**/*.rb TenantScope: current_account TenantOwnedModels: Document: documents RestrictedMethods: - find - find_by - where
42. What about problems that aren't linter-shaped?
43. Broken Migration db/migrate/ 20269141532_add_status_to_documents.rb 20260912143822_create_accounts.rb 20260913170541_add_role_to_users.rb
44. AGENTS.md? When creating a migration: - Prefix the filename with a 14-digit UTC timestamp in `YYYYMMDDHHMMSS` format. - Ensure the migration number follows the existing migrations. - Use a snake_case filename whose name matches the migration class.
45. Prompt and Pray Prompt Model Code
46. How would you solve this by hand?
47. AGENTS.md directive Always create migrations with `bin/rails generate migration`.
48. Claude hook "PreToolUse": [ { "matcher": "Write", "hooks": [ { "type": "command", "command": "ruby \"$CLAUDE_PROJECT_DIR/.claude/hooks/block-db-migratewrites.rb\"", "statusMessage": "Checking migration path" } ] } ]
49. Steer with the Error Message DENIAL_REASON = <<~MSG Creating files under db/migrate/ directly is blocked in this project. Use the Rails generator instead: bin/rails generate migration AddPartNumberToProducts part_number:string It picks the timestamped filename and matching class name. Once the file exists you can edit it freely to refine the migration — only creating it from scratch is blocked. Run `bin/rails generate migration --help` for the column and reference syntax it accepts. MSG
50. Be ambitious!
51. Fancier generators bin/rails generate endpoint Admin::Invoices app/controllers/admin/invoices_controller.rb app/policies/admin/invoice_policy.rb test/controllers/admin/invoices_controller_test.rb test/policies/admin/invoice_policy_test.rb
52. When Determinism Fails Branching mixed with doing Behavior on wrong object Special cases hide abstraction
53. AGENTS.md? - Separate branching from doing. - Prefer behavior on domain objects. - Avoid special cases through better abstractions.
54. Prompt and Pray Prompt Model Code
55. LLM Review Skills /structural-review /method-review /hierararchical-documentation
56. Human plumbing 1 4 1 You: Invoke skills 2 Sub-agent: review code 3 You: Share review 2 You have to copy-paste this 4 3 Main Agent: Make the fixes
57. Orchestration via /implement Implement and verify the requested work following this workflow: 1. Implement the requested work. 2. Have a sub-agent run `/structural-review`; make appropriate fixes. 3. Have a sub-agent run `/method-review`; make appropriate fixes. 4. Have a sub-agent run `/hierarchical-documentation`; make appropriate fixes.
58. Isn't this prompt and pray?
61. IV Putting Things in Perspective
62. MENTAL MODEL Harness Engineering for Coding Agent Users Birgitta Böckler martinfowler.com/articles/harness-engineering.html
63. The 2 Dimensions Feedback Feed forward Computational Inferential
64. Feedback/Computational Computational Feedback Feed forward Rubocop Inferential
65. Feed forward/Computational Computational Rubocop Feedback Feed forward Generators Inferential
66. Feedback/Inferential Computational Rubocop Feedback Feed forward Generators Agentic Review Inferential
67. Feed forward/Inferential Computational Rubocop Feedback Feed forward Generators Agentic Review ??? Inferential
68. Feed forward/Inferential Computational Rubocop Feedback Feed forward Generators Prompts Agentic Review AGENTS.md Inferential
69. Move judgement out of feed forward/inferential
70. Let's be creative! Computational Rubocop Generators Flog Logs Playwrite Prompts Agentic Review AGENTS.md Inferential Security Scanner Tests Feedback Feed forward Reek Code Climate
71. Let's be creative! Computational Rubocop Generators Templates Reek Setup Scripts Playwrite Prompts Agentic Review AGENTS.md Inferential Security Scanner Tests Feedback Feed forward Flog Logs Code Climate
72. Let's be creative! Computational Rubocop Generators Templates Reek Setup Scripts Security Scanner Playwrite Prompts Agentic Review AGENTS.md Tests Agentic Security Scan Intelligent code analysis Inferential Feedback Feed forward Flog Logs Code Climate
73. Let's be creative! Computational Rubocop Generators Reek Setup Scripts Playwrite Prompts Agentic Review AGENTS.md Nearby Source Code Security Scanner Tests Agentic Security Scan Intelligent code analysis Inferential Feedback Feed forward Templates Static Docs Flog Logs Code Climate
74. When a measure becomes a target it ceases to be a good measure GOODHART'S LAW
75. V Fixing a Document Processing System
76. Documents Automation Rate 5%
78. Document Pipeline is a Funnel Parse Text Extract IDs Parse Error Blank Doc Match Publish No IDs No Match Found Wrong Match
79. Stable defintions make populations comparable
81. Single bucket is a good starting point
82. Domain-specific scripts are good for humans too!
84. TOOLING Works with with Rake or Rails shebang scripts
85. VI Continous Improvment
86. So many mechanisms Computational Rubocop Generators Reek Setup Scripts Playwrite Prompts Agentic Review AGENTS.md Nearby Source Code Security Scanner Tests Agentic Security Scan Intelligent code analysis Inferential Feedback Feed forward Templates Static Docs Flog Logs Code Climate
87. How could this be the last time?
88. Human plumbing 1 4 1 You: Invoke tool 2 Tool: gives output 3 You: share output with LLM 4 LLM: process and suggest next tool invocation 2 3
89. Questions to improve your harness Where did I have to intervene? What signals were missing? Was there anything I had to manually copy back and forth? Is there anything in my AGENTS.md that could be better done deterministically? Did the conversation degrade? Could this have been split? Are there any mistakes that have happened multiple times?
90. /retro skill analyzes history and suggests automations
92. No More Prompt and Pray! Prompt Model Code
93. Don't fix the mistake, fix the system!
94. Joël Quenneville @joelquen thoughtbot co-host Principal Developer bikeshed.fm