[SF Ruby 2026] Ruby AI and AI for Ruby

Transcript

2. Ruby for AI
3. Ruby for AI Applications
4. Ruby for AI Applications Start your new company with Rails and AI—you know why!
5. rails The batteries-included framework Ruby for AI Applications
6. rails The batteries-included framework Ruby for AI Applications ruby_llm #sfruby2026 The AI framework Python should envy
7. rails The batteries-included framework Ruby for AI Applications ruby_llm #sfruby2026 The AI framework Python should envy activeagent MVC for AI interactions and more
8. kino async Take advantage of all Our apps are calling for Ractors! that waiting time! Ruby for AI Applications rails The batteries-included framework ruby_llm #sfruby2026 The AI framework Python should envy activeagent MVC for AI interactions and more
9. kino Our apps are calling async Take advantage of all rails for Ractors! The batteries-included that waiting time! framework Ruby for AI Applications spinel When Master meets Claude roundhouse Rails as an intermediate representation ruby_llm #sfruby2026 The AI framework Python should envy activeagent MVC for AI interactions and more
10. kino Our apps are calling async Take advantage of all rails for Ractors! The batteries-included that waiting time! framework Ruby for AI Applications spinel When Master meets Claude roundhouse Rails as an intermediate representation ruby_llm #sfruby2026 The AI framework Python should envy activeagent MVC for AI interactions and more
11. Ruby for AI
12. AI for Ruby
13. AI for Ruby Engineers
14. The purpose of a tool is to bolster one's abilities, not to off-load one's work entirely. —B.Chambers, A Psalm for the Wild-Built AI for Ruby Engineers
15. The purpose of a tool is to bolster one's abilities, not to off-load one's work entirely. —B.Chambers, A Psalm for the Wild-Built AI for Ruby Engineers agentnear /ˈeɪdʒənt-nɪr/ 1. a person whom an AI agent occasionally consults for instructions or for approval of certain operations • yesterday, my agentnear said 'thank you' to me
16. GENERATION IS NOT A PROBLEM
17. GENERATION IS NOT A PROBLEM @nateberkopec Generation is not the problem. Acceptance and verification is the only problem that matters now.
18. GENERATION IS NOT A PROBLEM @jorgemanru We have unlimited effort at our disposal, but the challenge is providing enough direction, context, and verification without becoming the bottleneck. @nateberkopec Generation is not the problem. Acceptance and verification is the only problem that matters now.
19. GENERATION IS NOT A PROBLEM @jorgemanru We have unlimited effort at our disposal, #sfruby2026 but the challenge is providing enough @skryukov direction, context, and verification Agents scale, humans don't without becoming the bottleneck.
20. GENERATION IS NOT A PROBLEM @jorgemanru We have unlimited effort at our disposal, #sfruby2026 but the challenge is providing enough @skryukov DIRECTION, CONTEXT and VERIFICATION direction, context, verification Agents scale, humans don't without becoming the bottleneck.
21. DIRECTION CONTEXT VERIFICATION
22. DESIGNING VERIFICATION IS MORE IMPORTANT THAN UNDERSTANDING CODE
23. THE PYRAMID Multiple layers of verification (architecture, static, runtime, gentime) The more deterministic the better ↳ Humans stop being a bottleneck
24. Ruby for AI for Ruby
25. evilmartians.com
26. evilmartians.com/clients
27. rubyonrails.org/ai
28. rubyonrails.org/ai
29. Evals
30. Gap Evals Gap
31. Hypothesis Gap Evals DIRECTION Hypothesis Gap CONTEXT Hypothesis VERIFICATION
32. SFRUBY EVALS
33. SFRUBY EVALS Rails AI suite as a baseline Task variants: same instruction, same grading + additional agent capabilities (tools, linters, context) 14 tasks, 21 variants, 7 models, ~400 trials
34. SFRUBY EVALS: GOALS What works and how to configure it properly What's missing or what we as a community must invest our tokens into
35. SFRUBY EVALS: VARIANTS Types Tools & Guards (static and runtime linters, mutation testing) Context (docs, code search, reference apps) ↳ Inspired by personal experience and myths and (un-)verified claims found on the Internet
36. MYTH / FACT: AGENTS NEED TYPES
37. AKITA ON RAILS AND TYPES 7 languages, 4 models, 1 task: no language won "Strong typing spared no repair: there was no type repair to spare" "The models’ effort went to verification and understanding the contract" akitaonrails.com
38. THE AGENTS ON RAILS FILES 2 stacks (Rails, Rust), 2 tasks (Campfire, a greenfield app) evilmartians.com/chronicles
39. THE AGENTS ON RAILS FILES 2 stacks (Rails, Rust), 2 tasks (Campfire, a greenfield app) "Frontier models are stackagnostic" "Cheaper models work well with a strong framework" evilmartians.com/chronicles
40. THE AGENTS ON RAILS FILES 2 stacks (Rails, Rust), 2 tasks (Campfire, a greenfield app) "Frontier models are stackagnostic" "Cheaper models work well with a strong framework" Will a strong framework become even stronger with types?
41. VARIANTS: #RBS, #SORBET 3 Fizzy tasks, 4 models #rbs: inline RBS, rbs_rails, Steep #sorbet: inline RBS syntax, Tapioca, Sorbet TYPES.md: type-writing and checking skill bit.ly/aityped-ruby # app/models/card.rb @fizzy class Card < ApplicationRecord # ... #: () -> String def to_param number.to_s end #: (Board) -> void def move_to(new_board) # ... end #: () -> bool def filled? title.present? || description.present? end end
42. VARIANTS: #RBS, #SORBET Types do not change score score tokens time Model plain #rbs #sorbet Sol 6.1 xhigh 6/8 6/8 7/8 Luna 6 max 4/8 4/8 4/8 DeepSeek v4.1 Flash max 4/8 5/8 4/8 GLM 5.3 Flash max 3/8 3/8 3/8
43. VARIANTS: #RBS, #SORBET Types do not change score Communicating and maintaining types costs tokens score tokens time Model plain #rbs #sorbet Sol 6.1 xhigh 1.0M 1.41x 1.30x Luna 6 max 6.9M 1.44x 1.41x DeepSeek v4.1 Flash max 40.9M 1.20x 0.90x GLM 5.3 Flash max 23.2M 1.11x 0.91x
44. VARIANTS: #RBS, #SORBET Types do not change score Communicating and maintaining types costs tokens Type checking takes time (and Steep is slow) score tokens time Model plain #rbs #sorbet Sol 6.1 xhigh 12m 43s 1.10x 1.18x Luna 6 max 18m 6s 1.51x 1.15x DeepSeek v4.1 Flash max 21m 31s 1.81x 0.98x GLM 5.3 Flash max 26m 25s 0.95x 1.00x
45. VARIANT INSIGHTS: #RBS, #SORBET Agents treat types as a secondary gate to pass Agents rely on tests first Frontier models write correctly typed code themselves (weaker overuse untyped though) @matz (circa 2019) I hate type annotations
46. VARIANT INSIGHTS: #RBS, #SORBET Agents treat types as a secondary gate to pass Agents rely on tests first Frontier models write correctly typed code themselves (weaker overuse untyped though) Type systems are limited (low coverage, no ERB support) ↳ Models can generate typed Ruby but they don't benefit from types—tests win
47. VARIANT INSIGHTS: #RBS, #SORBET Agents treat types as a secondary gate to pass Agents rely on tests first Frontier models write correctly typed code themselves (weaker overuse untyped though) Type systems are limited (low coverage, no ERB support) ↳ Models can generate typed Ruby but they don't benefit from types—tests win @davidteren Not sure if your tests assert enough? Ask mutants!
48. MUTINEER github.com/davidteren/mutineer Mutation testing for Rails (OSS) $ bin/mutineer app/models/user ... Mutate code, run tests. No failures? Mutant survives and points to a potential bug or just a lack of coverage Fizzy case: mutants caught bugs in 3 recent PRs (merged and later fixed) Mutation score: 66.7% (killed / (killed + survived) Surviving Mutants ----------------app/models/user/role.rb User::Role#sole_owner? (role.rb:26) Operator: operand_removal (owner? && account.user (account.users.owner.excluding(self).none?)) - owner? && account.users.owner.excluding(self + (account.users.owner.excluding(self).none?)
49. VARIANT INSIGHTS: #MUTATION Reproduction trials did't confirm the hypothesis: no model left the bug in the code Mutation testing isn't fast (eats ~40% of trials time) One positive (or is it?) side-effect is agents writing more tests ↳ Mutation testing* adds little for modern models— they rarely make unit-level mistakes * Modulo Mutineer: it's in active development phase and hasn't reached its full potential yet
50. MYTH / FACT: ALL YOU NEED IS grep
51. AGENTS VS. CODE SEARCH Every task starts with exploration: locate, inspect, history Agents work in loops and eventually locate everything needed for the task (so they think)
52. AGENTS VS. CODE SEARCH grep is a (fast) brute-force search Every miss is a waste of time and context grep knows no semantics and no connections between system entities ↳ Can a specialized search tool help an agent find a (shorter) path to the solution?
53. CODE SEARCH VARIANTS 2 tasks, 4 models 4 tools: ‣#codegraph ‣#rubydex ‣#lancet ‣#sorbet
54. TASK: ft-board-publish-public-boundary Fizzy Task: Add board publishing (secret public URL) and un-publishing. Caveat: card attachments must has the board's access level (published or private). Authorization hook is already implemented in an Active Storage extension.
55. TASK: ms-retire-proposal $ rails stats Consul Democracy: an open source Rails 8.0 monolith, est. 2015 +-----------------+--------+--------+---------+- | Controllers | 8815 | 7073 | 201 | Task: Retire the citizen proposals feature in three steps: code, database schema, finalize. | Helpers | 1111 | 948 | 0 | | Models | 10241 | 8213 | 177 | | Mailers | 244 | 194 | 4 | | Views | 13987 | 12222 | 0 | | JavaScripts | 3132 | 2808 | 0 | | Stylesheets | 13567 | 10638 | 0 | | ... | | | | Caveat: the feature covers not only Ruby code but templates, YAMLs, assets. | Name | Lines | LOC | Classes | +-----------------+--------+--------+---------+- +-----------------+--------+--------+---------+| Total | 128802 | 100656 | 385 | +-----------------+--------+--------+---------+Code LOC: 42270 Test LOC: 58386 Code to Test
56. SORBET The application type model can be used for code exploration (LSP) $ spoom srb lsp find publicly_accessible? Symbols matching publicly_accessible?: def Card::Accessible#publicly_accessible? (/app/models/card/accessible.rb:12:2-12:26) def Comment#publicly_accessible? (/app/models/comment.rb:24:2-24:79) def User::Avatar#publicly_accessible? (/app/models/user/avatar.rb:42:2-42:26) def ActiveStorage::Attachment#publicly_accessible? (/sorbet/rbi/shims/rails_ext.rbi:45:2-45:26) def Board::Publishable#publicly_accessible? (/app/models/board/publishable.rb:22:15-22:36) def ActiveStorage::Authorize#publicly_accessible_blob? (/lib/rails_ext/active_storage_authorization.rb:46:6-46:35)
57. SORBET The application type model can be used for code exploration (LSP) ↳ But agents prefer grep-ing RBI files $ spoom srb lsp find publicly_accessible? Symbols matching publicly_accessible?: def Card::Accessible#publicly_accessible? (/app/models/card/accessible.rb:12:2-12:26) def Comment#publicly_accessible? (/app/models/comment.rb:24:2-24:79) def User::Avatar#publicly_accessible? (/app/models/user/avatar.rb:42:2-42:26) def ActiveStorage::Attachment#publicly_accessible? (/sorbet/rbi/shims/rails_ext.rbi:45:2-45:26) def Board::Publishable#publicly_accessible? (/app/models/board/publishable.rb:22:15-22:36) def ActiveStorage::Authorize#publicly_accessible_blob? (/lib/rails_ext/active_storage_authorization.rb:46:6-46:35)
58. SORBET Type checker can quickly form a target list for refactoring ↳ But its completion is questionable.. so we grep again $ rm app/models/proposal.rb && tapioca dsl && srb tc sorbet/rbi/dsl/management/proposals_controller.rbi:16: Unable to `ProposalDashboardActionsHelper` sorbet/rbi/dsl/admin/dashboard/actions_controller.rbi:17: Unable `ProposalDashboardActionsHelper` ... app/components/management/menu_component.rb:72: Method `managemen does not exist on `Management::MenuComponent` app/components/management/menu_component.rb:109: Method `print_management_proposals_path` does not exist on `Management:: app/components/moderation/menu_component.rb:9: Method `proposals_ on `Moderation::MenuComponent` ... `Shared::RecommendedIndexComponent` Errors: 27
59. CODEGRAPH github.com/colbymchenry/codegraph Code knowledge graph: extracts symbols, recognizes (some static) relations, SQLite FTS5 $ codegraph explore "board publication public anonymous board col authentication" | head -180 **Exploration: board publication public anonymous board columns c authentication** Found 41 symbols across 6 files. **Relationships** **extends:** - Columns::Cards::Drops::ColumnsController → ApplicationControlle - Boards::Columns::CardsController → ApplicationController - CardsController → ApplicationController - ... and 3 more
60. CODEGRAPH github.com/colbymchenry/codegraph Code knowledge graph: extracts symbols, recognizes (some static) relations, SQLite FTS5 ↳ Poor Ruby understanding, no ERB, agents' grep beats FTS $ codegraph explore "board publication public anonymous board col authentication" | head -180 **Exploration: board publication public anonymous board columns c authentication** Found 41 symbols across 6 files. **Relationships** **extends:** - Columns::Cards::Drops::ColumnsController → ApplicationControlle - Boards::Columns::CardsController → ApplicationController - CardsController → ApplicationController - ... and 3 more
61. RUBYDEX github.com/Shopify/rubydex A static index of Ruby declarations and constant references $ rubydex get_declaration --name Proposal {"name":"Proposal","kind":"Class", "definitions": [{"path":"app/models/proposal.rb","line":1,"comm "ancestors":[ {"name":"Proposal","kind":"Class"}, {"name":"Globalizable","kind":"Module"}, {"name":"ActsAsParanoidAliases","kind":"Module"}, {"name":"SDG::Relatable","kind":"Module"}, {"name":"Randomizable","kind":"Module"}, {"name":"Milestoneable","kind":"Module"}, ... {"name":"ApplicationRecord","kind":"Class"}, ... ] }
62. RUBYDEX github.com/Shopify/rubydex A static index of Ruby declarations and constant references ↳ The best (static) navigator in the Ruby world but not beyond $ rubydex find_constant_references --name Proposal | ruby extract app/components/admin/proposals/index_component.rb app/components/proposals/supports_component.rb app/controllers/admin/hidden_proposals_controller.rb ... spec/models/tag_cloud_spec.rb spec/support/common_actions/proposals.rb total 64 # but we need to modify/delete 266 files !!!
63. RUBYDEX github.com/Shopify/rubydex A static index of Ruby declarations and constant references ↳ Removing a constant makes leftover references invisible $ rubydex find_constant_references --name Proposal { "error":"not_found", "message":"Declaration 'Proposal' not found", ... }
64. LANCET github.com/izhanov/lancet Ollama-generated embeddings and text chunks stored in LanceDB; tuned for Rails $ lt search --limit 8 "active storage authorization attachments" 1 0.96 lib/rails_ext/active_storage_authorization.rb ActiveSupport.on_load :active_storage_blob do def access attachments.includes(:record).any? { |attachment| attachment.acce || attachments.none? end def publicly_accessi… 2 0.50 script/import-sqlite-database.rb class Models > method attachments def attachments @attachments ||= Class.new(application_r self.table_name = "active_storage_attachments" end end ... 5 0.48 ... test/controllers/active_storage/direct_uploads_controlle
65. LANCET github.com/izhanov/lancet Smarter models know how to google, weaker can't come up with good queries ↳ Semantic search skills separate Luna and Sol in this task ft-board-publish-public-boundary Model plain #lancet Sol 6.1 xhigh 0.50 0.94 Luna 6 max 0.50 0.50
66. LANCET github.com/izhanov/lancet Text search can beat grep by being opinionated ↳ grep-ing is also a skill a model must have ms-retire-proposals Model plain #lancet Sol 6.1 xhigh 0.88 0.95 $ grep -RInE '\bProposal\b|\bproposals\b' app \ --exclude-dir=proposals \ --exclude='*proposal*' | grep -v legislation # miss relevant results $ lt search --mode text 'proposals' ... app/views/admin/legislation/proposals/_proposals.html.erb
67. CODE SEARCH INSIGHTS Variants required many iterations: models hesitated to use tools without direction and hooks No tool narrowed the trajectories but semantic search nudged smarter models towards more complete solution
68. CODE SEARCH INSIGHTS Even with dozens of years of grep's knowledge models can omit a flag and result in incomplete coverage (do you teach agents to use grep efficiently) Agents search (grep) more often through logs, tool output, other artifacts than code. They rely on the format assumptions that could be turned into facts via stricter AI-native interfaces
69. CODE SEARCH INSIGHTS A tool that doesn't fit the stack causes the context bloat for nothing (e.g., codegraph printing useless source) A specialized index (e.g., rubydex) costs no extra and can be efficient as a part of the search toolbox ↳ There is no a sliver bullet code search tool; mix, match, build a custom specialized tool—and verify
70. TASK: ft-i18n-support Task: Make Fizzy multilingual! Caveat: plain English lives in Ruby, YAMLs, but mostly in ERB that know search tool treats more than just text
71. TASK: ft-i18n-support Historical (Rails AI) trials: i18n coverage Model .rb .erb Sol 6.1 xhigh 97% 99% Luna 6 max 26% 31% DeepSeek v4.1 Flash max 61% 50% GLM 5.3 Flash max 64% 71%
72. TASK: ft-i18n-support Historical (Rails AI) trials: Some Swiss pal i18n coverage Model .rb .erb Sol 6.1 xhigh 97% 99% Luna 6 max 26% 31% DeepSeek v4.1 Flash max 61% 50% GLM 5.3 Flash max 64% 71% #sfruby2026 Have you heard about Herb?
73. VARIANTS: #HERB, #ERBX #herb Give models Herb and tell how it can be used to extract data from ERB #erbx Build an ERB indexer, erbx, with some ready-made commands $ erbx find --kind text app/views/account/exports/show.html.erb:17 <div>Your export app/views/account/exports/show.html.erb:24 <div>That downloa ... $ erbx find --readable app/views/account account/exports/show.html.erb:2 string @page_title= Download account/exports/show.html.erb:9 string back_link_to#0 Accoun account/exports/show.html.erb:17 account/imports/new.html.erb:7 text <div> Your export is text <div> instead? account/settings/_users.html.erb:14 attr <input placeholder> ...
74. VARIANT INSIGHTS: #HERB, #ERBX Models don't know Herb A custom indexing tool (erbx) gave models a reliable checklist to work with ↳ Agents need more tools than humans i18n coverage (.erb) Model plain #erbx Luna 6 max 31% 98% DeepSeek v4.1 Flash max 61% 98% GLM 5.3 Flash max 64% 98%
75. TOOLS BUILT FOR HUMANS BECOME GUARDRAILS
76. GUARD RAILS packwerk / packs database_consistency / database_validations strong_migrations isolator n_plus_one_control bullet / prosopite i18n-tasks brakeman ssrf_filter
77. GUARD RAILS packwerk / packs database_consistency / database_validations strong_migrations isolator n_plus_one_control bullet / prosopite i18n-tasks brakeman ssrf_filter evilmartians.com/chronicles
78. GUARD RAILS NEW ERA packwerk / packs archspec database_consistency / database_validations enola-rb strong_migrations isolator n_plus_one_control bullet / prosopite i18n-tasks brakeman ssrf_filter rubydex linter herb-lint ui_guardrails design-lint !
79. ISOLATOR github.com/palkan/isolator Task: ms-campfire-bots Kimi K3 failed this task (0/2) only due to this bug class Room < ApplicationRecord def self.create_for(attributes, users:) transaction do room = create!(attributes) room.memberships.grant_to(users) users.select(&:bot?).each do |bot| Bots::JoinedJob.perform_later(bot, room) end room end end end Room.create_for(...) # == Background worker logs == # # [ActiveJob] Enqueued Bots::JoinedJob... # [ActiveJob] Error: ActiveJob::DeserializationError # (Couldn't find Rooms::Closed with 'id'="2026"...
80. ISOLATOR github.com/palkan/isolator Task: ms-campfire-bots Kimi K3 failed this task (0/2) only due to this bug With isolator, it would score 100% # Gemfile gem "isolator", group: [:development, :test] Isolator.config.raise_exceptions = false # true in test env Room.create_for(...) " Isolator::BackgroundJobError: You are trying to enqueue background job inside db transaction. In case of transaction failure, this may lead to data inconsistency and unexpected bugs Details: Bots::JoinedJob (#<User:..>, #<Room:..>)
81. SSRF_FILTER github.com/arkadiyt/ssrf_filter Task: ms-campfire-bots 17/42 trials failed only due to SSRF vulnerability (despite RestrictedHTTP in the codebase) $ bin/rails test ... Error: Bot::WebhookTest#test_a_mention_posts_the_payload: Adding a DIY isolator-ssrf made all pass Isolator::SSRF::UnfilteredRequestError: ↳ A human or an agent could spot the problem during review; deterministic guardrails more reliable and provide instant feedback Send the request with SsrfFilter (SsrfFilter.get, .pos POST https://bot.example.com/hook did not use SsrfFi at app/models/bot/webhook.rb:4 to .isolator_ssrf_ignore.yml.
82. GUARDRAILS BEAT PROSE
83. PROSE STILL NEEDED
84. TASKS: at-* Writebook, atomic tasks Rails API recall gives most of the credit Variants: ‣#context7 ‣#raw-docs Rails guides ‣#docs-lancet
85. DOCS INSIGHTS Docs teach models right API for the job Docs make memory verifiable Searchable docs are better than static Models follow the docs' answer even when they ask the wrong question Model plain #rawdocs #docslancet #context7 Sol 6.1 xhigh 0.32 0.55 0.52 0.52 Luna 6 max 0.32 0.55 0.78 0.78 DeepSeek v4.1 Flash max 0.70 - - 0.75 GLM 5.3 Flash max 0.33 - - 0.78 Opus 5.5 medium 0.25 - 0.78 1.0 Astra 6 medium 0.25 - 0.55 0.33
86. TASKS: at-announce-once A book announcement must be sent once (on initial publication) # app/models/book/announceable.rb module Book::Announceable extend ActiveSupport::Concern included do after_update_commit :announce_publication, if: :saved_change_to_published? end end # Tricky edge case: undo (announcement goes off) Book.transaction do @book.update!(published: true) @book.update!(published: false) end
87. TASKS: at-announce-once A book announcement must be sent once (on initial publication) Models often come up with the right design # app/models/book/announceable.rb module Book::Announceable extend ActiveSupport::Concern included do after_save :remember_publication, if: -> { saved_change_to_published?( from: false, to: true) } after_commit :announce_publication, if: :announce_publication? after_rollback :forget_publication end def announce_publication? = @announce_publication && published? end
88. TASKS: at-announce-once Models use docs to verify their idea and hit the enqueue_after_transaction_commit DeepSeek: "This is definitely the standard approach: use enqueue_after_transaction_commit." # app/models/book/announceable.rb module Book::Announceable extend ActiveSupport::Concern included do after_save :remember_publication, if: -> { saved_change_to_published?( from: false, to: true) } after_commit :announce_publication, if: :announce_publication? after_rollback :forget_publication end def announce_publication? = @announce_publication && published? end
89. TASKS: at-announce-once Models use docs to verify their idea and hit the enqueue_after_transaction_commit Their refactor their solution fully assured that it's correct (it isn't) ↳ Trials that tested the undowithin-transaction case passed # app/jobs/application_job.rb class ApplicationJob < ActiveJob::Base self.enqueue_after_transaction_commit = true end # app/models/book/announceable.rb module Book::Announceable extend ActiveSupport::Concern included do after_save :announce_publication, if: -> { saved_change_to_published?( from: false, to: true) } end end
90. TASK: ms-mastodon-tests Mastodon: 7.5k specs, ~10min test run Task: Make model and request specs at least 2x faster (in three steps) Caveat: we not only measure the performance but also assess best practices
91. VARIANTS: #DOCS, #DOCS-PRO Plain trials demonstrated good knowledge of TestProf (and StackProf) among most models #docs: full TestProf docs (a bunch of .md-s) #docs-pro: same + secret sauce
92. VARIANT INSIGHTS: #DOCS, #DOCS-PRO Raw documentation helps only some models Model plain #docs #docs-pro Sol 6.1 xhigh 0.28 0.56 0.83 secret sauce makes cheaper Luna 6 max 0.25 0.00 0.43 (and faster) models suitable for performance work DeepSeek v4.1 Flash max 0.00 0.00 0.57 GLM 5.3 Flash max 0.30 - 0.92 Opus 5.5 medium 0.91 0.91 0.94
93. VARIANT INSIGHTS: #DOCS, #DOCS-PRO Raw documentation helps only some models Speedup (avg / max) Model plain #docs #docs-pro 2.48 / 2.48 2.51 / 2.51 2.45 / 2.45 secret sauce makes cheaper Sol 6.1 xhigh Luna 6 max 2.26 / 2.26 - 2.20 / 2.20 (and faster) models suitable for performance work DeepSeek v4.1 Flash max - - 3.22 / 3.23 GLM 5.3 Flash max 3.06 / 3.06 - 3.27 / 3.43 Opus 5.5 medium 2.84 / 3.02 2.92 / 3.05 3.23 / 3.30 secret sauce maximizes the final test suite speedup
94. secret sauce
95. secret sauce DIRECTION CONTEXT VERIFICATION
96. TestPro SOON test-prof.evilmartians.io/pro INSTRUCTIONS DOCS PLAYBOOKS VERIFICATION INDEX EXAMPLES
97. Hypothesis Gap Evals DIRECTION Hypothesis Gap CONTEXT Hypothesis VERIFICATION
98. Hypothesis Gap Evals DIRECTION Hypothesis Gap CONTEXT Hypothesis VERIFICATION
99. DON'T TRUST BENCHMARKS
100. DON'T TRUST BENCHMARKS RUN YOUR OWN EVALS
101. LEMANS github.com/rails/lemans CLI and a Ruby library Sandbox management (Docker, Daytona) Agents (miniswen, Claude Code, Codex) The flow: checkout, seal, run agents, verify, collect artifacts, generate reports Fair play guardrails evilmartians.com/chronicles
102. LEMANS github.com/rails/lemans Quick start in minutes $ gem install lemans $ lemans init my-bench
103. LEMANS github.com/rails/lemans Quick start in minutes You already have a Dockerfile my-bench/ ├── bench.yml ├── README.md ├── environment/Dockerfile └── tasks/ ├── hello-world/ │ ├── instruction.md │ ├── environment/Dockerfile │ ├── environment.patch │ ├── verification_test.rb │ └── solution.patch └── example-task/
104. LEMANS github.com/rails/lemans Quick start in minutes You already have a Dockerfile You already have a long history of tickets/issues/pull requests my-bench/ ├── bench.yml ├── README.md ├── environment/Dockerfile └── tasks/ ├── hello-world/ │ ├── instruction.md │ ├── environment/Dockerfile │ ├── environment.patch │ ├── verification_test.rb │ └── solution.patch └── example-task/
105. LEMANS github.com/rails/lemans Quick start in minutes You already have a Dockerfile You already have a long history of tickets/issues/pull requests You already have tests my-bench/ ├── bench.yml ├── README.md ├── environment/Dockerfile └── tasks/ ├── hello-world/ │ ├── instruction.md │ ├── environment/Dockerfile │ ├── environment.patch │ ├── verification_test.rb │ └── solution.patch └── example-task/
106. LEMANS github.com/rails/lemans Quick start in minutes You already have a Dockerfile You already have a long history of tickets/issues/pull requests You already have tests ↳ Your codebase is the benchmark that matters $ lemans report runs -A task-model task model score credit hire-martians claude-opus-5.5 3/3 1 hire-martians gpt-6.1-sol#xhigh 3/3 1 layered-design-boundaries claude-opus-5.5 1/3 0.53 layered-design-boundaries gpt-6.1-sol#xhigh 3/3 0.88 fix-flaky-tests claude-opus-5.5 3/3 0.93 fix-flaky-tests gpt-6.1-sol#xhigh 2/3 0.77 resumeable-bg-jobs claude-opus-5.5 3/3 0.9 resumeable-bg-jobs gpt-6.1-sol#xhigh 1/3 0.4 reversible-migrations claude-opus-5.5 2/3 0.67 reversible-migrations gpt-6.1-sol#xhigh 0/3 0.07 claude-opus-5.5 15 trials: 15 scored, 0 invalid, 12 solved gpt-6.1-sol#xhigh 15 trials: 15 scored, 0 invalid, 9 solved total 60 trials: 59 scored, 1 invalid, 40 solved
107. DESIGNING VERIFICATION IS BECOMING OUR PRIMARY JOB
108. VERIFICATION IS THE NEW PLUMBING
109. VERIFICATION IS THE NEW PLUMBING @garrytan Convention over configuration meant we spent our time on product instead of on plumbing
110. CONVENTIONAL VERIFICATION
111. NEXT BIG THING? CONVENTION OVER VERIFICATION
112. NEXT BIG THING? CONVENTION OVER VERIFICATION Look at all the things I'm not verifying when using Rails!
113. GAME IS NOT OVER evilmartians.com github.com/palkan x.com/palkan_tula