Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Ruby for AI

Slide 3

Slide 3 text

Ruby for AI Applications

Slide 4

Slide 4 text

Ruby for AI Applications Start your new company with Rails and AI—you know why!

Slide 5

Slide 5 text

rails The batteries-included framework Ruby for AI Applications

Slide 6

Slide 6 text

rails The batteries-included framework Ruby for AI Applications ruby_llm #sfruby2026 The AI framework Python should envy

Slide 7

Slide 7 text

rails The batteries-included framework Ruby for AI Applications ruby_llm #sfruby2026 The AI framework Python should envy activeagent MVC for AI interactions and more

Slide 8

Slide 8 text

kino async Take advantage of all Our apps are calling for Ractors! that waiting time! Ruby for AI Applications rails The batteries-included framework ruby_llm #sfruby2026 The AI framework Python should envy activeagent MVC for AI interactions and more

Slide 9

Slide 9 text

kino Our apps are calling async Take advantage of all rails for Ractors! The batteries-included that waiting time! framework Ruby for AI Applications spinel When Master meets Claude roundhouse Rails as an intermediate representation ruby_llm #sfruby2026 The AI framework Python should envy activeagent MVC for AI interactions and more

Slide 10

Slide 10 text

kino Our apps are calling async Take advantage of all rails for Ractors! The batteries-included that waiting time! framework Ruby for AI Applications spinel When Master meets Claude roundhouse Rails as an intermediate representation ruby_llm #sfruby2026 The AI framework Python should envy activeagent MVC for AI interactions and more

Slide 11

Slide 11 text

Ruby for AI

Slide 12

Slide 12 text

AI for Ruby

Slide 13

Slide 13 text

AI for Ruby Engineers

Slide 14

Slide 14 text

The purpose of a tool is to bolster one's abilities, not to off-load one's work entirely. —B.Chambers, A Psalm for the Wild-Built AI for Ruby Engineers

Slide 15

Slide 15 text

The purpose of a tool is to bolster one's abilities, not to off-load one's work entirely. —B.Chambers, A Psalm for the Wild-Built AI for Ruby Engineers agentnear /ˈeɪdʒənt-nɪr/ 1. a person whom an AI agent occasionally consults for instructions or for approval of certain operations • yesterday, my agentnear said 'thank you' to me

Slide 16

Slide 16 text

GENERATION IS NOT A PROBLEM

Slide 17

Slide 17 text

GENERATION IS NOT A PROBLEM @nateberkopec Generation is not the problem. Acceptance and verification is the only problem that matters now.

Slide 18

Slide 18 text

GENERATION IS NOT A PROBLEM @jorgemanru We have unlimited effort at our disposal, but the challenge is providing enough direction, context, and verification without becoming the bottleneck. @nateberkopec Generation is not the problem. Acceptance and verification is the only problem that matters now.

Slide 19

Slide 19 text

GENERATION IS NOT A PROBLEM @jorgemanru We have unlimited effort at our disposal, #sfruby2026 but the challenge is providing enough @skryukov direction, context, and verification Agents scale, humans don't without becoming the bottleneck.

Slide 20

Slide 20 text

GENERATION IS NOT A PROBLEM @jorgemanru We have unlimited effort at our disposal, #sfruby2026 but the challenge is providing enough @skryukov DIRECTION, CONTEXT and VERIFICATION direction, context, verification Agents scale, humans don't without becoming the bottleneck.

Slide 21

Slide 21 text

DIRECTION CONTEXT VERIFICATION

Slide 22

Slide 22 text

DESIGNING VERIFICATION IS MORE IMPORTANT THAN UNDERSTANDING CODE

Slide 23

Slide 23 text

THE PYRAMID Multiple layers of verification (architecture, static, runtime, gentime) The more deterministic the better ↳ Humans stop being a bottleneck

Slide 24

Slide 24 text

Ruby for AI for Ruby

Slide 25

Slide 25 text

evilmartians.com

Slide 26

Slide 26 text

evilmartians.com/clients

Slide 27

Slide 27 text

rubyonrails.org/ai

Slide 28

Slide 28 text

rubyonrails.org/ai

Slide 29

Slide 29 text

Evals

Slide 30

Slide 30 text

Gap Evals Gap

Slide 31

Slide 31 text

Hypothesis Gap Evals DIRECTION Hypothesis Gap CONTEXT Hypothesis VERIFICATION

Slide 32

Slide 32 text

SFRUBY EVALS

Slide 33

Slide 33 text

SFRUBY EVALS Rails AI suite as a baseline Task variants: same instruction, same grading + additional agent capabilities (tools, linters, context) 14 tasks, 21 variants, 7 models, ~400 trials

Slide 34

Slide 34 text

SFRUBY EVALS: GOALS What works and how to configure it properly What's missing or what we as a community must invest our tokens into

Slide 35

Slide 35 text

SFRUBY EVALS: VARIANTS Types Tools & Guards (static and runtime linters, mutation testing) Context (docs, code search, reference apps) ↳ Inspired by personal experience and myths and (un-)verified claims found on the Internet

Slide 36

Slide 36 text

MYTH / FACT: AGENTS NEED TYPES

Slide 37

Slide 37 text

AKITA ON RAILS AND TYPES 7 languages, 4 models, 1 task: no language won "Strong typing spared no repair: there was no type repair to spare" "The models’ effort went to verification and understanding the contract" akitaonrails.com

Slide 38

Slide 38 text

THE AGENTS ON RAILS FILES 2 stacks (Rails, Rust), 2 tasks (Campfire, a greenfield app) evilmartians.com/chronicles

Slide 39

Slide 39 text

THE AGENTS ON RAILS FILES 2 stacks (Rails, Rust), 2 tasks (Campfire, a greenfield app) "Frontier models are stackagnostic" "Cheaper models work well with a strong framework" evilmartians.com/chronicles

Slide 40

Slide 40 text

THE AGENTS ON RAILS FILES 2 stacks (Rails, Rust), 2 tasks (Campfire, a greenfield app) "Frontier models are stackagnostic" "Cheaper models work well with a strong framework" Will a strong framework become even stronger with types?

Slide 41

Slide 41 text

VARIANTS: #RBS, #SORBET 3 Fizzy tasks, 4 models #rbs: inline RBS, rbs_rails, Steep #sorbet: inline RBS syntax, Tapioca, Sorbet TYPES.md: type-writing and checking skill bit.ly/aityped-ruby # app/models/card.rb @fizzy class Card < ApplicationRecord # ... #: () -> String def to_param number.to_s end #: (Board) -> void def move_to(new_board) # ... end #: () -> bool def filled? title.present? || description.present? end end

Slide 42

Slide 42 text

VARIANTS: #RBS, #SORBET Types do not change score score tokens time Model plain #rbs #sorbet Sol 6.1 xhigh 6/8 6/8 7/8 Luna 6 max 4/8 4/8 4/8 DeepSeek v4.1 Flash max 4/8 5/8 4/8 GLM 5.3 Flash max 3/8 3/8 3/8

Slide 43

Slide 43 text

VARIANTS: #RBS, #SORBET Types do not change score Communicating and maintaining types costs tokens score tokens time Model plain #rbs #sorbet Sol 6.1 xhigh 1.0M 1.41x 1.30x Luna 6 max 6.9M 1.44x 1.41x DeepSeek v4.1 Flash max 40.9M 1.20x 0.90x GLM 5.3 Flash max 23.2M 1.11x 0.91x

Slide 44

Slide 44 text

VARIANTS: #RBS, #SORBET Types do not change score Communicating and maintaining types costs tokens Type checking takes time (and Steep is slow) score tokens time Model plain #rbs #sorbet Sol 6.1 xhigh 12m 43s 1.10x 1.18x Luna 6 max 18m 6s 1.51x 1.15x DeepSeek v4.1 Flash max 21m 31s 1.81x 0.98x GLM 5.3 Flash max 26m 25s 0.95x 1.00x

Slide 45

Slide 45 text

VARIANT INSIGHTS: #RBS, #SORBET Agents treat types as a secondary gate to pass Agents rely on tests first Frontier models write correctly typed code themselves (weaker overuse untyped though) @matz (circa 2019) I hate type annotations

Slide 46

Slide 46 text

VARIANT INSIGHTS: #RBS, #SORBET Agents treat types as a secondary gate to pass Agents rely on tests first Frontier models write correctly typed code themselves (weaker overuse untyped though) Type systems are limited (low coverage, no ERB support) ↳ Models can generate typed Ruby but they don't benefit from types—tests win

Slide 47

Slide 47 text

VARIANT INSIGHTS: #RBS, #SORBET Agents treat types as a secondary gate to pass Agents rely on tests first Frontier models write correctly typed code themselves (weaker overuse untyped though) Type systems are limited (low coverage, no ERB support) ↳ Models can generate typed Ruby but they don't benefit from types—tests win @davidteren Not sure if your tests assert enough? Ask mutants!

Slide 48

Slide 48 text

MUTINEER github.com/davidteren/mutineer Mutation testing for Rails (OSS) $ bin/mutineer app/models/user ... Mutate code, run tests. No failures? Mutant survives and points to a potential bug or just a lack of coverage Fizzy case: mutants caught bugs in 3 recent PRs (merged and later fixed) Mutation score: 66.7% (killed / (killed + survived) Surviving Mutants ----------------app/models/user/role.rb User::Role#sole_owner? (role.rb:26) Operator: operand_removal (owner? && account.user (account.users.owner.excluding(self).none?)) - owner? && account.users.owner.excluding(self + (account.users.owner.excluding(self).none?)

Slide 49

Slide 49 text

VARIANT INSIGHTS: #MUTATION Reproduction trials did't confirm the hypothesis: no model left the bug in the code Mutation testing isn't fast (eats ~40% of trials time) One positive (or is it?) side-effect is agents writing more tests ↳ Mutation testing* adds little for modern models— they rarely make unit-level mistakes * Modulo Mutineer: it's in active development phase and hasn't reached its full potential yet

Slide 50

Slide 50 text

MYTH / FACT: ALL YOU NEED IS grep

Slide 51

Slide 51 text

AGENTS VS. CODE SEARCH Every task starts with exploration: locate, inspect, history Agents work in loops and eventually locate everything needed for the task (so they think)

Slide 52

Slide 52 text

AGENTS VS. CODE SEARCH grep is a (fast) brute-force search Every miss is a waste of time and context grep knows no semantics and no connections between system entities ↳ Can a specialized search tool help an agent find a (shorter) path to the solution?

Slide 53

Slide 53 text

CODE SEARCH VARIANTS 2 tasks, 4 models 4 tools: ‣#codegraph ‣#rubydex ‣#lancet ‣#sorbet

Slide 54

Slide 54 text

TASK: ft-board-publish-public-boundary Fizzy Task: Add board publishing (secret public URL) and un-publishing. Caveat: card attachments must has the board's access level (published or private). Authorization hook is already implemented in an Active Storage extension.

Slide 55

Slide 55 text

TASK: ms-retire-proposal $ rails stats Consul Democracy: an open source Rails 8.0 monolith, est. 2015 +-----------------+--------+--------+---------+- | Controllers | 8815 | 7073 | 201 | Task: Retire the citizen proposals feature in three steps: code, database schema, finalize. | Helpers | 1111 | 948 | 0 | | Models | 10241 | 8213 | 177 | | Mailers | 244 | 194 | 4 | | Views | 13987 | 12222 | 0 | | JavaScripts | 3132 | 2808 | 0 | | Stylesheets | 13567 | 10638 | 0 | | ... | | | | Caveat: the feature covers not only Ruby code but templates, YAMLs, assets. | Name | Lines | LOC | Classes | +-----------------+--------+--------+---------+- +-----------------+--------+--------+---------+| Total | 128802 | 100656 | 385 | +-----------------+--------+--------+---------+Code LOC: 42270 Test LOC: 58386 Code to Test

Slide 56

Slide 56 text

SORBET The application type model can be used for code exploration (LSP) $ spoom srb lsp find publicly_accessible? Symbols matching publicly_accessible?: def Card::Accessible#publicly_accessible? (/app/models/card/accessible.rb:12:2-12:26) def Comment#publicly_accessible? (/app/models/comment.rb:24:2-24:79) def User::Avatar#publicly_accessible? (/app/models/user/avatar.rb:42:2-42:26) def ActiveStorage::Attachment#publicly_accessible? (/sorbet/rbi/shims/rails_ext.rbi:45:2-45:26) def Board::Publishable#publicly_accessible? (/app/models/board/publishable.rb:22:15-22:36) def ActiveStorage::Authorize#publicly_accessible_blob? (/lib/rails_ext/active_storage_authorization.rb:46:6-46:35)

Slide 57

Slide 57 text

SORBET The application type model can be used for code exploration (LSP) ↳ But agents prefer grep-ing RBI files $ spoom srb lsp find publicly_accessible? Symbols matching publicly_accessible?: def Card::Accessible#publicly_accessible? (/app/models/card/accessible.rb:12:2-12:26) def Comment#publicly_accessible? (/app/models/comment.rb:24:2-24:79) def User::Avatar#publicly_accessible? (/app/models/user/avatar.rb:42:2-42:26) def ActiveStorage::Attachment#publicly_accessible? (/sorbet/rbi/shims/rails_ext.rbi:45:2-45:26) def Board::Publishable#publicly_accessible? (/app/models/board/publishable.rb:22:15-22:36) def ActiveStorage::Authorize#publicly_accessible_blob? (/lib/rails_ext/active_storage_authorization.rb:46:6-46:35)

Slide 58

Slide 58 text

SORBET Type checker can quickly form a target list for refactoring ↳ But its completion is questionable.. so we grep again $ rm app/models/proposal.rb && tapioca dsl && srb tc sorbet/rbi/dsl/management/proposals_controller.rbi:16: Unable to `ProposalDashboardActionsHelper` sorbet/rbi/dsl/admin/dashboard/actions_controller.rbi:17: Unable `ProposalDashboardActionsHelper` ... app/components/management/menu_component.rb:72: Method `managemen does not exist on `Management::MenuComponent` app/components/management/menu_component.rb:109: Method `print_management_proposals_path` does not exist on `Management:: app/components/moderation/menu_component.rb:9: Method `proposals_ on `Moderation::MenuComponent` ... `Shared::RecommendedIndexComponent` Errors: 27

Slide 59

Slide 59 text

CODEGRAPH github.com/colbymchenry/codegraph Code knowledge graph: extracts symbols, recognizes (some static) relations, SQLite FTS5 $ codegraph explore "board publication public anonymous board col authentication" | head -180 **Exploration: board publication public anonymous board columns c authentication** Found 41 symbols across 6 files. **Relationships** **extends:** - Columns::Cards::Drops::ColumnsController → ApplicationControlle - Boards::Columns::CardsController → ApplicationController - CardsController → ApplicationController - ... and 3 more

Slide 60

Slide 60 text

CODEGRAPH github.com/colbymchenry/codegraph Code knowledge graph: extracts symbols, recognizes (some static) relations, SQLite FTS5 ↳ Poor Ruby understanding, no ERB, agents' grep beats FTS $ codegraph explore "board publication public anonymous board col authentication" | head -180 **Exploration: board publication public anonymous board columns c authentication** Found 41 symbols across 6 files. **Relationships** **extends:** - Columns::Cards::Drops::ColumnsController → ApplicationControlle - Boards::Columns::CardsController → ApplicationController - CardsController → ApplicationController - ... and 3 more

Slide 61

Slide 61 text

RUBYDEX github.com/Shopify/rubydex A static index of Ruby declarations and constant references $ rubydex get_declaration --name Proposal {"name":"Proposal","kind":"Class", "definitions": [{"path":"app/models/proposal.rb","line":1,"comm "ancestors":[ {"name":"Proposal","kind":"Class"}, {"name":"Globalizable","kind":"Module"}, {"name":"ActsAsParanoidAliases","kind":"Module"}, {"name":"SDG::Relatable","kind":"Module"}, {"name":"Randomizable","kind":"Module"}, {"name":"Milestoneable","kind":"Module"}, ... {"name":"ApplicationRecord","kind":"Class"}, ... ] }

Slide 62

Slide 62 text

RUBYDEX github.com/Shopify/rubydex A static index of Ruby declarations and constant references ↳ The best (static) navigator in the Ruby world but not beyond $ rubydex find_constant_references --name Proposal | ruby extract app/components/admin/proposals/index_component.rb app/components/proposals/supports_component.rb app/controllers/admin/hidden_proposals_controller.rb ... spec/models/tag_cloud_spec.rb spec/support/common_actions/proposals.rb total 64 # but we need to modify/delete 266 files !!!

Slide 63

Slide 63 text

RUBYDEX github.com/Shopify/rubydex A static index of Ruby declarations and constant references ↳ Removing a constant makes leftover references invisible $ rubydex find_constant_references --name Proposal { "error":"not_found", "message":"Declaration 'Proposal' not found", ... }

Slide 64

Slide 64 text

LANCET github.com/izhanov/lancet Ollama-generated embeddings and text chunks stored in LanceDB; tuned for Rails $ lt search --limit 8 "active storage authorization attachments" 1 0.96 lib/rails_ext/active_storage_authorization.rb ActiveSupport.on_load :active_storage_blob do def access attachments.includes(:record).any? { |attachment| attachment.acce || attachments.none? end def publicly_accessi… 2 0.50 script/import-sqlite-database.rb class Models > method attachments def attachments @attachments ||= Class.new(application_r self.table_name = "active_storage_attachments" end end ... 5 0.48 ... test/controllers/active_storage/direct_uploads_controlle

Slide 65

Slide 65 text

LANCET github.com/izhanov/lancet Smarter models know how to google, weaker can't come up with good queries ↳ Semantic search skills separate Luna and Sol in this task ft-board-publish-public-boundary Model plain #lancet Sol 6.1 xhigh 0.50 0.94 Luna 6 max 0.50 0.50

Slide 66

Slide 66 text

LANCET github.com/izhanov/lancet Text search can beat grep by being opinionated ↳ grep-ing is also a skill a model must have ms-retire-proposals Model plain #lancet Sol 6.1 xhigh 0.88 0.95 $ grep -RInE '\bProposal\b|\bproposals\b' app \ --exclude-dir=proposals \ --exclude='*proposal*' | grep -v legislation # miss relevant results $ lt search --mode text 'proposals' ... app/views/admin/legislation/proposals/_proposals.html.erb

Slide 67

Slide 67 text

CODE SEARCH INSIGHTS Variants required many iterations: models hesitated to use tools without direction and hooks No tool narrowed the trajectories but semantic search nudged smarter models towards more complete solution

Slide 68

Slide 68 text

CODE SEARCH INSIGHTS Even with dozens of years of grep's knowledge models can omit a flag and result in incomplete coverage (do you teach agents to use grep efficiently) Agents search (grep) more often through logs, tool output, other artifacts than code. They rely on the format assumptions that could be turned into facts via stricter AI-native interfaces

Slide 69

Slide 69 text

CODE SEARCH INSIGHTS A tool that doesn't fit the stack causes the context bloat for nothing (e.g., codegraph printing useless source) A specialized index (e.g., rubydex) costs no extra and can be efficient as a part of the search toolbox ↳ There is no a sliver bullet code search tool; mix, match, build a custom specialized tool—and verify

Slide 70

Slide 70 text

TASK: ft-i18n-support Task: Make Fizzy multilingual! Caveat: plain English lives in Ruby, YAMLs, but mostly in ERB that know search tool treats more than just text

Slide 71

Slide 71 text

TASK: ft-i18n-support Historical (Rails AI) trials: i18n coverage Model .rb .erb Sol 6.1 xhigh 97% 99% Luna 6 max 26% 31% DeepSeek v4.1 Flash max 61% 50% GLM 5.3 Flash max 64% 71%

Slide 72

Slide 72 text

TASK: ft-i18n-support Historical (Rails AI) trials: Some Swiss pal i18n coverage Model .rb .erb Sol 6.1 xhigh 97% 99% Luna 6 max 26% 31% DeepSeek v4.1 Flash max 61% 50% GLM 5.3 Flash max 64% 71% #sfruby2026 Have you heard about Herb?

Slide 73

Slide 73 text

VARIANTS: #HERB, #ERBX #herb Give models Herb and tell how it can be used to extract data from ERB #erbx Build an ERB indexer, erbx, with some ready-made commands $ erbx find --kind text app/views/account/exports/show.html.erb:17
Your export app/views/account/exports/show.html.erb:24
That downloa ... $ erbx find --readable app/views/account account/exports/show.html.erb:2 string @page_title= Download account/exports/show.html.erb:9 string back_link_to#0 Accoun account/exports/show.html.erb:17 account/imports/new.html.erb:7 text
Your export is text
instead? account/settings/_users.html.erb:14 attr ...

Slide 74

Slide 74 text

VARIANT INSIGHTS: #HERB, #ERBX Models don't know Herb A custom indexing tool (erbx) gave models a reliable checklist to work with ↳ Agents need more tools than humans i18n coverage (.erb) Model plain #erbx Luna 6 max 31% 98% DeepSeek v4.1 Flash max 61% 98% GLM 5.3 Flash max 64% 98%

Slide 75

Slide 75 text

TOOLS BUILT FOR HUMANS BECOME GUARDRAILS

Slide 76

Slide 76 text

GUARD RAILS packwerk / packs database_consistency / database_validations strong_migrations isolator n_plus_one_control bullet / prosopite i18n-tasks brakeman ssrf_filter

Slide 77

Slide 77 text

GUARD RAILS packwerk / packs database_consistency / database_validations strong_migrations isolator n_plus_one_control bullet / prosopite i18n-tasks brakeman ssrf_filter evilmartians.com/chronicles

Slide 78

Slide 78 text

GUARD RAILS NEW ERA packwerk / packs archspec database_consistency / database_validations enola-rb strong_migrations isolator n_plus_one_control bullet / prosopite i18n-tasks brakeman ssrf_filter rubydex linter herb-lint ui_guardrails design-lint !

Slide 79

Slide 79 text

ISOLATOR github.com/palkan/isolator Task: ms-campfire-bots Kimi K3 failed this task (0/2) only due to this bug class Room < ApplicationRecord def self.create_for(attributes, users:) transaction do room = create!(attributes) room.memberships.grant_to(users) users.select(&:bot?).each do |bot| Bots::JoinedJob.perform_later(bot, room) end room end end end Room.create_for(...) # == Background worker logs == # # [ActiveJob] Enqueued Bots::JoinedJob... # [ActiveJob] Error: ActiveJob::DeserializationError # (Couldn't find Rooms::Closed with 'id'="2026"...

Slide 80

Slide 80 text

ISOLATOR github.com/palkan/isolator Task: ms-campfire-bots Kimi K3 failed this task (0/2) only due to this bug With isolator, it would score 100% # Gemfile gem "isolator", group: [:development, :test] Isolator.config.raise_exceptions = false # true in test env Room.create_for(...) " Isolator::BackgroundJobError: You are trying to enqueue background job inside db transaction. In case of transaction failure, this may lead to data inconsistency and unexpected bugs Details: Bots::JoinedJob (#, #)

Slide 81

Slide 81 text

SSRF_FILTER github.com/arkadiyt/ssrf_filter Task: ms-campfire-bots 17/42 trials failed only due to SSRF vulnerability (despite RestrictedHTTP in the codebase) $ bin/rails test ... Error: Bot::WebhookTest#test_a_mention_posts_the_payload: Adding a DIY isolator-ssrf made all pass Isolator::SSRF::UnfilteredRequestError: ↳ A human or an agent could spot the problem during review; deterministic guardrails more reliable and provide instant feedback Send the request with SsrfFilter (SsrfFilter.get, .pos POST https://bot.example.com/hook did not use SsrfFi at app/models/bot/webhook.rb:4 to .isolator_ssrf_ignore.yml.

Slide 82

Slide 82 text

GUARDRAILS BEAT PROSE

Slide 83

Slide 83 text

PROSE STILL NEEDED

Slide 84

Slide 84 text

TASKS: at-* Writebook, atomic tasks Rails API recall gives most of the credit Variants: ‣#context7 ‣#raw-docs Rails guides ‣#docs-lancet

Slide 85

Slide 85 text

DOCS INSIGHTS Docs teach models right API for the job Docs make memory verifiable Searchable docs are better than static Models follow the docs' answer even when they ask the wrong question Model plain #rawdocs #docslancet #context7 Sol 6.1 xhigh 0.32 0.55 0.52 0.52 Luna 6 max 0.32 0.55 0.78 0.78 DeepSeek v4.1 Flash max 0.70 - - 0.75 GLM 5.3 Flash max 0.33 - - 0.78 Opus 5.5 medium 0.25 - 0.78 1.0 Astra 6 medium 0.25 - 0.55 0.33

Slide 86

Slide 86 text

TASKS: at-announce-once A book announcement must be sent once (on initial publication) # app/models/book/announceable.rb module Book::Announceable extend ActiveSupport::Concern included do after_update_commit :announce_publication, if: :saved_change_to_published? end end # Tricky edge case: undo (announcement goes off) Book.transaction do @book.update!(published: true) @book.update!(published: false) end

Slide 87

Slide 87 text

TASKS: at-announce-once A book announcement must be sent once (on initial publication) Models often come up with the right design # app/models/book/announceable.rb module Book::Announceable extend ActiveSupport::Concern included do after_save :remember_publication, if: -> { saved_change_to_published?( from: false, to: true) } after_commit :announce_publication, if: :announce_publication? after_rollback :forget_publication end def announce_publication? = @announce_publication && published? end

Slide 88

Slide 88 text

TASKS: at-announce-once Models use docs to verify their idea and hit the enqueue_after_transaction_commit DeepSeek: "This is definitely the standard approach: use enqueue_after_transaction_commit." # app/models/book/announceable.rb module Book::Announceable extend ActiveSupport::Concern included do after_save :remember_publication, if: -> { saved_change_to_published?( from: false, to: true) } after_commit :announce_publication, if: :announce_publication? after_rollback :forget_publication end def announce_publication? = @announce_publication && published? end

Slide 89

Slide 89 text

TASKS: at-announce-once Models use docs to verify their idea and hit the enqueue_after_transaction_commit Their refactor their solution fully assured that it's correct (it isn't) ↳ Trials that tested the undowithin-transaction case passed # app/jobs/application_job.rb class ApplicationJob < ActiveJob::Base self.enqueue_after_transaction_commit = true end # app/models/book/announceable.rb module Book::Announceable extend ActiveSupport::Concern included do after_save :announce_publication, if: -> { saved_change_to_published?( from: false, to: true) } end end

Slide 90

Slide 90 text

TASK: ms-mastodon-tests Mastodon: 7.5k specs, ~10min test run Task: Make model and request specs at least 2x faster (in three steps) Caveat: we not only measure the performance but also assess best practices

Slide 91

Slide 91 text

VARIANTS: #DOCS, #DOCS-PRO Plain trials demonstrated good knowledge of TestProf (and StackProf) among most models #docs: full TestProf docs (a bunch of .md-s) #docs-pro: same + secret sauce

Slide 92

Slide 92 text

VARIANT INSIGHTS: #DOCS, #DOCS-PRO Raw documentation helps only some models Model plain #docs #docs-pro Sol 6.1 xhigh 0.28 0.56 0.83 secret sauce makes cheaper Luna 6 max 0.25 0.00 0.43 (and faster) models suitable for performance work DeepSeek v4.1 Flash max 0.00 0.00 0.57 GLM 5.3 Flash max 0.30 - 0.92 Opus 5.5 medium 0.91 0.91 0.94

Slide 93

Slide 93 text

VARIANT INSIGHTS: #DOCS, #DOCS-PRO Raw documentation helps only some models Speedup (avg / max) Model plain #docs #docs-pro 2.48 / 2.48 2.51 / 2.51 2.45 / 2.45 secret sauce makes cheaper Sol 6.1 xhigh Luna 6 max 2.26 / 2.26 - 2.20 / 2.20 (and faster) models suitable for performance work DeepSeek v4.1 Flash max - - 3.22 / 3.23 GLM 5.3 Flash max 3.06 / 3.06 - 3.27 / 3.43 Opus 5.5 medium 2.84 / 3.02 2.92 / 3.05 3.23 / 3.30 secret sauce maximizes the final test suite speedup

Slide 94

Slide 94 text

secret sauce

Slide 95

Slide 95 text

secret sauce DIRECTION CONTEXT VERIFICATION

Slide 96

Slide 96 text

TestPro SOON test-prof.evilmartians.io/pro INSTRUCTIONS DOCS PLAYBOOKS VERIFICATION INDEX EXAMPLES

Slide 97

Slide 97 text

Hypothesis Gap Evals DIRECTION Hypothesis Gap CONTEXT Hypothesis VERIFICATION

Slide 98

Slide 98 text

Hypothesis Gap Evals DIRECTION Hypothesis Gap CONTEXT Hypothesis VERIFICATION

Slide 99

Slide 99 text

DON'T TRUST BENCHMARKS

Slide 100

Slide 100 text

DON'T TRUST BENCHMARKS RUN YOUR OWN EVALS

Slide 101

Slide 101 text

LEMANS github.com/rails/lemans CLI and a Ruby library Sandbox management (Docker, Daytona) Agents (miniswen, Claude Code, Codex) The flow: checkout, seal, run agents, verify, collect artifacts, generate reports Fair play guardrails evilmartians.com/chronicles

Slide 102

Slide 102 text

LEMANS github.com/rails/lemans Quick start in minutes $ gem install lemans $ lemans init my-bench

Slide 103

Slide 103 text

LEMANS github.com/rails/lemans Quick start in minutes You already have a Dockerfile my-bench/ ├── bench.yml ├── README.md ├── environment/Dockerfile └── tasks/ ├── hello-world/ │ ├── instruction.md │ ├── environment/Dockerfile │ ├── environment.patch │ ├── verification_test.rb │ └── solution.patch └── example-task/

Slide 104

Slide 104 text

LEMANS github.com/rails/lemans Quick start in minutes You already have a Dockerfile You already have a long history of tickets/issues/pull requests my-bench/ ├── bench.yml ├── README.md ├── environment/Dockerfile └── tasks/ ├── hello-world/ │ ├── instruction.md │ ├── environment/Dockerfile │ ├── environment.patch │ ├── verification_test.rb │ └── solution.patch └── example-task/

Slide 105

Slide 105 text

LEMANS github.com/rails/lemans Quick start in minutes You already have a Dockerfile You already have a long history of tickets/issues/pull requests You already have tests my-bench/ ├── bench.yml ├── README.md ├── environment/Dockerfile └── tasks/ ├── hello-world/ │ ├── instruction.md │ ├── environment/Dockerfile │ ├── environment.patch │ ├── verification_test.rb │ └── solution.patch └── example-task/

Slide 106

Slide 106 text

LEMANS github.com/rails/lemans Quick start in minutes You already have a Dockerfile You already have a long history of tickets/issues/pull requests You already have tests ↳ Your codebase is the benchmark that matters $ lemans report runs -A task-model task model score credit hire-martians claude-opus-5.5 3/3 1 hire-martians gpt-6.1-sol#xhigh 3/3 1 layered-design-boundaries claude-opus-5.5 1/3 0.53 layered-design-boundaries gpt-6.1-sol#xhigh 3/3 0.88 fix-flaky-tests claude-opus-5.5 3/3 0.93 fix-flaky-tests gpt-6.1-sol#xhigh 2/3 0.77 resumeable-bg-jobs claude-opus-5.5 3/3 0.9 resumeable-bg-jobs gpt-6.1-sol#xhigh 1/3 0.4 reversible-migrations claude-opus-5.5 2/3 0.67 reversible-migrations gpt-6.1-sol#xhigh 0/3 0.07 claude-opus-5.5 15 trials: 15 scored, 0 invalid, 12 solved gpt-6.1-sol#xhigh 15 trials: 15 scored, 0 invalid, 9 solved total 60 trials: 59 scored, 1 invalid, 40 solved

Slide 107

Slide 107 text

DESIGNING VERIFICATION IS BECOMING OUR PRIMARY JOB

Slide 108

Slide 108 text

VERIFICATION IS THE NEW PLUMBING

Slide 109

Slide 109 text

VERIFICATION IS THE NEW PLUMBING @garrytan Convention over configuration meant we spent our time on product instead of on plumbing

Slide 110

Slide 110 text

CONVENTIONAL VERIFICATION

Slide 111

Slide 111 text

NEXT BIG THING? CONVENTION OVER VERIFICATION

Slide 112

Slide 112 text

NEXT BIG THING? CONVENTION OVER VERIFICATION Look at all the things I'm not verifying when using Rails!

Slide 113

Slide 113 text

GAME IS NOT OVER evilmartians.com github.com/palkan x.com/palkan_tula