Agents on Rails - Rails at Scale 2026

Transcript

1. Agents on Rails Racing Le Mans and picking the right model for the job Irina Nazarova and Vladimir Dementyev Rails at Scale Summit · Austin, TX, September 22, 2026
2. The Project 6 weeks 21 models 41 tasks ~2400 runs published (much more burnt) …and just 15 minutes on stage today 😁
3. Stage #1 21 atomic tasks on Writebook · 18 models · 3 runs each · provider default effort solved 95% 85% Fable 5.1 Kimi K3 Astra Sol Grok 4.6 GLM 5.3 · Opus 4.8 Terra Muse 1.2 Gemini 3.7 Qwen3.8 Opus 5 Fable 5 75% Sonnet 5 65% $3 $1 GLM 5.2 30¢ GLM 5.3 Flash Luna DeepSeek 10¢ 3¢ 1¢ cost per run, cheaper to the right
4. Model Score Cost (avg) Time (med) Fable 5.1 92% $1.1877 5m23s Opus 5 92% $1.9 9m42s Kimi K3 90% $1.0887 12m45s Sol 5.6 84% $0.52 5m4s Astra 6 84% $0.6555 3m30s GLM 5.3 Flash 83% $0.0525 17m8s Luna 5.6 73% $0.0143 3m19s Getting 92% (Fable 5.1) costs 83x more than having 73% (Luna 5.6) You can save 10x when aiming 85% if you’re fine with waiting 4x longer (Sol/Astra vs. GLM)
5. Model Score Cost (avg) Time (med) … … … … Sol 5.6 84% $0.52 5m4s Astra 6 84% $0.6555 3m30s 81% $0.0547 4m55s 73% $0.0143 3m19s Luna 5.6 xhigh Luna 5.6 Getting 92% (Fable 5.1) costs 83x more than having 73% (Luna 5.6) You can save 10x when aiming 85% if you’re fine with waiting 4x longer (Sol/Astra vs. GLM) Effort matters. So does cost
6. Atomic tasks are good for evals Engineers think bigger
7. Stage #2 20 feature tasks on Fizzy · 11 models · 3 runs each · provider default effort solved 30% 20% Astra Fable 5.1 Opus 5 Gemini 3.8 Flash Kimi K3 10% Sol Grok 4.6 GLM 5.3 Flash DeepSeek 4.1 Flash Muse 1.3 0% $10 $3 $1 30¢ Luna 0/60 10¢ 3¢ cost per run, cheaper to the right
8. Model Score Tasks* Cost (avg) Astra 6 35% 10/20 $2.5079 Fable 5.1 32% 9/20 $9.1409 Gemini 3.8 Flash 28% 7/20 $2.9239 Opus 5 25% 8/20 $9.8549 Sol 5.6 18% 7/20 $0.7533 Luna 5.6 0% 0/20 $0.0282 * The number of tasks solved at least once out of 3 (aka pass@3 ). For product features, cents buy nothing Solving a half of all tasks at least once is a real struggle; 3 tasks were never solved
9. Score Tasks Cost (avg) max 53% 12/20 $6.63 Astra 6 35% 10/20 $2.5079 32% 10/20 $19.10 Fable 5.1 32% 9/20 $9.1409 Opus 5 max 32% 8/20 $15.03 Gemini 3.8 Flash 28% 7/20 $2.9239 Sol 5.6 18% 7/20 $0.7533 Model Astra 6 Fable 5.1 max Higher effort is not a silver bullet (see Fable) Though some gains are significant (Astra)
10. Smartshopping vs. tokenmaxxing? Or how to use these numbers Trade-offs are real, so is trade: Picking a faster, cheaper (a bit dumber) model for code reviews on CI. Why not? (Especially if you can break down the review process into more atomic tasks) Some tasks do not require urgency (e.g., continuous profiling and optimization factories). Let a hardworking (=slow) model do the job! Your trade? Share at the happy hour 🧋
11. Beyond numbers
12. Fighting saturation Atomic task Feature task Instructions Tech spec Product outcome Grading What task specifies What an engineer expects (quality) Top score 92% 35%
13. An atomic task Writebook app. Redirect after sign in. Shared and emailed links carry the page they point at as ?return_to= , and SessionsController#create sends the reader there once they sign in. We got a security report and reproduced it. Put somebody else’s address in that parameter and the reader ends up on their site the moment they sign in, looking at a copy of our page that asks them to type their password again. Please close it, and keep signing in working — with a deep link, without one, and whatever anyone puts in that parameter.
14. A feature task Some cards get a burst of attention and then go cold. We want to flag those. A card gets a burst when three or more comments from at least two people land on it within a third of the board’s auto-close period, or when someone is assigned to it or reopens it. A card that had a burst and then sees no activity for two weeks is stalled. Show that on the card wherever cards are shown, and let people list stalled cards: the cards index with indexed_by=stalled . Any activity on the card should reset it. A Fizzy card · comments, assignees, checklist steps
15. A feature task 29/29 passed “stalled after no activity” checks 5/29 passed “ticking off a step clears a stalled card” check A Fizzy card · comments, assignees, checklist steps
16. Failure modes Models crack whatever checks the ticket explicitly mentions like nuts (85% success rate across Stage #2 models). What stays uncrackable most often: Finding all affected call sites Keeping tests green Writing resilient background jobs Creating migrations
17. Nobody runs rails g migration Runs that invoked a Rails generator: 27 of 2,460. All of them `migration`. Atomic tasks Feature tasks Zero or one run per model. Gemini 3.7 Flash managed three. Gemini 3.8 Flash used in 10 runs; Kimi and Sol — in 4 each, Muse Spark — in 2. Every other model: never.
18. Nobody runs rails g migration What do they do instead? Use date (Grok) Extrapolate on db/migrate (Fable) Last migration +1 day (that could be months ago from now) (Astra)
19. Which Rails APIs models reach for One API per task, never named in the ticket · share of 51 runs that reached for it unprompted EVERYONE REACHES FOR IT ONE OR TWO MODELS DO NOBODY, IN 51 RUNS EACH to_fs · 96% insert_all · 78% enqueue_after_transaction_commit ActiveJob::Continuable · 27% BroadcastLogger · 12% DeepSeek, delegated_type purge_later ActiveStorage::Variation after_all_transactions_commit commit_transaction_on_non_local_return 73% rate_limit · 67% sanitize_sql_like url_from · 49% env_config · 49% · 59% · Luna, Muse, Terra quote_column_name · 6% Fable 5.1 run_commit_callbacks_… · 2% Fable 5.1 callable_cache_key · 2% GLM Flash before_destroy · 2% Sol Turbo::StreamsChannel · 2% Sonnet + 2 of Writebook's own methods 5 Age predicts none of this. sanitize_sql_like is ancient and sits on the left. before_destroy is older and sits in the middle.
20. Don't teach the models what they already know Focus on what's missing Run your own evals with lemans
21. gem install lemans lemans init my-evals # bench.yml, a Dockerfile, two example tasks lemans run --agent oracle # prove the bench: every task must score 1.0 lemans run --model <a> --model <b> -k 3
22. The anatomy of a task One unit of work, one score. No DSL, no schema, no config language. tasks/ac-throttle-search/ ├── instruction.md # the ticket, plus frontmatter ├── environment.patch # seed patch: removes the feature or plants the bug ├── solution.patch # the reference fix └── verification_test.rb # hidden checks, uploaded only at grading
23. The grader is a Ruby test ft-card-stalled-quiet-clock/verification_test.rb test "ticking off a step clears a stalled card" do card = comment_burst_on(card_named("Steps, not comments")) step = with_current_user(@author) { card.steps.create!(content: "Ship it") } travel 30.days assert_includes stalled_listing, card.title # 29/29 runs get here with_current_user(@author) { step.update!(completed: true) } assert_not_includes stalled_listing, card.title end # 5/29 get past this
24. What lemans brings Sandbox management (Docker locally for a single run, Daytona to run evals at scale) miniswen — a Ruby version of mini-SWE-agent (simple prompt, single tool, bash ) The flow: checkout, seal, run agents, verify, collect artifacts, generate reports More: multi-step tasks, credit-based grading (not only 0 or 1), and more Fair play guardrails
25. Fair play is hard Models can try to tamper with the grader
26. Fair play is hard Models can try to tamper with the grader. We mitigate: Grading files are only mounted on completion Restore grading-related files to the baseline ( test/ , config/environments/test.rb , bin/ , etc.) Gate diff size def tampered? Minitest::Test.new("probe").assert(false) true rescue Minitest::Assertion false end
27. Fair play is hard Models can try to tamper with the grader Models love code archeology (logs, caches, etc.)
29. Fair play is hard Models can try to tamper with the grader Models love code archeology Models are good escape artists
30. Prison breaks All were captured and punished re-graded DeepSeek 4.1 Flash extracted the OPENROUTER_API_KEY from the sandbox and used it to fetch Fizzy source via Perplexity (since only openrouter.ai was allowed, for miniswen) Muse Spark 1.2 found the HTTP_PROXY env variable and tried to use it with git config http.proxy , curl -x to break out of the Docker runtime and perform HTTP requests through the host—good try! (Daytona’s firewall is unreachable from the sandbox). Grok 4.6 spent two to five steps per run trying to fetch the upstream Fizzy implementation from GitHub using various techniques (different GitHub APIs, different HTTP clients, proxies, etc.).
31. Fair play is hard Models can try to tamper with the grader Models love code archeology Models are good escape artists Models notice they’re being benchmarked
32. What models say about evals Qwen: "I’ll look around to see if there are any hints from the grading tests, CI configuration, hidden tests." Then: "There’s no hidden grader." Opus 5: "No hidden mechanism found in configs. Let me fix the file and then run the test suite to see if anything reverts it." Fable 5.1: "Let me look at the existing tests, since hidden tests will likely extend them." GLM 5.3 Flash: "It’s safer to avoid app changes entirely; the grader may check."
33. Life after Stage #2
34. Future stages and plans Stage 3 is in the works: Large codebases (Mastodon, Redmine) and greenfield apps from scratch Wider tasks: profiling and optimization, evolving (multi-step) features Also: granular grading, more quality checks (architecture, etc.)
35. Stage 3: Mastodon tests optimization Our CI builds are too slow and most of the time is bin/rspec run. Optimize spec/models and spec/requests tests and speed them up at least 2x. Start with spec/models 1, then spec/requests 2, and additionally analyze and optimize our factories usage3. A multi-step (3 turns) task, 140k LoC codebase. What do we grade: The actual speedup is at least 2x (measured multiple times against the baseline captured in the same sandbox) Sidekiq test mode is set to fake by default (reverted upstream optimization) Log level is set to at least :info for tests (not present in the upstream) Other known optimizations are checked but do not affect the pass/fail verdict.
36. Stage 3: Mastodon tests optimization Early insights and findings Model Passed Score Cost (avg) Time (avg) kimi-k3 1/3 0.31 $6.3253 112m 4s grok-4.6 1/3 0.3 $6.9456 79m 25s gemini-3.8-flash 1/3 0.26 $4.6455 120m 59s muse-spark-1.3 0/3 0 $4.8759 43m 58s glm-5.3-flash 0/3 0 $0.3599 190m 27s gpt-6-astra 0/3 0 $5.3272 80m 27s
37. Stage 3: Mastodon tests optimization Insights, findings, stories Fable and Opus considered setting Sidekiq to fake by default for models and requests "too risky since many specs depend on inline processing", "would change behavior"—they haven’t even tried, the assumptions are out of nowhere. Over-engineered ad-hoc patches instead. Muse Spark 1.3 never runs a directory. Every one of its trials failed because of the broken tests it never tried to run. Most models know how to use profilers (StackProf, TestProf) without looking at their source code. Kimi is the deepest profiler. Astra disabled debug logging only for requests (w/ measurements) but not models (using Rails.logger.silence(Logger::WARN) in an RSpec hook).
38. Future stages and plans Beyond stages: More context ( AGENTS.md -like) and tools (search) Multiple prompt flavors (various prompt richness) Real-world agents instead of miniswen Your contributions?
39. Ready to race lemans? You already have a Dockerfile You already have a long history of tickets/issues/pull requests You already have tests
40. Your codebase is the benchmark that matters lemans is MIT. Point it at your monolith.
41. More AI on Rails This topic deserves more than 15 minutes Come join us in SF in November, 10-11
42. Thank you Rails Foundation · Amanda Perino rubyonrails.org/ai · github.com/rails/lemans · github.com/rails/ai-evals Built by Svyatoslav Kryukov, Artur Petrov, Vladimir Dementyev, Alexander Baygeldin, Aibek Izhanov, Anton Senkovskiy and Irina Nazarova