use them They write some of my code 0% 0 0% 0 They write all my code; I read it all 0% 0 They write all my code; I read some of it 0% 0 They write all my code; I don't read it 0% 0 0 CONNECTED · 0 RESPONDED all-hands-martians.netlify.app/quiz.html
simple Ruby coding agent built with RubyLLM. One tool: bash. Ruby benchmarking tooling. Runs miniswen in Daytona sandboxes. Tasks and published run data. Logs, patches, and results. miniswen rails/lemans rails/ai-evals 3 runs per task, per model, at the provider’s default effort.
· 3 runs per task · provider default effort Runs solved 95% Fable 5 Opus 5 85% 75% Muse 1.2 4 / 4 Fable 5.1 Terra Sol Sonnet 5 GLM 5.3 · Opus 4.8 GLM 5.2 Grok 4.6 Qwen3.8 65% $3 Another tie, half the cost Kimi K3 $1 • Default effort ┄ Best score at each price Kimi K3 matches Fable 5 on solved runs for less than half the cost. GLM 5.3 Flash DeepSeek Luna Gemini 3.7 30¢ 10¢ 3¢ 1¢ Average cost per run · cheaper → Kimi K3 90.5% Fable 5 90.5% $1.09 / run $2.32 / run
the ticket as the model reads it TICKET · WRITEBOOK Shared and emailed links carry the page they point at as ?return_to= , and SessionsController#create sends the reader there once they sign in. We got a security report and reproduced it. Put somebody else’s address in that parameter and the reader ends up on their site the moment they sign in, looking at a copy of our page that asks them to type their password again. Please close it, and keep signing in working: with a deep link, without one, and whatever anyone puts in that parameter.
burst of attention and then go cold. We want to flag those. A card gets a burst when three or more comments from at least two people land on it within a third of the board’s auto-close period, or when someone is assigned to it or reopens it. A card that had a burst and then sees no activity for two weeks is stalled. Show that on the card wherever cards are shown, and let people list stalled cards: the cards index with indexed_by=stalled . Any activity on the card should reset it. A Fizzy card · comments, assignees, checklist steps
default batch · 29 runs reached grading # ft-card-stalled-quiet-clock/verification_test.rb test "ticking off a step clears a stalled card" do card = comment_burst_on(card_named("Steps, not comments")) step = with_current_user(@author) do card.steps.create!(content: "Ship it") end travel 30.days assert_includes stalled_listing, card.title with_current_user(@author) { step.update!(completed: true) } assert_not_includes stalled_listing, card.title end 29/29 stalled after 30 quiet days 5/29 completing a step clears the stalled status
· 3 runs per task · default effort, then higher effort Runs solved Fable 5.1 More effort is no guarantee Astra max Sol 5.6 max Gemini 3.8 Flash Astra Opus 5.5 Muse max 50% Fable max 40% Opus 5 max 30% 5 / 5 Luna 5.6 max Gemini high GPT-6 Sol GPT-6 Luna Muse 1.3 GLM 5.3 Flash Kimi K3 Grok 4.7 DeepSeek max 10% Sol 5.6 DeepSeek 4.1 Flash Luna 5.6 0/60 Grok 4.6 xhigh 20% Opus 5 Grok 4.6 0% $10 $3 $1 30¢ 10¢ 3¢ Average cost per run · cheaper → • Default effort ◆ Higher effort ┄ Best score at each price The full sweep: gains, flat scores, and a drop. Measure the trade on your own tasks. Astra max 53.3% Fable max 31.7% Gemini high 23.3% $6.63 / run $19.10 / run $3.18 / run
Organizer of the SF Ruby meetup and conference Trained as a C++ engineer Master's in economics and data analysis Former equity research analyst Former startup founder Also: Murka's human. With Marco and Vipul, hiking at Stanford
when people and companies run into issues with them. SHARE PRIMITIVES, CONVENTIONS AND TOOLS Ruby and Rails core and the ecosystem. SHARE MENTAL MODELS How to think about hard or new things. Vipul and Marco at the SF Ruby meetup
Less joy in the work. Research, then build Separate modules by risk. Build the high-risk code and the harness. Own evals, observability, skills, linters, hooks for the whole team.
What actually happened? What could explain it? Make a testable prediction. 4. TEST 5. ANALYZE 6. CONCLUDE Run an experiment. Compare the result with your prediction. Use what you learned. Repeat. We can find joy in figuring out why.
activity feed API Grok 4.7 stalled cards Claude Opus 5.5 entropy sweep 2/3 runs ended with an empty patch “I couldn’t reproduce the slowdown, so I haven’t changed any code.” The honest answer, and still a failed ticket: the sweep keeps reading every card on every run.
of 2,670 public runs. Fake a timestamp 541/873 migration-writing runs The newest timestamp + 1–2 days, even months later. Delete valid rows 27/64 account-ID safety checks The backfill missed an owner, then deleted the row as an “orphan.” Test up → down → up on seeded data. Check that the data survives. Block rollback 9/64 account-ID safety checks The migration raised IrreversibleMigration .
my-evals # bench.yml, a Dockerfile, two example tasks lemans run --agent oracle # prove the bench: every task must score 1.0 lemans run --model <a> --model <b> -k 3 tasks/ac-throttle-search/ ├── instruction.md # the ticket, plus frontmatter ├── environment.patch # seed patch: removes the feature or plants the bug ├── solution.patch # the reference fix └── verification_test.rb # hidden checks, uploaded only at grading
tasks · pick the hard ones A bug or feature you shipped → tasks/<name>/ The commit before the fix → environment/Dockerfile · git checkout FIX^ The issue, word for word → instruction.md The tests that cover it today → verification_test.rb hidden until grading The fix you merged → solution.patch the oracle must score 1.0 lemans run --agent oracle → 1.0 · --agent nop → 0 · then --model a --model b -k 3
OF CAMPFIRE 37signals' chat app. 3,993 lines of Ruby in app/ and lib/ . AI-generated, Rails-shaped. 50,818 lines of Rust: 12.7× the code. Existing app · chat bots in 5 steps · shared required behavior tests · October 5, unpublished
METRIC RAILS RUST RATIO Cost $5.67 $20.52 3.6× Cost $7.14 $14.17 2.0× Agent time 35 min 85 min 2.4× Agent time 27 min 59 min 2.2× Completion score 0.86 0.88 n/a Completion score 0.88 0.84 n/a One completed run per model per stack · ratio = Rust ÷ Rails · completion score includes optional checks
configurations × 2 stacks · one trial each RAILS RUST Start with the framework. Build a conference quiz app. Start with an empty Cargo app. Choose the web stack. Author quizzes → import/export → live play → scoring → timer Same five steps. Same external behavior tests.
Unclear agent build speed 50% / 75% completion rate · Rust / Rails All 8 configurations per stack · one trial each · completion = every required check passed Speed comparison: Rust had twice the CPU and memory.
throughput. Rails can be cheaper or have a higher completion rate. 03 Alternatives to a “Rust rewrite” 01 The advantage depends on the model and task. Rewrite a service decoupled from business logic. Compile Rails to a high-performance target with Roundhouse or Spinel (WIP). Switch to Roda, Sinatra, or Rage.
· x.com/inazarova Rails Foundation · Amanda Perino Built by Svyatoslav Kryukov, Artur Petrov, Vladimir Dementyev, Alexander Baygeldin, Aibek Izhanov, Anton Senkovskiy and Irina Nazarova
install lemans A provider outage at step 4 of 5 no longer costs the race: restart from the last completed step --recover picks up the agent’s own session mid-step --reverify grades again with your new tests, no agent rerun Allowlist network on Docker: the agent reaches the model and nothing else gem install lemans # 1.5.1 lemans restart RUN --recover lemans restart RUN --reverify lemans report -A model -S score-credit Multi-step runs get a progress column: steps completed out of the task’s steps
· use cases Domain models · scopes Infrastructure database · HTTP Account enumeration 53% Morph wipes what you type 40% Every screen translated 12% .count on a preload 62% The migration deletes rows Rails wrote for you Exactly-once after a restart 7% Cleanup without retry 25% N+1 past the preload 12% touch on the wrong column 15% to_plain_text in the view only 11% Recount instead of a counter 25% SSRF on a push URL 11% SSRF on a webhook URL 47% A sweep no index serves 31% Backfill that survives a rollback 25% 25% of evaluated runs pass this check Rails keeps attachments and rich text in its own tables, linked to your records in ways your models don’t show. A backfill that only knows your models calls those rows orphans and deletes them. TOP MODELS Astra scored zero on all six runs of this ticket. Fable and Opus 5.5 pass two in three.
irinanazarova
davidcarrasco
axbom
mclloyd
addyosmani
ks91
philnash
jonyablonski
coloredviolet
thirion
signer
tammyeverts
mkilby
