Ruby on Ruby: ruby/spec, MRI Tests, and JRuby and Living on the Edge

Ruby has multiple implementations. CRuby (the canonical "MRI") is the reference, but JRuby (now at Ruby 4.0 on the JVM) and TruffleRuby (at 3.4, with 4.0 underway on GraalVM) are real, production-grade alternatives. They aim to run the same Ruby code, so your gems, your Rails apps, your scripts. But they're written in different languages, on different VMs, with different internal data structures. So how do they agree on what Ruby actually is?

Transcript

1. Ruby on Ruby ruby/spec, MRI Tests, and JRuby — Living on the Edge Sampo Kuokkanen · Evil Martians · COSCUP 2026
2. Sampo Kuokkanen Head of Evil Martians Japan Rails developer Osaka evilmartians.com · evilmartians.jp Evil Martians paid for the flight that got me here.
3. @ioquatix, 19 April — three days before RubyKaigi
4. On the way up we stopped at a preserved tsunami ruin in Miyako and the tsunami museum. That same afternoon, a magnitude 7.7 earthquake struck 100 km off that coast.
5. RubyKaigi is three days. The #KaigiEffect starts on the drive home.
6. RubyKaigi ended April 24th. My first JRuby pull request was April 28th.
7. How many Rubies are there? (But not all of them!)
8. CRuby / MRI — the reference. Written in C. What you get when you install Ruby. JRuby — runs on the JVM. Written in Java. Now targeting Ruby 4.0. TruffleRuby — runs on GraalVM. At 3.4, with 4.0 underway.
9. "hello".upcase # => "HELLO" All three must agree on this. CRuby JRuby TruffleRuby struct RString { long len; char *ptr; char ary[1]; } class RubyString extends RubyObject { ByteList value; } class RubyString { AbstractTruffleString tstring; } a Java object a TruffleString a C struct NOW SHARED STILL NOT SHARED The parser — Prism What any of it means all three, same C library every semantic in this talk
10. So how do they agree on what Ruby actually is?
11. MRI's test suite ruby/spec test/ruby/*.rb spec/ruby/**/*.rb Written for CRuby, by CRuby. Written to be shared. Uses test-unit , knows about C internals, checks GC and ObjectSpace. RSpec-ish syntax, no internals, version-guarded. The de facto spec. The intentional spec.
12. 2006 — starts inside Rubinius. JRuby contributes early. 2008 — splits out on its own, as RubySpec. 2014 — Brian Shirai ends it. Not enough uptake. 2015 — Charles Nutter asks ruby-core what happens now. Benoit Daloze revives it as ruby/spec. today — ~15,000 commits all told. TruffleRuby over half of everything since 2014; CRuby a steady fifth every year.
13. Part One: JRuby
14. JRuby vendors both suites: spec/ruby/ — a copy of ruby/spec test/mri/ — a copy of MRI's own tests And then it runs them against the JVM.
15. He's making a list, And checking it twice, Gonna find out, Who's naughty and nice
16. test/mri/excludes/TestDefined.rb exclude :test_defined_matchdata, "work in progress" exclude :test_defined_method_single_call, "work in progress" exclude :test_defined_paren_void_stmts, "needs investigation" exclude :test_defined_protected_method, "work in progress" exclude :test_defined_refined_call_with_using, "work in progress" exclude :test_defined_refined_vcall_with_using, "work in progress" exclude :test_defined_undefined_argument, "work in progress"
17. 272 2,215 exclude files excluded tests 634 437 "needs investigation" "work in progress"
18. WILL NEVER BE FIXED EXPLICITLY OPEN ~273 1,071 "GC is not configurable" "needs investigation" "no working assert_no_memory_leak" "work in progress" "uses ObjectSpace.memsize_of" "dumping is largely a CRuby-specific feature" × 634 × 437 Nobody has looked properly yet. "tests CRuby internals via a special File extension" "Continuations are not supported" TruffleRuby keeps 2,645 of these, for the same reasons. This list can never reach zero. It isn't supposed to.
19. Half of that list is permanent. The other half is a to-do list.
20. Start small
21. JRuby # ruby -w should NOT warn about these: [[1, 2]].each do |(a, b)| # `a` and `b` unused — but destructured, # so no "assigned but unused variable" end # JRuby warned. MRI didn't.
22. jruby/jruby#9400 — merged
23. 1. Pick a line from excludes/ 2. Run that one MRI test. Watch it fail. 3. Lift the Ruby out of the test method. 4. Run it on both Rubies. 5. Fix JRuby. 6. Delete the exclude line.
24. ruby/ruby#16885 — my first CRuby patch def test_freeze_inside_sort_bang array = [1, 2, 3, 4, 5] assert_raise(FrozenError) do count = 0 array.sort! do |a, b| - array.freeze if (count += 1) == 6 + array.freeze if (count += 1) == 3 b <=> a end end end
25. jruby/jruby#9421 — the fast path forgot # JRuby 10.1.0.0 [1, 2, 3].freeze.reverse! # FrozenError ✓ correct [1, 2].freeze.reverse! # => [2, 1] ✗ silently mutated [1].freeze.reverse! # => [1] ✗ silently mutated # org/jruby/specialized/RubyArrayOneObject.java +1 # org/jruby/specialized/RubyArrayTwoObject.java +3
26. Part Two: ruby/spec
27. 273 of MRI's tests ask about garbage collection, object sizes and continuations. They were written to test MRI.
28. A JRuby fix helps JRuby. A ruby/spec commit helps every Ruby, including the ones that aren't written yet.
29. ruby/spec — Method#original_name through an alias chain class C def a; end alias_method :b, :a alias_method :c, :b alias_method :d, :c end describe "Method#original_name" do it "returns the original name through a chain of aliases" do C.new.method(:d).original_name.should == :a end end
30. Part Three: finding an actual bug
31. test/mri/excludes/TestDefined.rb exclude :test_defined_matchdata, "work in progress" exclude :test_defined_method_single_call, "work in progress" exclude :test_defined_paren_void_stmts, "needs investigation" exclude :test_defined_protected_method, "work in progress" exclude :test_defined_refined_call_with_using, "work in progress" exclude :test_defined_refined_vcall_with_using, "work in progress" exclude :test_defined_undefined_argument, "work in progress"
32. Five lines of Ruby class Foo def foo; end protected :foo def bar(f) defined?(f.foo) end end class Sub < Foo; end Foo.new.bar(Foo.new) # MRI: "method" JRuby: "method" ✓ Foo.new.bar(Sub.new) # MRI: "method" JRuby: nil ✗
33. JRuby Visibility visibility = method.getVisibility(); if (visibility != Visibility.PRIVATE && (visibility != Visibility.PROTECTED || - metaClass.getRealClass().isInstance(self)) + method.getImplementationClass().isInstance(self)) && !method.isUndefined()) { return definedMessage; }
34. jruby/jruby#9448 — one line of Java
35. Then I wondered: what if the protected method is in a module?
36. Same question, via a module module M def m; :m; end protected :m def call_m(o) = o.m # actually call it def defined_m(o) = defined?(o.m) # ask if we could end class A; include M; end class B; include M; end a, b = A.new, B.new a.call_m(b) # => :m the call WORKS a.defined_m(b) # => nil ...but defined? says no
37. I went looking for a JRuby bug. I found one in the reference implementation.
38. Bug #22076 — bugs.ruby-lang.org/issues/22076 defined? returns nil for protected methods defined in a module even when callable Nobuyoshi Nakada: The expectation and the fix seem reasonable both. Backport — 3.4: DONE · 4.0: DONE
39. bugs.ruby-lang.org/issues/22076
40. So why?
41. vm_insnhelper.c — vm_defined(), before case DEFINED_METHOD: { VALUE klass = CLASS_OF(v); const rb_method_entry_t *me = rb_method_entry_with_refinements(klass, SYM2ID(obj), NULL); if (me) { switch (METHOD_ENTRY_VISI(me)) { case METHOD_VISI_PROTECTED: if (!rb_obj_is_kind_of(GET_SELF(), rb_class_real(me->defined_class))) { break; } /* → returns nil */
42. vm_insnhelper.c — vm_defined(), after VALUE klass = CLASS_OF(v); + + const rb_method_entry_t *me = rb_method_entry_with_refinements(klass, SYM2ID(obj), NULL); const rb_callable_method_entry_t *cme = rb_callable_method_entry_with_refinements(klass, SYM2ID(obj), NULL); if (cme) { switch (METHOD_ENTRY_VISI(cme)) { case METHOD_VISI_PROTECTED: if (!rb_obj_is_kind_of(GET_SELF(), - rb_class_real(me->defined_class))) { + vm_defined_class_for_protected_call(cme))) {
43. ruby/ruby#17069 — merged by nobu on May 21
44. ruby/spec class ProtectedBase def m; end protected :m def defined_on(o) = defined?(o.m) end class ProtectedSubclass < ProtectedBase; end
45. ruby/spec module ProtectedInModule def m; end protected :m def defined_on(o) = defined?(o.m) end class ProtectedIncluderA include ProtectedInModule end class ProtectedIncluderB include ProtectedInModule end
46. ruby/spec describe "for a protected method" do it "returns 'method' when the receiver is a subclass instance" do DefinedSpecs::ProtectedBase.new .defined_on(DefinedSpecs::ProtectedSubclass.new).should == "method" end ruby_bug "#22076", ""..."4.1" do it "returns 'method' when the receiver is a sibling class instance " \ "via a shared included module" do DefinedSpecs::ProtectedIncluderA.new .defined_on(DefinedSpecs::ProtectedIncluderB.new).should == "method" end end end
47. ruby/spec#1362 — merged
48. One bug. Three repositories. Three days. MAY 20 MAY 21 MAY 22 jruby/jruby #9448 Bug #22076 → ruby/ruby #17069 ruby/spec #1362 Use implementation class for the protected check. Ticket first, then the fix for defined? . Specs for defined? on protected methods. 1 line of Java, −1 exclusion still open 5 lines of C + a test merged in 48 minutes merged 27 May vendored by all three Shipped in Ruby 3.4.10 and 4.0.6. 4.0.1 → nil · 4.0.6 → "method"
49. Living on the Edge
50. The gap between implementations is where the bugs live. The reproduction is usually already written. Making the tests pass brings the Rubies closer together. Fix it, then write a PR. You can help.
51. How to start tonight
52. Tonight, if you want # 1. Clone JRuby, and go read the queue. git clone https://github.com/jruby/jruby grep -rn "work in progress" jruby/test/mri/excludes | shuf | head -5 # 2. Pick one. Run just that file. # (JRuby: ./mvnw, then jruby -Itest test/mri/ruby/test_foo.rb) # 3. Strip the assertions off it. Run the bare Ruby on both. # 4. If MRI and JRuby disagree — you have a pull request. # If MRI looks wrong too — you have a bugs.ruby-lang.org ticket.
53. I didn't invent any of this. Stan Lo — from here, in Taiwan. CRuby committer, ruby/debug , IRB. Start small: improve tests, work the old TODO comments, clear the deprecation warnings. "The problem is letting bad English stop you from speaking." st0012.dev — Building a Career with Open Source, April 2026
54. @ioquatix, 26 April — omiyage stop #1
55. @ioquatix, 26 April — the drive home
56. It was already open. I just hadn't looked.
57. 謝謝 COSCUP Sampo Kuokkanen · @sampokuokkanen Bug #22076 · jruby/jruby · ruby/spec · ruby/ruby evilmartians.jp · evilmartians.com/chronicles
58. Bonus: MRI's own tests
59. test/ruby/test_hash.rb — test_each_value def test_each_value res = [] @cls[].each_value { |v| res << v } assert_equal(0, [].length) @h.each_value { |v| res << v } assert_equal(0, [].length) # ... end
60. ruby/ruby def test_each_value res = [] @cls[].each_value { |v| res << v } assert_equal(0, res.length) @h.each_value { |v| res << v } assert_equal(@h.size, res.length) # ... end
61. test/ruby/test_class.rb — test_metaclass assert_nothing_raised { metametaclass_of_bar.metaclass_method_c } assert_nothing_raised { metametaclass_of_bar.metametaclass_method_o } assert_nothing_raised { metametaclass_of_bar.metametaclass_method_f } # before — duplicates an assertion made 40 lines earlier: assert_raise(NoMethodError) { metametaclass_of_bar.metaclass_method_b } # after — the case this test exists to cover: assert_raise(NoMethodError) { metametaclass_of_bar.metametaclass_method_b }
62. A passing test is one small step for the programmer, but a giant one for Ruby.