Slide 1

Slide 1 text

Automatically generating types by running tests RubyKaigi 2025 Day1

Slide 2

Slide 2 text

About me • Takumi Shotoku • @sinsokuʢGitHubʣ • @sinsoku_listyʢXʣ • Timee, Inc. and mov inc. • Omotesando.rb, Asakusa-bashi.rbs 2

Slide 3

Slide 3 text

3

Slide 4

Slide 4 text

Today's talk 1. Introducing rbs-trace 2. Implementation details 3. Future work 4

Slide 5

Slide 5 text

5

Slide 6

Slide 6 text

The API is very simple trace = RBS:"Trace.new # Enable tracing to call methods trace.enable # Call methods user = User.new("Yukihiro", "Matsumoto") user.say_hello # Disable tracing trace.disable # Save RBS type declarations as comments trace.save_comments 6

Slide 7

Slide 7 text

Insert RBS comments to Ruby files class User # @rbs (String, String) -" void def initialize(first_name, last_name) @first_name = first_name @last_name = last_name end # @rbs () -" String def full_name "#%@first_name} #%@last_name}" end # @rbs () -" void def say_hello puts "hi, #%full_name}." end end 7

Slide 8

Slide 8 text

Integration with RSpec Add the following code to spec_helper.rb. if ENV["RBS_TRACE"] RSpec.configure do |config| trace = RBS:#Trace.new config.before(:suite) { trace.enable } config.after(:suite) do trace.disable trace.save_comments end end end 8

Slide 9

Slide 9 text

Integration with Minitest Add the following code to test_helper.rb. if ENV["RBS_TRACE"] trace = RBS:"Trace.new trace.enable Minitest.after_run do trace.disable trace.save_comments end end 9

Slide 10

Slide 10 text

Then, simply run the tests $ RBS_TRACE=1 bin/rails test Automatically inserts RBS comments into Ruby files. 10

Slide 11

Slide 11 text

Questions • ❓ What is type generation from running tests? • ❓ Can it be used practically in Rails apps? • ❓ What about performance? 11

Slide 12

Slide 12 text

Tests and type declarations are aligned • No tests, no type declarations • Not enough tests, not enough type declarations • Tests are wrong, type declarations are wrong 12

Slide 13

Slide 13 text

Only executed methods RSpec.design User do describe "#foo" do it { .." } end end class User # @rbs () -" void def foo end def bar end end 13

Slide 14

Slide 14 text

Only arguments passed in tests RSpec.describe "Calculator" do describe ".sum" do context "when args are Integer" subject { Calculator.sum(1, 2) } it { is_expected.to eq 3 } end end end class Calculator # @rbs (Integer, Integer) -" Integer def self.sum(x, y) x.to_i + y.to_i end end 14

Slide 15

Slide 15 text

Two test cases can be generated RSpec.describe "Calculator" do describe ".sum" do context "when args are Integer" subject { Calculator.sum(1, 2) } it { is_expected.to eq 3 } end context "when args are String" subject { Calculator.sum("1", "2") } it { is_expected.to eq 3 } end end end class Calculator # @rbs (Integer|String, Integer|String) -" Integer def self.sum(x, y) x.to_i + y.to_i end end 15

Slide 16

Slide 16 text

Type declarations are generated as tests RSpec.describe "User" do describe "#initialize" do subject { User.new(nil) } it { is_expected.to be } end end class User # @rbs (nil) -" void def initialize(name) @name = name end end 16

Slide 17

Slide 17 text

Questions • ✅ What is type generation from running tests? • ❓ Can it be used practically in Rails apps? • ❓ What about performance? 17

Slide 18

Slide 18 text

It works in the following two Rails apps RED MINE flexible project management 18

Slide 19

Slide 19 text

Are type declarations correct? Looks good to me. 19

Slide 20

Slide 20 text

See for yourself • Redmine • https://github.com/sinsoku/redmine/pull/2 • Mastodon • https://github.com/sinsoku/mastodon/pull/4 20

Slide 21

Slide 21 text

Questions • ✅ What is type generation from running tests? • ✅ Can it be used practically in Rails apps? • ❓ What about performance? 21

Slide 22

Slide 22 text

Test execution time on local machine1 • Redmine - parallelize(workers: 1) • Simply run bin/rails test • Mastodon - using flatware • Run all tests in parallel 1 MacBook Pro (14-inch, 2021) M1 Max 64GB 22

Slide 23

Slide 23 text

Comparison results before after di! slower Redmine 2m14s 12m 7s 9m53s 5.43x Mastodon 45s 2m54s 2m 9s 3.87x 23

Slide 24

Slide 24 text

Organization's Rails apps jobs before after di! slower Timee, Inc.2 35 7m16s 10m40s 3m24s 1.47x mov inc.3 20 15m52s 22m16s 6m24s 1.40x 3 mov: Actions, RSpec, r7kamura/split-tests-by-timings 2 Timee: Actions, RSpec, https://github.com/mtsmfm/split-test 24

Slide 25

Slide 25 text

rbs-trace only needs to be run once. The performance trade-off is acceptable. 25

Slide 26

Slide 26 text

Questions • ✅ What is type generation from running tests? • ✅ Can it be used practically in Rails apps? • ✅ What about performance? If you have other questions, ask me after this talk! 26

Slide 27

Slide 27 text

Today's talk 1. Introducing rbs-trace 2. Implementation details 3. Future work 27

Slide 28

Slide 28 text

Core design 1. Record class name for arguments and return values 2. Perform this process on all method calls 3. Insert RBS comments above the method definition 28

Slide 29

Slide 29 text

Core design 1. Record class name for arguments and return values 2. Perform this process on all method calls 3. Insert RBS comments above the method definition This can be solved with TracePoint. 29

Slide 30

Slide 30 text

Record argument types with TracePoint class User def initialize(first_name, last_name) p("initialize") end end TracePoint.trace(:call) do |tp| p(defined_class: tp.defined_class, method_id: tp.method_id, parameters: tp.parameters) end User.new("Yukihiro", "Matsumoto") # {defined_class: User, method_id: :initialize, parameters: [[:req, :first_name], [:req, :last_name]]} # "initialize" 30

Slide 31

Slide 31 text

Record argument types with TracePoint class User def initialize(first_name, last_name) p("initialize") end end TracePoint.trace(:call) do |tp| tp.parameters.each do |_type, name| # [[:req, :first_name], [:req, :last_name]] value = tp.binding.local_variable_get(name) p(name:, value:, class_name: value.class) end end User.new("Yukihiro", "Matsumoto") # {name: :first_name, value: "Yukihiro", class: String} # {name: :last_name, value: "Matsumoto", class: String} # "initialize" 31

Slide 32

Slide 32 text

Record the return type with TracePoint class Calculator def self.sum(x, y) p("sum") x + y end end TracePoint.trace(:return) do |tp| p(return_value: tp.return_value, class: tp.return_value.class) end Calculator.sum(1, 2) # "sum" # {return_value: 3, class: Integer} 32

Slide 33

Slide 33 text

It looks like it can be implemented. And I tested it. 33

Slide 34

Slide 34 text

Issues • NoMethodError occurs in the class method • The class name becomes ActiveRecord:"Relation • Does not support void type • Does not work in parallel testing 34

Slide 35

Slide 35 text

The class method missing • Redmine has a class that extends BasicObject • BasicObject does not have a class method class A < BasicObject end obj = A.new p(class: obj.class) # undefined method 'class' for an instance of A (NoMethodError) 35

Slide 36

Slide 36 text

Solved with UnboundMethod class A < BasicObject end obj = A.new unbound_class = Object.instance_method(:class) p(class: unbound_class.bind_call(obj)) # {class: A} 36

Slide 37

Slide 37 text

Issues • ✅ NoMethodError occurs in the class method • The class name becomes ActiveRecord:"Relation • Does not support void type • Does not work in parallel testing 37

Slide 38

Slide 38 text

The name method is often overridden users = User.all users.class.name #"> "ActiveRecord:$Relation" users.class.to_s #"> "User:$ActiveRecord_Relation" 38

Slide 39

Slide 39 text

The name method is often overridden4 class X end X.method(:name) #"> #(Module)#name()> users = User.all users.class.method(:name) #"> # # (ActiveRecord:$Delegation:$ClassSpecificRelation:$ClassMethods)#name() 4 https://github.com/rails/rails/blob/v8.0.2/activerecord/lib/active_record/relation/delegation.rb#L112- L114 39

Slide 40

Slide 40 text

Also resolved by UnboundMethod users = User.all unbound_name = Class.instance_method(:name) unbound_name.bind_call(users.class) #"> "User:$ActiveRecord_Relation" 40

Slide 41

Slide 41 text

How to get the correct class name UNBOUND_CLASS = Object.instance_method(:class) UNBOUND_NAME = Class.instance_method(:name) TracePoint.trace(:return) do |tp| klass = UNBOUND_CLASS.bind_call(tp.return_value) class_name = UNBOUND_NAME.bind_call(klass) end 41

Slide 42

Slide 42 text

Issues • ✅ NoMethodError occurs in the class method • ✅ The class name becomes ActiveRecord:"Relation • Does not support void type • Does not work in parallel testing 42

Slide 43

Slide 43 text

What is void type? class UsersController < ApplicationController def new user = User.new user.set_password render :ok end end class User # @rbs () -" String def set_password @password = SecureRandom.hex(20) end end 43

Slide 44

Slide 44 text

What is void type? class UsersController < ApplicationController def new user = User.new user.set_password render :ok end end class User # @rbs () -" void def set_password @password = SecureRandom.hex(20) end end 44

Slide 45

Slide 45 text

Use caller_locations class A def foo bar end def bar nil end end TracePoint.trace(:call) do |tp| p(method_id: tp.method_id, caller_locations:) end A.new.foo # {method_id: :foo, caller_locations: ["example.rb:3:in 'A#foo'", # "example.rb:13:in ''"]} # {method_id: :bar, caller_locations: ["example.rb:6:in 'A#bar'", # "example.rb:3:in 'A#foo'", # "example.rb:13:in ''"]} 45

Slide 46

Slide 46 text

Parse the caller's file with Prism 46

Slide 47

Slide 47 text

How to determine void type The conditions for void type are as follows. 1. Method call is directly under the def keyword 2. But exclude the last method call def example foo bar buz # the return value is used end 47

Slide 48

Slide 48 text

Example of parsing with Prism 48

Slide 49

Slide 49 text

Example of parsing with Prism 49

Slide 50

Slide 50 text

Example of parsing with Prism 50

Slide 51

Slide 51 text

Example of parsing with Prism 51

Slide 52

Slide 52 text

Example of parsing with Prism 52

Slide 53

Slide 53 text

Example of parsing with Prism 53

Slide 54

Slide 54 text

Example of parsing with Prism 54

Slide 55

Slide 55 text

Example of parsing with Prism 55

Slide 56

Slide 56 text

Issues • ✅ NoMethodError occurs in the class method • ✅ The class name becomes ActiveRecord:"Relation • ✅ Does not support void type • Does not work in parallel testing 56

Slide 57

Slide 57 text

57

Slide 58

Slide 58 text

58

Slide 59

Slide 59 text

59

Slide 60

Slide 60 text

60

Slide 61

Slide 61 text

Configure for parallel testing RSpec.configure do |config| trace = RBS:#Trace.new config.before(:suite) { trace.enable } config.after(:suite) do trace.disable trace.save_files(out_dir: "tmp/sig-#&ENV.fetch('TEST_ENV_NUMBER', nil)}") end end 61

Slide 62

Slide 62 text

Execute the following commands # Run parallel testings $ bin/flatware rspec -r ./spec/flatware_helper.rb # Merge RBS files into one $ bundle exec rbs-trace merge -$sig-dir='tmp/sig-*' > tmp/sig/merged.rbs # Insert RBS comments from the merged file $ bundle exec rbs-trace inline -$sig-dir=tmp/sig -$rb-dir=app -$rb-dir=lib 62

Slide 63

Slide 63 text

! RBS files can be generated by CI 1. Add rbs-trace to Gemfile 2. Configure spec_helper.rb 3. Save RBS files on CI6 4. Download RBS files from CI 5. Use merge and inline commands locally 6 https://github.com/actions/upload-artifact ͳͲ 63

Slide 64

Slide 64 text

Issues • ✅ NoMethodError occurs in the class method • ✅ The class name becomes ActiveRecord:"Relation • ✅ Does not support void type • ✅ Does not work in parallel testing 64

Slide 65

Slide 65 text

Today's talk 1. Introducing rbs-trace 2. Implementation details 3. Future work 65

Slide 66

Slide 66 text

Future work 1. Support more types • Block arguments, Generics types, Interface types • ...etc 2. Update RBS comments each time tests are run 3. Save RBS files for gems not in gem_rbs_collection 66

Slide 67

Slide 67 text

Conclusion • It generally works and is practical • TracePoint, Prism, caller_locations are very useful • Read the code and the generated type declarations Let's generate type declarations! 67

Slide 68

Slide 68 text

Appendix https://github.com/sinsoku/redmine/pull/2 https://github.com/sinsoku/mastodon/pull/4 68