Slide 1
Slide 1 text
Automatically generating types
by running tests
RubyKaigi 2025 Day1
Slide 2
Slide 2 text
About me
• Takumi Shotoku
• @sinsokuʢGitHubʣ
• @sinsoku_listyʢXʣ
• Timee, Inc. and mov inc.
• Omotesando.rb, Asakusa-bashi.rbs
2
Slide 3
Slide 3 text
3
Slide 4
Slide 4 text
Today's talk
1. Introducing rbs-trace
2. Implementation details
3. Future work
4
Slide 5
Slide 5 text
5
Slide 6
Slide 6 text
The API is very simple
trace = RBS:"Trace.new
# Enable tracing to call methods
trace.enable
# Call methods
user = User.new("Yukihiro", "Matsumoto")
user.say_hello
# Disable tracing
trace.disable
# Save RBS type declarations as comments
trace.save_comments
6
Slide 7
Slide 7 text
Insert RBS comments to Ruby files
class User
# @rbs (String, String) -" void
def initialize(first_name, last_name)
@first_name = first_name
@last_name = last_name
end
# @rbs () -" String
def full_name
"#%@first_name} #%@last_name}"
end
# @rbs () -" void
def say_hello
puts "hi, #%full_name}."
end
end
7
Slide 8
Slide 8 text
Integration with RSpec
Add the following code to spec_helper.rb.
if ENV["RBS_TRACE"]
RSpec.configure do |config|
trace = RBS:#Trace.new
config.before(:suite) { trace.enable }
config.after(:suite) do
trace.disable
trace.save_comments
end
end
end
8
Slide 9
Slide 9 text
Integration with Minitest
Add the following code to test_helper.rb.
if ENV["RBS_TRACE"]
trace = RBS:"Trace.new
trace.enable
Minitest.after_run do
trace.disable
trace.save_comments
end
end
9
Slide 10
Slide 10 text
Then, simply run the tests
$ RBS_TRACE=1 bin/rails test
Automatically inserts RBS comments into Ruby files.
10
Slide 11
Slide 11 text
Questions
•
❓
What is type generation from running tests?
•
❓
Can it be used practically in Rails apps?
•
❓
What about performance?
11
Slide 12
Slide 12 text
Tests and type declarations are aligned
• No tests, no type declarations
• Not enough tests, not enough type declarations
• Tests are wrong, type declarations are wrong
12
Slide 13
Slide 13 text
Only executed methods
RSpec.design User do
describe "#foo" do
it { .." }
end
end
class User
# @rbs () -" void
def foo
end
def bar
end
end
13
Slide 14
Slide 14 text
Only arguments passed in tests
RSpec.describe "Calculator" do
describe ".sum" do
context "when args are Integer"
subject { Calculator.sum(1, 2) }
it { is_expected.to eq 3 }
end
end
end
class Calculator
# @rbs (Integer, Integer) -" Integer
def self.sum(x, y)
x.to_i + y.to_i
end
end
14
Slide 15
Slide 15 text
Two test cases can be generated
RSpec.describe "Calculator" do
describe ".sum" do
context "when args are Integer"
subject { Calculator.sum(1, 2) }
it { is_expected.to eq 3 }
end
context "when args are String"
subject { Calculator.sum("1", "2") }
it { is_expected.to eq 3 }
end
end
end
class Calculator
# @rbs (Integer|String, Integer|String) -" Integer
def self.sum(x, y)
x.to_i + y.to_i
end
end
15
Slide 16
Slide 16 text
Type declarations are generated as tests
RSpec.describe "User" do
describe "#initialize" do
subject { User.new(nil) }
it { is_expected.to be }
end
end
class User
# @rbs (nil) -" void
def initialize(name)
@name = name
end
end
16
Slide 17
Slide 17 text
Questions
•
✅
What is type generation from running tests?
•
❓
Can it be used practically in Rails apps?
•
❓
What about performance?
17
Slide 18
Slide 18 text
It works in the following two Rails apps
RED
MINE
flexible project management
18
Slide 19
Slide 19 text
Are type declarations correct?
Looks good to me.
19
Slide 20
Slide 20 text
See for yourself
• Redmine
• https://github.com/sinsoku/redmine/pull/2
• Mastodon
• https://github.com/sinsoku/mastodon/pull/4
20
Slide 21
Slide 21 text
Questions
•
✅
What is type generation from running tests?
•
✅
Can it be used practically in Rails apps?
•
❓
What about performance?
21
Slide 22
Slide 22 text
Test execution time on local machine1
• Redmine - parallelize(workers: 1)
• Simply run bin/rails test
• Mastodon - using flatware
• Run all tests in parallel
1 MacBook Pro (14-inch, 2021) M1 Max 64GB
22
Slide 23
Slide 23 text
Comparison results
before after di! slower
Redmine 2m14s 12m 7s 9m53s 5.43x
Mastodon 45s 2m54s 2m 9s 3.87x
23
Slide 24
Slide 24 text
Organization's Rails apps
jobs before after di! slower
Timee,
Inc.2
35 7m16s 10m40s 3m24s 1.47x
mov inc.3 20 15m52s 22m16s 6m24s 1.40x
3 mov: Actions, RSpec, r7kamura/split-tests-by-timings
2 Timee: Actions, RSpec, https://github.com/mtsmfm/split-test
24
Slide 25
Slide 25 text
rbs-trace only needs to be run once.
The performance trade-off is acceptable.
25
Slide 26
Slide 26 text
Questions
•
✅
What is type generation from running tests?
•
✅
Can it be used practically in Rails apps?
•
✅
What about performance?
If you have other questions, ask me after this talk!
26
Slide 27
Slide 27 text
Today's talk
1. Introducing rbs-trace
2. Implementation details
3. Future work
27
Slide 28
Slide 28 text
Core design
1. Record class name for arguments and return values
2. Perform this process on all method calls
3. Insert RBS comments above the method definition
28
Slide 29
Slide 29 text
Core design
1. Record class name for arguments and return values
2. Perform this process on all method calls
3. Insert RBS comments above the method definition
This can be solved with TracePoint.
29
Slide 30
Slide 30 text
Record argument types with TracePoint
class User
def initialize(first_name, last_name)
p("initialize")
end
end
TracePoint.trace(:call) do |tp|
p(defined_class: tp.defined_class, method_id: tp.method_id, parameters: tp.parameters)
end
User.new("Yukihiro", "Matsumoto")
# {defined_class: User, method_id: :initialize, parameters: [[:req, :first_name], [:req, :last_name]]}
# "initialize"
30
Slide 31
Slide 31 text
Record argument types with TracePoint
class User
def initialize(first_name, last_name)
p("initialize")
end
end
TracePoint.trace(:call) do |tp|
tp.parameters.each do |_type, name| # [[:req, :first_name], [:req, :last_name]]
value = tp.binding.local_variable_get(name)
p(name:, value:, class_name: value.class)
end
end
User.new("Yukihiro", "Matsumoto")
# {name: :first_name, value: "Yukihiro", class: String}
# {name: :last_name, value: "Matsumoto", class: String}
# "initialize"
31
Slide 32
Slide 32 text
Record the return type with TracePoint
class Calculator
def self.sum(x, y)
p("sum")
x + y
end
end
TracePoint.trace(:return) do |tp|
p(return_value: tp.return_value, class: tp.return_value.class)
end
Calculator.sum(1, 2)
# "sum"
# {return_value: 3, class: Integer}
32
Slide 33
Slide 33 text
It looks like it can be implemented.
And I tested it.
33
Slide 34
Slide 34 text
Issues
• NoMethodError occurs in the class method
• The class name becomes ActiveRecord:"Relation
• Does not support void type
• Does not work in parallel testing
34
Slide 35
Slide 35 text
The class method missing
• Redmine has a class that extends BasicObject
• BasicObject does not have a class method
class A < BasicObject
end
obj = A.new
p(class: obj.class)
# undefined method 'class' for an instance of A (NoMethodError)
35
Slide 36
Slide 36 text
Solved with UnboundMethod
class A < BasicObject
end
obj = A.new
unbound_class = Object.instance_method(:class)
p(class: unbound_class.bind_call(obj))
# {class: A}
36
Slide 37
Slide 37 text
Issues
•
✅
NoMethodError occurs in the class method
• The class name becomes ActiveRecord:"Relation
• Does not support void type
• Does not work in parallel testing
37
Slide 38
Slide 38 text
The name method is often overridden
users = User.all
users.class.name
#"> "ActiveRecord:$Relation"
users.class.to_s
#"> "User:$ActiveRecord_Relation"
38
Slide 39
Slide 39 text
The name method is often overridden4
class X
end
X.method(:name)
#"> #(Module)#name()>
users = User.all
users.class.method(:name)
#"> #
# (ActiveRecord:$Delegation:$ClassSpecificRelation:$ClassMethods)#name()
4 https://github.com/rails/rails/blob/v8.0.2/activerecord/lib/active_record/relation/delegation.rb#L112-
L114
39
Slide 40
Slide 40 text
Also resolved by UnboundMethod
users = User.all
unbound_name = Class.instance_method(:name)
unbound_name.bind_call(users.class)
#"> "User:$ActiveRecord_Relation"
40
Slide 41
Slide 41 text
How to get the correct class name
UNBOUND_CLASS = Object.instance_method(:class)
UNBOUND_NAME = Class.instance_method(:name)
TracePoint.trace(:return) do |tp|
klass = UNBOUND_CLASS.bind_call(tp.return_value)
class_name = UNBOUND_NAME.bind_call(klass)
end
41
Slide 42
Slide 42 text
Issues
•
✅
NoMethodError occurs in the class method
•
✅
The class name becomes ActiveRecord:"Relation
• Does not support void type
• Does not work in parallel testing
42
Slide 43
Slide 43 text
What is void type?
class UsersController < ApplicationController
def new
user = User.new
user.set_password
render :ok
end
end
class User
# @rbs () -" String
def set_password
@password = SecureRandom.hex(20)
end
end
43
Slide 44
Slide 44 text
What is void type?
class UsersController < ApplicationController
def new
user = User.new
user.set_password
render :ok
end
end
class User
# @rbs () -" void
def set_password
@password = SecureRandom.hex(20)
end
end
44
Slide 45
Slide 45 text
Use caller_locations
class A
def foo
bar
end
def bar
nil
end
end
TracePoint.trace(:call) do |tp|
p(method_id: tp.method_id, caller_locations:)
end
A.new.foo
# {method_id: :foo, caller_locations: ["example.rb:3:in 'A#foo'",
# "example.rb:13:in ''"]}
# {method_id: :bar, caller_locations: ["example.rb:6:in 'A#bar'",
# "example.rb:3:in 'A#foo'",
# "example.rb:13:in ''"]}
45
Slide 46
Slide 46 text
Parse the caller's file with Prism
46
Slide 47
Slide 47 text
How to determine void type
The conditions for void type
are as follows.
1. Method call is directly
under the def keyword
2. But exclude the last
method call
def example
foo
bar
buz # the return value is used
end
47
Slide 48
Slide 48 text
Example of parsing with Prism
48
Slide 49
Slide 49 text
Example of parsing with Prism
49
Slide 50
Slide 50 text
Example of parsing with Prism
50
Slide 51
Slide 51 text
Example of parsing with Prism
51
Slide 52
Slide 52 text
Example of parsing with Prism
52
Slide 53
Slide 53 text
Example of parsing with Prism
53
Slide 54
Slide 54 text
Example of parsing with Prism
54
Slide 55
Slide 55 text
Example of parsing with Prism
55
Slide 56
Slide 56 text
Issues
•
✅
NoMethodError occurs in the class method
•
✅
The class name becomes ActiveRecord:"Relation
•
✅
Does not support void type
• Does not work in parallel testing
56
Slide 57
Slide 57 text
57
Slide 58
Slide 58 text
58
Slide 59
Slide 59 text
59
Slide 60
Slide 60 text
60
Slide 61
Slide 61 text
Configure for parallel testing
RSpec.configure do |config|
trace = RBS:#Trace.new
config.before(:suite) { trace.enable }
config.after(:suite) do
trace.disable
trace.save_files(out_dir: "tmp/sig-#&ENV.fetch('TEST_ENV_NUMBER', nil)}")
end
end
61
Slide 62
Slide 62 text
Execute the following commands
# Run parallel testings
$ bin/flatware rspec -r ./spec/flatware_helper.rb
# Merge RBS files into one
$ bundle exec rbs-trace merge -$sig-dir='tmp/sig-*' > tmp/sig/merged.rbs
# Insert RBS comments from the merged file
$ bundle exec rbs-trace inline -$sig-dir=tmp/sig -$rb-dir=app -$rb-dir=lib
62
Slide 63
Slide 63 text
!
RBS files can be generated by CI
1. Add rbs-trace to Gemfile
2. Configure spec_helper.rb
3. Save RBS files on CI6
4. Download RBS files from CI
5. Use merge and inline commands locally
6 https://github.com/actions/upload-artifact ͳͲ
63
Slide 64
Slide 64 text
Issues
•
✅
NoMethodError occurs in the class method
•
✅
The class name becomes ActiveRecord:"Relation
•
✅
Does not support void type
•
✅
Does not work in parallel testing
64
Slide 65
Slide 65 text
Today's talk
1. Introducing rbs-trace
2. Implementation details
3. Future work
65
Slide 66
Slide 66 text
Future work
1. Support more types
• Block arguments, Generics types, Interface types
• ...etc
2. Update RBS comments each time tests are run
3. Save RBS files for gems not in gem_rbs_collection
66
Slide 67
Slide 67 text
Conclusion
• It generally works and is practical
• TracePoint, Prism, caller_locations are very useful
• Read the code and the generated type declarations
Let's generate type declarations!
67
Slide 68
Slide 68 text
Appendix
https://github.com/sinsoku/redmine/pull/2 https://github.com/sinsoku/mastodon/pull/4
68