A Type-level Ruby Interpreter
for Testing and Understanding
Yusuke Endoh
RubyKaigi 2019 (2019/04/18)
Slide 2
Slide 2 text
PR: Cookpad Booth (3F)
•Cookpad Daily Ruby Puzzles
•Get a problem sheet
• Complete "Hello world" by
adding minimum letters and
get a prize!
def foo
"Hello world" if
false
end
puts foo
def foo
"Hello world" if
!false
end
puts foo
!
Slide 3
Slide 3 text
This talk is about "Type Profiler"
Slide 4
Slide 4 text
The plan towards Ruby 3 static analysis
Sorbet
Steep
RDL
Library code
type signature
Type error
warnings
Application code
type signature
type signature
Type Profiler
Type error
warnings
This talk
Slide 5
Slide 5 text
This talk is about "Type Profiler"
•A type analyzer for Ruby 3
applicable to a non-annotated Ruby code
• Level-1 type checking
•Type signature prototyping
•Note: The analysis is not "sound"
• It may miss bugs and print wrong signature
Slide 6
Slide 6 text
Agenda
➔What "Type Profiler" is
•Demos
•Implementation
•Evaluation
•Conclusion
Slide 7
Slide 7 text
What Type Profiler is
•Target: A normal Ruby code
•No type signature/annotation required
•Objectives: Testing and understanding
• Testing: Warns possible errors of Ruby code
•Understanding: Prototypes type signature
Slide 8
Slide 8 text
Type Profiler for Testing
•Finds NoMethodError, TypeError, etc.
def foo(n)
if n < 10
n.timees {|x|
}
end
end
foo(42)
Type
Profiler
t.rb:3: [error] undefined method:
Integer#timees
Typo
Slide 9
Slide 9 text
Type Profiler for Understanding
•Generates a prototype of type definition
def foo(n)
n.to_s
end
foo(42)
Type
Profiler
Object#foo ::
(Integer) -> String
Slide 10
Slide 10 text
How Type Profiler does
•Runs a Ruby code in "type-level"
Normal interpreter
def foo(n)
n.to_s
end
foo(42)
Calls w/
42
Returns
"42"
Type profiler
def foo(n)
n.to_s
end
foo(42)
Calls w/
Integer
Returns
String
Object#foo ::
(Integer) -> String
Slide 11
Slide 11 text
How does TP handle a branch?
•"Forks" the execution
def foo(n)
if n < 10
n
else
"error"
end
end
foo(42)
Fork!
Now here;
We cannot tell
taken true or false
as we just know
n is Integer
Returns
Integer
Returns
String
Object#foo ::
(Integer) ->
(Integer | String)
Slide 12
Slide 12 text
Is a method executed at every call?
•No, the result is reused if possible
def foo(n)
n.to_s
end
x=foo(42)
y=foo(43)
z=foo(42.0)
Calls w/
Integer Returns
String
We already know
foo::(Integer)->String;
Immediately returns String
Calls w/
Float
Returns
String Object#foo :: (Integer)->String
Object#foo :: (Float)->String
Slide 13
Slide 13 text
Is Type Profiler a type checker?
Normal type checker
is intra-procedural
def foo(n:int):str
n.to_s
end
ret = foo(42)
Assume
Integer Check if
String
Check if Integer
Assume String
Type profiler
is inter-procedural
def foo(n)
n.to_s
end
foo(42)
Calls w/
Integer
Returns
String
Slide 14
Slide 14 text
What this technique is?
Easy to scale
but restrictive
Flexible
but hard to scale
Type
checking
Abstract interpretation
Symbolic execution
Type profiler...?
Flow analysis
Steep
Sorbet
RDL
Excuse: This figure is just my personal opinion
Demos
•You can see the demo programs
•https://github.com/mame/ruby-type-profiler
•But the spec is still under consideration
• The output format (and even behavior) may
change in future
Demo: Block
def foo(x)
yield 42
end
s = "str"
foo(1) do |x|
s
end
Type
Profiler
Object#foo ::
(Integer, &Proc[(Integer) -> String])
-> String
Slide 21
Slide 21 text
Demo: Tuple-like array
def swap(a)
[a[1], a[0]]
end
a = [42, "str"]
swap(a)
Type
Profiler
Object#swap ::
([Integer, String])
-> [String, Integer]
Slide 22
Slide 22 text
Demo: Sequence-like array
def foo
[1] + ["str"]
end
foo
Type
Profiler
Object#foo ::
Array[Integer | String]
Slide 23
Slide 23 text
Demo: Recursive method
def fib(n)
if n > 1
fib(n-1) + fib(n-2)
else
n
end
end
fib(10000)
Type
Profiler
Object#fib ::
(Integer) -> Integer
Slide 24
Slide 24 text
Demo: Unanalyzable method
a = eval("1 + 1")
a.foobar
Type
Profiler
t.rb:1: cannot handle eval;
the result is any
A call is not warned
when the receiver is any
Slide 25
Slide 25 text
Looks good?
•I think it would be good enough
•To be fair, Type Profiler is never perfect
•Can tell lies (false positive)
• Requires tests (false negative)
•Cannot handle some Ruby features
• May be very slow (state explosion)
Slide 26
Slide 26 text
Problem: False positives
if n < 10
x = 42
else
x = "str"
end
if n < 10
x += 1
end
String
+
Integer?
Error!
This path is not feasible
because of
the conditions
Possible workaround:
Please don't write
such a untypeable code!
Slide 27
Slide 27 text
Problem: A test is required
Possible workaround:
• Write a test!
• TP may be improved in some cases
def foo(x)
x.timees
end
Type
Profiler
(nothing)
Type Profiler cannot guess
the argument type......
Slide 28
Slide 28 text
Problem: Intractable features
•Object#send
•Singleton class
• TP abstracts all values as a type (class)
• But a singleton class is unique to the value
•Workaround: Difficult... (TP plugin?)
send(method_name)
Type
Profiler
Type Profiler cannot identify
the method being called!
Slide 29
Slide 29 text
Problem: State explosion
a=b=c=d=e=nil
a = 42 if n < 10
b = 42 if n < 10
c = 42 if n < 10
d = 42 if n < 10
e = 42 if n < 10
Fork!
Fork!
Fork!
Fork!
Fork!
2
4
8
16
32
The number
of states
Possible workaround:
• Write an annotation...? → a::NilClass|Integer
• Merge a pair of similar states (not implemented yet)
Slide 30
Slide 30 text
Problems
•Type Profiler is never perfect
•But "better than nothing"
• Only one choice for no-type Ruby lovers
•You can use a type checker if you want a type
• You may use Type Profiler to get a prototype
of type signatures
•I'm still thinking of the improvement
• Stay tuned...
Slide 31
Slide 31 text
Agenda
•What "Type Profiler" is
•Demos and Problems
➔Implementation
•Evaluation
•Conclusion
Slide 32
Slide 32 text
Implementation overview
•Core: A Type-level Ruby VM
•Runs a YARV bytecode
• A variable has a type instead of a value
•A branch copies the state for each target
•Profiling features
• Reports all type errors during execution
• Records all method arguments and return
values (for type signature prototype)
Slide 33
Slide 33 text
Some implementation details
•State enumeration
•Method return
•Three method types
Slide 34
Slide 34 text
State enumeration
•Enumerate all reachable states
•From the first line of the entry program
• Until fixed-point
•The same states are merged
• To avoid redundant execution
• TODO: merge "similar" states
if n < 10
a=1+1
else
a=2+2
end
...
The two states
are the same
Slide 35
Slide 35 text
Method return
•TP state has no call stack
•To avoid state explosion
• Cannot identify return address
•Returns to all calls
•This handles recursive calls
elegantly
def foo
...
return 42
end
foo()
foo()
foo()
foo()
foo()
Slide 36
Slide 36 text
Three method types
1. User-defined method
• When called,
TP enters its method body
2. Type-defined method
• TP just checks argument types
and returns its return type
• For built-in methods or libraries
3. Custom special method
• TP executes custom behavior
• (TP plugin can exploit this?)
def foo(n)
...
end
Integer#+ ::
(Integer)->Integer
Object#require???
Slide 37
Slide 37 text
Agenda
•What "Type Profiler" is
•Demos and Problems
•Implementation
➔(Very preliminary) Evaluation
•Conclusion
Slide 38
Slide 38 text
Excuse: TP is very preliminary...
•Currently supports
• Basic language features
• Limited built-in classes (shown in Demos)
•No-support-yet
• Almost all built-in classes including Hash
• Complex arguments (optional, rest, keywords)
• Exceptions
• Modules (Mix-in)
• etc etc.
Experiment 1: Self-profiling
•Apply Type Profiler to itself
•TP statistics: 2167 LOC, 221 methods
•Quantitative result:
•Reached 91 methods in 10 minutes (!)
•Qualitative result: Found an actual bug
•TP said "a method receives NilClass"
• It is not intended, and turned out a bug
Slide 41
Slide 41 text
Why many methods not reached?
•Because of not-implemented-yet features
•For example, Array#<< is not implemented
•Some methods are defined but unused
•Idea: virtually call all methods with "any"
arguments?
a = []
a << Foo.new
a[0].bar
Foo#bar
cannot be reached
Slide 42
Slide 42 text
Type Profiler code coverage
42
A method State#run
is not reached
state is any
for state.run
state = states.pop
caused nil
because Array#pop
was not implemented yet
Slide 43
Slide 43 text
Experiment 2: optcarrot
•Apply Type Profiler to optcarrot
• 8bit hardware emulator written in Ruby
• statistics: 4476 LOC, 394 methods
•Result:
• Reached 40 methods in 3 minutes?
• Object#send and Fiber are not implemented yet
• CPU uses Object#send for instruction dispatch
• GPU uses Fiber for state machine
Slide 44
Slide 44 text
Why did it took so long?
•State explosion
•A method returns
Array or Integer
•Calling it forks
the state
•Idea: Merge similar states more intelligently
foo()
foo()
foo()
foo()
foo()
Fork!
Fork!
Fork!
Fork!
Fork!
foo :: () ->
(Array[Integer] | Integer)
Slide 45
Slide 45 text
Agenda
•What "Type Profiler" is
•Demos and Problems
•Implementation
•Very preliminary evaluation
➔(Related work and) Conclusion
Slide 46
Slide 46 text
Related Work
•mruby-meta-circular (Hideki Miura)
• A very similar approach for mruby JIT
• This inspired Type Profiler (Thanks!)
•HPC Ruby (Koichi Nakamura)
• Convert Ruby to C for HPC by abstract interp.
•pytype (Google's unofficial project)
• Python abstract interpreter for type analysis
• More concrete than TP
• Limits the stack depth to three
Slide 47
Slide 47 text
Acknowledgement
•Hideki Miura
•Matz, Akr, Ko1
•PPL paper co-authors
• Soutaro Matsumoto
• Katsuhiro Ueno
• Eijiro Sumii
•Stripe team & Jeff Foster
•And many people
Slide 48
Slide 48 text
Conclusion
•A yet another type analyzer for Ruby 3
applicable to a non-annotated Ruby code
• Based on abstract interpretation technique
• Little change for Ruby programming experience
•Contribution is really welcome!
•The development is very preliminary
• https://github.com/mame/ruby-type-profiler
Slide 49
Slide 49 text
(A lot of) Future work
• Support language features
• Almost all built-in classes
including Hash
• Complex arguments
(optional, rest, keywords)
• Exceptions
• Modules (Mix-in)
• etc etc.
• Write a type definition for
built-in classes/methods
• Read type definition file
• Improve performance
• Make it useful
• Code coverage
• Flow-sensitive analysis
• Heuristic type aggregation
• Diagnosis feature
• Incremental type profiling
• etc etc.