A Type-level Ruby Interpreter for Testing and Understanding

Rubykaigi 2019

https://rubykaigi.org/2019/presentations/mametter.html

Transcript

1. A Type-level Ruby Interpreter for Testing and Understanding Yusuke Endoh RubyKaigi 2019 (2019/04/18)
2. PR: Cookpad Booth (3F) •Cookpad Daily Ruby Puzzles •Get a problem sheet • Complete "Hello world" by adding minimum letters and get a prize! def foo "Hello world" if false end puts foo def foo "Hello world" if !false end puts foo !
3. This talk is about "Type Profiler"
4. The plan towards Ruby 3 static analysis Sorbet Steep RDL Library code type signature Type error warnings Application code type signature type signature Type Profiler Type error warnings This talk
5. This talk is about "Type Profiler" •A type analyzer for Ruby 3 applicable to a non-annotated Ruby code • Level-1 type checking •Type signature prototyping •Note: The analysis is not "sound" • It may miss bugs and print wrong signature
6. Agenda ➔What "Type Profiler" is •Demos •Implementation •Evaluation •Conclusion
7. What Type Profiler is •Target: A normal Ruby code •No type signature/annotation required •Objectives: Testing and understanding • Testing: Warns possible errors of Ruby code •Understanding: Prototypes type signature
8. Type Profiler for Testing •Finds NoMethodError, TypeError, etc. def foo(n) if n < 10 n.timees {|x| } end end foo(42) Type Profiler t.rb:3: [error] undefined method: Integer#timees Typo
9. Type Profiler for Understanding •Generates a prototype of type definition def foo(n) n.to_s end foo(42) Type Profiler Object#foo :: (Integer) -> String
10. How Type Profiler does •Runs a Ruby code in "type-level" Normal interpreter def foo(n) n.to_s end foo(42) Calls w/ 42 Returns "42" Type profiler def foo(n) n.to_s end foo(42) Calls w/ Integer Returns String Object#foo :: (Integer) -> String
11. How does TP handle a branch? •"Forks" the execution def foo(n) if n < 10 n else "error" end end foo(42) Fork! Now here; We cannot tell taken true or false as we just know n is Integer Returns Integer Returns String Object#foo :: (Integer) -> (Integer | String)
12. Is a method executed at every call? •No, the result is reused if possible def foo(n) n.to_s end x=foo(42) y=foo(43) z=foo(42.0) Calls w/ Integer Returns String We already know foo::(Integer)->String; Immediately returns String Calls w/ Float Returns String Object#foo :: (Integer)->String Object#foo :: (Float)->String
13. Is Type Profiler a type checker? Normal type checker is intra-procedural def foo(n:int):str n.to_s end ret = foo(42) Assume Integer Check if String Check if Integer Assume String Type profiler is inter-procedural def foo(n) n.to_s end foo(42) Calls w/ Integer Returns String
14. What this technique is? Easy to scale but restrictive Flexible but hard to scale Type checking Abstract interpretation Symbolic execution Type profiler...? Flow analysis Steep Sorbet RDL Excuse: This figure is just my personal opinion
15. Agenda •What "Type Profiler" is ➔Demos (and Problems) •Implementation •Evaluation •Conclusion
16. Demos •You can see the demo programs •https://github.com/mame/ruby-type-profiler •But the spec is still under consideration • The output format (and even behavior) may change in future
17. Demo: Overloading def my_to_s(x) x.to_s end my_to_s(42) my_to_s("STR") my_to_s(:sym) Type Profiler Object#my_to_s :: (Integer) -> String Object#my_to_s :: (String) -> String Object#my_to_s :: (Symbol) -> String
18. Demo: User-defined classes class Foo end class Bar def make_foo Foo.new end end Bar.new.make_foo Type Profiler Bar#make_foo :: () -> Foo
19. Demo: Instance variables class Foo attr_accessor :ivar end Foo.new.ivar = 42 Foo.new.ivar = "STR" Foo.new.ivar Type Profiler Foo#@ivar :: Integer | String Foo#ivar= :: (Integer) -> Integer Foo#ivar= :: (String) -> String Foo#ivar :: () -> (String | Integer)
20. Demo: Block def foo(x) yield 42 end s = "str" foo(1) do |x| s end Type Profiler Object#foo :: (Integer, &Proc[(Integer) -> String]) -> String
21. Demo: Tuple-like array def swap(a) [a[1], a[0]] end a = [42, "str"] swap(a) Type Profiler Object#swap :: ([Integer, String]) -> [String, Integer]
22. Demo: Sequence-like array def foo [1] + ["str"] end foo Type Profiler Object#foo :: Array[Integer | String]
23. Demo: Recursive method def fib(n) if n > 1 fib(n-1) + fib(n-2) else n end end fib(10000) Type Profiler Object#fib :: (Integer) -> Integer
24. Demo: Unanalyzable method a = eval("1 + 1") a.foobar Type Profiler t.rb:1: cannot handle eval; the result is any A call is not warned when the receiver is any
25. Looks good? •I think it would be good enough •To be fair, Type Profiler is never perfect •Can tell lies (false positive) • Requires tests (false negative) •Cannot handle some Ruby features • May be very slow (state explosion)
26. Problem: False positives if n < 10 x = 42 else x = "str" end if n < 10 x += 1 end String + Integer? Error! This path is not feasible because of the conditions Possible workaround: Please don't write such a untypeable code!
27. Problem: A test is required Possible workaround: • Write a test! • TP may be improved in some cases def foo(x) x.timees end Type Profiler (nothing) Type Profiler cannot guess the argument type......
28. Problem: Intractable features •Object#send •Singleton class • TP abstracts all values as a type (class) • But a singleton class is unique to the value •Workaround: Difficult... (TP plugin?) send(method_name) Type Profiler Type Profiler cannot identify the method being called!
29. Problem: State explosion a=b=c=d=e=nil a = 42 if n < 10 b = 42 if n < 10 c = 42 if n < 10 d = 42 if n < 10 e = 42 if n < 10 Fork! Fork! Fork! Fork! Fork! 2 4 8 16 32 The number of states Possible workaround: • Write an annotation...? → a::NilClass|Integer • Merge a pair of similar states (not implemented yet)
30. Problems •Type Profiler is never perfect •But "better than nothing" • Only one choice for no-type Ruby lovers •You can use a type checker if you want a type • You may use Type Profiler to get a prototype of type signatures •I'm still thinking of the improvement • Stay tuned...
31. Agenda •What "Type Profiler" is •Demos and Problems ➔Implementation •Evaluation •Conclusion
32. Implementation overview •Core: A Type-level Ruby VM •Runs a YARV bytecode • A variable has a type instead of a value •A branch copies the state for each target •Profiling features • Reports all type errors during execution • Records all method arguments and return values (for type signature prototype)
33. Some implementation details •State enumeration •Method return •Three method types
34. State enumeration •Enumerate all reachable states •From the first line of the entry program • Until fixed-point •The same states are merged • To avoid redundant execution • TODO: merge "similar" states if n < 10 a=1+1 else a=2+2 end ... The two states are the same
35. Method return •TP state has no call stack •To avoid state explosion • Cannot identify return address •Returns to all calls •This handles recursive calls elegantly def foo ... return 42 end foo() foo() foo() foo() foo()
36. Three method types 1. User-defined method • When called, TP enters its method body 2. Type-defined method • TP just checks argument types and returns its return type • For built-in methods or libraries 3. Custom special method • TP executes custom behavior • (TP plugin can exploit this?) def foo(n) ... end Integer#+ :: (Integer)->Integer Object#require???
37. Agenda •What "Type Profiler" is •Demos and Problems •Implementation ➔(Very preliminary) Evaluation •Conclusion
38. Excuse: TP is very preliminary... •Currently supports • Basic language features • Limited built-in classes (shown in Demos) •No-support-yet • Almost all built-in classes including Hash • Complex arguments (optional, rest, keywords) • Exceptions • Modules (Mix-in) • etc etc.
39. Experiments •Experiment 1: Self Type-profiling •Experiment 2: Type-profiling optcarrot
40. Experiment 1: Self-profiling •Apply Type Profiler to itself •TP statistics: 2167 LOC, 221 methods •Quantitative result: •Reached 91 methods in 10 minutes (!) •Qualitative result: Found an actual bug •TP said "a method receives NilClass" • It is not intended, and turned out a bug
41. Why many methods not reached? •Because of not-implemented-yet features •For example, Array#<< is not implemented •Some methods are defined but unused •Idea: virtually call all methods with "any" arguments? a = [] a << Foo.new a[0].bar Foo#bar cannot be reached
42. Type Profiler code coverage 42 A method State#run is not reached state is any for state.run state = states.pop caused nil because Array#pop was not implemented yet
43. Experiment 2: optcarrot •Apply Type Profiler to optcarrot • 8bit hardware emulator written in Ruby • statistics: 4476 LOC, 394 methods •Result: • Reached 40 methods in 3 minutes? • Object#send and Fiber are not implemented yet • CPU uses Object#send for instruction dispatch • GPU uses Fiber for state machine
44. Why did it took so long? •State explosion •A method returns Array or Integer •Calling it forks the state •Idea: Merge similar states more intelligently foo() foo() foo() foo() foo() Fork! Fork! Fork! Fork! Fork! foo :: () -> (Array[Integer] | Integer)
45. Agenda •What "Type Profiler" is •Demos and Problems •Implementation •Very preliminary evaluation ➔(Related work and) Conclusion
46. Related Work •mruby-meta-circular (Hideki Miura) • A very similar approach for mruby JIT • This inspired Type Profiler (Thanks!) •HPC Ruby (Koichi Nakamura) • Convert Ruby to C for HPC by abstract interp. •pytype (Google's unofficial project) • Python abstract interpreter for type analysis • More concrete than TP • Limits the stack depth to three
47. Acknowledgement •Hideki Miura •Matz, Akr, Ko1 •PPL paper co-authors • Soutaro Matsumoto • Katsuhiro Ueno • Eijiro Sumii •Stripe team & Jeff Foster •And many people
48. Conclusion •A yet another type analyzer for Ruby 3 applicable to a non-annotated Ruby code • Based on abstract interpretation technique • Little change for Ruby programming experience •Contribution is really welcome! •The development is very preliminary • https://github.com/mame/ruby-type-profiler
49. (A lot of) Future work • Support language features • Almost all built-in classes including Hash • Complex arguments (optional, rest, keywords) • Exceptions • Modules (Mix-in) • etc etc. • Write a type definition for built-in classes/methods • Read type definition file • Improve performance • Make it useful • Code coverage • Flow-sensitive analysis • Heuristic type aggregation • Diagnosis feature • Incremental type profiling • etc etc.