A Plan towards Ruby 3 Types

EuRuKo 2019

https://2019.euruko.org/#schedule

Transcript

1. A Plan towards Ruby 3 Types Yusuke Endoh EuRuKo '19 - 22rd Jun. 2019 1
2. Yusuke Endoh (@mametter) •A full-time Ruby committer at Cookpad Inc. • with @ko1 (Ruby's speed freak) 2
3. PR: Cookpad Inc. •Mission: "Make Everyday Cooking Fun" •cookpad.com: A recipe sharing service • Monthly Average Users: 96 million 3
4. PR: Cookpad Inc. We're hiring! HQ is in Bristol, UK Aim to be No.1 in 100 countries (Now in 29 Languages and 72 Countries) 4
5. Yusuke Endoh (@mametter) •A full-time Ruby committer at Cookpad Inc. • with @ko1 (Ruby's speed freak) •My contributions: • Keyword arguments since Ruby 2.0 • Test coverage measurement • Optcarrot: a benckmark for Ruby 3x3 • Recent: Beginless/endless range: (1..), (..1) 5
6. Ruby 3 goals •Performance (Ruby 3x3) •Concurrency •Static analysis  My mission 6
7. This Talk •Matz's plan for Ruby 3 Types •An introduction of Type Profiler 7
8. Matz's plan for Ruby 3 Types 8
9. Ruby 3 Types' •Objective: Find a possible type bug • To help development •Requirement: Keep Ruby's experience • Type annotation is optional 9 # s is String def foo(s) ... end type annotation
10. Items for Ruby 3 Static Analysis 1. Standard type signature format 2. Level-1 type checker without signatures + Type inference to suggest type signature for non-annotated Ruby code 3. Level-2 type checker with signatures 10
11. 1. Type Signature Format (.rbi) •What types a method accepts/returns class Array[A] include Enumerable def []: (Integer) -> A? def each: { (A)->void } -> self ... end mix-in generics option type interface any type Proposal: ruby-signature 11
12. 2. Type Checker without Signature •Finds possible NoMethodError/TypeErrors •Work without application's type signature • May report false positive def foo(s) s.gsuub!(//, "") s + 42 end foo("foo") NoMethodError? TypeError? Proposals: • Type Profiler • mruby's JIT compiler 12
13. 2'. A kind of Type Inference •Suggest a prototype of type signature for non-annotated Ruby code • Possible approach • virtually simulating the code in type-level def foo(s) s.to_s end foo("foo") foo(42) (String | Integer) -> String ? Proposals: • Type Profiler • mruby's JIT compiler 13
14. 3. Type Checker with Signatures •Conservatively find NoMethod/TypeErrors •Verifies that the code complies with signature • Possible implementation • Gradual type checking • Tool-defined annotation def foo(s) s.gsuub!(//, "") s + 42 end TypeError! def foo: (String) -> Integer Proposals: • Steep • Sorbet • RDL NoMethodError! 14
15. Items for Ruby 3 Static Analysis Library code type signature Sorbet Steep RDL Type error warnings Application code Type Profiler (mruby JIT) Type error warnings type signature 15
16. Use Cases • I want to find a possible bug • Write a code →Use 2 (level-1 type checker without signature) • I want to verify my code • Write a code → Use 2' (type inference) → Use 3 (level-2 type checker with signature) • I want to write a code in "type-driven" style • Hand-write a signature and then a code → Use 3 (level-2 type checker with signature) 16
17. Current Development Status •ruby-signature: discussed by tool developers • Any comment is welcome: https://github.com/ruby/ruby-signature •Type-Profiler: Very experimental... •Steep: Trial use in Sider, Inc. • Good for duck typing / requires many annotations •Sorbet: Trial use in some companies • Gradually applicable / less support for duck typing 17
18. What Ruby 3 will Ship •Matz wants to bundle "type signatures" •A parser library for the format • Type signature files for stdlibs •RubyGems with type signature support •No plan to bundle the checkers • They will be released as external gems 18
19. An introduction of Type Profiler 19
20. Type Profiler for Checking •Finds NoMethodError, TypeError, etc. def foo(n) if n < 10 n.timees {|x| } end end foo(42) Type Profiler t.rb:3: [error] undefined method: Integer#timees Typo 20
21. Type Profiler for Inference •Generates a prototype of type definition def foo(n) n.to_s end foo(42) Type Profiler Object#foo :: (Integer) -> String 21
22. Demo: Overloading def my_to_s(x) x.to_s end my_to_s(42) my_to_s("STR") my_to_s(:sym) Type Profiler Object#my_to_s :: (Integer) -> String Object#my_to_s :: (String) -> String Object#my_to_s :: (Symbol) -> String 22
23. Demo: Recursive Method def fib(n) if n > 1 fib(n-1) + fib(n-2) else n end end fib(10000) Type Profiler Object#fib :: (Integer) -> Integer 23
24. How Type Profiler Does •Runs a Ruby code in "type-level" Normal interpreter def foo(n) n.to_s end foo(42) Calls w/ 42 Returns "42" Type profiler def foo(n) n.to_s end foo(42) Calls w/ Integer Returns String Object#foo :: (Integer) -> String 24
25. Type Profiler and Branch •"Forks" the execution def foo(n) if n < 10 n else "error" end end foo(42) Fork! Now here We cannot tell if n<10 or not Returns Integer Returns String Object#foo :: (Integer) -> (Integer | String) 25
26. The Problem: State Explosion •Analysis time (at April, at RubyKaigi) •Analyzing the code of Type Profiler: 10 min. • Analyzing optcarrot: 3 min. a=b=c=d=e=nil a = 42 if n < 10 b = 42 if n < 10 c = 42 if n < 10 d = 42 if n < 10 e = 42 if n < 10 Fork! Fork! Fork! Fork! Fork! 2 4 8 16 32 State numbers 26
27. The Progress since RubyKaigi •Revamped the analysis algorithm •"state merging" technique called in symbolic execution 27
28. Basic Analysis Algorithm •"Environment": A map from variable to type • Example: •TP runs each instruction iteratively • by propagating the environments •until no environments are updated 28 x = 1 x = 1 x = 1 x y { int, str } { int } x y { nil } { int } ∅ ∅ x y { nil } { int } { int } { int }
29. Example of Analysis 1: def foo(a) 2: if a < 10 3: b = 42 4: else 5: b = "str" 6: end 7: c = b 8: c 9: end 10: 11: ret = foo(42) Line# a b c 1 ∅ ∅ ∅ 2 ∅ ∅ ∅ 3 ∅ ∅ ∅ 4 - - - 5 ∅ ∅ ∅ 6 - 7 ∅ ∅ ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 ∅ ∅ ∅ 3 ∅ ∅ ∅ 4 - - - 5 ∅ ∅ ∅ 6 - 7 ∅ ∅ ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 { Int } ∅ ∅ 3 ∅ ∅ ∅ 4 - - - 5 ∅ ∅ ∅ 6 - 7 ∅ ∅ ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 { Int } ∅ ∅ 3 { Int } ∅ ∅ 4 - - - 5 { Int } ∅ ∅ 6 - 7 ∅ ∅ ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 { Int } ∅ ∅ 3 { Int } ∅ ∅ 4 - - - 5 { Int } ∅ ∅ 6 - 7 { Int } { Int } ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 { Int } ∅ ∅ 3 { Int } ∅ ∅ 4 - - - 5 { Int } ∅ ∅ 6 - 7 { Int } { Int } ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 { Int } ∅ ∅ 3 { Int } ∅ ∅ 4 - - - 5 { Int } ∅ ∅ 6 - 7 { Int } { Int, Str } ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 { Int } ∅ ∅ 3 { Int } ∅ ∅ 4 - - - 5 { Int } ∅ ∅ 6 - 7 { Int } { Int, Str } ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 { Int } ∅ ∅ 3 { Int } ∅ ∅ 4 - - - 5 { Int } ∅ ∅ 6 - 7 { Int } { Int, Str } ∅ 8 { Int } { Int, Str } { Int, Str } Object#foo :: (Int) -> (Int|Str) 29
30. Experiment result •Type Profiler: 10 min. ➔ 2.5 sec. •optcarrot: 3 min. ➔ 6 sec. 30
31. Analysis Time of Type Profiler 0 200 400 600 800 self-profiling optcarrot seconds old new 31
32. Other Problems of Type Profiler •It requires a test •False positive and false suggestion •Some Ruby features cannot be handled • e.g., Object#send, singleton classes # b: Integer or String c = b # c: Integer or String # We lose the correspond between b and c b + c # "May call Integer#+(String)!" 32
33. Development Progress: Done •Design the basic analysis algorithm •Based on abstract interpretation •Support basic language features • Variables, methods, user-defined classes, etc. •Blocks and arrays (maybe) • A limited set of built-in classes 33
34. Development Progress: To Do •Support more features •More built-in classes (Hash!) • Complex arguments (optional/rest/keyword) •Exception • Modules •Input/output type signature format •A lot of improvements for practical use... 34
35. Acknowledgement •Hideki Miura •Matz, Akr, Ko1 •PPL paper co-authors • Soutaro Matsumoto • Katsuhiro Ueno • Eijiro Sumii •Stripe team & Jeff Foster •And many people 35
36. Conclusion •Explained Matz's plan for Ruby 3 static analysis •Introduced Type Profiler • A type analyzer for Ruby 3 applicable to a non-annotated Ruby code • Based on abstract interpretation technique • Little change for Ruby programming experience •Any comments and/or contribution are welcome! • https://github.com/mame/ruby-type-profiler 36