A Plan towards Ruby 3 Types Yusuke Endoh EuRuKo '19 - 22rd Jun. 2019 1

Yusuke Endoh (@mametter) •A full-time Ruby committer at Cookpad Inc. • with @ko1 (Ruby's speed freak) 2

PR: Cookpad Inc. •Mission: "Make Everyday Cooking Fun" •cookpad.com: A recipe sharing service • Monthly Average Users: 96 million 3

PR: Cookpad Inc. We're hiring! HQ is in Bristol, UK Aim to be No.1 in 100 countries (Now in 29 Languages and 72 Countries) 4

Yusuke Endoh (@mametter) •A full-time Ruby committer at Cookpad Inc. • with @ko1 (Ruby's speed freak) •My contributions: • Keyword arguments since Ruby 2.0 • Test coverage measurement • Optcarrot: a benckmark for Ruby 3x3 • Recent: Beginless/endless range: (1..), (..1) 5

Ruby 3 goals •Performance (Ruby 3x3) •Concurrency •Static analysis  My mission 6

This Talk •Matz's plan for Ruby 3 Types •An introduction of Type Profiler 7

Matz's plan for Ruby 3 Types 8

Ruby 3 Types' •Objective: Find a possible type bug • To help development •Requirement: Keep Ruby's experience • Type annotation is optional 9 # s is String def foo(s) ... end type annotation

Items for Ruby 3 Static Analysis 1. Standard type signature format 2. Level-1 type checker without signatures + Type inference to suggest type signature for non-annotated Ruby code 3. Level-2 type checker with signatures 10

1. Type Signature Format (.rbi) •What types a method accepts/returns class Array[A] include Enumerable def []: (Integer) -> A? def each: { (A)->void } -> self ... end mix-in generics option type interface any type Proposal: ruby-signature 11

2. Type Checker without Signature •Finds possible NoMethodError/TypeErrors •Work without application's type signature • May report false positive def foo(s) s.gsuub!(//, "") s + 42 end foo("foo") NoMethodError? TypeError? Proposals: • Type Profiler • mruby's JIT compiler 12

2'. A kind of Type Inference •Suggest a prototype of type signature for non-annotated Ruby code • Possible approach • virtually simulating the code in type-level def foo(s) s.to_s end foo("foo") foo(42) (String | Integer) -> String ? Proposals: • Type Profiler • mruby's JIT compiler 13

3. Type Checker with Signatures •Conservatively find NoMethod/TypeErrors •Verifies that the code complies with signature • Possible implementation • Gradual type checking • Tool-defined annotation def foo(s) s.gsuub!(//, "") s + 42 end TypeError! def foo: (String) -> Integer Proposals: • Steep • Sorbet • RDL NoMethodError! 14

Items for Ruby 3 Static Analysis Library code type signature Sorbet Steep RDL Type error warnings Application code Type Profiler (mruby JIT) Type error warnings type signature 15

Use Cases • I want to find a possible bug • Write a code →Use 2 (level-1 type checker without signature) • I want to verify my code • Write a code → Use 2' (type inference) → Use 3 (level-2 type checker with signature) • I want to write a code in "type-driven" style • Hand-write a signature and then a code → Use 3 (level-2 type checker with signature) 16

Current Development Status •ruby-signature: discussed by tool developers • Any comment is welcome: https://github.com/ruby/ruby-signature •Type-Profiler: Very experimental... •Steep: Trial use in Sider, Inc. • Good for duck typing / requires many annotations •Sorbet: Trial use in some companies • Gradually applicable / less support for duck typing 17

What Ruby 3 will Ship •Matz wants to bundle "type signatures" •A parser library for the format • Type signature files for stdlibs •RubyGems with type signature support •No plan to bundle the checkers • They will be released as external gems 18

An introduction of Type Profiler 19

Type Profiler for Checking •Finds NoMethodError, TypeError, etc. def foo(n) if n < 10 n.timees {|x| } end end foo(42) Type Profiler t.rb:3: [error] undefined method: Integer#timees Typo 20

Type Profiler for Inference •Generates a prototype of type definition def foo(n) n.to_s end foo(42) Type Profiler Object#foo :: (Integer) -> String 21

Demo: Overloading def my_to_s(x) x.to_s end my_to_s(42) my_to_s("STR") my_to_s(:sym) Type Profiler Object#my_to_s :: (Integer) -> String Object#my_to_s :: (String) -> String Object#my_to_s :: (Symbol) -> String 22

Demo: Recursive Method def fib(n) if n > 1 fib(n-1) + fib(n-2) else n end end fib(10000) Type Profiler Object#fib :: (Integer) -> Integer 23

How Type Profiler Does •Runs a Ruby code in "type-level" Normal interpreter def foo(n) n.to_s end foo(42) Calls w/ 42 Returns "42" Type profiler def foo(n) n.to_s end foo(42) Calls w/ Integer Returns String Object#foo :: (Integer) -> String 24

Type Profiler and Branch •"Forks" the execution def foo(n) if n < 10 n else "error" end end foo(42) Fork! Now here We cannot tell if n<10 or not Returns Integer Returns String Object#foo :: (Integer) -> (Integer | String) 25

The Problem: State Explosion •Analysis time (at April, at RubyKaigi) •Analyzing the code of Type Profiler: 10 min. • Analyzing optcarrot: 3 min. a=b=c=d=e=nil a = 42 if n < 10 b = 42 if n < 10 c = 42 if n < 10 d = 42 if n < 10 e = 42 if n < 10 Fork! Fork! Fork! Fork! Fork! 2 4 8 16 32 State numbers 26

The Progress since RubyKaigi •Revamped the analysis algorithm •"state merging" technique called in symbolic execution 27

Basic Analysis Algorithm •"Environment": A map from variable to type • Example: •TP runs each instruction iteratively • by propagating the environments •until no environments are updated 28 x = 1 x = 1 x = 1 x y { int, str } { int } x y { nil } { int } ∅ ∅ x y { nil } { int } { int } { int }

Example of Analysis 1: def foo(a) 2: if a < 10 3: b = 42 4: else 5: b = "str" 6: end 7: c = b 8: c 9: end 10: 11: ret = foo(42) Line# a b c 1 ∅ ∅ ∅ 2 ∅ ∅ ∅ 3 ∅ ∅ ∅ 4 - - - 5 ∅ ∅ ∅ 6 - 7 ∅ ∅ ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 ∅ ∅ ∅ 3 ∅ ∅ ∅ 4 - - - 5 ∅ ∅ ∅ 6 - 7 ∅ ∅ ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 { Int } ∅ ∅ 3 ∅ ∅ ∅ 4 - - - 5 ∅ ∅ ∅ 6 - 7 ∅ ∅ ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 { Int } ∅ ∅ 3 { Int } ∅ ∅ 4 - - - 5 { Int } ∅ ∅ 6 - 7 ∅ ∅ ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 { Int } ∅ ∅ 3 { Int } ∅ ∅ 4 - - - 5 { Int } ∅ ∅ 6 - 7 { Int } { Int } ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 { Int } ∅ ∅ 3 { Int } ∅ ∅ 4 - - - 5 { Int } ∅ ∅ 6 - 7 { Int } { Int } ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 { Int } ∅ ∅ 3 { Int } ∅ ∅ 4 - - - 5 { Int } ∅ ∅ 6 - 7 { Int } { Int, Str } ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 { Int } ∅ ∅ 3 { Int } ∅ ∅ 4 - - - 5 { Int } ∅ ∅ 6 - 7 { Int } { Int, Str } ∅ 8 ∅ ∅ ∅ Line# a b c 1 { Int } ∅ ∅ 2 { Int } ∅ ∅ 3 { Int } ∅ ∅ 4 - - - 5 { Int } ∅ ∅ 6 - 7 { Int } { Int, Str } ∅ 8 { Int } { Int, Str } { Int, Str } Object#foo :: (Int) -> (Int|Str) 29

Experiment result •Type Profiler: 10 min. ➔ 2.5 sec. •optcarrot: 3 min. ➔ 6 sec. 30

Analysis Time of Type Profiler 0 200 400 600 800 self-profiling optcarrot seconds old new 31

Other Problems of Type Profiler •It requires a test •False positive and false suggestion •Some Ruby features cannot be handled • e.g., Object#send, singleton classes # b: Integer or String c = b # c: Integer or String # We lose the correspond between b and c b + c # "May call Integer#+(String)!" 32

Development Progress: Done •Design the basic analysis algorithm •Based on abstract interpretation •Support basic language features • Variables, methods, user-defined classes, etc. •Blocks and arrays (maybe) • A limited set of built-in classes 33

Development Progress: To Do •Support more features •More built-in classes (Hash!) • Complex arguments (optional/rest/keyword) •Exception • Modules •Input/output type signature format •A lot of improvements for practical use... 34

Acknowledgement •Hideki Miura •Matz, Akr, Ko1 •PPL paper co-authors • Soutaro Matsumoto • Katsuhiro Ueno • Eijiro Sumii •Stripe team & Jeff Foster •And many people 35

Conclusion •Explained Matz's plan for Ruby 3 static analysis •Introduced Type Profiler • A type analyzer for Ruby 3 applicable to a non-annotated Ruby code • Based on abstract interpretation technique • Little change for Ruby programming experience •Any comments and/or contribution are welcome! • https://github.com/mame/ruby-type-profiler 36