Tweet
Tweet

Slide 1

Slide 1 text

Type Checking Ruby Programs with Annotations Soutaro Matsumoto
 (@soutaro)

Slide 2

Slide 2 text

Soutaro Matsumoto • GitHub, Twitter @soutaro

Slide 3

Slide 3 text

Soutaro Matsumoto • GitHub, Twitter @soutaro • I have implemented four type checkers for Ruby

Slide 4

Slide 4 text

Soutaro Matsumoto • GitHub, Twitter @soutaro • I have implemented four type checkers for Ruby • 2005, Type inference, structural subtyping

Slide 5

Slide 5 text

Soutaro Matsumoto • GitHub, Twitter @soutaro • I have implemented four type checkers for Ruby • 2005, Type inference, structural subtyping • 2007, Type inference, polymorphic record types

Slide 6

Slide 6 text

Soutaro Matsumoto • GitHub, Twitter @soutaro • I have implemented four type checkers for Ruby • 2005, Type inference, structural subtyping • 2007, Type inference, polymorphic record types • 2009, Control flow analysis

Slide 7

Slide 7 text

Soutaro Matsumoto • GitHub, Twitter @soutaro • I have implemented four type checkers for Ruby • 2005, Type inference, structural subtyping • 2007, Type inference, polymorphic record types • 2009, Control flow analysis • 2017, Local type inference, structural subtyping [New]

Slide 8

Slide 8 text

Why do we want a type checker?

Slide 9

Slide 9 text

Benefits • Find bugs • Verifiable documentation • Auto completion • Easier refactoring • For advanced program analysis

Slide 10

Slide 10 text

Type Checking for Ruby • People have tried for this at least for 12 years • Static Type Inference for Ruby [Furr, 2008] • Type Inference for Ruby Programs Based on Polymorphic Record Types [Matsumoto, 2007] • They had tried to infer types of Ruby programs, because Ruby is an untyped language

Slide 11

Slide 11 text

Static Type Inference for Ruby • Implementation is available as Diamondback Ruby • Based on structural subtyping • This means it cannot infer polymorphic types

Slide 12

Slide 12 text

Type Inference for Ruby Programs Based on Polymorphic Record Types • RubyKaigi 2008 • Based on ML type inference and polymorphic record types • Infers polymorphic types • Cannot give types to some Ruby builtin • Polymorphic recursion (Array cannot be polymorphic) • Non regular types (Array#map)

Slide 13

Slide 13 text

Type Checking for Ruby Furr, 2008 Matsumoto, 2007 Type System Structural Subtyping Polymorphic Record Types Type Inference Constraint based ML Type Inference Correctness Maybe (not proved) Limitations Cannot infer polymorphic types Cannot type some builtin

Slide 14

Slide 14 text

Type Checking for Ruby Furr, 2008 Matsumoto, 2007 Type System Structural Subtyping Polymorphic Record Types Type Inference Constraint based ML Type Inference Correctness Maybe (not proved) Limitations Cannot infer polymorphic types Cannot type some builtin

Slide 15

Slide 15 text

The Conclusion • We cannot construct type inference for Ruby programs • If we choose subtyping, no polymorphic types inferred • If we choose polymorphic type inference, some builtins cannot be typed

Slide 16

Slide 16 text

Requirements • Correctness: if type checker says ok, no type error during execution • Static: without execution • No annotation: type inference

Slide 17

Slide 17 text

Relaxing Requirements • Correctness → Forget correctness • Static → Defer type checking to runtime • No annotation → Let programmers write types

Slide 18

Slide 18 text

Forget Correctness • Incorrect type checking may still help programmers • TypeScript accepts unsound co-variant subtyping on function parameters • Lint tools • RuboCop, Brakeman, Querly • Set of ad-hoc bad program patterns, but helps detecting bugs

Slide 19

Slide 19 text

Type Checking at Runtime • Just-in-Time Static Type Checking for Dynamic Languages [Ren, 2016] • Run type check for method body at the beginning of the execution of the method • Not before starting execution • Before Ruby raising NoMethodError • Support meta-programming 1 def foo(x) 2 "".bar if x 3 end 4 5 foo(false)

Slide 20

Slide 20 text

Annotate Ruby Programs • This is my latest static type checker

Slide 21

Slide 21 text

Steep • Gradual typing for Ruby • https:/ /github.com/soutaro/steep $ gem install steep --pre

Slide 22

Slide 22 text

Key Ideas • Gradual Typing • If you don't annotate your program, it won't type check • Programmers annotate their Ruby programs • Local type inference to minimize annotation effort • Another language to define types

Slide 23

Slide 23 text

Example • Type annotations are given as comment • ClassName is a type for instance of that class • ClassName.class is a type for class itself • Local variable types can be inferred from its value # @type var x: String # @type const Pathname: Pathname.class path = Pathname.new(x)

Slide 24

Slide 24 text

Example • Type annotations are given as comment • ClassName is a type for instance of that class • ClassName.class is a type for class itself • Local variable types can be inferred from its value # @type var x: String # @type const Pathname: Pathname.class path = Pathname.new(x)

Slide 25

Slide 25 text

Example • Type annotations are given as comment • ClassName is a type for instance of that class • ClassName.class is a type for class itself • Local variable types can be inferred from its value # @type var x: String # @type const Pathname: Pathname.class path = Pathname.new(x)

Slide 26

Slide 26 text

Example • Type annotations are given as comment • ClassName is a type for instance of that class • ClassName.class is a type for class itself • Local variable types can be inferred from its value # @type var x: String # @type const Pathname: Pathname.class path = Pathname.new(x)

Slide 27

Slide 27 text

Annotating Constants? • In Ruby, constants are similar to method • Inheritance • Module nest • Dynamic class Foo def foo p Foo end end Foo.new.foo # => Foo Foo::Foo = "Hello World" Foo.new.foo # => "Hello World"

Slide 28

Slide 28 text

Annotating Constants? • In Ruby, constants are similar to method • Inheritance • Module nest • Dynamic class Foo def foo p Foo end end Foo.new.foo # => Foo Foo::Foo = "Hello World" Foo.new.foo # => "Hello World"

Slide 29

Slide 29 text

Annotating Constants? • In Ruby, constants are similar to method • Inheritance • Module nest • Dynamic class Foo def foo p Foo end end Foo.new.foo # => Foo Foo::Foo = "Hello World" Foo.new.foo # => "Hello World"

Slide 30

Slide 30 text

Annotating Constants? • In Ruby, constants are similar to method • Inheritance • Module nest • Dynamic class Foo def foo p Foo end end Foo.new.foo # => Foo Foo::Foo = "Hello World" Foo.new.foo # => "Hello World"

Slide 31

Slide 31 text

Type Definitions • Types are defined in another language by programmers • Not extracted from Ruby programs • Like C headers

Slide 32

Slide 32 text

Type Definition interface _StringConvertible def to_str: -> String end class String def split: (_StringConvertible, ?Integer) -> Array | (Regexp, ?Integer) -> Array ... end class Person <: Object def initialize: (name: String) -> any def name: -> String end

Slide 33

Slide 33 text

Type Definition interface _StringConvertible def to_str: -> String end class String def split: (_StringConvertible, ?Integer) -> Array | (Regexp, ?Integer) -> Array ... end class Person <: Object def initialize: (name: String) -> any def name: -> String end

Slide 34

Slide 34 text

Type Definition interface _StringConvertible def to_str: -> String end class String def split: (_StringConvertible, ?Integer) -> Array | (Regexp, ?Integer) -> Array ... end class Person <: Object def initialize: (name: String) -> any def name: -> String end

Slide 35

Slide 35 text

Type Definition interface _StringConvertible def to_str: -> String end class String def split: (_StringConvertible, ?Integer) -> Array | (Regexp, ?Integer) -> Array ... end class Person <: Object def initialize: (name: String) -> any def name: -> String end

Slide 36

Slide 36 text

Type Definition interface _WithEach<'a> def each: { ('a) -> any } -> instance end module Enumerable<'a> : _WithEach<'a> def map: <'b> { ('a) -> 'b } -> Array<'b> ... end class Array<'a> include Enumerable<'a> def each: { ('a) -> any } -> instance ... end

Slide 37

Slide 37 text

Type Definition interface _WithEach<'a> def each: { ('a) -> any } -> instance end module Enumerable<'a> : _WithEach<'a> def map: <'b> { ('a) -> 'b } -> Array<'b> ... end class Array<'a> include Enumerable<'a> def each: { ('a) -> any } -> instance ... end

Slide 38

Slide 38 text

Type Definition interface _WithEach<'a> def each: { ('a) -> any } -> instance end module Enumerable<'a> : _WithEach<'a> def map: <'b> { ('a) -> 'b } -> Array<'b> ... end class Array<'a> include Enumerable<'a> def each: { ('a) -> any } -> instance ... end

Slide 39

Slide 39 text

Type Definition interface _WithEach<'a> def each: { ('a) -> any } -> instance end module Enumerable<'a> : _WithEach<'a> def map: <'b> { ('a) -> 'b } -> Array<'b> ... end class Array<'a> include Enumerable<'a> def each: { ('a) -> any } -> instance ... end

Slide 40

Slide 40 text

Type Definition interface _WithEach<'a> def each: { ('a) -> any } -> instance end module Enumerable<'a> : _WithEach<'a> def map: <'b> { ('a) -> 'b } -> Array<'b> ... end class Array<'a> include Enumerable<'a> def each: { ('a) -> any } -> instance ... end

Slide 41

Slide 41 text

Open Class • Use another signature construct called extension • Extension adds methods to existing class/module • The name is from C#, Swift, or Objective C • This is also flow insensitive extension Object (try) def try: (Symbol) -> any | <'a> { (instance) -> 'a } -> 'a end

Slide 42

Slide 42 text

What is Signature? • Interface is the core of the type system • Classes and modules are utility constructs • Defines interfaces expanding inheritance and mixin • Person class signature is not a type of Person.new • Specify type of Person constant by annotation explicitly

Slide 43

Slide 43 text

Signature Code Separation • There are at least two Object class definition • If the types are Ruby classes, when you write a type Object, which one does that mean? • To avoid the confusion, Steep uses another signature language class Object ... end class Object def try(...) ... end ... end

Slide 44

Slide 44 text

Signature Code Separation • There are at least two Object class definition • If the types are Ruby classes, when you write a type Object, which one does that mean? • To avoid the confusion, Steep uses another signature language class Object ... end class Object def try(...) ... end ... end

Slide 45

Slide 45 text

Signature Code Separation • There are at least two Object class definition • If the types are Ruby classes, when you write a type Object, which one does that mean? • To avoid the confusion, Steep uses another signature language class Object ... end class Object def try(...) ... end ... end

Slide 46

Slide 46 text

Signature Code Separation • There are at least two Object class definition • If the types are Ruby classes, when you write a type Object, which one does that mean? • To avoid the confusion, Steep uses another signature language class Object ... end class Object def try(...) ... end ... end

Slide 47

Slide 47 text

Steep • Local type inference & structural subtyping (from TypeScript) • Polymorphism • Union types • Method overloading • Signature code separation & extension (from Objective C) • No need to type meta-programming • Monkey patching

Slide 48

Slide 48 text

Conclusion • Type inference for Ruby is impossible • I'm working for a type checker with type annotations • Local type inference & structural subtyping • I hope Steep be a good material to explorer the static type checker for Ruby

Slide 49

Slide 49 text

Slides below were skipped in my presentation

Slide 50

Slide 50 text

References • [Matsumoto, 2007] S. Matsumoto and Y. Minamide. Type Inference for Ruby Programs based on Polymorphic Record Types • [Furr, 2009] M. Furr, J. hoon (David) An, J. S. Foster, and M. Hicks. Static Type Inference for Ruby • [Ren, 2016] B. M. Ren and J. S. Foster. Just-in-Time Static Type Checking for Dynamic Languages

Slide 51

Slide 51 text

Using Steep 1. Declare types 2. Implement and annotate the Ruby program 3. Run the type checker You can find examples from some of Steep source code and its tests

Slide 52

Slide 52 text

Declare Types class Contact def initialize: (name: String, address: Address) -> any def name: -> String def address: -> Address end class ContactList def contacts: -> Array def filter: { (Contact) -> _Boolean } -> ContactList end

Slide 53

Slide 53 text

Declare Types # @type const ContactList: ContactList.class list = ContactList.new list.contacts << 3 list = list.contacts.filter {|contact| contact.name == "Matsumoto" } $ steep check -I contact.rbi test.rb test.rb:9:0: ArgumentTypeMismatch: type=Array, method=<< test.rb:10:0: IncompatibleAssignment: lhs_type=ContactList, rhs_type=Array

Slide 54

Slide 54 text

Declare Types # @type const ContactList: ContactList.class list = ContactList.new list.contacts << 3 list = list.contacts.filter {|contact| contact.name == "Matsumoto" } $ steep check -I contact.rbi test.rb test.rb:9:0: ArgumentTypeMismatch: type=Array, method=<< test.rb:10:0: IncompatibleAssignment: lhs_type=ContactList, rhs_type=Array

Slide 55

Slide 55 text

Declare Types # @type const ContactList: ContactList.class list = ContactList.new list.contacts << 3 list = list.contacts.filter {|contact| contact.name == "Matsumoto" } $ steep check -I contact.rbi test.rb test.rb:9:0: ArgumentTypeMismatch: type=Array, method=<< test.rb:10:0: IncompatibleAssignment: lhs_type=ContactList, rhs_type=Array

Slide 56

Slide 56 text

... 8 9 list.contacts << 3 10 list = list.contacts.filter {|contact| contact.name == "Matsumoto" } Type Check $ steep check -I address.rbi test.rb test.rb:9:0: ArgumentTypeMismatch: type=Array, method=<< test.rb:10:0: IncompatibleAssignment: lhs_type=ContactList, rhs_type=Array

Slide 57

Slide 57 text

Annotate Ruby Program class Contact # @implements Contact attr_reader :name attr_reader :address def initialize(name:, address:) @name = name @address = address end end

Slide 58

Slide 58 text

Type Check • There is no defs in the class for name or address • Add annotation to tell steep that it does not have to check existence of that method definitions $ steep check -I contact.rbi contact.rb contact.rb:1:0: MethodDefinitionMissing: module=Contact, method=name contact.rb:1:0: MethodDefinitionMissing: module=Contact, method=address

Slide 59

Slide 59 text

Annotate Ruby Program class Contact # @implements Contact # @dynamic name attr_reader :name # @dynamic address attr_reader :address def initialize(name:, address:) @name = name @address = address end end

Slide 60

Slide 60 text

Annotate Ruby Programs class ContactList # @implements ContactList # @dynamic contacts attr_reader :contacts def initialize; @contacts = []; end def filter contacts.select do |contact| yield contact end end end

Slide 61

Slide 61 text

Type Check $ steep check -I contact.rbi contact.rb contact.rb:25:2: MethodBodyTypeMismatch: method=filter, expected=ContactList, actual=Array def filter contacts.select do |contact| yield contact end end contacts is Array and #select returns Array

Slide 62

Slide 62 text

Fix Implementation class ContactList # @implements ContactList ... def filter copy = ContactList.new contacts.each do |contact| copy.contacts << contact if yield(contact) end copy end end

Slide 63

Slide 63 text

Type Check • Is that sure? • I'm afraid if there is unexpected fallback to any type $ steep check -I contact.rbi contact.rb $

Slide 64

Slide 64 text

Type Check $ steep check --fallback-any-is-error -I contact.rbi contact.rb contact.rb:26:11: FallbackAny 26 copy = ContactList.new

Slide 65

Slide 65 text

Annotate Ruby Programs class ContactList # @implements ContactList # @type const ContactList: ContactList.class ... end

Slide 66

Slide 66 text

Type Check $ steep check --fallback-any-is-error -I contact.rbi contact.rb $

Slide 67

Slide 67 text

Future Work • Support typical Ruby programming styles • Instead of adding annotations to all constants, infer their types from signatures • String::String=3, ([Integer, String].sample)::Foo=3 • Type system improvements • Typing rule enhancements & bug fixes • Access control (public/private) • Integration with Ruby

Slide 68

Slide 68 text

Integration with Ruby • Some features cannot be implemented without extending Ruby • Integrating annotations to Ruby syntax • Update: Matz rejected adding typing syntax yesterday • Dynamic type testing • Ruby only has is_a? inheritance relation testing operator • Want structural subtyping relation testing operator • Not Module#conform (because the params and return type should be checked)

Slide 69

Slide 69 text

Structural Subtyping Failure class A def initialize: (name: String) -> any end class B <: A def initialize: (year: Integer) -> any def print: () -> any end A.class == { new: (name: String) -> A, ... } B.class == { new: (year: Integer) -> B, ... } A == { class: -> A.class, ... } B == { class: -> B.class, ... }

Slide 70

Slide 70 text

Array#zip class Array<'a> ... def zip: <'b> (Array<'b>) -> Array end

Slide 71

Slide 71 text

Array#zip • Tuple types ('a * 'b) • Type variable extraction difficulty def zip: <'b> (Array<'b>) -> Array<'a * 'b> # @type var x: Array ["a"].zip(x) -> Array # @type var y: Foo ["a"].zip(y) -> Array

Slide 72

Slide 72 text

Array#zip • Tuple types ('a * 'b) • Type variable extraction difficulty def zip: <'b> (Array<'b>) -> Array<'a * 'b> # @type var x: Array ["a"].zip(x) -> Array # @type var y: Foo ["a"].zip(y) -> Array

Slide 73

Slide 73 text

Array#zip • Tuple types ('a * 'b) • Type variable extraction difficulty def zip: <'b> (Array<'b>) -> Array<'a * 'b> # @type var x: Array ["a"].zip(x) -> Array # @type var y: Foo ["a"].zip(y) -> Array ???