The challenges behind Ruby type checking

RubyKaigi 2019, Fukuoka.

Transcript

1. The challenges behind
   Ruby type checking
   Soutaro Matsumoto
   
   @soutaro
   

   View Slide

2. Steep changelog
   • 10 releases since last RubyKaigi. (0.10.0, Mar 3, 2019)
   
   • Many bugfixes and type system improvements.
   
   • User experience improvements: $ steep watch
   • Working for editor integration using LSP.
   
   • Done: On the fly type checking, expression type hover.
   
   • WIP: Jump to definition/declaration, completion.
   $ gem install steep
   

   View Slide

3. Outline
   • Ruby type checking difficulties review.
   
   • Introduction to the Ruby signature language.
   

   View Slide

4. Duck typing
   

   View Slide

5. Duck typing
   rubber duck
   

   View Slide

6. def write(io, message)
   io << message
   end
   

   View Slide

7. def write(io, message)
   io << message
   end
   write(StringIO.new, "Hello World")
   write(STDOUT, "Hello World")
   

   View Slide

8. def write(io, message)
   io << message
   end
   write(StringIO.new, "Hello World")
   write(STDOUT, "Hello World")
   write([], "Hello World")
   write(1, 8)
   

   View Slide

9. Duck typing
   • An untyped programming.
   
   • Assumes an interface (a set of methods) implicitly.
   
   • An object conforms to an interface if the object has all of the methods
   listed in the interface.
   

   View Slide

10. Duck Scale
    Interface Implementation Languages
    Declared explicitly Declare in class definition Java
    Declared explicitly Declare after class definition Objective C, Swift, Haskell, Sorbet, ...
    Declared explicitly Conforms implicitly TypeScript, Steep
    Declared implicitly Conforms implicitly OCaml, C++, JavaScript, Ruby
    Less Duck
    More Duck
    

    View Slide

11. Metaprogramming
    It means that a program can be designed to read,
    generate, analyze or transform other programs, and
    even modify itself while running.
    Metaprogramming, Wikipedia
    

    View Slide

12. Metaprogramming
    eval(gets)
    

    View Slide

13. Metaprogramming
    eval(gets)
    define_method :foo do ... end
    

    View Slide

14. Metaprogramming
    class User < ApplicationRecord
    has_many :articles
    end
    eval(gets)
    define_method :foo do ... end
    

    View Slide

15. Metaprogramming
    class User < ApplicationRecord
    has_many :articles
    end
    include Enumerable
    eval(gets)
    define_method :foo do ... end
    

    View Slide

16. Metaprogramming
    class User < ApplicationRecord
    has_many :articles
    end
    include Enumerable
    private
    eval(gets)
    define_method :foo do ... end
    

    View Slide

17. Metaprogramming
    require "pathname"
    load "./entrypoint.rb"
    class User < ApplicationRecord
    has_many :articles
    end
    include Enumerable
    private
    eval(gets)
    define_method :foo do ... end
    

    View Slide

18. Metaprogramming
    require "pathname"
    load "./entrypoint.rb"
    StrongJSON.new do
    let :user, object(name: string,
    email: string?)
    end
    class User < ApplicationRecord
    has_many :articles
    end
    include Enumerable
    private
    eval(gets)
    define_method :foo do ... end
    

    View Slide

19. • The metaprogramming primitives in Ruby are methods, which is subject to
    re-definition while execution.
    
    • What can we do?
    
    • Let users write annotations.
    
    • Simulate metaprogramming primitives in type checkers.
    
    • Using runtime information.
    
    • Typing rules plug-in?
    

    View Slide

20. case x
    when String
    puts "x is a String"
    when Integer
    puts "x is a Integer"
    end
    

    View Slide

21. case x
    when String
    puts "x is a String"
    when Integer
    puts "x is a Integer"
    end
    a = [1, "2", :3, 4.0]
    x, y*, z = a
    

    View Slide

22. case x
    when String
    puts "x is a String"
    when Integer
    puts "x is a Integer"
    end
    a = [1, "2", :3, 4.0]
    x, y*, z = a
    spawn("rm", "-rf", "/")
    spawn("rm -rf /")
    spawn({"ENV"=>"/bin"}, "rm -rf /")
    spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")
    

    View Slide

23. case x
    when String
    puts "x is a String"
    when Integer
    puts "x is a Integer"
    end
    a = [1, "2", :3, 4.0]
    x, y*, z = a
    1.instance_eval do
    self + 3
    end
    spawn("rm", "-rf", "/")
    spawn("rm -rf /")
    spawn({"ENV"=>"/bin"}, "rm -rf /")
    spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")
    

    View Slide

24. case x
    when String
    puts "x is a String"
    when Integer
    puts "x is a Integer"
    end
    a = [1, "2", :3, 4.0]
    x, y*, z = a
    1.instance_eval do
    self + 3
    end
    1.tap do
    break "foo"
    end
    spawn("rm", "-rf", "/")
    spawn("rm -rf /")
    spawn({"ENV"=>"/bin"}, "rm -rf /")
    spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")
    

    View Slide

25. case x
    when String
    puts "x is a String"
    when Integer
    puts "x is a Integer"
    end
    a = [1, "2", :3, 4.0]
    x, y*, z = a
    1.instance_eval do
    self + 3
    end
    1.tap do
    break "foo"
    end
    [a,b].sort
    [1,2,3].map!(&:to_s)
    spawn("rm", "-rf", "/")
    spawn("rm -rf /")
    spawn({"ENV"=>"/bin"}, "rm -rf /")
    spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")
    

    View Slide

26. • We extend the type system.
    
    • Union types.
    
    • Tuple types, record types.
    
    • Method overloading.
    
    • Adding self type on block types.
    
    • Adding break type on block types?
    
    • Conditional types on method types??
    

    View Slide

27. No Ruby library has type
    definition.
    

    View Slide

28. Overview
    • Define the standard language to describe signature of Ruby program.
    
    • Ruby will ship with stdlib signatures.
    
    • Gems will ship with their signatures.
    
    • Type checkers will use the signature to know the type of libraries.
    

    View Slide

29. Goals
    1. Define the syntax of signature language.
    
    2. Define the semantics of the signature language.
    
    3. Implement a library to manipulate Ruby signatures.
    
    4. Provide the signature of Ruby standard library.
    
    5. Encourage another type checker development.
    

    View Slide

30. https://github.com/soutaro/ruby-signature
    

    View Slide

31. $ rbi list
    $ rbi methods --singleton ::Object
    $ rbi -r pathname method ::Object Pathname
    

    View Slide

32. View Slide

33. class Array[A]
    def map: [X] { (A) -> X } -> Array[X]
    | -> Enumerable[A, self]
    ...
    end
    

    View Slide

34. Signature language
    • Defines structure of Ruby programs statically.
    
    • No program execution, no metaprogramming.
    
    • Expressive enough to describe type of Ruby methods.
    
    • Easy to read/write and intuitive.
    

    View Slide

35. class Set[A]
    def initialize: (_Each[A] objects) -> void
    def add: (A) -> self
    def add?: (A) -> self?
    include Enumerable[A, void]
    def each: { (A) -> void } -> Set[A]
    def self.`[]`: [X] (*X) -> Set[X]
    ...
    end
    interface _Each[A]
    def each: { (A) -> void } -> any
    end
    

    View Slide

36. class Set[A]
    def initialize: (_Each[A] objects) -> void
    def add: (A) -> self
    def add?: (A) -> self?
    include Enumerable[A, void]
    def each: { (A) -> void } -> Set[A]
    def self.`[]`: [X] (*X) -> Set[X]
    ...
    end
    interface _Each[A]
    def each: { (A) -> void } -> any
    end
    

    View Slide

37. class Set[A]
    def initialize: (_Each[A] objects) -> void
    def add: (A) -> self
    def add?: (A) -> self?
    include Enumerable[A, void]
    def each: { (A) -> void } -> Set[A]
    def self.`[]`: [X] (*X) -> Set[X]
    ...
    end
    interface _Each[A]
    def each: { (A) -> void } -> any
    end
    

    View Slide

38. class Set[A]
    def initialize: (_Each[A] objects) -> void
    def add: (A) -> self
    def add?: (A) -> self?
    include Enumerable[A, void]
    def each: { (A) -> void } -> Set[A]
    def self.`[]`: [X] (*X) -> Set[X]
    ...
    end
    interface _Each[A]
    def each: { (A) -> void } -> any
    end
    

    View Slide

39. class Set[A]
    def initialize: (_Each[A] objects) -> void
    def add: (A) -> self
    def add?: (A) -> self?
    include Enumerable[A, void]
    def each: { (A) -> void } -> Set[A]
    def self.`[]`: [X] (*X) -> Set[X]
    ...
    end
    interface _Each[A]
    def each: { (A) -> void } -> any
    end
    

    View Slide

40. class Set[A]
    def initialize: (_Each[A] objects) -> void
    def add: (A) -> self
    def add?: (A) -> self?
    include Enumerable[A, void]
    def each: { (A) -> void } -> Set[A]
    def self.`[]`: [X] (*X) -> Set[X]
    ...
    end
    interface _Each[A]
    def each: { (A) -> void } -> any
    end
    

    View Slide

41. class Set[A]
    def initialize: (_Each[A] objects) -> void
    def add: (A) -> self
    def add?: (A) -> self?
    include Enumerable[A, void]
    def each: { (A) -> void } -> Set[A]
    def self.`[]`: [X] (*X) -> Set[X]
    ...
    end
    interface _Each[A]
    def each: { (A) -> void } -> any
    end
    

    View Slide

42. class Set[A]
    def initialize: (_Each[A] objects) -> void
    def add: (A) -> self
    def add?: (A) -> self?
    include Enumerable[A, void]
    def each: { (A) -> void } -> Set[A]
    def self.`[]`: [X] (*X) -> Set[X]
    ...
    end
    interface _Each[A]
    def each: { (A) -> void } -> any
    end
    

    View Slide

43. Ruby 㱻 Signature
    class ClassInstanceType
    include Application
    def sub(s)
    self.class.new(
    name: name,
    args: args.map {|arg| arg.sub(s) }
    )
    end
    end
    class ClassInstanceType
    include Application
    def sub: (_Substitution) -> self
    end
    interface _Substitution
    def sub: (type) -> type
    end
    

    View Slide

44. Mixin?
    • include/prepend/extend in rbi are syntax, not method calls.
    
    • No confusion by re-definition.
    
    • Almost compatible semantics with Ruby (Module).
    
    • We have attr_reader/attr_writer/attr_accessor, and they are
    syntax too.
    
    • And private/public
    

    View Slide

45. Types
    Integer
    singleton(Integer)
    Class instance/singleton
    

    View Slide

46. Types
    Integer
    singleton(Integer)
    Class instance/singleton
    Array[Integer]
    Hash[Symbol, String]
    Generic class type
    

    View Slide

47. Types
    Integer
    singleton(Integer)
    Class instance/singleton
    Array[Integer]
    Hash[Symbol, String]
    Generic class type
    Integer | String
    Union types
    

    View Slide

48. Types
    Integer
    singleton(Integer)
    Class instance/singleton
    Array[Integer]
    Hash[Symbol, String]
    Generic class type
    Integer | String
    Union types
    any
    Dynamic type
    

    View Slide

49. Types
    Integer
    singleton(Integer)
    Class instance/singleton
    Array[Integer]
    Hash[Symbol, String]
    Generic class type
    Integer | String
    Union types
    any
    Dynamic type
    nil
    bool
    void
    Base types
    

    View Slide

50. Types
    Integer
    singleton(Integer)
    Class instance/singleton
    Array[Integer]
    Hash[Symbol, String]
    Generic class type
    Integer | String
    Union types
    any
    Dynamic type
    nil
    bool
    void
    Base types
    Integer?
    Optional type
    

    View Slide

51. Types
    Integer
    singleton(Integer)
    Class instance/singleton
    Array[Integer]
    Hash[Symbol, String]
    Generic class type
    Integer | String
    Union types
    any
    Dynamic type
    nil
    bool
    void
    Base types
    Integer?
    Optional type
    1
    :hello
    "world"
    Singleton types
    

    View Slide

52. Types
    Integer
    singleton(Integer)
    Class instance/singleton
    Array[Integer]
    Hash[Symbol, String]
    Generic class type
    Integer | String
    Union types
    any
    Dynamic type
    nil
    bool
    void
    Base types
    Integer?
    Optional type
    1
    :hello
    "world"
    Singleton types
    [Integer, String]
    { name: String, email: String? }
    Tuple and record types
    

    View Slide

53. Types
    Integer
    singleton(Integer)
    Class instance/singleton
    Array[Integer]
    Hash[Symbol, String]
    Generic class type
    Integer | String
    Union types
    any
    Dynamic type
    nil
    bool
    void
    Base types
    Integer?
    Optional type
    1
    :hello
    "world"
    Singleton types
    [Integer, String]
    { name: String, email: String? }
    Tuple and record types
    ^(Integer, String) -> void
    Proc type
    

    View Slide

54. Method types
    (Integer, ?bool, *String, any, name: String, ?email: String?, **String) -> S
    (String name, ?email: String? email) -> void
    

    View Slide

55. Method types
    (Integer, ?bool, *String, any, name: String, ?email: String?, **String) -> S
    (String name, ?email: String? email) -> void
    (Integer) { (Integer) -> void } -> String
    (Integer) ?{ (Integer) -> void } -> String
    

    View Slide

56. Method types
    (Integer, ?bool, *String, any, name: String, ?email: String?, **String) -> S
    (String name, ?email: String? email) -> void
    (Integer) { (Integer) -> void } -> String
    (Integer) ?{ (Integer) -> void } -> String
    [X] { (A) -> X } -> Array[X]
    | -> Enumerable[A, self]
    

    View Slide

57. def self.open: (String | Integer path,
    ?String | Integer mode,
    ?Integer perm,
    ?external_encoding: Encoding | String,
    ?internal_encoding: Encoding | String,
    ?encoding: Encoding | String,
    ?mode: String | Integer,
    ?textmode: bool,
    ?binmode: bool,
    ?autoclose: bool) -> IO
    | [X] (String | Integer path,
    ?String | Integer mode,
    ?Integer perm,
    ?external_encoding: Encoding | String,
    ?internal_encoding: Encoding | String,
    ?encoding: Encoding | String,
    ?mode: String | Integer,
    ?textmode: bool,
    ?binmode: bool,
    ?autoclose: bool) { (IO) -> X } -> X
    

    View Slide

58. Open class
    • Add methods to existing classes/modules using extension construct.
    
    • Keep class/module declarations closed in signatures.
    
    • Does this define new class, or modify an existing class?
    extension Kernel (Pathname)
    def Pathname: (String) -> Pathname
    end
    

    View Slide

59. Local things
    • You can declare the instance variables
    and private methods in signature.
    
    • Ruby allows accessing them through
    mixin/inheritance.
    
    • If it is a part of external API, write them.
    class User
    @name: String
    @email: String?
    @phone: String?
    def send: (String) -> void
    private
    def send_email: (String)-> void
    def send_sms: (String) -> void
    end
    

    View Slide

60. Integration with type checkers
    • There are at least four type checking tools with incompatible type
    systems.
    
    • Sorbet, RDL: Nominal subtyping
    
    • Steep: Structural subtyping.
    
    • type-profiler: No subtyping yet.
    
    • Parametrized type checking semantics with axioms.
    

    View Slide

61. T <: void
    T <: bool
    S <: T | S
    T <: T | S
    [T, S] <: ::Array[T | S]
    S <: T
    Axiom of union types
    Axiom of void and bool types
    Axiom of tuple types?
    Type checkers will define their own subtyping relations,
    
    but should follow the axioms.
    S <: S?
    nil <: S?
    Axiom of optional types
    ::String <: ::Object
    Axiom of subclassing
    

    View Slide

62. T <: void
    T <: bool
    S <: T | S
    T <: T | S
    [T, S] <: ::Array[T | S]
    S <: T
    Axiom of union types
    Axiom of void and bool types
    Axiom of tuple types?
    Type checkers will define their own subtyping relations,
    
    but should follow the axioms.
    S <: S?
    nil <: S?
    Axiom of optional types
    ::String <: ::Object
    Axiom of subclassing
    

    View Slide

63. Missing features
    • Mechanism to reuse parameter types.
    
    • Mechanism to share type of existing gems.
    

    View Slide

64. Sharing gem types
    • What can we do for existing gems without signatures? (if the authors don't
    ship with signatures?)
    
    • TypeScript (DefinitelyTyped)
    
    • Community managed type definitions of npm packages.
    source "https://ruby-signatures.org" do
    gem "rails"
    end
    

    View Slide

65. Sharing gem types
    • What can we do for existing gems without signatures? (if the authors don't
    ship with signatures?)
    
    • TypeScript (DefinitelyTyped)
    
    • Community managed type definitions of npm packages.
    source "https://ruby-signatures.org" do
    gem "rails"
    end
    A prefix to avoid name conflict.ʢୄʣ
    

    View Slide

66. Recap
    • Type signatures of existing Ruby libraries are essential.
    
    • We define the standard type signature for Ruby libraries.
    
    • You can write types of your library.
    
    • Type checking is optional, but the signature helps developers to
    understand precisely how the API of your library is.
    
    • Give us your feedbacks!
    

    View Slide