Pro Yearly is on sale from $80 to $50! »

The challenges behind Ruby type checking

The challenges behind Ruby type checking

RubyKaigi 2019, Fukuoka.

1fab9d01b25e99522f3dfd01e3d4cb51?s=128

Soutaro Matsumoto

April 20, 2019
Tweet

Transcript

  1. The challenges behind Ruby type checking Soutaro Matsumoto @soutaro

  2. Steep changelog • 10 releases since last RubyKaigi. (0.10.0, Mar

    3, 2019) • Many bugfixes and type system improvements. • User experience improvements: $ steep watch • Working for editor integration using LSP. • Done: On the fly type checking, expression type hover. • WIP: Jump to definition/declaration, completion. $ gem install steep
  3. Outline • Ruby type checking difficulties review. • Introduction to

    the Ruby signature language.
  4. Duck typing

  5. Duck typing rubber duck

  6. def write(io, message) io << message end

  7. def write(io, message) io << message end write(StringIO.new, "Hello World")

    write(STDOUT, "Hello World")
  8. def write(io, message) io << message end write(StringIO.new, "Hello World")

    write(STDOUT, "Hello World") write([], "Hello World") write(1, 8)
  9. Duck typing • An untyped programming. • Assumes an interface

    (a set of methods) implicitly. • An object conforms to an interface if the object has all of the methods listed in the interface.
  10. Duck Scale Interface Implementation Languages Declared explicitly Declare in class

    definition Java Declared explicitly Declare after class definition Objective C, Swift, Haskell, Sorbet, ... Declared explicitly Conforms implicitly TypeScript, Steep Declared implicitly Conforms implicitly OCaml, C++, JavaScript, Ruby Less Duck More Duck
  11. Metaprogramming It means that a program can be designed to

    read, generate, analyze or transform other programs, and even modify itself while running. Metaprogramming, Wikipedia
  12. Metaprogramming eval(gets)

  13. Metaprogramming eval(gets) define_method :foo do ... end

  14. Metaprogramming class User < ApplicationRecord has_many :articles end eval(gets) define_method

    :foo do ... end
  15. Metaprogramming class User < ApplicationRecord has_many :articles end include Enumerable

    eval(gets) define_method :foo do ... end
  16. Metaprogramming class User < ApplicationRecord has_many :articles end include Enumerable

    private eval(gets) define_method :foo do ... end
  17. Metaprogramming require "pathname" load "./entrypoint.rb" class User < ApplicationRecord has_many

    :articles end include Enumerable private eval(gets) define_method :foo do ... end
  18. Metaprogramming require "pathname" load "./entrypoint.rb" StrongJSON.new do let :user, object(name:

    string, email: string?) end class User < ApplicationRecord has_many :articles end include Enumerable private eval(gets) define_method :foo do ... end
  19. • The metaprogramming primitives in Ruby are methods, which is

    subject to re-definition while execution. • What can we do? • Let users write annotations. • Simulate metaprogramming primitives in type checkers. • Using runtime information. • Typing rules plug-in?
  20. case x when String puts "x is a String" when

    Integer puts "x is a Integer" end
  21. case x when String puts "x is a String" when

    Integer puts "x is a Integer" end a = [1, "2", :3, 4.0] x, y*, z = a
  22. case x when String puts "x is a String" when

    Integer puts "x is a Integer" end a = [1, "2", :3, 4.0] x, y*, z = a spawn("rm", "-rf", "/") spawn("rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")
  23. case x when String puts "x is a String" when

    Integer puts "x is a Integer" end a = [1, "2", :3, 4.0] x, y*, z = a 1.instance_eval do self + 3 end spawn("rm", "-rf", "/") spawn("rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")
  24. case x when String puts "x is a String" when

    Integer puts "x is a Integer" end a = [1, "2", :3, 4.0] x, y*, z = a 1.instance_eval do self + 3 end 1.tap do break "foo" end spawn("rm", "-rf", "/") spawn("rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")
  25. case x when String puts "x is a String" when

    Integer puts "x is a Integer" end a = [1, "2", :3, 4.0] x, y*, z = a 1.instance_eval do self + 3 end 1.tap do break "foo" end [a,b].sort [1,2,3].map!(&:to_s) spawn("rm", "-rf", "/") spawn("rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")
  26. • We extend the type system. • Union types. •

    Tuple types, record types. • Method overloading. • Adding self type on block types. • Adding break type on block types? • Conditional types on method types??
  27. No Ruby library has type definition.

  28. Overview • Define the standard language to describe signature of

    Ruby program. • Ruby will ship with stdlib signatures. • Gems will ship with their signatures. • Type checkers will use the signature to know the type of libraries.
  29. Goals 1. Define the syntax of signature language. 2. Define

    the semantics of the signature language. 3. Implement a library to manipulate Ruby signatures. 4. Provide the signature of Ruby standard library. 5. Encourage another type checker development.
  30. https://github.com/soutaro/ruby-signature

  31. $ rbi list $ rbi methods --singleton ::Object $ rbi

    -r pathname method ::Object Pathname
  32. None
  33. class Array[A] def map: [X] { (A) -> X }

    -> Array[X] | -> Enumerable[A, self] ... end
  34. Signature language • Defines structure of Ruby programs statically. •

    No program execution, no metaprogramming. • Expressive enough to describe type of Ruby methods. • Easy to read/write and intuitive.
  35. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
  36. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
  37. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
  38. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
  39. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
  40. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
  41. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
  42. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
  43. Ruby 㱻 Signature class ClassInstanceType include Application def sub(s) self.class.new(

    name: name, args: args.map {|arg| arg.sub(s) } ) end end class ClassInstanceType include Application def sub: (_Substitution) -> self end interface _Substitution def sub: (type) -> type end
  44. Mixin? • include/prepend/extend in rbi are syntax, not method calls.

    • No confusion by re-definition. • Almost compatible semantics with Ruby (Module). • We have attr_reader/attr_writer/attr_accessor, and they are syntax too. • And private/public
  45. Types Integer singleton(Integer) Class instance/singleton

  46. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type
  47. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type Integer | String Union types
  48. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type Integer | String Union types any Dynamic type
  49. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type Integer | String Union types any Dynamic type nil bool void Base types
  50. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type Integer | String Union types any Dynamic type nil bool void Base types Integer? Optional type
  51. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type Integer | String Union types any Dynamic type nil bool void Base types Integer? Optional type 1 :hello "world" Singleton types
  52. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type Integer | String Union types any Dynamic type nil bool void Base types Integer? Optional type 1 :hello "world" Singleton types [Integer, String] { name: String, email: String? } Tuple and record types
  53. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type Integer | String Union types any Dynamic type nil bool void Base types Integer? Optional type 1 :hello "world" Singleton types [Integer, String] { name: String, email: String? } Tuple and record types ^(Integer, String) -> void Proc type
  54. Method types (Integer, ?bool, *String, any, name: String, ?email: String?,

    **String) -> S (String name, ?email: String? email) -> void
  55. Method types (Integer, ?bool, *String, any, name: String, ?email: String?,

    **String) -> S (String name, ?email: String? email) -> void (Integer) { (Integer) -> void } -> String (Integer) ?{ (Integer) -> void } -> String
  56. Method types (Integer, ?bool, *String, any, name: String, ?email: String?,

    **String) -> S (String name, ?email: String? email) -> void (Integer) { (Integer) -> void } -> String (Integer) ?{ (Integer) -> void } -> String [X] { (A) -> X } -> Array[X] | -> Enumerable[A, self]
  57. def self.open: (String | Integer path, ?String | Integer mode,

    ?Integer perm, ?external_encoding: Encoding | String, ?internal_encoding: Encoding | String, ?encoding: Encoding | String, ?mode: String | Integer, ?textmode: bool, ?binmode: bool, ?autoclose: bool) -> IO | [X] (String | Integer path, ?String | Integer mode, ?Integer perm, ?external_encoding: Encoding | String, ?internal_encoding: Encoding | String, ?encoding: Encoding | String, ?mode: String | Integer, ?textmode: bool, ?binmode: bool, ?autoclose: bool) { (IO) -> X } -> X
  58. Open class • Add methods to existing classes/modules using extension

    construct. • Keep class/module declarations closed in signatures. • Does this define new class, or modify an existing class? extension Kernel (Pathname) def Pathname: (String) -> Pathname end
  59. Local things • You can declare the instance variables and

    private methods in signature. • Ruby allows accessing them through mixin/inheritance. • If it is a part of external API, write them. class User @name: String @email: String? @phone: String? def send: (String) -> void private def send_email: (String)-> void def send_sms: (String) -> void end
  60. Integration with type checkers • There are at least four

    type checking tools with incompatible type systems. • Sorbet, RDL: Nominal subtyping • Steep: Structural subtyping. • type-profiler: No subtyping yet. • Parametrized type checking semantics with axioms.
  61. T <: void T <: bool S <: T |

    S T <: T | S [T, S] <: ::Array[T | S] S <: T Axiom of union types Axiom of void and bool types Axiom of tuple types? Type checkers will define their own subtyping relations, but should follow the axioms. S <: S? nil <: S? Axiom of optional types ::String <: ::Object Axiom of subclassing
  62. T <: void T <: bool S <: T |

    S T <: T | S [T, S] <: ::Array[T | S] S <: T Axiom of union types Axiom of void and bool types Axiom of tuple types? Type checkers will define their own subtyping relations, but should follow the axioms. S <: S? nil <: S? Axiom of optional types ::String <: ::Object Axiom of subclassing
  63. Missing features • Mechanism to reuse parameter types. • Mechanism

    to share type of existing gems.
  64. Sharing gem types • What can we do for existing

    gems without signatures? (if the authors don't ship with signatures?) • TypeScript (DefinitelyTyped) • Community managed type definitions of npm packages. source "https://ruby-signatures.org" do gem "rails" end
  65. Sharing gem types • What can we do for existing

    gems without signatures? (if the authors don't ship with signatures?) • TypeScript (DefinitelyTyped) • Community managed type definitions of npm packages. source "https://ruby-signatures.org" do gem "rails" end A prefix to avoid name conflict.ʢୄʣ
  66. Recap • Type signatures of existing Ruby libraries are essential.

    • We define the standard type signature for Ruby libraries. • You can write types of your library. • Type checking is optional, but the signature helps developers to understand precisely how the API of your library is. • Give us your feedbacks!