The challenges behind Ruby type checking

Transcript

1. The challenges behind Ruby type checking Soutaro Matsumoto @soutaro

2. Steep changelog • 10 releases since last RubyKaigi. (0.10.0, Mar

   3, 2019) • Many bugfixes and type system improvements. • User experience improvements: $ steep watch • Working for editor integration using LSP. • Done: On the fly type checking, expression type hover. • WIP: Jump to definition/declaration, completion. $ gem install steep

3. Outline • Ruby type checking difficulties review. • Introduction to

   the Ruby signature language.

4. Duck typing

5. Duck typing rubber duck

6. def write(io, message) io << message end

7. def write(io, message) io << message end write(StringIO.new, "Hello World")

   write(STDOUT, "Hello World")

8. def write(io, message) io << message end write(StringIO.new, "Hello World")

   write(STDOUT, "Hello World") write([], "Hello World") write(1, 8)

9. Duck typing • An untyped programming. • Assumes an interface

   (a set of methods) implicitly. • An object conforms to an interface if the object has all of the methods listed in the interface.

10. Duck Scale Interface Implementation Languages Declared explicitly Declare in class

    definition Java Declared explicitly Declare after class definition Objective C, Swift, Haskell, Sorbet, ... Declared explicitly Conforms implicitly TypeScript, Steep Declared implicitly Conforms implicitly OCaml, C++, JavaScript, Ruby Less Duck More Duck

11. Metaprogramming It means that a program can be designed to

    read, generate, analyze or transform other programs, and even modify itself while running. Metaprogramming, Wikipedia

12. Metaprogramming eval(gets)

13. Metaprogramming eval(gets) define_method :foo do ... end

14. Metaprogramming class User < ApplicationRecord has_many :articles end eval(gets) define_method

    :foo do ... end

15. Metaprogramming class User < ApplicationRecord has_many :articles end include Enumerable

    eval(gets) define_method :foo do ... end

16. Metaprogramming class User < ApplicationRecord has_many :articles end include Enumerable

    private eval(gets) define_method :foo do ... end

17. Metaprogramming require "pathname" load "./entrypoint.rb" class User < ApplicationRecord has_many

    :articles end include Enumerable private eval(gets) define_method :foo do ... end

18. Metaprogramming require "pathname" load "./entrypoint.rb" StrongJSON.new do let :user, object(name:

    string, email: string?) end class User < ApplicationRecord has_many :articles end include Enumerable private eval(gets) define_method :foo do ... end

19. • The metaprogramming primitives in Ruby are methods, which is

    subject to re-definition while execution. • What can we do? • Let users write annotations. • Simulate metaprogramming primitives in type checkers. • Using runtime information. • Typing rules plug-in?

20. case x when String puts "x is a String" when

    Integer puts "x is a Integer" end

21. case x when String puts "x is a String" when

    Integer puts "x is a Integer" end a = [1, "2", :3, 4.0] x, y*, z = a

22. case x when String puts "x is a String" when

    Integer puts "x is a Integer" end a = [1, "2", :3, 4.0] x, y*, z = a spawn("rm", "-rf", "/") spawn("rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")

23. case x when String puts "x is a String" when

    Integer puts "x is a Integer" end a = [1, "2", :3, 4.0] x, y*, z = a 1.instance_eval do self + 3 end spawn("rm", "-rf", "/") spawn("rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")

24. case x when String puts "x is a String" when

    Integer puts "x is a Integer" end a = [1, "2", :3, 4.0] x, y*, z = a 1.instance_eval do self + 3 end 1.tap do break "foo" end spawn("rm", "-rf", "/") spawn("rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")

25. case x when String puts "x is a String" when

    Integer puts "x is a Integer" end a = [1, "2", :3, 4.0] x, y*, z = a 1.instance_eval do self + 3 end 1.tap do break "foo" end [a,b].sort [1,2,3].map!(&:to_s) spawn("rm", "-rf", "/") spawn("rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")

26. • We extend the type system. • Union types. •

    Tuple types, record types. • Method overloading. • Adding self type on block types. • Adding break type on block types? • Conditional types on method types??

27. No Ruby library has type definition.

28. Overview • Define the standard language to describe signature of

    Ruby program. • Ruby will ship with stdlib signatures. • Gems will ship with their signatures. • Type checkers will use the signature to know the type of libraries.

29. Goals 1. Define the syntax of signature language. 2. Define

    the semantics of the signature language. 3. Implement a library to manipulate Ruby signatures. 4. Provide the signature of Ruby standard library. 5. Encourage another type checker development.

30. https://github.com/soutaro/ruby-signature

31. $ rbi list $ rbi methods --singleton ::Object $ rbi

    -r pathname method ::Object Pathname
32. None

33. class Array[A] def map: [X] { (A) -> X }

    -> Array[X] | -> Enumerable[A, self] ... end

34. Signature language • Defines structure of Ruby programs statically. •

    No program execution, no metaprogramming. • Expressive enough to describe type of Ruby methods. • Easy to read/write and intuitive.

35. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end

36. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end

37. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end

38. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end

39. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end

40. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end

41. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end

42. class Set[A] def initialize: (_Each[A] objects) -> void def add:

    (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end

43. Ruby 㱻 Signature class ClassInstanceType include Application def sub(s) self.class.new(

    name: name, args: args.map {|arg| arg.sub(s) } ) end end class ClassInstanceType include Application def sub: (_Substitution) -> self end interface _Substitution def sub: (type) -> type end

44. Mixin? • include/prepend/extend in rbi are syntax, not method calls.

    • No confusion by re-definition. • Almost compatible semantics with Ruby (Module). • We have attr_reader/attr_writer/attr_accessor, and they are syntax too. • And private/public

45. Types Integer singleton(Integer) Class instance/singleton

46. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type

47. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type Integer | String Union types

48. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type Integer | String Union types any Dynamic type

49. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type Integer | String Union types any Dynamic type nil bool void Base types

50. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type Integer | String Union types any Dynamic type nil bool void Base types Integer? Optional type

51. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type Integer | String Union types any Dynamic type nil bool void Base types Integer? Optional type 1 :hello "world" Singleton types

52. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type Integer | String Union types any Dynamic type nil bool void Base types Integer? Optional type 1 :hello "world" Singleton types [Integer, String] { name: String, email: String? } Tuple and record types

53. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class

    type Integer | String Union types any Dynamic type nil bool void Base types Integer? Optional type 1 :hello "world" Singleton types [Integer, String] { name: String, email: String? } Tuple and record types ^(Integer, String) -> void Proc type

54. Method types (Integer, ?bool, *String, any, name: String, ?email: String?,

    **String) -> S (String name, ?email: String? email) -> void

55. Method types (Integer, ?bool, *String, any, name: String, ?email: String?,

    **String) -> S (String name, ?email: String? email) -> void (Integer) { (Integer) -> void } -> String (Integer) ?{ (Integer) -> void } -> String

56. Method types (Integer, ?bool, *String, any, name: String, ?email: String?,

    **String) -> S (String name, ?email: String? email) -> void (Integer) { (Integer) -> void } -> String (Integer) ?{ (Integer) -> void } -> String [X] { (A) -> X } -> Array[X] | -> Enumerable[A, self]

57. def self.open: (String | Integer path, ?String | Integer mode,

    ?Integer perm, ?external_encoding: Encoding | String, ?internal_encoding: Encoding | String, ?encoding: Encoding | String, ?mode: String | Integer, ?textmode: bool, ?binmode: bool, ?autoclose: bool) -> IO | [X] (String | Integer path, ?String | Integer mode, ?Integer perm, ?external_encoding: Encoding | String, ?internal_encoding: Encoding | String, ?encoding: Encoding | String, ?mode: String | Integer, ?textmode: bool, ?binmode: bool, ?autoclose: bool) { (IO) -> X } -> X

58. Open class • Add methods to existing classes/modules using extension

    construct. • Keep class/module declarations closed in signatures. • Does this define new class, or modify an existing class? extension Kernel (Pathname) def Pathname: (String) -> Pathname end

59. Local things • You can declare the instance variables and

    private methods in signature. • Ruby allows accessing them through mixin/inheritance. • If it is a part of external API, write them. class User @name: String @email: String? @phone: String? def send: (String) -> void private def send_email: (String)-> void def send_sms: (String) -> void end

60. Integration with type checkers • There are at least four

    type checking tools with incompatible type systems. • Sorbet, RDL: Nominal subtyping • Steep: Structural subtyping. • type-profiler: No subtyping yet. • Parametrized type checking semantics with axioms.

61. T <: void T <: bool S <: T |

    S T <: T | S [T, S] <: ::Array[T | S] S <: T Axiom of union types Axiom of void and bool types Axiom of tuple types? Type checkers will define their own subtyping relations, but should follow the axioms. S <: S? nil <: S? Axiom of optional types ::String <: ::Object Axiom of subclassing

62. T <: void T <: bool S <: T |

    S T <: T | S [T, S] <: ::Array[T | S] S <: T Axiom of union types Axiom of void and bool types Axiom of tuple types? Type checkers will define their own subtyping relations, but should follow the axioms. S <: S? nil <: S? Axiom of optional types ::String <: ::Object Axiom of subclassing

63. Missing features • Mechanism to reuse parameter types. • Mechanism

    to share type of existing gems.

64. Sharing gem types • What can we do for existing

    gems without signatures? (if the authors don't ship with signatures?) • TypeScript (DefinitelyTyped) • Community managed type definitions of npm packages. source "https://ruby-signatures.org" do gem "rails" end

65. Sharing gem types • What can we do for existing

    gems without signatures? (if the authors don't ship with signatures?) • TypeScript (DefinitelyTyped) • Community managed type definitions of npm packages. source "https://ruby-signatures.org" do gem "rails" end A prefix to avoid name conflict.ʢୄʣ

66. Recap • Type signatures of existing Ruby libraries are essential.

    • We define the standard type signature for Ruby libraries. • You can write types of your library. • Type checking is optional, but the signature helps developers to understand precisely how the API of your library is. • Give us your feedbacks!