The challenges behind Ruby type checking

RubyKaigi 2019, Fukuoka.

Transcript

1. The challenges behind Ruby type checking Soutaro Matsumoto @soutaro
2. Steep changelog • 10 releases since last RubyKaigi. (0.10.0, Mar 3, 2019) • Many bugfixes and type system improvements. • User experience improvements: $ steep watch • Working for editor integration using LSP. • Done: On the fly type checking, expression type hover. • WIP: Jump to definition/declaration, completion. $ gem install steep
3. Outline • Ruby type checking difficulties review. • Introduction to the Ruby signature language.
4. Duck typing
5. Duck typing rubber duck
6. def write(io, message) io << message end
7. def write(io, message) io << message end write(StringIO.new, "Hello World") write(STDOUT, "Hello World")
8. def write(io, message) io << message end write(StringIO.new, "Hello World") write(STDOUT, "Hello World") write([], "Hello World") write(1, 8)
9. Duck typing • An untyped programming. • Assumes an interface (a set of methods) implicitly. • An object conforms to an interface if the object has all of the methods listed in the interface.
10. Duck Scale Interface Implementation Languages Declared explicitly Declare in class definition Java Declared explicitly Declare after class definition Objective C, Swift, Haskell, Sorbet, ... Declared explicitly Conforms implicitly TypeScript, Steep Declared implicitly Conforms implicitly OCaml, C++, JavaScript, Ruby Less Duck More Duck
11. Metaprogramming It means that a program can be designed to read, generate, analyze or transform other programs, and even modify itself while running. Metaprogramming, Wikipedia
12. Metaprogramming eval(gets)
13. Metaprogramming eval(gets) define_method :foo do ... end
14. Metaprogramming class User < ApplicationRecord has_many :articles end eval(gets) define_method :foo do ... end
15. Metaprogramming class User < ApplicationRecord has_many :articles end include Enumerable eval(gets) define_method :foo do ... end
16. Metaprogramming class User < ApplicationRecord has_many :articles end include Enumerable private eval(gets) define_method :foo do ... end
17. Metaprogramming require "pathname" load "./entrypoint.rb" class User < ApplicationRecord has_many :articles end include Enumerable private eval(gets) define_method :foo do ... end
18. Metaprogramming require "pathname" load "./entrypoint.rb" StrongJSON.new do let :user, object(name: string, email: string?) end class User < ApplicationRecord has_many :articles end include Enumerable private eval(gets) define_method :foo do ... end
19. • The metaprogramming primitives in Ruby are methods, which is subject to re-definition while execution. • What can we do? • Let users write annotations. • Simulate metaprogramming primitives in type checkers. • Using runtime information. • Typing rules plug-in?
20. case x when String puts "x is a String" when Integer puts "x is a Integer" end
21. case x when String puts "x is a String" when Integer puts "x is a Integer" end a = [1, "2", :3, 4.0] x, y*, z = a
22. case x when String puts "x is a String" when Integer puts "x is a Integer" end a = [1, "2", :3, 4.0] x, y*, z = a spawn("rm", "-rf", "/") spawn("rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")
23. case x when String puts "x is a String" when Integer puts "x is a Integer" end a = [1, "2", :3, 4.0] x, y*, z = a 1.instance_eval do self + 3 end spawn("rm", "-rf", "/") spawn("rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")
24. case x when String puts "x is a String" when Integer puts "x is a Integer" end a = [1, "2", :3, 4.0] x, y*, z = a 1.instance_eval do self + 3 end 1.tap do break "foo" end spawn("rm", "-rf", "/") spawn("rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")
25. case x when String puts "x is a String" when Integer puts "x is a Integer" end a = [1, "2", :3, 4.0] x, y*, z = a 1.instance_eval do self + 3 end 1.tap do break "foo" end [a,b].sort [1,2,3].map!(&:to_s) spawn("rm", "-rf", "/") spawn("rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf /") spawn({"ENV"=>"/bin"}, "rm -rf .", chdir: "/")
26. • We extend the type system. • Union types. • Tuple types, record types. • Method overloading. • Adding self type on block types. • Adding break type on block types? • Conditional types on method types??
27. No Ruby library has type definition.
28. Overview • Define the standard language to describe signature of Ruby program. • Ruby will ship with stdlib signatures. • Gems will ship with their signatures. • Type checkers will use the signature to know the type of libraries.
29. Goals 1. Define the syntax of signature language. 2. Define the semantics of the signature language. 3. Implement a library to manipulate Ruby signatures. 4. Provide the signature of Ruby standard library. 5. Encourage another type checker development.
30. https://github.com/soutaro/ruby-signature
31. $ rbi list $ rbi methods --singleton ::Object $ rbi -r pathname method ::Object Pathname
33. class Array[A] def map: [X] { (A) -> X } -> Array[X] | -> Enumerable[A, self] ... end
34. Signature language • Defines structure of Ruby programs statically. • No program execution, no metaprogramming. • Expressive enough to describe type of Ruby methods. • Easy to read/write and intuitive.
35. class Set[A] def initialize: (_Each[A] objects) -> void def add: (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
36. class Set[A] def initialize: (_Each[A] objects) -> void def add: (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
37. class Set[A] def initialize: (_Each[A] objects) -> void def add: (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
38. class Set[A] def initialize: (_Each[A] objects) -> void def add: (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
39. class Set[A] def initialize: (_Each[A] objects) -> void def add: (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
40. class Set[A] def initialize: (_Each[A] objects) -> void def add: (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
41. class Set[A] def initialize: (_Each[A] objects) -> void def add: (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
42. class Set[A] def initialize: (_Each[A] objects) -> void def add: (A) -> self def add?: (A) -> self? include Enumerable[A, void] def each: { (A) -> void } -> Set[A] def self.`[]`: [X] (*X) -> Set[X] ... end interface _Each[A] def each: { (A) -> void } -> any end
43. Ruby 㱻 Signature class ClassInstanceType include Application def sub(s) self.class.new( name: name, args: args.map {|arg| arg.sub(s) } ) end end class ClassInstanceType include Application def sub: (_Substitution) -> self end interface _Substitution def sub: (type) -> type end
44. Mixin? • include/prepend/extend in rbi are syntax, not method calls. • No confusion by re-definition. • Almost compatible semantics with Ruby (Module). • We have attr_reader/attr_writer/attr_accessor, and they are syntax too. • And private/public
45. Types Integer singleton(Integer) Class instance/singleton
46. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class type
47. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class type Integer | String Union types
48. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class type Integer | String Union types any Dynamic type
49. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class type Integer | String Union types any Dynamic type nil bool void Base types
50. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class type Integer | String Union types any Dynamic type nil bool void Base types Integer? Optional type
51. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class type Integer | String Union types any Dynamic type nil bool void Base types Integer? Optional type 1 :hello "world" Singleton types
52. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class type Integer | String Union types any Dynamic type nil bool void Base types Integer? Optional type 1 :hello "world" Singleton types [Integer, String] { name: String, email: String? } Tuple and record types
53. Types Integer singleton(Integer) Class instance/singleton Array[Integer] Hash[Symbol, String] Generic class type Integer | String Union types any Dynamic type nil bool void Base types Integer? Optional type 1 :hello "world" Singleton types [Integer, String] { name: String, email: String? } Tuple and record types ^(Integer, String) -> void Proc type
54. Method types (Integer, ?bool, *String, any, name: String, ?email: String?, **String) -> S (String name, ?email: String? email) -> void
55. Method types (Integer, ?bool, *String, any, name: String, ?email: String?, **String) -> S (String name, ?email: String? email) -> void (Integer) { (Integer) -> void } -> String (Integer) ?{ (Integer) -> void } -> String
56. Method types (Integer, ?bool, *String, any, name: String, ?email: String?, **String) -> S (String name, ?email: String? email) -> void (Integer) { (Integer) -> void } -> String (Integer) ?{ (Integer) -> void } -> String [X] { (A) -> X } -> Array[X] | -> Enumerable[A, self]
57. def self.open: (String | Integer path, ?String | Integer mode, ?Integer perm, ?external_encoding: Encoding | String, ?internal_encoding: Encoding | String, ?encoding: Encoding | String, ?mode: String | Integer, ?textmode: bool, ?binmode: bool, ?autoclose: bool) -> IO | [X] (String | Integer path, ?String | Integer mode, ?Integer perm, ?external_encoding: Encoding | String, ?internal_encoding: Encoding | String, ?encoding: Encoding | String, ?mode: String | Integer, ?textmode: bool, ?binmode: bool, ?autoclose: bool) { (IO) -> X } -> X
58. Open class • Add methods to existing classes/modules using extension construct. • Keep class/module declarations closed in signatures. • Does this define new class, or modify an existing class? extension Kernel (Pathname) def Pathname: (String) -> Pathname end
59. Local things • You can declare the instance variables and private methods in signature. • Ruby allows accessing them through mixin/inheritance. • If it is a part of external API, write them. class User @name: String @email: String? @phone: String? def send: (String) -> void private def send_email: (String)-> void def send_sms: (String) -> void end
60. Integration with type checkers • There are at least four type checking tools with incompatible type systems. • Sorbet, RDL: Nominal subtyping • Steep: Structural subtyping. • type-profiler: No subtyping yet. • Parametrized type checking semantics with axioms.
61. T <: void T <: bool S <: T | S T <: T | S [T, S] <: ::Array[T | S] S <: T Axiom of union types Axiom of void and bool types Axiom of tuple types? Type checkers will define their own subtyping relations, but should follow the axioms. S <: S? nil <: S? Axiom of optional types ::String <: ::Object Axiom of subclassing
62. T <: void T <: bool S <: T | S T <: T | S [T, S] <: ::Array[T | S] S <: T Axiom of union types Axiom of void and bool types Axiom of tuple types? Type checkers will define their own subtyping relations, but should follow the axioms. S <: S? nil <: S? Axiom of optional types ::String <: ::Object Axiom of subclassing
63. Missing features • Mechanism to reuse parameter types. • Mechanism to share type of existing gems.
64. Sharing gem types • What can we do for existing gems without signatures? (if the authors don't ship with signatures?) • TypeScript (DefinitelyTyped) • Community managed type definitions of npm packages. source "https://ruby-signatures.org" do gem "rails" end
65. Sharing gem types • What can we do for existing gems without signatures? (if the authors don't ship with signatures?) • TypeScript (DefinitelyTyped) • Community managed type definitions of npm packages. source "https://ruby-signatures.org" do gem "rails" end A prefix to avoid name conflict.ʢୄʣ
66. Recap • Type signatures of existing Ruby libraries are essential. • We define the standard type signature for Ruby libraries. • You can write types of your library. • Type checking is optional, but the signature helps developers to understand precisely how the API of your library is. • Give us your feedbacks!