Automated Type Contracts Generation

Beauty and power of Ruby and Rails pays us back when it comes to finding bugs in large codebases. Static analysis is hindered by magic DSLs and patches.

We may annotate the code with YARD which also enables improved tooling such as code completion. Sadly, the benefits of this process rarely compensate for the effort.

In this session we’ll see a new approach to type annotations generation.
We'll learn how to obtain this data from runtime, to cope with DSLs and monkey patching, propose some tooling beyond YARD and create contracts like `(String, T) -> T`

YARV hacking and minimized DFAs included.

Transcript

1. Automated
 Type Contracts Generation — Valentin Fondaratov Hiroshima, September 2017 A tale about better code analysis @valich @fondarat
2. Why Types Matter An IDE Perspective
3. An IDE Perspective — • Where does the method go? (aka Resolution)
4. An IDE Perspective — • Where does the method go? (aka Resolution) • Bug prediction (aka NameError)
5. An IDE Perspective — • Where does the method go? (aka Resolution) • Bug prediction (aka NameError) • IDE goodness (aka Speed)
6. An IDE Perspective — • Where does the method go? (aka Resolution) • Bug prediction (aka NameError) • IDE goodness (aka Speed) • Rename refactoring (aka Safety),
7. An IDE Perspective — • Where does the method go? (aka Resolution) • Bug prediction (aka NameError) • IDE goodness (aka Speed) • Rename refactoring (aka Safety),
8. Why Types Matter A non-IDE Perspective
9. RuboCop
 is an industry standard solution — © http://batsov.com/articles/2014/09/05/rubocop-logo/ Inspecting 2184 files ....................................................................................................................... ..............................................................................................................W........ ....................................................................................................................... ....................................................................................................................... ....................................................................................................................... ....................................................................................................................... ....................................................................................................................... ....................................................................................................................... ....................................................................................................................... ....................................................................................................................... ....................................................................................................................... ....................................................................................................................... ....................................................................................................................... ....................................................................................................................... ....................................W.................................................................................. ....................................................................................................................... ........................ Offenses: actionpack/lib/action_dispatch/system_test_case.rb:109:7: C: Use 2 (not 11) spaces for indentation. SystemTesting::Browser.new(using, screen_size) ^^^^^^^^^^^ actionpack/lib/action_dispatch/system_test_case.rb:112:16: W: end at 112, 15 is not aligned with driver = if at 108, 6. end ^^^ guides/rails_guides/markdown/renderer.rb:104:18: W: end at 104, 17 is not aligned with path = case at 97, 10. end ^^^ guides/rails_guides/markdown/renderer.rb:106:1: C: Extra blank line detected. 2184 files inspected, 4 offenses detected
10. Missed errors — RuboCop does its job quite well suggesting following Ruby Code Style.
 
 The real error on line 5 is missed, though.
 (downcase is not a method of Hash) Inspecting 1 file C Offenses: rubocop_fails.rb:1:1: C: Missing frozen string literal comment. x = "123" ^ rubocop_fails.rb:1:5: C: Prefer single-quoted strings when you don't need string interpolation or special symbols. x = "123" ^^^^^ rubocop_fails.rb:4:5: C: Space inside { missing. x = {:a => '1', :b => '2', :c => '3'} ^ rubocop_fails.rb:4:6: C: Use the new Ruby 1.9 hash syntax. x = {:a => '1', :b => '2', :c => '3'} ^^^^^ rubocop_fails.rb:4:17: C: Use the new Ruby 1.9 hash syntax. x = {:a => '1', :b => '2', :c => '3'} ^^^^^ rubocop_fails.rb:4:28: C: Use the new Ruby 1.9 hash syntax. x = {:a => '1', :b => '2', :c => '3'} ^^^^^ rubocop_fails.rb:4:37: C: Space inside } missing. x = {:a => '1', :b => '2', :c => '3'} ^ 1 file inspected, 7 offenses detected
11. Static analysis can detect more — RubyMine, for example, can detect such errors. Notice that this unresolved warning 
 is not screaming red. We’ll see why.
12. Ruby DSLs
 are hard
 to analyse — Ruby core features, readability and elegance, have a price. Inability
 to properly verify the programs 
 is one of the compromises.
 What type does @photo variable have?
13. “Beware of bugs in the above code;
 I have only proved it correct, not tried it” Donald E. Knuth
14. “Beware of bugs in the above code;
 I have only proved it correct, not tried it” Donald E. Knuth “Program testing can be used 
 to show the presence of bugs, 
 but never to show their absence!” Edsger W. Dijkstra
15. Coverage is a lie — Any composition of 2+ branching methods requires cross product of branches tests. Without static analysis and proper inspections we are limited by • Running tests (uncaught bugs we call “regressions”), • Looking through the code with debugger and verifying manually (after each change??), • Testing with users in production :)
16. There is so much more
 we can have by running
 the tests Than just checking the answers
17. Analysing RSpec::Matchers —
18. Analysing RSpec::Matchers —
19. Analysing RSpec::Matchers —
20. //demo/gif —
21. //demo/gif —
22. How it works — The process behind the magic:
23. How it works — The process behind the magic: 1. Attach to Ruby VM to collect the types for all calls,
24. How it works — The process behind the magic: 1. Attach to Ruby VM to collect the types for all calls, 2. Transform raw call data into type contracts,
25. How it works — The process behind the magic: 1. Attach to Ruby VM to collect the types for all calls, 2. Transform raw call data into type contracts, 3. Collect and share the data.
26. How it works — The process behind the magic: 1. Attach to Ruby VM to collect the types for all calls, 2. Transform raw call data into type contracts, 3. Collect and share the data.
27. Retrieving the data with TracePoint API — TracePoint is a class allowing 
 to hook several Ruby VM events
 like method calls and returns 
 and get any data through Binding
28. Retrieving the data with TracePoint API — TracePoint is a class allowing 
 to hook several Ruby VM events
 like method calls and returns 
 and get any data through Binding
29. Retrieving the data with TracePoint API — TracePoint is a class allowing 
 to hook several Ruby VM events
 like method calls and returns 
 and get any data through Binding
30. Retrieving the data with TracePoint API — TracePoint is a class allowing 
 to hook several Ruby VM events
 like method calls and returns 
 and get any data through Binding
31. Retrieving the data with TracePoint API — TracePoint is a class allowing 
 to hook several Ruby VM events
 like method calls and returns 
 and get any data through Binding
32. Unspecified arguments — One can’t distinguish default parameter values from the passed ones.
 If a method is defined dynamically, there is no way to derive which types
 will be passed.
 
 What type does foo() return? (Int) -> Int (String) -> String foo() = ?
33. Optional parameters — YARV compiles code into the bytecode. Note that instructions for filling in 
 the default values are present, 
 independent on usages. When optional parameters are passed,
 VM just skips these instructions.
34. Optional parameters — YARV compiles code into the bytecode. Note that instructions for filling in 
 the default values are present, 
 independent on usages. When optional parameters are passed,
 VM just skips these instructions.
35. Optional parameters — YARV compiles code into the bytecode. Note that instructions for filling in 
 the default values are present, 
 independent on usages. When optional parameters are passed,
 VM just skips these instructions.
36. Optional parameters — YARV compiles code into the bytecode. Note that instructions for filling in 
 the default values are present, 
 independent on usages. When optional parameters are passed,
 VM just skips these instructions.
37. Optional parameters — rb_control_frame_t const VALUE *pc; const rb_iseq_t *iseq; PC
38. Optional parameters — rb_control_frame_t const VALUE *pc; const rb_iseq_t *iseq; PC
41. Resources — http://patshaughnessy.net/ruby-under-a-microscope
 
 https://silverhammermba.github.io/emberb/c/
 
 https://github.com/ruby/ruby
42. How it works — The process behind the magic: 1. Attach to Ruby VM to collect the types for all calls, 2. Transform raw call data into type contracts, 3. Collect and share the data.
43. Type Tuples — str.split(pattern=nil, [limit]) -> anArray
44. Type Tuples — str.split(pattern=nil, [limit]) -> anArray
45. Type Tuples — str.split(pattern=nil, [limit]) -> anArray Method calls Type tuples
46. Too much data — str.split(pattern=nil, [limit]) -> anArray
47. Too much data — str.split(pattern=nil, [limit]) -> anArray
48. Too much data — str.split(pattern=nil, [limit]) -> anArray https://boardgamegeek.com/image/1955740/game-goose http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg
49. Too much data — str.split(pattern=nil, [limit]) -> anArray https://boardgamegeek.com/image/1955740/game-goose http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg
50. Too much data — str.split(pattern=nil, [limit]) -> anArray https://boardgamegeek.com/image/1955740/game-goose http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg
51. Too much data — str.split(pattern=nil, [limit]) -> anArray https://boardgamegeek.com/image/1955740/game-goose http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg
52. Too much data — str.split(pattern=nil, [limit]) -> anArray split(<String>, nil) ? ! https://boardgamegeek.com/image/1955740/game-goose http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg
53. Too much data — str.split(pattern=nil, [limit]) -> anArray ? ? split(<String>, <String>) https://boardgamegeek.com/image/1955740/game-goose http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg
54. Too much data —
55. Too much data —
56. Too much data —
57. A worse example —
58. Template automatons —
59. Equality masks —
60. Equality masks — Param0 Param1 Param2 Equals to Param0 Equals to Param1 {} {1} {11}
61. Equality masks —
62. Merge —
63. Merge —
64. Merge —
65. Merge — + Merge
66. Merge — + Merge Quack inference?
67. //demo/ —
68. //demo/ —
69. How it works — The process behind the magic: 1. Attach to Ruby VM to collect the types for all calls, 2. Transform raw call data into type contracts, 3. Collect and share the data.
71. Q. How do I collect the data?
72. Q. How do I collect the data? A. Run tests.
73. Q. How do I collect the data? A. Run tests.
74. Q. How do I collect the data? A. Run tests. Q. How do I collect more data?
75. Q. How do I collect the data? A. Run tests. Q. How do I collect more data? A. Run more tests.
76. Q. How do I collect the data? A. Run tests. Q. How do I collect more data? A. Run more tests.
77. Q. How do I collect the data? A. Run tests. Q. How do I collect more data? A. Run more tests. Q. How do I collect so much data that the type contracts obtain exhaustiveness, i.e. become true?
78. Q. How do I collect the data? A. Run tests. Q. How do I collect more data? A. Run more tests. Q. How do I collect so much data that the type contracts obtain exhaustiveness, i.e. become true? A. Cooperate with others to run even MORE TESTS.
79. Community effort — Project1 Project2 … ProjectN
80. Community effort — Project1 Project2 … ProjectN
81. Community effort — Project1 Project2 … ProjectN Contracts1 Contracts2 ContractsN Spec Spec Spec
82. Community effort — Project1 Project2 … ProjectN Contracts1 Contracts2 ContractsN Spec Spec Spec “Devise Annotated”, ‘4.2’ M E R G E
83. Community effort — Contract
 Diffs Rails 5 4.2 4.1 capybara
 +selenium 2.12.0 2.11.0 “Ruby weekly”.tar.gz Rubyists Cloud Storage
84. Community effort — Contract
 Diffs Rails 5 4.2 4.1 capybara
 +selenium 2.12.0 2.11.0 “Ruby weekly”.tar.gz Rubyists Cloud Storage
85. A single team may not have
 100% test coverage. A community is likely to have.
87. Tooling —
88. Tooling — • IDE goodness
89. Tooling — • IDE goodness • Code verification
90. Tooling — • IDE goodness • Code verification • Guided Optimization
91. Contribute — This might be a viable alternative to explicit
 type annotations which is contradictory.
 
 We have a chance to make Ruby 
 much more “static” for analysis
 while preserving its
 power and beauty.

92. jetbrains.com Thank you for your attention — jetbrains.com/ruby/ github.com/valich github.com/jetbrains/ruby-type-inference