Automated Type Contracts Generation

Beauty and power of Ruby and Rails pays us back when it comes to finding bugs in large codebases. Static analysis is hindered by magic DSLs and patches.

We may annotate the code with YARD which also enables improved tooling such as code completion. Sadly, the benefits of this process rarely compensate for the effort.

In this session we’ll see a new approach to type annotations generation.
We'll learn how to obtain this data from runtime, to cope with DSLs and monkey patching, propose some tooling beyond YARD and create contracts like `(String, T) -> T`

YARV hacking and minimized DFAs included.

Transcript

1. Automated

   Type Contracts
   Generation
   —
   Valentin Fondaratov
   Hiroshima, September 2017
   A tale about better code analysis
   @valich
   @fondarat
   

   View Slide

2. Why Types Matter
   An IDE Perspective
   

   View Slide

3. An IDE Perspective
   —
   • Where does the method go? (aka Resolution)
   

   View Slide

4. An IDE Perspective
   —
   • Where does the method go? (aka Resolution)
   • Bug prediction (aka NameError)
   

   View Slide

5. An IDE Perspective
   —
   • Where does the method go? (aka Resolution)
   • Bug prediction (aka NameError)
   • IDE goodness (aka Speed)
   

   View Slide

6. An IDE Perspective
   —
   • Where does the method go? (aka Resolution)
   • Bug prediction (aka NameError)
   • IDE goodness (aka Speed)
   • Rename refactoring (aka Safety),
   

   View Slide

7. An IDE Perspective
   —
   • Where does the method go? (aka Resolution)
   • Bug prediction (aka NameError)
   • IDE goodness (aka Speed)
   • Rename refactoring (aka Safety),
   

   View Slide

8. Why Types Matter
   A non-IDE
   Perspective
   

   View Slide

9. RuboCop

   is an industry standard
   solution
   —
   © http://batsov.com/articles/2014/09/05/rubocop-logo/
   Inspecting 2184 files
   .......................................................................................................................
   ..............................................................................................................W........
   .......................................................................................................................
   .......................................................................................................................
   .......................................................................................................................
   .......................................................................................................................
   .......................................................................................................................
   .......................................................................................................................
   .......................................................................................................................
   .......................................................................................................................
   .......................................................................................................................
   .......................................................................................................................
   .......................................................................................................................
   .......................................................................................................................
   ....................................W..................................................................................
   .......................................................................................................................
   ........................
   Offenses:
   actionpack/lib/action_dispatch/system_test_case.rb:109:7: C: Use 2 (not 11) spaces for indentation.
   SystemTesting::Browser.new(using, screen_size)
   ^^^^^^^^^^^
   actionpack/lib/action_dispatch/system_test_case.rb:112:16: W: end at 112, 15 is not aligned with driver = if at 108, 6.
   end
   ^^^
   guides/rails_guides/markdown/renderer.rb:104:18: W: end at 104, 17 is not aligned with path = case at 97, 10.
   end
   ^^^
   guides/rails_guides/markdown/renderer.rb:106:1: C: Extra blank line detected.
   2184 files inspected, 4 offenses detected
   

   View Slide

10. Missed errors
    —
    RuboCop does its job quite well
    suggesting following Ruby Code Style.

    

    The real error on line 5 is missed,
    though.

    (downcase is not a method of Hash)
    Inspecting 1 file
    C
    Offenses:
    rubocop_fails.rb:1:1: C: Missing frozen string literal comment.
    x = "123"
    ^
    rubocop_fails.rb:1:5: C: Prefer single-quoted strings when you don't need string interpolation or special symbols.
    x = "123"
    ^^^^^
    rubocop_fails.rb:4:5: C: Space inside { missing.
    x = {:a => '1', :b => '2', :c => '3'}
    ^
    rubocop_fails.rb:4:6: C: Use the new Ruby 1.9 hash syntax.
    x = {:a => '1', :b => '2', :c => '3'}
    ^^^^^
    rubocop_fails.rb:4:17: C: Use the new Ruby 1.9 hash syntax.
    x = {:a => '1', :b => '2', :c => '3'}
    ^^^^^
    rubocop_fails.rb:4:28: C: Use the new Ruby 1.9 hash syntax.
    x = {:a => '1', :b => '2', :c => '3'}
    ^^^^^
    rubocop_fails.rb:4:37: C: Space inside } missing.
    x = {:a => '1', :b => '2', :c => '3'}
    ^
    1 file inspected, 7 offenses detected
    

    View Slide

11. Static analysis
    can detect more
    —
    RubyMine, for example, can detect
    such errors.
    Notice that this unresolved warning 

    is not screaming red. We’ll see why.
    

    View Slide

12. Ruby DSLs

    are hard

    to analyse
    —
    Ruby core features, readability and
    elegance, have a price. Inability

    to properly verify the programs 

    is one of the compromises.

    What type does @photo variable have?
    

    View Slide

13. “Beware of bugs in the above code;

    I have only proved it correct, not tried it”
    Donald E. Knuth
    

    View Slide

14. “Beware of bugs in the above code;

    I have only proved it correct, not tried it”
    Donald E. Knuth
    “Program testing can be used 

    to show the presence of bugs, 

    but never to show their absence!”
    Edsger W. Dijkstra
    

    View Slide

15. Coverage is a lie
    —
    Any composition of 2+ branching methods requires cross product of branches tests.
    Without static analysis and proper inspections we are limited by
    • Running tests (uncaught bugs we call “regressions”),
    • Looking through the code with debugger and verifying manually (after each change??),
    • Testing with users in production :)
    

    View Slide

16. There is so much more

    we can have by running

    the tests
    Than just checking the answers
    

    View Slide

17. Analysing
    RSpec::Matchers
    —
    

    View Slide

18. Analysing
    RSpec::Matchers
    —
    

    View Slide

19. Analysing
    RSpec::Matchers
    —
    

    View Slide

20. //demo/gif
    —
    

    View Slide

21. //demo/gif
    —
    

    View Slide

22. How it works
    —
    The process behind the magic:
    

    View Slide

23. How it works
    —
    The process behind the magic:
    1. Attach to Ruby VM to collect the types for all calls,
    

    View Slide

24. How it works
    —
    The process behind the magic:
    1. Attach to Ruby VM to collect the types for all calls,
    2. Transform raw call data into type contracts,
    

    View Slide

25. How it works
    —
    The process behind the magic:
    1. Attach to Ruby VM to collect the types for all calls,
    2. Transform raw call data into type contracts,
    3. Collect and share the data.
    

    View Slide

26. How it works
    —
    The process behind the magic:
    1. Attach to Ruby VM to collect the types for all calls,
    2. Transform raw call data into type contracts,
    3. Collect and share the data.
    

    View Slide

27. Retrieving the data
    with TracePoint API
    —
    TracePoint is a class allowing 

    to hook several Ruby VM events

    like method calls and returns 

    and get any data through Binding
    

    View Slide

28. Retrieving the data
    with TracePoint API
    —
    TracePoint is a class allowing 

    to hook several Ruby VM events

    like method calls and returns 

    and get any data through Binding
    

    View Slide

29. Retrieving the data
    with TracePoint API
    —
    TracePoint is a class allowing 

    to hook several Ruby VM events

    like method calls and returns 

    and get any data through Binding
    

    View Slide

30. Retrieving the data
    with TracePoint API
    —
    TracePoint is a class allowing 

    to hook several Ruby VM events

    like method calls and returns 

    and get any data through Binding
    

    View Slide

31. Retrieving the data
    with TracePoint API
    —
    TracePoint is a class allowing 

    to hook several Ruby VM events

    like method calls and returns 

    and get any data through Binding
    

    View Slide

32. Unspecified arguments
    —
    One can’t distinguish default parameter
    values from the passed ones.

    If a method is defined dynamically,
    there is no way to derive which types

    will be passed.

    

    What type does foo() return?
    (Int) -> Int
    (String) -> String
    foo() = ?
    

    View Slide

33. Optional parameters
    —
    YARV compiles code into the bytecode.
    Note that instructions for filling in 

    the default values are present, 

    independent on usages.
    When optional parameters are passed,

    VM just skips these instructions.
    

    View Slide

34. Optional parameters
    —
    YARV compiles code into the bytecode.
    Note that instructions for filling in 

    the default values are present, 

    independent on usages.
    When optional parameters are passed,

    VM just skips these instructions.
    

    View Slide

35. Optional parameters
    —
    YARV compiles code into the bytecode.
    Note that instructions for filling in 

    the default values are present, 

    independent on usages.
    When optional parameters are passed,

    VM just skips these instructions.
    

    View Slide

36. Optional parameters
    —
    YARV compiles code into the bytecode.
    Note that instructions for filling in 

    the default values are present, 

    independent on usages.
    When optional parameters are passed,

    VM just skips these instructions.
    

    View Slide

37. Optional parameters
    —
    rb_control_frame_t
    const VALUE *pc;
    const rb_iseq_t *iseq;
    PC
    

    View Slide

38. Optional parameters
    —
    rb_control_frame_t
    const VALUE *pc;
    const rb_iseq_t *iseq;
    PC
    

    View Slide

39. View Slide

40. View Slide

41. Resources
    —
    http://patshaughnessy.net/ruby-under-a-microscope

    

    https://silverhammermba.github.io/emberb/c/

    

    https://github.com/ruby/ruby
    

    View Slide

42. How it works
    —
    The process behind the magic:
    1. Attach to Ruby VM to collect the types for all calls,
    2. Transform raw call data into type contracts,
    3. Collect and share the data.
    

    View Slide

43. Type Tuples
    —
    str.split(pattern=nil, [limit]) -> anArray
    

    View Slide

44. Type Tuples
    —
    str.split(pattern=nil, [limit]) -> anArray
    

    View Slide

45. Type Tuples
    —
    str.split(pattern=nil, [limit]) -> anArray
    Method calls Type tuples
    

    View Slide

46. Too much data
    —
    str.split(pattern=nil, [limit]) -> anArray
    

    View Slide

47. Too much data
    —
    str.split(pattern=nil, [limit]) -> anArray
    

    View Slide

48. Too much data
    —
    str.split(pattern=nil, [limit]) -> anArray
    https://boardgamegeek.com/image/1955740/game-goose
    http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg
    

    View Slide

49. Too much data
    —
    str.split(pattern=nil, [limit]) -> anArray
    https://boardgamegeek.com/image/1955740/game-goose
    http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg
    

    View Slide

50. Too much data
    —
    str.split(pattern=nil, [limit]) -> anArray
    https://boardgamegeek.com/image/1955740/game-goose
    http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg
    

    View Slide

51. Too much data
    —
    str.split(pattern=nil, [limit]) -> anArray
    https://boardgamegeek.com/image/1955740/game-goose
    http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg
    

    View Slide

52. Too much data
    —
    str.split(pattern=nil, [limit]) -> anArray
    split(, nil)
    ?
    !
    https://boardgamegeek.com/image/1955740/game-goose
    http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg
    

    View Slide

53. Too much data
    —
    str.split(pattern=nil, [limit]) -> anArray
    ?
    ?
    split(, )
    https://boardgamegeek.com/image/1955740/game-goose
    http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg
    

    View Slide

54. Too much data
    —
    

    View Slide

55. Too much data
    —
    

    View Slide

56. Too much data
    —
    

    View Slide

57. A worse example
    —
    

    View Slide

58. Template automatons
    —
    

    View Slide

59. Equality masks
    —
    

    View Slide

60. Equality masks
    —
    Param0 Param1 Param2
    Equals to Param0
    Equals to Param1
    {} {1} {11}
    

    View Slide

61. Equality masks
    —
    

    View Slide

62. Merge
    —
    

    View Slide

63. Merge
    —
    

    View Slide

64. Merge
    —
    

    View Slide

65. Merge
    —
    +
    Merge
    

    View Slide

66. Merge
    —
    +
    Merge
    Quack inference?
    

    View Slide

67. //demo/
    —
    

    View Slide

68. //demo/
    —
    

    View Slide

69. How it works
    —
    The process behind the magic:
    1. Attach to Ruby VM to collect the types for all calls,
    2. Transform raw call data into type contracts,
    3. Collect and share the data.
    

    View Slide

70. View Slide

71. Q. How do I collect the data?
    

    View Slide

72. Q. How do I collect the data?
    A. Run tests.
    

    View Slide

73. Q. How do I collect the data?
    A. Run tests.
    

    View Slide

74. Q. How do I collect the data?
    A. Run tests.
    Q. How do I collect more data?
    

    View Slide

75. Q. How do I collect the data?
    A. Run tests.
    Q. How do I collect more data?
    A. Run more tests.
    

    View Slide

76. Q. How do I collect the data?
    A. Run tests.
    Q. How do I collect more data?
    A. Run more tests.
    

    View Slide

77. Q. How do I collect the data?
    A. Run tests.
    Q. How do I collect more data?
    A. Run more tests.
    Q. How do I collect so much data that the type contracts
    obtain exhaustiveness, i.e. become true?
    

    View Slide

78. Q. How do I collect the data?
    A. Run tests.
    Q. How do I collect more data?
    A. Run more tests.
    Q. How do I collect so much data that the type contracts
    obtain exhaustiveness, i.e. become true?
    A. Cooperate with others to run even MORE TESTS.
    

    View Slide

79. Community effort
    —
    Project1 Project2 … ProjectN
    

    View Slide

80. Community effort
    —
    Project1 Project2 … ProjectN
    

    View Slide

81. Community effort
    —
    Project1 Project2 … ProjectN
    Contracts1 Contracts2 ContractsN
    Spec Spec Spec
    

    View Slide

82. Community effort
    —
    Project1 Project2 … ProjectN
    Contracts1 Contracts2 ContractsN
    Spec Spec Spec
    “Devise Annotated”, ‘4.2’
    M E R G E
    

    View Slide

83. Community effort
    —
    Contract

    Diffs
    Rails 5
    4.2
    4.1
    capybara

    +selenium
    2.12.0
    2.11.0
    “Ruby weekly”.tar.gz
    Rubyists
    Cloud Storage
    

    View Slide

84. Community effort
    —
    Contract

    Diffs
    Rails 5
    4.2
    4.1
    capybara

    +selenium
    2.12.0
    2.11.0
    “Ruby weekly”.tar.gz
    Rubyists
    Cloud Storage
    

    View Slide

85. A single team may not have

    100% test coverage.
    A community is likely to have.
    

    View Slide

86. View Slide

87. Tooling
    —
    

    View Slide

88. Tooling
    —
    • IDE goodness
    

    View Slide

89. Tooling
    —
    • IDE goodness
    • Code verification
    

    View Slide

90. Tooling
    —
    • IDE goodness
    • Code verification
    • Guided Optimization
    

    View Slide

91. Contribute
    —
    This might be a viable alternative to explicit

    type annotations which is contradictory.

    

    We have a chance to make Ruby 

    much more “static” for analysis

    while preserving its

    power and beauty.

    

    View Slide

92. jetbrains.com
    Thank you
    for your attention
    —
    jetbrains.com/ruby/
    github.com/valich
    github.com/jetbrains/ruby-type-inference
    

    View Slide