$30 off During Our Annual Pro Sale. View Details »

Automated Type Contracts Generation

Automated Type Contracts Generation

Beauty and power of Ruby and Rails pays us back when it comes to finding bugs in large codebases. Static analysis is hindered by magic DSLs and patches.

We may annotate the code with YARD which also enables improved tooling such as code completion. Sadly, the benefits of this process rarely compensate for the effort.

In this session we’ll see a new approach to type annotations generation.
We'll learn how to obtain this data from runtime, to cope with DSLs and monkey patching, propose some tooling beyond YARD and create contracts like `(String, T) -> T`

YARV hacking and minimized DFAs included.

Valentin Fondaratov

September 19, 2017
Tweet

Other Decks in Programming

Transcript

  1. Automated

    Type Contracts
    Generation

    Valentin Fondaratov
    Hiroshima, September 2017
    A tale about better code analysis
    @valich
    @fondarat

    View Slide

  2. Why Types Matter
    An IDE Perspective

    View Slide

  3. An IDE Perspective

    • Where does the method go? (aka Resolution)

    View Slide

  4. An IDE Perspective

    • Where does the method go? (aka Resolution)
    • Bug prediction (aka NameError)

    View Slide

  5. An IDE Perspective

    • Where does the method go? (aka Resolution)
    • Bug prediction (aka NameError)
    • IDE goodness (aka Speed)

    View Slide

  6. An IDE Perspective

    • Where does the method go? (aka Resolution)
    • Bug prediction (aka NameError)
    • IDE goodness (aka Speed)
    • Rename refactoring (aka Safety),

    View Slide

  7. An IDE Perspective

    • Where does the method go? (aka Resolution)
    • Bug prediction (aka NameError)
    • IDE goodness (aka Speed)
    • Rename refactoring (aka Safety),

    View Slide

  8. Why Types Matter
    A non-IDE
    Perspective

    View Slide

  9. RuboCop

    is an industry standard
    solution

    © http://batsov.com/articles/2014/09/05/rubocop-logo/
    Inspecting 2184 files
    .......................................................................................................................
    ..............................................................................................................W........
    .......................................................................................................................
    .......................................................................................................................
    .......................................................................................................................
    .......................................................................................................................
    .......................................................................................................................
    .......................................................................................................................
    .......................................................................................................................
    .......................................................................................................................
    .......................................................................................................................
    .......................................................................................................................
    .......................................................................................................................
    .......................................................................................................................
    ....................................W..................................................................................
    .......................................................................................................................
    ........................
    Offenses:
    actionpack/lib/action_dispatch/system_test_case.rb:109:7: C: Use 2 (not 11) spaces for indentation.
    SystemTesting::Browser.new(using, screen_size)
    ^^^^^^^^^^^
    actionpack/lib/action_dispatch/system_test_case.rb:112:16: W: end at 112, 15 is not aligned with driver = if at 108, 6.
    end
    ^^^
    guides/rails_guides/markdown/renderer.rb:104:18: W: end at 104, 17 is not aligned with path = case at 97, 10.
    end
    ^^^
    guides/rails_guides/markdown/renderer.rb:106:1: C: Extra blank line detected.
    2184 files inspected, 4 offenses detected

    View Slide

  10. Missed errors

    RuboCop does its job quite well
    suggesting following Ruby Code Style.


    The real error on line 5 is missed,
    though.

    (downcase is not a method of Hash)
    Inspecting 1 file
    C
    Offenses:
    rubocop_fails.rb:1:1: C: Missing frozen string literal comment.
    x = "123"
    ^
    rubocop_fails.rb:1:5: C: Prefer single-quoted strings when you don't need string interpolation or special symbols.
    x = "123"
    ^^^^^
    rubocop_fails.rb:4:5: C: Space inside { missing.
    x = {:a => '1', :b => '2', :c => '3'}
    ^
    rubocop_fails.rb:4:6: C: Use the new Ruby 1.9 hash syntax.
    x = {:a => '1', :b => '2', :c => '3'}
    ^^^^^
    rubocop_fails.rb:4:17: C: Use the new Ruby 1.9 hash syntax.
    x = {:a => '1', :b => '2', :c => '3'}
    ^^^^^
    rubocop_fails.rb:4:28: C: Use the new Ruby 1.9 hash syntax.
    x = {:a => '1', :b => '2', :c => '3'}
    ^^^^^
    rubocop_fails.rb:4:37: C: Space inside } missing.
    x = {:a => '1', :b => '2', :c => '3'}
    ^
    1 file inspected, 7 offenses detected

    View Slide

  11. Static analysis
    can detect more

    RubyMine, for example, can detect
    such errors.
    Notice that this unresolved warning 

    is not screaming red. We’ll see why.

    View Slide

  12. Ruby DSLs

    are hard

    to analyse

    Ruby core features, readability and
    elegance, have a price. Inability

    to properly verify the programs 

    is one of the compromises.

    What type does @photo variable have?

    View Slide

  13. “Beware of bugs in the above code;

    I have only proved it correct, not tried it”
    Donald E. Knuth

    View Slide

  14. “Beware of bugs in the above code;

    I have only proved it correct, not tried it”
    Donald E. Knuth
    “Program testing can be used 

    to show the presence of bugs, 

    but never to show their absence!”
    Edsger W. Dijkstra

    View Slide

  15. Coverage is a lie

    Any composition of 2+ branching methods requires cross product of branches tests.
    Without static analysis and proper inspections we are limited by
    • Running tests (uncaught bugs we call “regressions”),
    • Looking through the code with debugger and verifying manually (after each change??),
    • Testing with users in production :)

    View Slide

  16. There is so much more

    we can have by running

    the tests
    Than just checking the answers

    View Slide

  17. Analysing
    RSpec::Matchers

    View Slide

  18. Analysing
    RSpec::Matchers

    View Slide

  19. Analysing
    RSpec::Matchers

    View Slide

  20. //demo/gif

    View Slide

  21. //demo/gif

    View Slide

  22. How it works

    The process behind the magic:

    View Slide

  23. How it works

    The process behind the magic:
    1. Attach to Ruby VM to collect the types for all calls,

    View Slide

  24. How it works

    The process behind the magic:
    1. Attach to Ruby VM to collect the types for all calls,
    2. Transform raw call data into type contracts,

    View Slide

  25. How it works

    The process behind the magic:
    1. Attach to Ruby VM to collect the types for all calls,
    2. Transform raw call data into type contracts,
    3. Collect and share the data.

    View Slide

  26. How it works

    The process behind the magic:
    1. Attach to Ruby VM to collect the types for all calls,
    2. Transform raw call data into type contracts,
    3. Collect and share the data.

    View Slide

  27. Retrieving the data
    with TracePoint API

    TracePoint is a class allowing 

    to hook several Ruby VM events

    like method calls and returns 

    and get any data through Binding

    View Slide

  28. Retrieving the data
    with TracePoint API

    TracePoint is a class allowing 

    to hook several Ruby VM events

    like method calls and returns 

    and get any data through Binding

    View Slide

  29. Retrieving the data
    with TracePoint API

    TracePoint is a class allowing 

    to hook several Ruby VM events

    like method calls and returns 

    and get any data through Binding

    View Slide

  30. Retrieving the data
    with TracePoint API

    TracePoint is a class allowing 

    to hook several Ruby VM events

    like method calls and returns 

    and get any data through Binding

    View Slide

  31. Retrieving the data
    with TracePoint API

    TracePoint is a class allowing 

    to hook several Ruby VM events

    like method calls and returns 

    and get any data through Binding

    View Slide

  32. Unspecified arguments

    One can’t distinguish default parameter
    values from the passed ones.

    If a method is defined dynamically,
    there is no way to derive which types

    will be passed.


    What type does foo() return?
    (Int) -> Int
    (String) -> String
    foo() = ?

    View Slide

  33. Optional parameters

    YARV compiles code into the bytecode.
    Note that instructions for filling in 

    the default values are present, 

    independent on usages.
    When optional parameters are passed,

    VM just skips these instructions.

    View Slide

  34. Optional parameters

    YARV compiles code into the bytecode.
    Note that instructions for filling in 

    the default values are present, 

    independent on usages.
    When optional parameters are passed,

    VM just skips these instructions.

    View Slide

  35. Optional parameters

    YARV compiles code into the bytecode.
    Note that instructions for filling in 

    the default values are present, 

    independent on usages.
    When optional parameters are passed,

    VM just skips these instructions.

    View Slide

  36. Optional parameters

    YARV compiles code into the bytecode.
    Note that instructions for filling in 

    the default values are present, 

    independent on usages.
    When optional parameters are passed,

    VM just skips these instructions.

    View Slide

  37. Optional parameters

    rb_control_frame_t
    const VALUE *pc;
    const rb_iseq_t *iseq;
    PC

    View Slide

  38. Optional parameters

    rb_control_frame_t
    const VALUE *pc;
    const rb_iseq_t *iseq;
    PC

    View Slide

  39. View Slide

  40. View Slide

  41. Resources

    http://patshaughnessy.net/ruby-under-a-microscope


    https://silverhammermba.github.io/emberb/c/


    https://github.com/ruby/ruby

    View Slide

  42. How it works

    The process behind the magic:
    1. Attach to Ruby VM to collect the types for all calls,
    2. Transform raw call data into type contracts,
    3. Collect and share the data.

    View Slide

  43. Type Tuples

    str.split(pattern=nil, [limit]) -> anArray

    View Slide

  44. Type Tuples

    str.split(pattern=nil, [limit]) -> anArray

    View Slide

  45. Type Tuples

    str.split(pattern=nil, [limit]) -> anArray
    Method calls Type tuples

    View Slide

  46. Too much data

    str.split(pattern=nil, [limit]) -> anArray

    View Slide

  47. Too much data

    str.split(pattern=nil, [limit]) -> anArray

    View Slide

  48. Too much data

    str.split(pattern=nil, [limit]) -> anArray
    https://boardgamegeek.com/image/1955740/game-goose
    http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg

    View Slide

  49. Too much data

    str.split(pattern=nil, [limit]) -> anArray
    https://boardgamegeek.com/image/1955740/game-goose
    http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg

    View Slide

  50. Too much data

    str.split(pattern=nil, [limit]) -> anArray
    https://boardgamegeek.com/image/1955740/game-goose
    http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg

    View Slide

  51. Too much data

    str.split(pattern=nil, [limit]) -> anArray
    https://boardgamegeek.com/image/1955740/game-goose
    http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg

    View Slide

  52. Too much data

    str.split(pattern=nil, [limit]) -> anArray
    split(, nil)
    ?
    !
    https://boardgamegeek.com/image/1955740/game-goose
    http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg

    View Slide

  53. Too much data

    str.split(pattern=nil, [limit]) -> anArray
    ?
    ?
    split(, )
    https://boardgamegeek.com/image/1955740/game-goose
    http://www.megahowto.com/wp-content/uploads/2010/11/how-to-make-board-games.jpg

    View Slide

  54. Too much data

    View Slide

  55. Too much data

    View Slide

  56. Too much data

    View Slide

  57. A worse example

    View Slide

  58. Template automatons

    View Slide

  59. Equality masks

    View Slide

  60. Equality masks

    Param0 Param1 Param2
    Equals to Param0
    Equals to Param1
    {} {1} {11}

    View Slide

  61. Equality masks

    View Slide

  62. Merge

    View Slide

  63. Merge

    View Slide

  64. Merge

    View Slide

  65. Merge

    +
    Merge

    View Slide

  66. Merge

    +
    Merge
    Quack inference?

    View Slide

  67. //demo/

    View Slide

  68. //demo/

    View Slide

  69. How it works

    The process behind the magic:
    1. Attach to Ruby VM to collect the types for all calls,
    2. Transform raw call data into type contracts,
    3. Collect and share the data.

    View Slide

  70. View Slide

  71. Q. How do I collect the data?

    View Slide

  72. Q. How do I collect the data?
    A. Run tests.

    View Slide

  73. Q. How do I collect the data?
    A. Run tests.

    View Slide

  74. Q. How do I collect the data?
    A. Run tests.
    Q. How do I collect more data?

    View Slide

  75. Q. How do I collect the data?
    A. Run tests.
    Q. How do I collect more data?
    A. Run more tests.

    View Slide

  76. Q. How do I collect the data?
    A. Run tests.
    Q. How do I collect more data?
    A. Run more tests.

    View Slide

  77. Q. How do I collect the data?
    A. Run tests.
    Q. How do I collect more data?
    A. Run more tests.
    Q. How do I collect so much data that the type contracts
    obtain exhaustiveness, i.e. become true?

    View Slide

  78. Q. How do I collect the data?
    A. Run tests.
    Q. How do I collect more data?
    A. Run more tests.
    Q. How do I collect so much data that the type contracts
    obtain exhaustiveness, i.e. become true?
    A. Cooperate with others to run even MORE TESTS.

    View Slide

  79. Community effort

    Project1 Project2 … ProjectN

    View Slide

  80. Community effort

    Project1 Project2 … ProjectN

    View Slide

  81. Community effort

    Project1 Project2 … ProjectN
    Contracts1 Contracts2 ContractsN
    Spec Spec Spec

    View Slide

  82. Community effort

    Project1 Project2 … ProjectN
    Contracts1 Contracts2 ContractsN
    Spec Spec Spec
    “Devise Annotated”, ‘4.2’
    M E R G E

    View Slide

  83. Community effort

    Contract

    Diffs
    Rails 5
    4.2
    4.1
    capybara

    +selenium
    2.12.0
    2.11.0
    “Ruby weekly”.tar.gz
    Rubyists
    Cloud Storage

    View Slide

  84. Community effort

    Contract

    Diffs
    Rails 5
    4.2
    4.1
    capybara

    +selenium
    2.12.0
    2.11.0
    “Ruby weekly”.tar.gz
    Rubyists
    Cloud Storage

    View Slide

  85. A single team may not have

    100% test coverage.
    A community is likely to have.

    View Slide

  86. View Slide

  87. Tooling

    View Slide

  88. Tooling

    • IDE goodness

    View Slide

  89. Tooling

    • IDE goodness
    • Code verification

    View Slide

  90. Tooling

    • IDE goodness
    • Code verification
    • Guided Optimization

    View Slide

  91. Contribute

    This might be a viable alternative to explicit

    type annotations which is contradictory.


    We have a chance to make Ruby 

    much more “static” for analysis

    while preserving its

    power and beauty.


    View Slide

  92. jetbrains.com
    Thank you
    for your attention

    jetbrains.com/ruby/
    github.com/valich
    github.com/jetbrains/ruby-type-inference

    View Slide