Regression test for RuboCop

https://docs.google.com/presentation/d/1RxuTLFtK7XZNmyau09V8W3Kl8-jAXm6jBjbO2DYrWsM/edit?usp=sharing
https://heiseirb.github.io/kaigi01/

Transcript

1. Regression Test for RuboCop
   平成
   Ruby
   会議
   01
   14th Dec. 2019
   

   View Slide

2. pocke.display
   ●
   Masataka “Pocke” Kuwabara
   ●
   Work for Bit Journey, Inc. We develop Kibela.
   ●
   rurema team member
   ●
   RuboCop core team member
   

   View Slide

3. Kibela
   ●
   Documentation SaaS
   ○
   Access https://kibe.la!
   ●
   We use
   ○
   Ruby on Rails
   ○
   RuboCop
   ○
   GraphQL
   ○
   TypeScript
   ○
   React
   ○
   etc...
   

   View Slide

4. RuboCop core team member
   ●
   Many patches (recently a few patches)
   ●
   The author of
   ○
   rubocop-rake
   ○
   rubocop-extention-generator
   ○
   mry
   ○
   gry
   ○
   rubocop-rubycw
   ○
   rubocop-typed
   ○
   rubocop-regression-test
   ←
   Today’s topic
   

   View Slide

5. rurema team member
   ●
   The Ruby Reference Manual project.
   ○
   Work under https://github.com/rurema
   ●
   I write documents and review pull requests.
   

   View Slide

6. Rurema reviewing online meetup
   ●
   We review rurema pull requests every Tuesday!
   ○
   Next meetup is 17th Dec.
   ●
   Everyone can join the meetup!
   ○
   connpass:
   https://rurema-review.connpass.com/event/159182
   ○
   slack: https://ruby-jp.github.io/ #rurema channel
   

   View Slide

7. Agenda
   ●
   Why does RuboCop need the regression test?
   ●
   What’s the regression test?
   ●
   How to work.
   ●
   How to contribute to RuboCop.
   ●
   Future work.
   

   View Slide

8. Why does RuboCop need the regression test?
   

   View Slide

9. RuboCop breaks easily, with Ruby syntax
   ●
   Ruby has too complex syntax.
   ●
   Actually we do not understand Ruby syntax
   perfectly.
   ●
   A cop, implemented by us, will overlooks edge
   cases of Ruby syntax.
   ●
   Example: omitting parens
   

   View Slide

10. RuboCop breaks easily, with the configuration
    ●
    RuboCop has many options. You can configure
    RuboCop for your style.
    ●
    But it introduces complexity.
    ●
    Not default option is not maintained enough.
    

    View Slide

11. Conclusion of this section
    ●
    RuboCop breaks easily
    ○
    with Ruby syntax
    ○
    with configurations
    ●
    The regression test is necessary to find bugs
    before releasing RuboCop.
    

    View Slide

12. What’s the regression test?
    

    View Slide

13. Developing under here
    ●
    rubocop-hq/rubocop#6805
    ●
    https://github.com/pocke/rubocop-regression-test
    

    View Slide

14. What’s the regression test?
    ●
    In short, it executes RuboCop
    ○
    with all option combinations, and
    ○
    in real Ruby projects.
    ●
    Check exceptions.
    

    View Slide

15. Example
    ●
    In project-a
    ○
    rubocop --config .rubocop.1.yml
    ○
    rubocop --config .rubocop.2.yml
    ○
    rubocop --config .rubocop.3.yml
    ○
    …
    ●
    In project-b
    ○
    rubocop --config .rubocop.1.yml
    ○
    rubocop --config .rubocop.2.yml
    ○
    rubocop --config .rubocop.3.yml
    ○
    …
    ●
    In project-c
    ○
    ...
    

    View Slide

16. What does check the test?
    ●
    Check raised exceptions.
    ○
    Grep “An error occurred while ...” message.
    ○
    Do not check false positives.
    

    View Slide

17. How to work
    

    View Slide

18. What’s the regression test? (page 14)
    ●
    In short, it executes RuboCop
    ○
    with all option combinations, and
    ○
    in real Ruby projects.
    ●
    Check exceptions.
    

    View Slide

19. with all option
    combinations
    

    View Slide

20. with all option combinations
    ●
    Many cops has EnforcedStyle option to specify
    style.
    ●
    The style candidates is available as
    SupportedStyles.
    

    View Slide

21. Example of default configuration
    Style/StringLiterals:
    EnforcedStyle: single_quotes
    SupportedStyles:
    - single_quotes
    - double_quotes
    Layout/DotPosition:
    EnforcedStyle: leading
    SupportedStyles:
    - leading
    - trailing
    

    View Slide

22. Example of all options combinations
    # .rubocop.1.yml
    Style/StringLiterals:
    EnforcedStyle: single_quotes
    Layout/DotPosition:
    EnforcedStyle: leading
    # .rubocop.2.yml
    Style/StringLiterals:
    EnforcedStyle: double_quotes
    Layout/DotPosition:
    EnforcedStyle: trailing
    

    View Slide

23. in real Ruby
    projects
    

    View Slide

24. It tests RuboCop in real Ruby projects
    ●
    Unit test contains only a few code.
    ○
    It may overlook.
    ●
    Real Ruby projects contain many codes.
    ○
    We can test cop with many patterns.
    

    View Slide

25. It tests RuboCop in real Ruby projects
    ●
    Now it tests with:
    ○
    tric/trick2013, 2015 and 2018
    ○
    rubocop-hq/rubocop
    ●
    In the future, I plan adding test with:
    ○
    ruby/spec (pocke/rubocop-regressiontest#16)
    ○
    ruby/ruby
    ○
    rails/rails
    ○
    gitlabhq/gitlabhq
    ○
    discourse/discourse
    

    View Slide

26. How to contribute to RuboCop
    

    View Slide

27. How to contribute to RuboCop
    ●
    We have two contributing ways.
    ○
    Contribute to RuboCop
    ○
    Contribute to the regression test
    

    View Slide

28. Contribute to
    RuboCop
    

    View Slide

29. Contribute to RuboCop
    ●
    Find bugs with the test on your codebase
    ↓
    If you find bags, you can report them to
    RuboCop!
    

    View Slide

30. Execute the following command
    $ git clone
    https://github.com/pocke/rubocop-regression-t
    est
    $ cd rubocop-regression-test
    # Clone repo from GitHub and run the test
    $ ruby main.rb check OWNER/REPO
    # You can specify local directory also
    $ ruby main.rb check path/to/your/project/dir
    

    View Slide

31. Contribute to
    rubocop-regression
    -test
    

    View Slide

32. Contribute to RuboCop Regression Test
    ●
    You can add more repositories to the test.
    ○
    Add them to CircleCI configuration.
    ○
    If it is green on CI, I can merge it.
    ●
    Work for TODOs
    ○
    See the next section, “Future work”.
    

    View Slide

33. Future work
    

    View Slide

34. Test infinite loop in auto correction
    ●
    Currently it ignores infinite loop in auto
    correction.
    ○
    Because it is too noisy.
    ●
    But it also should detect and fix infinite loops.
    

    View Slide

35. More test for --auto-correct
    ●
    Auto correction should generate code that have
    same meaning if the cop is marked as “safe”.
    ●
    In short, test `rubocop -a && rspec`.
    

    View Slide

36. Test plugins
    ●
    RuboCop has many plugins under the
    rubocop-hq organization. It should test them
    also.
    ○
    rubocop-rails
    ○
    rubocop-performance
    ○
    rubocop-rspec
    ○
    rubocop-rake
    ○
    and so on.
    

    View Slide