Regression testing for RuboCop

Talked in Meguro.rb#25 https://megurorb.connpass.com/event/124735/
original: https://docs.google.com/presentation/d/1anN6aolrcEML2GBQ8Veib1E-fa4YkXMqp4BXpB0aciY/edit?usp=sharing

Transcript

1. Regression
   testing for
   RuboCop
   Meguro.rb #25
   28 Mar. 2019
   

   View Slide

2. pp self
   ● Masataka “Pocke” Kuwabara
   ● Work for Bit Journey, Inc.
   

   View Slide

3. Why do we need the regression test for
   RuboCop?
   

   View Slide

4. Because RuboCop is buggy, especially with:
   ● NOT being a default setting.
   ○ The core team and many users use the default
   settings, but other settings are used by less people.
   ● complex Ruby code.
   ○ Ruby is a complex language. There are mysterious
   codes in the world.
   ○ Cop’s author often overlooks edge cases.
   

   View Slide

5. So, we need tests for
   default settings and
   mysterious Ruby code!
   

   View Slide

6. What is the regression test?
   

   View Slide

7. What is the regression test?
   ● Proposed: rubocop-hq/rubocop#6805
   ● Under development:
   pocke/rubocop-regression-test
   

   View Slide

8. What is the regression test?
   ● It executes RuboCop
   ○ with all setting combinations, and
   ○ in real Ruby projects, that have many /
   mysterious / cool Ruby code.
   

   View Slide

9. With all setting
   combinations
   

   View Slide

10. With all setting combinations
    ● Many cops have EnforcedStyle and
    SupportedStyles attributes.
    ● The test configures all options.
    ● It is based on pocke/gry.
    

    View Slide

11. EnforcedStyle and SupportedStyles example
    Style/StringLiterals:
    EnforcedStyle: single_quotes
    SupportedStyles:
    - single_quotes
    - double_quotes
    

    View Slide

12. Combination hell: AlignHash
    Layout/AlignHash:
    EnforcedHashRocketStyle: key
    SupportedHashRocketStyles: [key, separator, table]
    EnforcedColonStyle: key
    SupportedColonStyles: [key, separator, table]
    EnforcedLastArgumentHashStyle: always_inspect
    SupportedLastArgumentHashStyles: [always_inspect,
    always_ignore, ignore_implicit, ignore_explicit]
    3 * 3 * 4 = 36
    patterns
    

    View Slide

13. Real Ruby projects
    

    View Slide

14. Real Ruby projects
    ● Test with edge cases
    ○ tric/trick2013, 2015 and 2018.
    ○ ruby/spec
    ● Test with large code base
    ○ ruby/ruby
    ○ gitlabhq/gitlabhq
    ○ etc.
    

    View Slide

15. Bug examples found by the test
    

    View Slide

16. rubocop-hq/rubocop#6802
    ● Bug of AutoCorrection
    ○ %I[#{foo}] => [:”#{foo}”]
    ● It was found with
    tric/trick2015/shinh/entry.rb
    

    View Slide

17. rubocop-hq/rubocop#6804, #6803
    ● Bug of AutoCorrection
    ○ -␣10000 => -10_000
    ○ 12345e6 => 12_345e6
    ● It was found with
    tric/trick2015/yoshi-taka/entry.rb
    

    View Slide

18. Current State
    

    View Slide

19. Current State
    ● The test passed on trick2013, 2015 and
    2018.
    ● It ran only on my local PC.
    

    View Slide

20. Production problems
    

    View Slide

21. Production problems
    ● It takes a long time.
    ○ Parallelize? Optimize? Machine power?
    ● CI configuration.
    ● How to report errors.
    

    View Slide

22. In The Future
    

    View Slide

23. In The Future
    ● Test AutoCorrection dynamically.
    ○ Currently it tests only SyntaxError, but I
    hope to make it test RuntimeError.
    ○ e.g. $ rubocop -a && rspec # $? == 0
    Thank you for listening!
    

    View Slide