Regression testing for RuboCop

7bc6612fa20296bf652f6b0357db81c1?s=47 pocke
March 28, 2019

Regression testing for RuboCop

7bc6612fa20296bf652f6b0357db81c1?s=128

pocke

March 28, 2019
Tweet

Transcript

  1. Regression testing for RuboCop Meguro.rb #25 28 Mar. 2019

  2. pp self • Masataka “Pocke” Kuwabara • Work for Bit

    Journey, Inc.
  3. Why do we need the regression test for RuboCop?

  4. Because RuboCop is buggy, especially with: • NOT being a

    default setting. ◦ The core team and many users use the default settings, but other settings are used by less people. • complex Ruby code. ◦ Ruby is a complex language. There are mysterious codes in the world. ◦ Cop’s author often overlooks edge cases.
  5. So, we need tests for default settings and mysterious Ruby

    code!
  6. What is the regression test?

  7. What is the regression test? • Proposed: rubocop-hq/rubocop#6805 • Under

    development: pocke/rubocop-regression-test
  8. What is the regression test? • It executes RuboCop ◦

    with all setting combinations, and ◦ in real Ruby projects, that have many / mysterious / cool Ruby code.
  9. With all setting combinations

  10. With all setting combinations • Many cops have EnforcedStyle and

    SupportedStyles attributes. • The test configures all options. • It is based on pocke/gry.
  11. EnforcedStyle and SupportedStyles example Style/StringLiterals: EnforcedStyle: single_quotes SupportedStyles: - single_quotes

    - double_quotes
  12. Combination hell: AlignHash Layout/AlignHash: EnforcedHashRocketStyle: key SupportedHashRocketStyles: [key, separator, table]

    EnforcedColonStyle: key SupportedColonStyles: [key, separator, table] EnforcedLastArgumentHashStyle: always_inspect SupportedLastArgumentHashStyles: [always_inspect, always_ignore, ignore_implicit, ignore_explicit] 3 * 3 * 4 = 36 patterns
  13. Real Ruby projects

  14. Real Ruby projects • Test with edge cases ◦ tric/trick2013,

    2015 and 2018. ◦ ruby/spec • Test with large code base ◦ ruby/ruby ◦ gitlabhq/gitlabhq ◦ etc.
  15. Bug examples found by the test

  16. rubocop-hq/rubocop#6802 • Bug of AutoCorrection ◦ %I[#{foo}] => [:”#{foo}”] •

    It was found with tric/trick2015/shinh/entry.rb
  17. rubocop-hq/rubocop#6804, #6803 • Bug of AutoCorrection ◦ -␣10000 => -10_000

    ◦ 12345e6 => 12_345e6 • It was found with tric/trick2015/yoshi-taka/entry.rb
  18. Current State

  19. Current State • The test passed on trick2013, 2015 and

    2018. • It ran only on my local PC.
  20. Production problems

  21. Production problems • It takes a long time. ◦ Parallelize?

    Optimize? Machine power? • CI configuration. • How to report errors.
  22. In The Future

  23. In The Future • Test AutoCorrection dynamically. ◦ Currently it

    tests only SyntaxError, but I hope to make it test RuntimeError. ◦ e.g. $ rubocop -a && rspec # $? == 0 Thank you for listening!