Regression testing for RuboCop

Talked in Meguro.rb#25 https://megurorb.connpass.com/event/124735/
original: https://docs.google.com/presentation/d/1anN6aolrcEML2GBQ8Veib1E-fa4YkXMqp4BXpB0aciY/edit?usp=sharing

Transcript

1. Regression testing for RuboCop Meguro.rb #25 28 Mar. 2019
2. pp self • Masataka “Pocke” Kuwabara • Work for Bit Journey, Inc.
3. Why do we need the regression test for RuboCop?
4. Because RuboCop is buggy, especially with: • NOT being a default setting. ◦ The core team and many users use the default settings, but other settings are used by less people. • complex Ruby code. ◦ Ruby is a complex language. There are mysterious codes in the world. ◦ Cop’s author often overlooks edge cases.
5. So, we need tests for default settings and mysterious Ruby code!
6. What is the regression test?
7. What is the regression test? • Proposed: rubocop-hq/rubocop#6805 • Under development: pocke/rubocop-regression-test
8. What is the regression test? • It executes RuboCop ◦ with all setting combinations, and ◦ in real Ruby projects, that have many / mysterious / cool Ruby code.
9. With all setting combinations
10. With all setting combinations • Many cops have EnforcedStyle and SupportedStyles attributes. • The test configures all options. • It is based on pocke/gry.
11. EnforcedStyle and SupportedStyles example Style/StringLiterals: EnforcedStyle: single_quotes SupportedStyles: - single_quotes - double_quotes
12. Combination hell: AlignHash Layout/AlignHash: EnforcedHashRocketStyle: key SupportedHashRocketStyles: [key, separator, table] EnforcedColonStyle: key SupportedColonStyles: [key, separator, table] EnforcedLastArgumentHashStyle: always_inspect SupportedLastArgumentHashStyles: [always_inspect, always_ignore, ignore_implicit, ignore_explicit] 3 * 3 * 4 = 36 patterns
13. Real Ruby projects
14. Real Ruby projects • Test with edge cases ◦ tric/trick2013, 2015 and 2018. ◦ ruby/spec • Test with large code base ◦ ruby/ruby ◦ gitlabhq/gitlabhq ◦ etc.
15. Bug examples found by the test
16. rubocop-hq/rubocop#6802 • Bug of AutoCorrection ◦ %I[#{foo}] => [:”#{foo}”] • It was found with tric/trick2015/shinh/entry.rb
17. rubocop-hq/rubocop#6804, #6803 • Bug of AutoCorrection ◦ -␣10000 => -10_000 ◦ 12345e6 => 12_345e6 • It was found with tric/trick2015/yoshi-taka/entry.rb
18. Current State
19. Current State • The test passed on trick2013, 2015 and 2018. • It ran only on my local PC.
20. Production problems
21. Production problems • It takes a long time. ◦ Parallelize? Optimize? Machine power? • CI configuration. • How to report errors.
22. In The Future
23. In The Future • Test AutoCorrection dynamically. ◦ Currently it tests only SyntaxError, but I hope to make it test RuntimeError. ◦ e.g. $ rubocop -a && rspec # $? == 0 Thank you for listening!