Slide 1
Slide 1 text
Regression
testing for
RuboCop
Meguro.rb #25
28 Mar. 2019
Slide 2
Slide 2 text
pp self
● Masataka “Pocke” Kuwabara
● Work for Bit Journey, Inc.
Slide 3
Slide 3 text
Why do we need the regression test for
RuboCop?
Slide 4
Slide 4 text
Because RuboCop is buggy, especially with:
● NOT being a default setting.
○ The core team and many users use the default
settings, but other settings are used by less people.
● complex Ruby code.
○ Ruby is a complex language. There are mysterious
codes in the world.
○ Cop’s author often overlooks edge cases.
Slide 5
Slide 5 text
So, we need tests for
default settings and
mysterious Ruby code!
Slide 6
Slide 6 text
What is the regression test?
Slide 7
Slide 7 text
What is the regression test?
● Proposed: rubocop-hq/rubocop#6805
● Under development:
pocke/rubocop-regression-test
Slide 8
Slide 8 text
What is the regression test?
● It executes RuboCop
○ with all setting combinations, and
○ in real Ruby projects, that have many /
mysterious / cool Ruby code.
Slide 9
Slide 9 text
With all setting
combinations
Slide 10
Slide 10 text
With all setting combinations
● Many cops have EnforcedStyle and
SupportedStyles attributes.
● The test configures all options.
● It is based on pocke/gry.
Slide 11
Slide 11 text
EnforcedStyle and SupportedStyles example
Style/StringLiterals:
EnforcedStyle: single_quotes
SupportedStyles:
- single_quotes
- double_quotes
Slide 12
Slide 12 text
Combination hell: AlignHash
Layout/AlignHash:
EnforcedHashRocketStyle: key
SupportedHashRocketStyles: [key, separator, table]
EnforcedColonStyle: key
SupportedColonStyles: [key, separator, table]
EnforcedLastArgumentHashStyle: always_inspect
SupportedLastArgumentHashStyles: [always_inspect,
always_ignore, ignore_implicit, ignore_explicit]
3 * 3 * 4 = 36
patterns
Slide 13
Slide 13 text
Real Ruby projects
Slide 14
Slide 14 text
Real Ruby projects
● Test with edge cases
○ tric/trick2013, 2015 and 2018.
○ ruby/spec
● Test with large code base
○ ruby/ruby
○ gitlabhq/gitlabhq
○ etc.
Slide 15
Slide 15 text
Bug examples found by the test
Slide 16
Slide 16 text
rubocop-hq/rubocop#6802
● Bug of AutoCorrection
○ %I[#{foo}] => [:”#{foo}”]
● It was found with
tric/trick2015/shinh/entry.rb
Slide 17
Slide 17 text
rubocop-hq/rubocop#6804, #6803
● Bug of AutoCorrection
○ -␣10000 => -10_000
○ 12345e6 => 12_345e6
● It was found with
tric/trick2015/yoshi-taka/entry.rb
Slide 18
Slide 18 text
Current State
Slide 19
Slide 19 text
Current State
● The test passed on trick2013, 2015 and
2018.
● It ran only on my local PC.
Slide 20
Slide 20 text
Production problems
Slide 21
Slide 21 text
Production problems
● It takes a long time.
○ Parallelize? Optimize? Machine power?
● CI configuration.
● How to report errors.
Slide 22
Slide 22 text
In The Future
Slide 23
Slide 23 text
In The Future
● Test AutoCorrection dynamically.
○ Currently it tests only SyntaxError, but I
hope to make it test RuntimeError.
○ e.g. $ rubocop -a && rspec # $? == 0
Thank you for listening!