Regression test for RuboCop

https://docs.google.com/presentation/d/1RxuTLFtK7XZNmyau09V8W3Kl8-jAXm6jBjbO2DYrWsM/edit?usp=sharing
https://heiseirb.github.io/kaigi01/

Transcript

1. Regression Test for RuboCop 平成 Ruby 会議 01 14th Dec. 2019
2. pocke.display • Masataka “Pocke” Kuwabara • Work for Bit Journey, Inc. We develop Kibela. • rurema team member • RuboCop core team member
3. Kibela • Documentation SaaS ◦ Access https://kibe.la! • We use ◦ Ruby on Rails ◦ RuboCop ◦ GraphQL ◦ TypeScript ◦ React ◦ etc...
4. RuboCop core team member • Many patches (recently a few patches) • The author of ◦ rubocop-rake ◦ rubocop-extention-generator ◦ mry ◦ gry ◦ rubocop-rubycw ◦ rubocop-typed ◦ rubocop-regression-test ← Today’s topic
5. rurema team member • The Ruby Reference Manual project. ◦ Work under https://github.com/rurema • I write documents and review pull requests.
6. Rurema reviewing online meetup • We review rurema pull requests every Tuesday! ◦ Next meetup is 17th Dec. • Everyone can join the meetup! ◦ connpass: https://rurema-review.connpass.com/event/159182 ◦ slack: https://ruby-jp.github.io/ #rurema channel
7. Agenda • Why does RuboCop need the regression test? • What’s the regression test? • How to work. • How to contribute to RuboCop. • Future work.
8. Why does RuboCop need the regression test?
9. RuboCop breaks easily, with Ruby syntax • Ruby has too complex syntax. • Actually we do not understand Ruby syntax perfectly. • A cop, implemented by us, will overlooks edge cases of Ruby syntax. • Example: omitting parens
10. RuboCop breaks easily, with the configuration • RuboCop has many options. You can configure RuboCop for your style. • But it introduces complexity. • Not default option is not maintained enough.
11. Conclusion of this section • RuboCop breaks easily ◦ with Ruby syntax ◦ with configurations • The regression test is necessary to find bugs before releasing RuboCop.
12. What’s the regression test?
13. Developing under here • rubocop-hq/rubocop#6805 • https://github.com/pocke/rubocop-regression-test
14. What’s the regression test? • In short, it executes RuboCop ◦ with all option combinations, and ◦ in real Ruby projects. • Check exceptions.
15. Example • In project-a ◦ rubocop --config .rubocop.1.yml ◦ rubocop --config .rubocop.2.yml ◦ rubocop --config .rubocop.3.yml ◦ … • In project-b ◦ rubocop --config .rubocop.1.yml ◦ rubocop --config .rubocop.2.yml ◦ rubocop --config .rubocop.3.yml ◦ … • In project-c ◦ ...
16. What does check the test? • Check raised exceptions. ◦ Grep “An error occurred while ...” message. ◦ Do not check false positives.
17. How to work
18. What’s the regression test? (page 14) • In short, it executes RuboCop ◦ with all option combinations, and ◦ in real Ruby projects. • Check exceptions.
19. with all option combinations
20. with all option combinations • Many cops has EnforcedStyle option to specify style. • The style candidates is available as SupportedStyles.
21. Example of default configuration Style/StringLiterals: EnforcedStyle: single_quotes SupportedStyles: - single_quotes - double_quotes Layout/DotPosition: EnforcedStyle: leading SupportedStyles: - leading - trailing
22. Example of all options combinations # .rubocop.1.yml Style/StringLiterals: EnforcedStyle: single_quotes Layout/DotPosition: EnforcedStyle: leading # .rubocop.2.yml Style/StringLiterals: EnforcedStyle: double_quotes Layout/DotPosition: EnforcedStyle: trailing
23. in real Ruby projects
24. It tests RuboCop in real Ruby projects • Unit test contains only a few code. ◦ It may overlook. • Real Ruby projects contain many codes. ◦ We can test cop with many patterns.
25. It tests RuboCop in real Ruby projects • Now it tests with: ◦ tric/trick2013, 2015 and 2018 ◦ rubocop-hq/rubocop • In the future, I plan adding test with: ◦ ruby/spec (pocke/rubocop-regressiontest#16) ◦ ruby/ruby ◦ rails/rails ◦ gitlabhq/gitlabhq ◦ discourse/discourse
26. How to contribute to RuboCop
27. How to contribute to RuboCop • We have two contributing ways. ◦ Contribute to RuboCop ◦ Contribute to the regression test
28. Contribute to RuboCop
29. Contribute to RuboCop • Find bugs with the test on your codebase ↓ If you find bags, you can report them to RuboCop!
30. Execute the following command $ git clone https://github.com/pocke/rubocop-regression-t est $ cd rubocop-regression-test # Clone repo from GitHub and run the test $ ruby main.rb check OWNER/REPO # You can specify local directory also $ ruby main.rb check path/to/your/project/dir
31. Contribute to rubocop-regression -test
32. Contribute to RuboCop Regression Test • You can add more repositories to the test. ◦ Add them to CircleCI configuration. ◦ If it is green on CI, I can merge it. • Work for TODOs ◦ See the next section, “Future work”.
33. Future work
34. Test infinite loop in auto correction • Currently it ignores infinite loop in auto correction. ◦ Because it is too noisy. • But it also should detect and fix infinite loops.
35. More test for --auto-correct • Auto correction should generate code that have same meaning if the cop is marked as “safe”. • In short, test `rubocop -a && rspec`.
36. Test plugins • RuboCop has many plugins under the rubocop-hq organization. It should test them also. ◦ rubocop-rails ◦ rubocop-performance ◦ rubocop-rspec ◦ rubocop-rake ◦ and so on.