Don't
 Repeat
 Your Code Review Suggestions Soutaro Matsumoto
 @soutaro Ruby X Elixir Conf Taiwan 2018

@soutaro • CTO at SideCI, Inc. • Interested in program analysis • Develops a type checking tool for Ruby called Steep • Will have a talk at RubyKaigi 2018

Code Review • Give feedback to code • To improve code structure and quality • To find bugs • To fix styling issues • Share knowledge about frameworks/projects • Using tools like Pull Request in GitHub

Code Review Example

Code Review Example

Code Review Example

Code Review Automation • LINT tools, like RuboCop, can make some review suggestions • We can automate some of the reviewing process using the tools

Code Review Automation Lint/InheritException: EnforcedStyle: standard_error

Code Review Automation

Code Review Automation

Project Specific Rules • General tools do not support them • Rules about APIs defined in the project • RuboCop cannot know about #with_octokit • Does not match/maybe conflicting with general best practices • Does anyone have .env.sample in your repo?

Let's make a RuboCop plugin!

Let's make a RuboCop plugin!

RuboCop Plugin • RuboCop provides plugin mechanism for custom rules • You can write arbitrary Ruby code as a plugin • We rarely develop the plugin for project specific rules • It needs a lot of work (high cost), but the plugin is only for a few people (low return)

RuboCop Plugin • RuboCop provides plugin mechanism for custom rules • You can write arbitrary Ruby code as a plugin • We rarely develop the plugin for project specific rules • It needs a lot of work (high cost), but the plugin is only for a few people (low return) What if we can make the development cost low enough?

Requirements • Encourages users to add rules • Writing rules • Reading rules • Testing rules • Rules to find out method calls and print messages to show possible improvements

Key Idea • Define a declarative language to define rules • A restricted language to define patterns of code detect • The language will be much easier to write/read than Ruby • Users will define their own rules using the language

• When the program refers ENV constant, print the message rules: - id: com.example.env_sample pattern: ENV message: > When you use env var, update .env.sample for other developers

rules: - id: com.example.octokit pattern: octokit() message: > To make a request to GitHub API, use #with_octokit to handle failures • When the program calls octokit method, print the message

$ gem install querly $ querly init $ querly check http://github.com/soutaro/querly $ querly check lib/main.rb:1:14 Exception You probably should use StandardError lib/main.rb:12:5 File.open("foo.txt") Use block to read/write file spec/main_spec.rb:6:4 pp 1+2 Delete debug print

Implementation • Pattern matching against Ruby AST • To handle ambiguity of Ruby grammar • Querly pattern parsing is done by a LALR(1) parser implemented using Racc Ruby source AST Querly configuration Pattern Pattern matching Output

Patterns self.pp(...) pp user: user, organization: organization string(:symbol:, !null: _) create_table(:users) do |t| t.string :email t.string :name, null: false end save!() [conditional] if user.save! render :show else render :error, locals: { user: user } end

Design Decisions • How flexible the patterns should be • Ruby DSL or different format • Docs and tests • False detections Principle: encourage users to write their own rules

Limited Flexibility • Flexibility is not important • The rules are project-specific ones • The patterns you can define in Querly is restricted to encourage adding new rules • Patterns you cannot write means it is out of the scope of Querly Rule flexibility Definition difficulty Querly RuboCop

Rule Definition in YAML • Different from Rakefile/Gemfile • Ruby is too flexible rules: - id: com.sample.non_null_migration pattern: "_.string(:symbol:, !null: _)" message: < Do you really want a nullable column?

Docs and Tests • Rule can include justification and examples to explain what the rule is fore more correctly • Examples are executable spec rules: - id: com.sample.non_null_migration pattern: "_.string(:symbol:, !null: _)" message: < Do you really want a nullable column? justification: - When you want a nullable column examples: - before: "t.string(:name)" after: "t.string(:name, null: true)"

Allow False Detections • Analysis of Ruby programs is really difficult • If we want more true detections, we have to allow false detections False detections True detections RuboCop Querly Extra layer to handle false detections

Detection Filtering • If number of detections you have to review at one time is not too big, false detections are ok • Hook pull requests! • Run Querly analysis when you open new pull request • Ignore issues from the code which is not changed in the pull request

Using OSS • Run Querly in CI and post the detected issues as comments to pull requests • Pronto & pronto-querly • $ gem i pronto
 $ gem i pronto-querly • reviewdog runner: querly: cmd: "bundle exec querly check" errorformat: - "%f:%l:%c\t%m"

Using SideCI • https://sideci.com • Managed service to run Querly and other LINT tools • Provides issue manager to review detected issues instead of posting comments to GitHub • Allows closing false detected issues Promoted

Experiments • With 5 developers using SideCI • Our config contains 57 rules • We casually add Querly rules to share knowledge about our project

Finding New Rules • When we are writing a review comment, we always think if we can write a Querly rule for that • When we are deprecating an API: octokit() • We can ship new API without completely replacing old one • When hidden requirements are uncovered:
 validates(:symbol:, ..., uniqueness: _, ...) • Better than asking your teammates to read all of the docs

Conclusion • Developed a tool for code review automation which covers project specific rules • Rules are defined by patterns • Used in my team to DRY up code review suggestions • Design decisions • Limited flexibility for ease of definitions • Handle false detections by extra layer to filter them