How to Write Better Code with Mutation Testing

Presentation for Railsconf and Fog City Ruby based on https://blog.blockscore.com/how-to-write-better-code-using-mutation-testing/.

Mutation testing can help you:

* Improve your test coverage
* Think about edge cases and catch more bugs before they break in production
* Learn more about Ruby, the libraries you depend on, and your own codebase

Transcript

1. How to Write Better Code with Mutation Testing John Backus - @backus - john@cognitohq.com
2. Talk Outline
3. Talk Outline Introduction
4. Talk Outline Introduction Improving coverage
5. Talk Outline Introduction Improving coverage Learning about Ruby and the code you rely on
6. Talk Outline Improving coverage X-Raying Legacy Code Learning about Ruby and the code you rely on
7. Talk Outline Incredible Dead Code Detection Learning about Ruby and the code you rely on X-Raying Legacy Code
8. Talk Outline Thorough Test Coverage X-Raying Legacy Code Incredible Dead Code Detection
9. Talk Outline Code Simplification Incredible Dead Code Detection Thorough Test Coverage
10. Talk Outline Practicality Thorough Test Coverage Code Simplification
11. Talk Outline Mutation Testing on the Job Code Simplification Practicality
12. Line Coverage Lines of Code Run by Tests Total Lines of Code ÷
13. Mutation Coverage How much of your code can I change without breaking your tests?
14. Mutation Testing by Hand
15. 1 class Gluttons 2 def initialize(twitter_client) 3 @twitter = twitter_client 4 end 5 6 def recent 7 query = @twitter.search('"I really enjoy #pizza"') 8 9 query.first(2).map { |tweet| "@#{tweet.author}" } 10 end 11 end 1 RSpec.describe Gluttons do 2 it 'lists the two most recent gluttonous tweeters' do 3 tweets = [double(author: 'John'), double(author: 'Jane')] 4 gluttons = Gluttons.new(double(search: tweets)) 5 6 expect(gluttons.recent).to eql(%w[@John @Jane]) 7 end 8 end
33. 1 class Gluttons 2 def recent 3 - query = @twitter.search('"I really enjoy #pizza"') 4 + query = @twitter.search('"I really enjoy #hotdogs"') 1 example, 0 failures
34. 1 class Gluttons 2 def recent 3 - query = @twitter.search('"I really enjoy #pizza"') 4 + query = @twitter.search('') 1 example, 0 failures
35. 1 class Gluttons 2 def recent 3 - query = @twitter.search('"I really enjoy #pizza"') 4 + query = @twitter.search 1 example, 0 failures
36. 1 class Gluttons 2 def recent 3 - query.first(2).map { |tweet| "@#{tweet.author}" } 4 + query.first(1).map { |tweet| "@#{tweet.author}" } Failure expected: ["@John", "@Jane"] got: ["@John"]
37. 1 - query.first(2).map { |tweet| "@#{tweet.author}" } 2 + query.first(3).map { |tweet| "@#{tweet.author}" } 3 end 1 example, 0 failures
38. Manual Mutation Testing Tedious
39. Manual Mutation Testing Hard to outsmart yourself
40. $ gem install mutest $ gem install mutant or
52. 1 def recent 2 - query = @twitter.search('"I really enjoy #pizza"') 3 + query = @twitter.search
53. 1 def recent 2 - query = @twitter.search('"I really enjoy #pizza"') 3 + query = @twitter.search(nil)
54. 1 - query.first(2).map { |tweet| "@#{tweet.author}" } 2 + query.last(2).map { |tweet| "@#{tweet.author}" } 3 end
55. 1 - query.first(2).map { |tweet| "@#{tweet.author}" } 2 + query.map { |tweet| "@#{tweet.author}" } 3 end
56. 1 it 'lists the two most recent gluttonous tweeters' do 2 tweets = [ 3 double(author: 'John'), 4 double(author: 'Jane'), 5 double(author: 'Devon') 6 ] 7 8 client = double('Client') 9 gluttons = Gluttons.new(client) 10 11 allow(client) 12 .to receive(:search) 13 .with('"I really enjoy #pizza"') 14 .and_return(tweets) 15 16 expect(gluttons.recent).to eql(%w[@John @Jane]) 17 end
57. Mutation Testing with mutest Automated
58. Probably more clever than you Mutation Testing with mutest
59. Example #1: Internal API
60. 1 it 'returns a user when given a valid id' do 2 expect(get(:show, id: 1)).to eq(id: 1, name: 'John') 3 end 4 5 it 'renders JSON error when given an invalid id' do 6 expect(get(:show, id: 0)) 7 .to eq(error: "Could not find User with 'id'=0") 8 end 1 class UsersController < ApplicationController 2 def show 3 render json: User.find(params[:id].to_i) 4 rescue User::RecordNotFound => error 5 render json: { error: error.to_s } 6 end 7 end
61. 1 def show 2 - render json: User.find(params[:id].to_i) 3 + render json: User.find(Integer(params[:id])) 4 rescue User::RecordNotFound => error 5 render json: { error: error.to_s } 6 end
62. 1 def show 2 - render json: User.find(params[:id].to_i) 3 + render json: User.find(params.fetch(:id).to_i) 4 rescue User::RecordNotFound => error 5 render json: { error: error.to_s } 6 end
63. 1 def show 2 - render json: User.find(params[:id].to_i) 3 + render json: User.find(Integer(params.fetch(:id))) 4 rescue User::RecordNotFound => error 5 render json: { error: error.to_s } 6 end
64. 1 class UsersController < ApplicationController 2 def created_after 3 after = Date.parse(params[:after]) 4 render json: User.recent(after) 5 end 6 end
65. 1 def created_after 2 - after = Date.parse(params[:after]) 3 + after = Date.iso8601(params[:after]) 4 render json: User.recent(after) 5 end
66. 1 def created_after 2 - after = Date.parse(params[:after]) 3 + after = Date.iso8601(params[:after]) 4 render json: User.recent(after) 5 end
67. 1 def created_after 2 - after = Date.parse(params[:after]) 3 + after = Date.iso8601(params[:after]) 4 render json: User.recent(after) 5 end “2017-05-01" "H29.05.01" "Tue May 01 00:00:00 2017" "Tue, 01 May 2017 00:00:00 +0000" "Tue, 01 May 2017 00:00:00 GMT" "May" "I may be complete garbage" “2017-05-01" Date.parse Date.iso8601
68. Example #2: Hardening Regular Expressions
69. 1 usernames.select do |username| 2 username =~ /^(John|Alain).+$/ 3 end
70. 1 usernames.select do |username| 2 - username =~ /^(John|Alain).+$/ 3 + username =~ /\A(John|Alain).+$/ 4 end 1 usernames.select do |username| 2 - username =~ /^(John|Alain).+$/ 3 + username =~ /^(John|Alain).+\z/ 4 end
71. 1 usernames.select do |username| 2 - username =~ /^(John|Alain).+$/ 3 + username =~ /^(Alain).+$/ 4 end 1 usernames.select do |username| 2 - username =~ /^(John|Alain).+$/ 3 + username =~ /^(John).+$/ 4 end
72. 1 usernames.select do |username| 2 - username =~ /^(John|Alain).+$/ 3 + username =~ /^(?:John|Alain).+$/ 4 end
73. 1 usernames.select do |username| 2 - username =~ /^(John|Alain).+$/ 3 + username.match?(/^(John|Alain).+$/) 4 end
74. 1 usernames.select do |username| 2 - username =~ /^(John|Alain).+$/ 3 + username.match?(/\A(?:John|Alain).+\z/) 4 end
75. Example #3: Learning about your HTTP Client
76. 1 def stars_for(repo) 2 url = "https://api.github.com/repos/#{repo}" 3 data = HTTParty.get(url).to_h 4 5 data['stargazers_count'] 6 end
77. 1 def stars_for(repo) 2 url = "https://api.github.com/repos/#{repo}" 3 - data = HTTParty.get(url).to_h 4 + data = HTTParty.get(url) 5 6 data['stargazers_count'] 7 end
78. X-Raying Legacy Code
79. 1 class UsersController < ApplicationController 2 def created_after 3 after = Date.parse(params[:after]) 4 render json: User.recent(after) 5 end 6 end
80. 1 def created_after 2 - after = Date.parse(params[:after]) 3 + after = Date.iso8601(params[:after]) 4 render json: User.recent(after) 5 end
81. 1 def created_after 2 - after = Date.parse(params[:after]) 3 + after = Date.iso8601(params[:after]) 4 render json: User.recent(after) 5 end “2017-05-01" "H29.05.01" "Tue May 01 00:00:00 2017" "Tue, 01 May 2017 00:00:00 +0000" "Tue, 01 May 2017 00:00:00 GMT" "May" "I may be complete garbage" “2017-05-01" Date.parse Date.iso8601
82. 1 def created_after 2 - after = Date.parse(params[:after]) 3 + after = Date.iso8601(params[:after]) 4 render json: User.recent(after) 5 end What was the author’s intent?
83. 1 def created_after 2 - after = Date.parse(params[:after]) 3 + after = Date.iso8601(params[:after]) 4 render json: User.recent(after) 5 end How is this code used today?
84. 1 def created_after 2 - after = Date.parse(params[:after]) 3 + after = Date.iso8601(params[:after]) 4 render json: User.recent(after) 5 end Should I update the code or the tests?
85. Thorough Test Coverage
86. 1 def can_buy_alcohol?(age) 2 age >= 21 3 end
87. 1 def can_buy_alcohol?(age) 2 - age >= 21 3 + age >= 22 4 end 1 def can_buy_alcohol?(age) 2 - age >= 21 3 + age >= 20 4 end
88. 1 def can_edit_post?(editor, post) 2 case editor.role 3 when :guest, :muted then false 4 when :user 5 editor == post.author && !post.locked? 6 when :moderator then !post.author.admin? 7 when :admin then true 8 end 9 end
89. 1 def can_edit_post?(editor, post) 2 case editor.role 3 when :guest, :muted then false 4 when :user 5 editor == post.author && !post.locked? 6 when :moderator then !post.author.admin? 7 when :admin then true 8 end 9 end when :guest 1
90. 1 def can_edit_post?(editor, post) 2 case editor.role 3 when :guest, :muted then false 4 when :user 5 editor == post.author && !post.locked? 6 when :moderator then !post.author.admin? 7 when :admin then true 8 end 9 end when :guest 1 :muted 2
91. 1 def can_edit_post?(editor, post) 2 case editor.role 3 when :guest, :muted then false 4 when :user 5 editor == post.author && !post.locked? 6 when :moderator then !post.author.admin? 7 when :admin then true 8 end 9 end when :user 3 when :guest 1 :muted 2
92. 1 def can_edit_post?(editor, post) 2 case editor.role 3 when :guest, :muted then false 4 when :user 5 editor == post.author && !post.locked? 6 when :moderator then !post.author.admin? 7 when :admin then true 8 end 9 end when :user 3 when :guest 1 :muted 2 !post.locked? 4
93. 1 def can_edit_post?(editor, post) 2 case editor.role 3 when :guest, :muted then false 4 when :user 5 editor == post.author && !post.locked? 6 when :moderator then !post.author.admin? 7 when :admin then true 8 end 9 end when :user 3 when :guest 1 :muted 2 && 5 !post.locked? 4
94. 1 def can_edit_post?(editor, post) 2 case editor.role 3 when :guest, :muted then false 4 when :user 5 editor == post.author && !post.locked? 6 when :moderator then !post.author.admin? 7 when :admin then true 8 end 9 end when :moderator 6 when :user 3 when :guest 1 :muted 2 && 5 !post.locked? 4
95. 1 def can_edit_post?(editor, post) 2 case editor.role 3 when :guest, :muted then false 4 when :user 5 editor == post.author && !post.locked? 6 when :moderator then !post.author.admin? 7 when :admin then true 8 end 9 end when :moderator 6 when :user 3 when :guest 1 :muted 2 && 5 !post.locked? 4 then !post.author.admin? 7
96. 1 def can_edit_post?(editor, post) 2 case editor.role 3 when :guest, :muted then false 4 when :user 5 editor == post.author && !post.locked? 6 when :moderator then !post.author.admin? 7 when :admin then true 8 end 9 end when :admin 8 when :moderator 6 when :user 3 when :guest 1 :muted 2 && 5 !post.locked? 4 then !post.author.admin? 7
97. guest muted user moderator admin guest muted editor different user moderator admin Editor’s Role Post Author’s Role
98. 1 def mailing_list(users) 2 users.map do |user| 3 next unless user.email && !user.unsubscribed? 4 5 user.email 6 end.compact 7 end
99. 1 it 'filters out users without emails' do 2 users = [good_user, user_without_email] 3 expect(mailing_list(users)).to eql([good_user.email]) 4 end 5 6 7 it 'filters out users unsubscribed users' do 8 users = [good_user, unsubscribed_user] 9 expect(mailing_list(users)).to eql([good_user.email]) 10 end
100. 1 def mailing_list(users) 2 users.map do |user| 3 - next unless user.email && !user.unsubscribed? 4 + break unless user.email && !user.unsubscribed? 5 6 user.email 7 end.compact 8 end
101. 1 it 'filters out users without emails' do 2 - users = [good_user, user_without_email] 3 + users = [user_without_email, good_user] 4 expect(mailing_list(users)).to eql(good_user.email) 5 end 6 7 8 it 'filters out users unsubscribed users' do 9 - users = [good_user, unsubscribed_user] 10 + users = [unsubscribed_user, good_user] 11 expect(mailing_list(users)).to eql(good_user.email) 12 end
102. Incredible Dead Code Detection
103. 1 class User < ActiveRecord::Base 2 def self.find_by_name(name) 3 find_by(name: name) 4 end 5 end
104. 1 class User < ActiveRecord::Base 2 def find_by_name(name) 3 - find_by(name: name) 4 + super 5 end 6 end
105. 1 class PostsController < ApplicationController 2 private 3 4 def authorized?(user = current_user) 5 # ... 6 end 7 end
106. 1 class PostsController < ApplicationController 2 def edit 3 return unauthorized unless authorized?(current_user) 4 end 5 6 def admin_edit 7 return unauthorized unless authorized?(current_user) 8 end 9 10 private 11 12 def authorized?(user = current_user) 13 # ... 14 end 15 end
107. 1 class PostsController < ApplicationController 2 - def authorized?(user = current_user) 3 + def authorized?(user) 4 # ... 5 end 6 end
108. 1 class PostsController < ApplicationController 2 def edit 3 return unauthorized unless authorized? 4 end 5 6 def admin_edit 7 return unauthorized unless authorized? 8 end 9 10 private 11 12 def authorized?(user = current_user) 13 # ... 14 end 15 end
109. 1 class PostsController < ApplicationController 2 def authorized?(user = current_user) 3 + user = current_user 4 # ... 5 end 6 end
110. 1 module MyApp 2 class User 3 def posted? 4 ::MyApp::Post.exists?(user_id: id) 5 end 6 end 7 end
111. 1 module MyApp 2 class User 3 def posted? 4 - ::MyApp::Post.exists?(user_id: id) 5 + Post.exists?(user_id: id) 6 end 7 end 8 end
112. 1 class PostsController < ApplicationController 2 def show 3 render json: Post.find(params[:id]), status: :ok 4 end 5 end
113. 1 class PostsController < ApplicationController 2 def show 3 - render json: Post.find(params[:id]), status: :ok 4 + render json: Post.find(params[:id]) 5 end 6 end
114. Code Simplification
115. 1 def show_many 2 User.find(*params[:user_ids]) 3 end
116. 1 def show_many 2 - User.find(*params[:user_ids]) 3 + User.find(params[:user_ids]) 4 end
117. 1 class UserDecorator 2 attr_reader :user 3 4 def initialize(user) 5 @user = user 6 end 7 8 def greeting 9 "Welcome, #{@user.name}!" 10 end 11 end
118. 1 class UserDecorator 2 attr_reader :user 3 4 def initialize(user) 5 @user = user 6 end 7 8 def greeting 9 - "Welcome, #{@user.name}!" 10 + "Welcome, #{user.name}!" 11 end 12 end
119. 1 def greeting 2 "Welcome, #{@usre.name}!" 3 end
120. 1 def greeting 2 "Welcome, #{@usre.name}!" 3 end NoMethodError: undefined method `name' for nil:NilClass
121. 1 def greeting 2 "Welcome, #{usre.name}!" 3 end
122. 1 def greeting 2 "Welcome, #{usre.name}!" 3 end NameError: undefined local variable or method `usre`
123. 1 def expand_home(path) 2 path.gsub(%r{\A~/}, ENV['HOME']) 3 end
124. 1 def expand_home(path) 2 - path.gsub(%r{\A~/}, ENV['HOME']) 3 + path.sub(%r{\A~/}, ENV['HOME']) 4 end
125. 1 def prune_inactive_images 2 images = Image.where("last_viewed_at > ?", 2.years.ago) 3 4 images.map do |image| 5 puts "Deleting image #{image.name}" 6 end 7 8 images.destroy_all 9 10 images.count 11 end
126. 1 def prune_inactive_images 2 images = Image.where("last_viewed_at > ?", 2.years.ago) 3 4 - images.map do |image| 5 + images.each do |image| 6 puts "Deleting image #{image.name}" 7 end 8 9 images.destroy_all 10 11 images.count 12 end
127. 1 require 'logger' 2 3 logger = Logger.new($stdout) 4 5 logger.formatter = 6 Proc.new do |severity, datetime, name, msg| 7 "[#{severity}] #{datetime} #{name} -- #{msg}\n" 8 end
128. 1 require 'logger' 2 3 logger = Logger.new($stdout) 4 5 logger.formatter = 6 - Proc.new do |severity, datetime, name, msg| 7 + lambda do |severity, datetime, name, msg| 8 "[#{severity}] #{datetime} #{name} -- #{msg}\n" 9 end
129. formatter = proc { |a, b| [a, b].inspect }
130. formatter = proc { |a, b| [a, b].inspect }
131. formatter = proc { |a, b| [a, b].inspect } formatter.call() # => "[nil, nil]" formatter.call(1) # => "[1, nil]" formatter.call(1, 2) # => "[1, 2]" formatter.call(1, 2, 3) # => "[1, 2]"
132. formatter = proc { |a, b| [a, b].inspect } formatter.call() # => "[nil, nil]" formatter.call(1) # => "[1, nil]" formatter.call(1, 2) # => "[1, 2]" formatter.call(1, 2, 3) # => "[1, 2]" formatter.call([]) # => "[nil, nil]" formatter.call([1]) # => "[1, nil]" formatter.call([1, 2]) # => "[1, 2]" formatter.call([1, 2, 3]) # => "[1, 2]"
133. formatter = lambda { |a, b| [a, b].inspect } formatter.call() # => ArgumentError formatter.call(1) # => ArgumentError formatter.call(1, 2) # => "[1, 2]" formatter.call(1, 2, 3) # => ArgumentError formatter.call([]) # => ArgumentError formatter.call([1]) # => ArgumentError formatter.call([1, 2]) # => ArgumentError formatter.call([1, 2, 3]) # => ArgumentError
134. Practicality
135. /^(John|Alain).+$/ /\A(John|Alain).+$/ /^(John|Alain).+\z/ /^(Alain).+$/ /^(John).+$/ /^(?:John|Alain).+$/
136. $ mutest --use rspec --since master
137. $ mutest \ --use rspec \ --since master \ ‘YourApp::User#validate_email’
138. 1 module YourApp 2 class User < ActiveRecord::Base 3 # Dozens of includes, scopes, class methods 4 5 def validate_email 6 # Simple method you're fixing 7 end 8 end 9 end
139. 1 RSpec.describe YourApp::User do 2 # 100s of tests and setup unrelated to your task 3 4 describe '#validate_email' do 5 # Half dozen tests you are focusing on 6 end 7 end
140. Become a better developer with mutest
141. Learn more about Ruby
142. Learn more about the code you rely on
143. Learn faster
144. Improve your testing skills
145. Ship fewer bugs
146. Introduce fewer regressions
147. Write simpler code
148. Mutation Testing on the Job
149. Run mutest before you push
150. Add mutest to CI
151. Love writing great code? Email jobs@cognitohq.com
152. mutest - Your Secret Weapon
153. Thanks! John Backus - @backus - john@cognitohq.com