Share
Embed
Crystalball: predicting test failures
A talk about Crystalball - a regression test selection library for Ruby (https://github.com/toptal/crystalball) for RubyKaigi 2019 (https://rubykaigi.org/2019/presentations/p0deje.html). The video of the talk is available at YouTube (https://www.youtube.com/watch?v=q2q-9Td71kE).
Alex Rodionov
April 19, 2019
Transcript
123 slides
-
17 Aaron Patterson blogged "Predicting test failures" Feb 2015 Running
tests is the worst. -
19 Aaron Patterson blogged "Predicting test failures" Feb 2015 It
takes forever, and by the time they’re all done running, I forgot what I was doing. -
20 Aaron Patterson blogged "Predicting test failures" Feb 2015 Pavel
Shutsin made proof-of-concept Mar 2017 -
21 Aaron Patterson blogged "Predicting test failures" Feb 2015 Pavel
Shutsin made proof-of-concept Mar 2017 Crystalball 0.5.0 was released Apr 2018 -
24 # spec/spec_helper.rb if ENV['CRYSTALBALL'] == 'true' require 'crystalball' Crystalball::MapGenerator.start!
do |config| config.register Crystalball::MapGenerator::CoverageStrategy.new end end -
25 $ env CRYSTALBALL=true bundle exec rspec … $ cat
tmp/crystalball_data.yml --- :type: Crystalball::ExecutionMap :commit: 647f096ed956bf6558b97364d8c1ee935d710109 :timestamp: 1551601518 :version: --- "./spec/controllers/home_controller_spec.rb[1:2]": - spec/controllers/home_controller_spec.rb - app/controllers/home_controller.rb - app/controllers/application_controller.rb … -
26 $ env CRYSTALBALL=true bundle exec rspec … $ cat
tmp/crystalball_data.yml --- :type: Crystalball::ExecutionMap :commit: 647f096ed956bf6558b97364d8c1ee935d710109 :timestamp: 1551601518 :version: --- "./spec/controllers/home_controller_spec.rb[1:2]": - spec/controllers/home_controller_spec.rb - app/controllers/home_controller.rb - app/controllers/application_controller.rb … -
27 $ env CRYSTALBALL=true bundle exec rspec … $ cat
tmp/crystalball_data.yml --- :type: Crystalball::ExecutionMap :commit: 647f096ed956bf6558b97364d8c1ee935d710109 :timestamp: 1551601518 :version: --- "./spec/controllers/home_controller_spec.rb[1:2]": - spec/controllers/home_controller_spec.rb - app/controllers/home_controller.rb - app/controllers/application_controller.rb … -
28 --- a/app/models/user.rb +++ b/app/models/user.rb @@ -5,7 +5,6 @@ class
User < ApplicationRecord validates :first_name, presence: true - validates :last_name, presence: true -
29 $ bundle exec crystalball I, [2019-04-02T10:17:11.595696 #92687] INFO --
: Crystalball starts to glow... I, [2019-04-02T10:17:11.626689 #92687] INFO -- : Starting RSpec. User ... should validate that :last_name cannot be empty/falsy (FAILED - 1) -
32 Generates a test-to-code map Predicts which tests should be
run Runs those tests -
38 1.Get coverage before test 2.Run test 3.Get coverage after
test 4.Compare Coverage -
39 Coverage require 'coverage' Coverage.start before = Coverage.peek_result yield example
after = Coverage.peek_result after.reject! do |file_name, after_coverage| before[file_name] == after_coverage end -
40 Coverage require 'coverage' Coverage.start before = Coverage.peek_result yield example
after = Coverage.peek_result after.reject! do |file_name, after_coverage| before[file_name] == after_coverage end -
41 Coverage require 'coverage' Coverage.start before = Coverage.peek_result yield example
after = Coverage.peek_result after.select! do |file_name, after_coverage| before[file_name] != after_coverage end -
42 Coverage "./spec/models/user_spec.rb[1:6]": - spec/models/user_spec.rb - app/models/user.rb - app/mailers/user_mailer.rb -
app/mailers/application_mailer.rb -
44 Allocated Objects 1. Add tracepoint for constant definition 2.
Load tests 3. Add tracepoint for object allocation 4. Run test 5. Get the list of objects allocated during test 6. Find which files define constants of these objects -
45 Allocated Objects TracePoint.new(:class) do |tp| mod = tp.self path
= tp.path constants_definition_paths[mod] ||= [] constants_definition_paths[mod] << path end.enable -
46 Allocated Objects TracePoint.new(:class) do |tp| mod = tp.self path
= tp.path constants_definition_paths[mod] ||= [] constants_definition_paths[mod] << path end.enable -
47 Allocated Objects TracePoint.new(:class) do |tp| mod = tp.self path
= tp.path constants_definition_paths[mod] ||= [] constants_definition_paths[mod] << path end.enable -
48 Allocated Objects TracePoint.new(:c_call) do |tp| next if tp.method_id !=
:new || tp.method_id != :allocate created_object_classes << tp.self end.enable(&example) -
49 Allocated Objects TracePoint.new(:c_call) do |tp| next if tp.method_id !=
:new || tp.method_id != :allocate created_object_classes << tp.self end.enable(&example) -
50 Allocated Objects TracePoint.new(:c_call) do |tp| next if tp.method_id !=
:new || tp.method_id != :allocate created_object_classes << tp.self end.enable(&example) -
53 Described Class 1. Add tracepoint for constant definition 2.
Load tests 3. Run test 4. Find file defining “described class” of the test -
54 Described Class RSpec.describe User do # ... end yield
example described_class = example.metadata[:described_class] constants_definition_paths[described_class] -
55 Described Class RSpec.describe User do # ... end yield
example described_class = example.metadata[:described_class] constants_definition_paths[described_class] -
56 Described Class RSpec.describe User do # ... end yield
example described_class = example.metadata[:described_class] constants_definition_paths[described_class] -
57 Described Class RSpec.describe User do # ... end yield
example described_class = example.metadata[:described_class] constants_definition_paths[described_class] -
60 Parser 1. Parse source code for constant definitions 2.
Run test 3. Parse files used by test 4. Search for calls to constants -
61 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)
do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end -
62 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)
do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end -
63 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)
do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end -
64 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)
do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end -
65 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)
do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end -
66 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)
do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end -
67 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)
do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end -
68 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)
do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end -
69 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)
do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end -
70 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)
do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end -
73 FactoryBot 1. Patch FactoryBot to collect all defined factories
2. Patch FactoryBot to collect factories used during test 3. Run test -
82 I18n 1. Patch I18n to collect all loaded translations
2. Patch I18n to collect all used translations 3. Run test -
87 Tables 1. Add tracepoint for constant definition 2. Run
test 3. Collect all table names from used constants -
94 Modified execution paths 1.Get modified files from Git 2.Find
related tests in generated map 3.Run tests -
95 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map
do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact -
96 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map
do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact -
97 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map
do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact -
98 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map
do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact -
99 Modified execution paths --- a/app/models/user.rb +++ b/app/models/user.rb @@ -5,7
+5,6 @@ class User < ApplicationRecord validates :first_name, presence: true - validates :last_name, presence: true -
100 Modified execution paths { "./spec/models/user_spec.rb" => [ "1:9", "1:2",
"1:3", "1:4", "1:5", "1:6", "1:7", "1:8", "1:1" ] } -
103 Modified specs --- a/spec/models/project_spec.rb +++ b/spec/models/project_spec.rb @@ -1,6 +1,7
@@ require 'rails_helper' RSpec.describe Project, type: :model do + it { is_expected.to be_a(Project) } -
106 Modified support specs 1. Get modified spec support files
from Git 2. Find tests using them from generated map 3. Run tests -
107 Modified support specs RSpec.describe TasksController do include_context "project setup"
… RSpec.describe ProjectsController do include_context "project setup" … -
108 Modified support specs RSpec.describe TasksController do include_context "project setup"
… RSpec.describe ProjectsController do include_context "project setup" … -
109 Modified support specs --- a/spec/support/contexts/project_setup.rb +++ b/spec/support/contexts/project_setup.rb @@ -1,5
+1,4 @@ RSpec.shared_context "project setup" do let(:user) { FactoryBot.create(:user) } let(:project) { FactoryBot.create(:project, owner: user) } - let(:task) { project.tasks.create!(name: "Test task") } end -
110 Modified support specs { "./spec/controllers/projects_controller_spec.rb" => ["1:1"], "./spec/controllers/tasks_controller_spec.rb" =>
[ "1:1:1", "1:2:1", "1:2:2", "1:2:3" ] } -
115 Modified schema 1. Get changed tables from schema diff
2. Find which models define those tables 3. Find which tests should be run for these models 4. Run tests -
116 Modified schema --- a/db/schema.rb +++ b/db/schema.rb create_table "tasks", force:
:cascade do |t| t.string "name" t.integer "project_id" t.boolean "completed" + t.string "title" end -
117 Modified schema { "./spec/controllers/tasks_controller_spec.rb" => [ "1:1:1", "1:2:1", "1:2:2",
], "./spec/models/task_spec.rb" => [ "1:1", "1:2", "1:3" ], "./spec/system/tasks_spec.rb" => ["1:1"] }