Crystalball: predicting test failures

A talk about Crystalball - a regression test selection library for Ruby (https://github.com/toptal/crystalball) for RubyKaigi 2019 (https://rubykaigi.org/2019/presentations/p0deje.html). The video of the talk is available at YouTube (https://www.youtube.com/watch?v=q2q-9Td71kE).

Transcript

1. CRYSTALBALL Predicting test failures
2. my name is Alex 2
3. my name is Alex 3
4. my name is Alex @p0deje 4
5. my name is Alex @p0deje 5
6. my name is Alex @p0deje 6
7. CRYSTALBALL Predicting test failures
8. 1. Problems of regression testing 2. Crystalball 3. Live demo
9. 1. Problems of regression testing 2. Crystalball 3. Live demo
10. 10 Tests are
11. 11 Tests are vital
12. 12 Tests are slow
13. 13 Tests are integrated
14. 14 Run all the tests on every change
15. 15 Matz hates tests
16. 16 Aaron Patterson blogged "Predicting test failures" Feb 2015
17. 17 Aaron Patterson blogged "Predicting test failures" Feb 2015 Running tests is the worst.
18. 18 Aaron Patterson blogged "Predicting test failures" Feb 2015 Seriously.
19. 19 Aaron Patterson blogged "Predicting test failures" Feb 2015 It takes forever, and by the time they’re all done running, I forgot what I was doing.
20. 20 Aaron Patterson blogged "Predicting test failures" Feb 2015 Pavel Shutsin made proof-of-concept Mar 2017
21. 21 Aaron Patterson blogged "Predicting test failures" Feb 2015 Pavel Shutsin made proof-of-concept Mar 2017 Crystalball 0.5.0 was released Apr 2018
22. 1. Problems of regression testing 2. Crystalball 3. Live demo
23. 23 Crystalball is a regression test selection library.
24. 24 # spec/spec_helper.rb if ENV['CRYSTALBALL'] == 'true' require 'crystalball' Crystalball::MapGenerator.start! do |config| config.register Crystalball::MapGenerator::CoverageStrategy.new end end
25. 25 $ env CRYSTALBALL=true bundle exec rspec … $ cat tmp/crystalball_data.yml --- :type: Crystalball::ExecutionMap :commit: 647f096ed956bf6558b97364d8c1ee935d710109 :timestamp: 1551601518 :version: --- "./spec/controllers/home_controller_spec.rb[1:2]": - spec/controllers/home_controller_spec.rb - app/controllers/home_controller.rb - app/controllers/application_controller.rb …
26. 26 $ env CRYSTALBALL=true bundle exec rspec … $ cat tmp/crystalball_data.yml --- :type: Crystalball::ExecutionMap :commit: 647f096ed956bf6558b97364d8c1ee935d710109 :timestamp: 1551601518 :version: --- "./spec/controllers/home_controller_spec.rb[1:2]": - spec/controllers/home_controller_spec.rb - app/controllers/home_controller.rb - app/controllers/application_controller.rb …
27. 27 $ env CRYSTALBALL=true bundle exec rspec … $ cat tmp/crystalball_data.yml --- :type: Crystalball::ExecutionMap :commit: 647f096ed956bf6558b97364d8c1ee935d710109 :timestamp: 1551601518 :version: --- "./spec/controllers/home_controller_spec.rb[1:2]": - spec/controllers/home_controller_spec.rb - app/controllers/home_controller.rb - app/controllers/application_controller.rb …
28. 28 --- a/app/models/user.rb +++ b/app/models/user.rb @@ -5,7 +5,6 @@ class User < ApplicationRecord validates :first_name, presence: true - validates :last_name, presence: true
29. 29 $ bundle exec crystalball I, [2019-04-02T10:17:11.595696 #92687] INFO -- : Crystalball starts to glow... I, [2019-04-02T10:17:11.626689 #92687] INFO -- : Starting RSpec. User ... should validate that :last_name cannot be empty/falsy (FAILED - 1)
30. 30 Generates a test-to-code map
31. 31 Generates a test-to-code map Predicts which tests should be run
32. 32 Generates a test-to-code map Predicts which tests should be run Runs those tests
33. 33 MapGenerator Predicts which tests should be run Runs those tests
34. 34 MapGenerator Predictor Runs those tests
35. 35 MapGenerator Predictor Runner
36. 36 MapGenerator Predictor Runner
37. 37 Coverage
38. 38 1.Get coverage before test 2.Run test 3.Get coverage after test 4.Compare Coverage
39. 39 Coverage require 'coverage' Coverage.start before = Coverage.peek_result yield example after = Coverage.peek_result after.reject! do |file_name, after_coverage| before[file_name] == after_coverage end
40. 40 Coverage require 'coverage' Coverage.start before = Coverage.peek_result yield example after = Coverage.peek_result after.reject! do |file_name, after_coverage| before[file_name] == after_coverage end
41. 41 Coverage require 'coverage' Coverage.start before = Coverage.peek_result yield example after = Coverage.peek_result after.select! do |file_name, after_coverage| before[file_name] != after_coverage end
42. 42 Coverage "./spec/models/user_spec.rb[1:6]": - spec/models/user_spec.rb - app/models/user.rb - app/mailers/user_mailer.rb - app/mailers/application_mailer.rb
43. 43 Allocated Objects
44. 44 Allocated Objects 1. Add tracepoint for constant definition 2. Load tests 3. Add tracepoint for object allocation 4. Run test 5. Get the list of objects allocated during test 6. Find which files define constants of these objects
45. 45 Allocated Objects TracePoint.new(:class) do |tp| mod = tp.self path = tp.path constants_definition_paths[mod] ||= [] constants_definition_paths[mod] << path end.enable
46. 46 Allocated Objects TracePoint.new(:class) do |tp| mod = tp.self path = tp.path constants_definition_paths[mod] ||= [] constants_definition_paths[mod] << path end.enable
47. 47 Allocated Objects TracePoint.new(:class) do |tp| mod = tp.self path = tp.path constants_definition_paths[mod] ||= [] constants_definition_paths[mod] << path end.enable
48. 48 Allocated Objects TracePoint.new(:c_call) do |tp| next if tp.method_id != :new || tp.method_id != :allocate created_object_classes << tp.self end.enable(&example)
49. 49 Allocated Objects TracePoint.new(:c_call) do |tp| next if tp.method_id != :new || tp.method_id != :allocate created_object_classes << tp.self end.enable(&example)
50. 50 Allocated Objects TracePoint.new(:c_call) do |tp| next if tp.method_id != :new || tp.method_id != :allocate created_object_classes << tp.self end.enable(&example)
51. 51 Allocated Objects "./spec/models/user_spec.rb[1:6]": - app/models/user.rb - app/models/application_record.rb
52. 52 Described Class
53. 53 Described Class 1. Add tracepoint for constant definition 2. Load tests 3. Run test 4. Find file defining “described class” of the test
54. 54 Described Class RSpec.describe User do # ... end yield example described_class = example.metadata[:described_class] constants_definition_paths[described_class]
55. 55 Described Class RSpec.describe User do # ... end yield example described_class = example.metadata[:described_class] constants_definition_paths[described_class]
56. 56 Described Class RSpec.describe User do # ... end yield example described_class = example.metadata[:described_class] constants_definition_paths[described_class]
57. 57 Described Class RSpec.describe User do # ... end yield example described_class = example.metadata[:described_class] constants_definition_paths[described_class]
58. 58 Described Class "./spec/models/user_spec.rb[1:6]": - app/models/user.rb
59. 59 Parser
60. 60 Parser 1. Parse source code for constant definitions
 2. Run test 3. Parse files used by test 4. Search for calls to constants
61. 61 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
62. 62 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
63. 63 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
64. 64 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
65. 65 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
66. 66 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
67. 67 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
68. 68 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
69. 69 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
70. 70 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
71. 71 Parser "./spec/models/user_spec.rb[1:6]": - app/models/user.rb - app/models/application_record.rb - app/mailers/user_mailer.rb - app/jobs/geocode_user_job.rb
72. 72 FactoryBot
73. 73 FactoryBot 1. Patch FactoryBot to collect all defined factories 2. Patch FactoryBot to collect factories used during test 3. Run test
74. 74 FactoryBot
75. 75 FactoryBot "./spec/models/user_spec.rb[1:1]": - spec/factories/users.rb "./spec/models/user_spec.rb[1:6]": []
76. 76 ActionView
77. 77 ActionView 1. Patch ActionView to collect all compiled views 2. Run test
78. 78 ActionView
79. 79 ActionView "./spec/models/user_spec.rb[1:6]": []
80. 80 ActionView "./spec/controllers/home_controller_spec.rb[1:2]": - app/views/home/index.html.erb - app/views/layouts/application.html.erb
81. 81 I18n
82. 82 I18n 1. Patch I18n to collect all loaded translations 2. Patch I18n to collect all used translations 3. Run test
83. 83 I18n
84. 84 I18n "./spec/models/user_spec.rb[1:6]": []
85. 85 I18n "./spec/system/sign_ins_spec.rb[1:1]": - config/locales/devise.en.yml
86. 86 Tables
87. 87 Tables 1. Add tracepoint for constant definition 2. Run test 3. Collect all table names from used constants
88. 88 Tables yield example ActiveRecord::Base.descendants.map do |constant| if constants_definition_paths[constant] constant.table_name end end.compact
89. 89 Tables yield example ActiveRecord::Base.descendants.map do |constant| if constants_definition_paths[constant] constant.table_name end end.compact
90. 90 Tables yield example ActiveRecord::Base.descendants.map do |constant| if constants_definition_paths[constant] constant.table_name end end.compact
91. 91 Tables users: - app/models/user.rb - app/models/application_record.rb
92. 92 MapGenerator Predictor Runner
93. 93 Modified execution paths
94. 94 Modified execution paths 1.Get modified files from Git 2.Find related tests in generated map 3.Run tests
95. 95 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact
96. 96 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact
97. 97 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact
98. 98 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact
99. 99 Modified execution paths --- a/app/models/user.rb +++ b/app/models/user.rb @@ -5,7 +5,6 @@ class User < ApplicationRecord validates :first_name, presence: true - validates :last_name, presence: true
100. 100 Modified execution paths { "./spec/models/user_spec.rb" => [ "1:9", "1:2", "1:3", "1:4", "1:5", "1:6", "1:7", "1:8", "1:1" ] }
101. 101 Modified specs
102. 102 Modified specs 1.Get modified spec files from Git 2.Run all of them
103. 103 Modified specs --- a/spec/models/project_spec.rb +++ b/spec/models/project_spec.rb @@ -1,6 +1,7 @@ require 'rails_helper' RSpec.describe Project, type: :model do + it { is_expected.to be_a(Project) }
104. 104 Modified specs ["spec/models/project_spec.rb"]
105. 105 Modified support specs
106. 106 Modified support specs 1. Get modified spec support files from Git 2. Find tests using them from generated map 3. Run tests
107. 107 Modified support specs RSpec.describe TasksController do include_context "project setup" … RSpec.describe ProjectsController do include_context "project setup" …
108. 108 Modified support specs RSpec.describe TasksController do include_context "project setup" … RSpec.describe ProjectsController do include_context "project setup" …
109. 109 Modified support specs --- a/spec/support/contexts/project_setup.rb +++ b/spec/support/contexts/project_setup.rb @@ -1,5 +1,4 @@ RSpec.shared_context "project setup" do let(:user) { FactoryBot.create(:user) } let(:project) { FactoryBot.create(:project, owner: user) } - let(:task) { project.tasks.create!(name: "Test task") } end
110. 110 Modified support specs { "./spec/controllers/projects_controller_spec.rb" => ["1:1"], "./spec/controllers/tasks_controller_spec.rb" => [ "1:1:1", "1:2:1", "1:2:2", "1:2:3" ] }
111. 111 Associated specs
112. 112 Associated specs 1.Manually associate source files with tests 2.Run associated tests
113. 113 Associated specs Crystalball::Predictor::AssociatedSpecs.new( from: %r{models/(.*).rb}, to: “./spec/models/%s_spec.rb" )
114. 114 Modified schema
115. 115 Modified schema 1. Get changed tables from schema diff 2. Find which models define those tables 3. Find which tests should be run for these models 4. Run tests
116. 116 Modified schema --- a/db/schema.rb +++ b/db/schema.rb create_table "tasks", force: :cascade do |t| t.string "name" t.integer "project_id" t.boolean "completed" + t.string "title" end
117. 117 Modified schema { "./spec/controllers/tasks_controller_spec.rb" => [ "1:1:1", "1:2:1", "1:2:2", ], "./spec/models/task_spec.rb" => [ "1:1", "1:2", "1:3" ], "./spec/system/tasks_spec.rb" => ["1:1"] }
118. 118 MapGenerator Predictor Runner
119. 119 RSpec
120. 120 RSpec 1.Used when you run Crystalball 2.Build predicted specs 3.Run specs
121. 1. Problems of regression testing 2. Crystalball 3. Live demo
122. 122 toptal.github.io/crystalball
123. Alex Rodionov @p0deje 123