Slide 1

Slide 1 text

CRYSTALBALL Predicting test failures

Slide 2

Slide 2 text

my name is Alex 2

Slide 3

Slide 3 text

my name is Alex 3

Slide 4

Slide 4 text

my name is Alex @p0deje 4

Slide 5

Slide 5 text

my name is Alex @p0deje 5

Slide 6

Slide 6 text

my name is Alex @p0deje 6

Slide 7

Slide 7 text

CRYSTALBALL Predicting test failures

Slide 8

Slide 8 text

1. Problems of regression testing 2. Crystalball 3. Live demo

Slide 9

Slide 9 text

1. Problems of regression testing 2. Crystalball 3. Live demo

Slide 10

Slide 10 text

10 Tests are

Slide 11

Slide 11 text

11 Tests are vital

Slide 12

Slide 12 text

12 Tests are slow

Slide 13

Slide 13 text

13 Tests are integrated

Slide 14

Slide 14 text

14 Run all the tests on every change

Slide 15

Slide 15 text

15 Matz hates tests

Slide 16

Slide 16 text

16 Aaron Patterson blogged "Predicting test failures" Feb 2015

Slide 17

Slide 17 text

17 Aaron Patterson blogged "Predicting test failures" Feb 2015 Running tests is the worst.

Slide 18

Slide 18 text

18 Aaron Patterson blogged "Predicting test failures" Feb 2015 Seriously.

Slide 19

Slide 19 text

19 Aaron Patterson blogged "Predicting test failures" Feb 2015 It takes forever, and by the time they’re all done running, I forgot what I was doing.

Slide 20

Slide 20 text

20 Aaron Patterson blogged "Predicting test failures" Feb 2015 Pavel Shutsin made proof-of-concept Mar 2017

Slide 21

Slide 21 text

21 Aaron Patterson blogged "Predicting test failures" Feb 2015 Pavel Shutsin made proof-of-concept Mar 2017 Crystalball 0.5.0 was released Apr 2018

Slide 22

Slide 22 text

1. Problems of regression testing 2. Crystalball 3. Live demo

Slide 23

Slide 23 text

23 Crystalball is a regression test selection library.

Slide 24

Slide 24 text

24 # spec/spec_helper.rb if ENV['CRYSTALBALL'] == 'true' require 'crystalball' Crystalball::MapGenerator.start! do |config| config.register Crystalball::MapGenerator::CoverageStrategy.new end end

Slide 25

Slide 25 text

25 $ env CRYSTALBALL=true bundle exec rspec … $ cat tmp/crystalball_data.yml --- :type: Crystalball::ExecutionMap :commit: 647f096ed956bf6558b97364d8c1ee935d710109 :timestamp: 1551601518 :version: --- "./spec/controllers/home_controller_spec.rb[1:2]": - spec/controllers/home_controller_spec.rb - app/controllers/home_controller.rb - app/controllers/application_controller.rb …

Slide 26

Slide 26 text

26 $ env CRYSTALBALL=true bundle exec rspec … $ cat tmp/crystalball_data.yml --- :type: Crystalball::ExecutionMap :commit: 647f096ed956bf6558b97364d8c1ee935d710109 :timestamp: 1551601518 :version: --- "./spec/controllers/home_controller_spec.rb[1:2]": - spec/controllers/home_controller_spec.rb - app/controllers/home_controller.rb - app/controllers/application_controller.rb …

Slide 27

Slide 27 text

27 $ env CRYSTALBALL=true bundle exec rspec … $ cat tmp/crystalball_data.yml --- :type: Crystalball::ExecutionMap :commit: 647f096ed956bf6558b97364d8c1ee935d710109 :timestamp: 1551601518 :version: --- "./spec/controllers/home_controller_spec.rb[1:2]": - spec/controllers/home_controller_spec.rb - app/controllers/home_controller.rb - app/controllers/application_controller.rb …

Slide 28

Slide 28 text

28 --- a/app/models/user.rb +++ b/app/models/user.rb @@ -5,7 +5,6 @@ class User < ApplicationRecord validates :first_name, presence: true - validates :last_name, presence: true

Slide 29

Slide 29 text

29 $ bundle exec crystalball I, [2019-04-02T10:17:11.595696 #92687] INFO -- : Crystalball starts to glow... I, [2019-04-02T10:17:11.626689 #92687] INFO -- : Starting RSpec. User ... should validate that :last_name cannot be empty/falsy (FAILED - 1)

Slide 30

Slide 30 text

30 Generates a test-to-code map

Slide 31

Slide 31 text

31 Generates a test-to-code map Predicts which tests should be run

Slide 32

Slide 32 text

32 Generates a test-to-code map Predicts which tests should be run Runs those tests

Slide 33

Slide 33 text

33 MapGenerator Predicts which tests should be run Runs those tests

Slide 34

Slide 34 text

34 MapGenerator Predictor Runs those tests

Slide 35

Slide 35 text

35 MapGenerator Predictor Runner

Slide 36

Slide 36 text

36 MapGenerator Predictor Runner

Slide 37

Slide 37 text

37 Coverage

Slide 38

Slide 38 text

38 1.Get coverage before test 2.Run test 3.Get coverage after test 4.Compare Coverage

Slide 39

Slide 39 text

39 Coverage require 'coverage' Coverage.start before = Coverage.peek_result yield example after = Coverage.peek_result after.reject! do |file_name, after_coverage| before[file_name] == after_coverage end

Slide 40

Slide 40 text

40 Coverage require 'coverage' Coverage.start before = Coverage.peek_result yield example after = Coverage.peek_result after.reject! do |file_name, after_coverage| before[file_name] == after_coverage end

Slide 41

Slide 41 text

41 Coverage require 'coverage' Coverage.start before = Coverage.peek_result yield example after = Coverage.peek_result after.select! do |file_name, after_coverage| before[file_name] != after_coverage end

Slide 42

Slide 42 text

42 Coverage "./spec/models/user_spec.rb[1:6]": - spec/models/user_spec.rb - app/models/user.rb - app/mailers/user_mailer.rb - app/mailers/application_mailer.rb

Slide 43

Slide 43 text

43 Allocated Objects

Slide 44

Slide 44 text

44 Allocated Objects 1. Add tracepoint for constant definition 2. Load tests 3. Add tracepoint for object allocation 4. Run test 5. Get the list of objects allocated during test 6. Find which files define constants of these objects

Slide 45

Slide 45 text

45 Allocated Objects TracePoint.new(:class) do |tp| mod = tp.self path = tp.path constants_definition_paths[mod] ||= [] constants_definition_paths[mod] << path end.enable

Slide 46

Slide 46 text

46 Allocated Objects TracePoint.new(:class) do |tp| mod = tp.self path = tp.path constants_definition_paths[mod] ||= [] constants_definition_paths[mod] << path end.enable

Slide 47

Slide 47 text

47 Allocated Objects TracePoint.new(:class) do |tp| mod = tp.self path = tp.path constants_definition_paths[mod] ||= [] constants_definition_paths[mod] << path end.enable

Slide 48

Slide 48 text

48 Allocated Objects TracePoint.new(:c_call) do |tp| next if tp.method_id != :new || tp.method_id != :allocate created_object_classes << tp.self end.enable(&example)

Slide 49

Slide 49 text

49 Allocated Objects TracePoint.new(:c_call) do |tp| next if tp.method_id != :new || tp.method_id != :allocate created_object_classes << tp.self end.enable(&example)

Slide 50

Slide 50 text

50 Allocated Objects TracePoint.new(:c_call) do |tp| next if tp.method_id != :new || tp.method_id != :allocate created_object_classes << tp.self end.enable(&example)

Slide 51

Slide 51 text

51 Allocated Objects "./spec/models/user_spec.rb[1:6]": - app/models/user.rb - app/models/application_record.rb

Slide 52

Slide 52 text

52 Described Class

Slide 53

Slide 53 text

53 Described Class 1. Add tracepoint for constant definition 2. Load tests 3. Run test 4. Find file defining “described class” of the test

Slide 54

Slide 54 text

54 Described Class RSpec.describe User do # ... end yield example described_class = example.metadata[:described_class] constants_definition_paths[described_class]

Slide 55

Slide 55 text

55 Described Class RSpec.describe User do # ... end yield example described_class = example.metadata[:described_class] constants_definition_paths[described_class]

Slide 56

Slide 56 text

56 Described Class RSpec.describe User do # ... end yield example described_class = example.metadata[:described_class] constants_definition_paths[described_class]

Slide 57

Slide 57 text

57 Described Class RSpec.describe User do # ... end yield example described_class = example.metadata[:described_class] constants_definition_paths[described_class]

Slide 58

Slide 58 text

58 Described Class "./spec/models/user_spec.rb[1:6]": - app/models/user.rb

Slide 59

Slide 59 text

59 Parser

Slide 60

Slide 60 text

60 Parser 1. Parse source code for constant definitions
 2. Run test 3. Parse files used by test 4. Search for calls to constants

Slide 61

Slide 61 text

61 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end

Slide 62

Slide 62 text

62 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end

Slide 63

Slide 63 text

63 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end

Slide 64

Slide 64 text

64 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end

Slide 65

Slide 65 text

65 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end

Slide 66

Slide 66 text

66 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end

Slide 67

Slide 67 text

67 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end

Slide 68

Slide 68 text

68 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end

Slide 69

Slide 69 text

69 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end

Slide 70

Slide 70 text

70 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node) do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end

Slide 71

Slide 71 text

71 Parser "./spec/models/user_spec.rb[1:6]": - app/models/user.rb - app/models/application_record.rb - app/mailers/user_mailer.rb - app/jobs/geocode_user_job.rb

Slide 72

Slide 72 text

72 FactoryBot

Slide 73

Slide 73 text

73 FactoryBot 1. Patch FactoryBot to collect all defined factories 2. Patch FactoryBot to collect factories used during test 3. Run test

Slide 74

Slide 74 text

74 FactoryBot

Slide 75

Slide 75 text

75 FactoryBot "./spec/models/user_spec.rb[1:1]": - spec/factories/users.rb "./spec/models/user_spec.rb[1:6]": []

Slide 76

Slide 76 text

76 ActionView

Slide 77

Slide 77 text

77 ActionView 1. Patch ActionView to collect all compiled views 2. Run test

Slide 78

Slide 78 text

78 ActionView

Slide 79

Slide 79 text

79 ActionView "./spec/models/user_spec.rb[1:6]": []

Slide 80

Slide 80 text

80 ActionView "./spec/controllers/home_controller_spec.rb[1:2]": - app/views/home/index.html.erb - app/views/layouts/application.html.erb

Slide 81

Slide 81 text

81 I18n

Slide 82

Slide 82 text

82 I18n 1. Patch I18n to collect all loaded translations 2. Patch I18n to collect all used translations 3. Run test

Slide 83

Slide 83 text

83 I18n

Slide 84

Slide 84 text

84 I18n "./spec/models/user_spec.rb[1:6]": []

Slide 85

Slide 85 text

85 I18n "./spec/system/sign_ins_spec.rb[1:1]": - config/locales/devise.en.yml

Slide 86

Slide 86 text

86 Tables

Slide 87

Slide 87 text

87 Tables 1. Add tracepoint for constant definition 2. Run test 3. Collect all table names from used constants

Slide 88

Slide 88 text

88 Tables yield example ActiveRecord::Base.descendants.map do |constant| if constants_definition_paths[constant] constant.table_name end end.compact

Slide 89

Slide 89 text

89 Tables yield example ActiveRecord::Base.descendants.map do |constant| if constants_definition_paths[constant] constant.table_name end end.compact

Slide 90

Slide 90 text

90 Tables yield example ActiveRecord::Base.descendants.map do |constant| if constants_definition_paths[constant] constant.table_name end end.compact

Slide 91

Slide 91 text

91 Tables users: - app/models/user.rb - app/models/application_record.rb

Slide 92

Slide 92 text

92 MapGenerator Predictor Runner

Slide 93

Slide 93 text

93 Modified execution paths

Slide 94

Slide 94 text

94 Modified execution paths 1.Get modified files from Git 2.Find related tests in generated map 3.Run tests

Slide 95

Slide 95 text

95 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact

Slide 96

Slide 96 text

96 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact

Slide 97

Slide 97 text

97 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact

Slide 98

Slide 98 text

98 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact

Slide 99

Slide 99 text

99 Modified execution paths --- a/app/models/user.rb +++ b/app/models/user.rb @@ -5,7 +5,6 @@ class User < ApplicationRecord validates :first_name, presence: true - validates :last_name, presence: true

Slide 100

Slide 100 text

100 Modified execution paths { "./spec/models/user_spec.rb" => [ "1:9", "1:2", "1:3", "1:4", "1:5", "1:6", "1:7", "1:8", "1:1" ] }

Slide 101

Slide 101 text

101 Modified specs

Slide 102

Slide 102 text

102 Modified specs 1.Get modified spec files from Git 2.Run all of them

Slide 103

Slide 103 text

103 Modified specs --- a/spec/models/project_spec.rb +++ b/spec/models/project_spec.rb @@ -1,6 +1,7 @@ require 'rails_helper' RSpec.describe Project, type: :model do + it { is_expected.to be_a(Project) }

Slide 104

Slide 104 text

104 Modified specs ["spec/models/project_spec.rb"]

Slide 105

Slide 105 text

105 Modified support specs

Slide 106

Slide 106 text

106 Modified support specs 1. Get modified spec support files from Git 2. Find tests using them from generated map 3. Run tests

Slide 107

Slide 107 text

107 Modified support specs RSpec.describe TasksController do include_context "project setup" … RSpec.describe ProjectsController do include_context "project setup" …

Slide 108

Slide 108 text

108 Modified support specs RSpec.describe TasksController do include_context "project setup" … RSpec.describe ProjectsController do include_context "project setup" …

Slide 109

Slide 109 text

109 Modified support specs --- a/spec/support/contexts/project_setup.rb +++ b/spec/support/contexts/project_setup.rb @@ -1,5 +1,4 @@ RSpec.shared_context "project setup" do let(:user) { FactoryBot.create(:user) } let(:project) { FactoryBot.create(:project, owner: user) } - let(:task) { project.tasks.create!(name: "Test task") } end

Slide 110

Slide 110 text

110 Modified support specs { "./spec/controllers/projects_controller_spec.rb" => ["1:1"], "./spec/controllers/tasks_controller_spec.rb" => [ "1:1:1", "1:2:1", "1:2:2", "1:2:3" ] }

Slide 111

Slide 111 text

111 Associated specs

Slide 112

Slide 112 text

112 Associated specs 1.Manually associate source files with tests 2.Run associated tests

Slide 113

Slide 113 text

113 Associated specs Crystalball::Predictor::AssociatedSpecs.new( from: %r{models/(.*).rb}, to: “./spec/models/%s_spec.rb" )

Slide 114

Slide 114 text

114 Modified schema

Slide 115

Slide 115 text

115 Modified schema 1. Get changed tables from schema diff 2. Find which models define those tables 3. Find which tests should be run for these models 4. Run tests

Slide 116

Slide 116 text

116 Modified schema --- a/db/schema.rb +++ b/db/schema.rb create_table "tasks", force: :cascade do |t| t.string "name" t.integer "project_id" t.boolean "completed" + t.string "title" end

Slide 117

Slide 117 text

117 Modified schema { "./spec/controllers/tasks_controller_spec.rb" => [ "1:1:1", "1:2:1", "1:2:2", ], "./spec/models/task_spec.rb" => [ "1:1", "1:2", "1:3" ], "./spec/system/tasks_spec.rb" => ["1:1"] }

Slide 118

Slide 118 text

118 MapGenerator Predictor Runner

Slide 119

Slide 119 text

119 RSpec

Slide 120

Slide 120 text

120 RSpec 1.Used when you run Crystalball 2.Build predicted specs 3.Run specs

Slide 121

Slide 121 text

1. Problems of regression testing 2. Crystalball 3. Live demo

Slide 122

Slide 122 text

122 toptal.github.io/crystalball

Slide 123

Slide 123 text

Alex Rodionov @p0deje 123