1. Problems of regression testing
2. Crystalball
3. Live demo
Slide 9
Slide 9 text
1. Problems of regression testing
2. Crystalball
3. Live demo
Slide 10
Slide 10 text
10
Tests are
Slide 11
Slide 11 text
11
Tests are vital
Slide 12
Slide 12 text
12
Tests are slow
Slide 13
Slide 13 text
13
Tests are integrated
Slide 14
Slide 14 text
14
Run all the tests on
every change
Slide 15
Slide 15 text
15
Matz hates tests
Slide 16
Slide 16 text
16
Aaron Patterson blogged
"Predicting test failures"
Feb 2015
Slide 17
Slide 17 text
17
Aaron Patterson blogged
"Predicting test failures"
Feb 2015
Running tests is
the worst.
Slide 18
Slide 18 text
18
Aaron Patterson blogged
"Predicting test failures"
Feb 2015
Seriously.
Slide 19
Slide 19 text
19
Aaron Patterson blogged
"Predicting test failures"
Feb 2015
It takes forever, and
by the time they’re all done
running, I forgot what I was
doing.
Slide 20
Slide 20 text
20
Aaron Patterson blogged
"Predicting test failures"
Feb 2015
Pavel Shutsin made
proof-of-concept
Mar 2017
Slide 21
Slide 21 text
21
Aaron Patterson blogged
"Predicting test failures"
Feb 2015
Pavel Shutsin made
proof-of-concept
Mar 2017
Crystalball 0.5.0
was released
Apr 2018
Slide 22
Slide 22 text
1. Problems of regression testing
2. Crystalball
3. Live demo
Slide 23
Slide 23 text
23
Crystalball is a regression
test selection library.
Slide 24
Slide 24 text
24
# spec/spec_helper.rb
if ENV['CRYSTALBALL'] == 'true'
require 'crystalball'
Crystalball::MapGenerator.start! do |config|
config.register Crystalball::MapGenerator::CoverageStrategy.new
end
end
29
$ bundle exec crystalball
I, [2019-04-02T10:17:11.595696 #92687] INFO -- : Crystalball starts to glow...
I, [2019-04-02T10:17:11.626689 #92687] INFO -- : Starting RSpec.
User
...
should validate that :last_name cannot be empty/falsy (FAILED - 1)
Slide 30
Slide 30 text
30
Generates a test-to-code map
Slide 31
Slide 31 text
31
Generates a test-to-code map
Predicts which tests should be run
Slide 32
Slide 32 text
32
Generates a test-to-code map
Predicts which tests should be run
Runs those tests
Slide 33
Slide 33 text
33
MapGenerator
Predicts which tests should be run
Runs those tests
Slide 34
Slide 34 text
34
MapGenerator
Predictor
Runs those tests
Slide 35
Slide 35 text
35
MapGenerator
Predictor
Runner
Slide 36
Slide 36 text
36
MapGenerator
Predictor
Runner
Slide 37
Slide 37 text
37
Coverage
Slide 38
Slide 38 text
38
1.Get coverage before test
2.Run test
3.Get coverage after test
4.Compare
Coverage
Slide 39
Slide 39 text
39
Coverage
require 'coverage'
Coverage.start
before = Coverage.peek_result
yield example
after = Coverage.peek_result
after.reject! do |file_name, after_coverage|
before[file_name] == after_coverage
end
Slide 40
Slide 40 text
40
Coverage
require 'coverage'
Coverage.start
before = Coverage.peek_result
yield example
after = Coverage.peek_result
after.reject! do |file_name, after_coverage|
before[file_name] == after_coverage
end
Slide 41
Slide 41 text
41
Coverage
require 'coverage'
Coverage.start
before = Coverage.peek_result
yield example
after = Coverage.peek_result
after.select! do |file_name, after_coverage|
before[file_name] != after_coverage
end
44
Allocated Objects
1. Add tracepoint for constant definition
2. Load tests
3. Add tracepoint for object allocation
4. Run test
5. Get the list of objects allocated during test
6. Find which files define constants of these objects
Slide 45
Slide 45 text
45
Allocated Objects
TracePoint.new(:class) do |tp|
mod = tp.self
path = tp.path
constants_definition_paths[mod] ||= []
constants_definition_paths[mod] << path
end.enable
Slide 46
Slide 46 text
46
Allocated Objects
TracePoint.new(:class) do |tp|
mod = tp.self
path = tp.path
constants_definition_paths[mod] ||= []
constants_definition_paths[mod] << path
end.enable
Slide 47
Slide 47 text
47
Allocated Objects
TracePoint.new(:class) do |tp|
mod = tp.self
path = tp.path
constants_definition_paths[mod] ||= []
constants_definition_paths[mod] << path
end.enable
Slide 48
Slide 48 text
48
Allocated Objects
TracePoint.new(:c_call) do |tp|
next if tp.method_id != :new ||
tp.method_id != :allocate
created_object_classes << tp.self
end.enable(&example)
Slide 49
Slide 49 text
49
Allocated Objects
TracePoint.new(:c_call) do |tp|
next if tp.method_id != :new ||
tp.method_id != :allocate
created_object_classes << tp.self
end.enable(&example)
Slide 50
Slide 50 text
50
Allocated Objects
TracePoint.new(:c_call) do |tp|
next if tp.method_id != :new ||
tp.method_id != :allocate
created_object_classes << tp.self
end.enable(&example)
53
Described Class
1. Add tracepoint for constant definition
2. Load tests
3. Run test
4. Find file defining “described class” of the test
Slide 54
Slide 54 text
54
Described Class
RSpec.describe User do
# ...
end
yield example
described_class = example.metadata[:described_class]
constants_definition_paths[described_class]
Slide 55
Slide 55 text
55
Described Class
RSpec.describe User do
# ...
end
yield example
described_class = example.metadata[:described_class]
constants_definition_paths[described_class]
Slide 56
Slide 56 text
56
Described Class
RSpec.describe User do
# ...
end
yield example
described_class = example.metadata[:described_class]
constants_definition_paths[described_class]
Slide 57
Slide 57 text
57
Described Class
RSpec.describe User do
# ...
end
yield example
described_class = example.metadata[:described_class]
constants_definition_paths[described_class]
Slide 58
Slide 58 text
58
Described Class
"./spec/models/user_spec.rb[1:6]":
- app/models/user.rb
Slide 59
Slide 59 text
59
Parser
Slide 60
Slide 60 text
60
Parser
1. Parse source code for constant definitions
2. Run test
3. Parse files used by test
4. Search for calls to constants
Slide 61
Slide 61 text
61
Parser
require 'parser/current'
node = Parser::CurrentRuby.parse(File.read(file_path))
constants_defined = recursively_map_children(node) do |child|
child.to_a.last.to_s if %i[const casgn].include?(child.type)
end
yield example
constants_called = used_files.flat_map do |file_path|
node = Parser::CurrentRuby.parse(File.read(file_path))
recursively_map_children(node) do |child|
if child.type == :send && child.children.detect { |c| c.type == :const }
child.to_a.first.to_a.last.to_s
end
end
end
Slide 62
Slide 62 text
62
Parser
require 'parser/current'
node = Parser::CurrentRuby.parse(File.read(file_path))
constants_defined = recursively_map_children(node) do |child|
child.to_a.last.to_s if %i[const casgn].include?(child.type)
end
yield example
constants_called = used_files.flat_map do |file_path|
node = Parser::CurrentRuby.parse(File.read(file_path))
recursively_map_children(node) do |child|
if child.type == :send && child.children.detect { |c| c.type == :const }
child.to_a.first.to_a.last.to_s
end
end
end
Slide 63
Slide 63 text
63
Parser
require 'parser/current'
node = Parser::CurrentRuby.parse(File.read(file_path))
constants_defined = recursively_map_children(node) do |child|
child.to_a.last.to_s if %i[const casgn].include?(child.type)
end
yield example
constants_called = used_files.flat_map do |file_path|
node = Parser::CurrentRuby.parse(File.read(file_path))
recursively_map_children(node) do |child|
if child.type == :send && child.children.detect { |c| c.type == :const }
child.to_a.first.to_a.last.to_s
end
end
end
Slide 64
Slide 64 text
64
Parser
require 'parser/current'
node = Parser::CurrentRuby.parse(File.read(file_path))
constants_defined = recursively_map_children(node) do |child|
child.to_a.last.to_s if %i[const casgn].include?(child.type)
end
yield example
constants_called = used_files.flat_map do |file_path|
node = Parser::CurrentRuby.parse(File.read(file_path))
recursively_map_children(node) do |child|
if child.type == :send && child.children.detect { |c| c.type == :const }
child.to_a.first.to_a.last.to_s
end
end
end
Slide 65
Slide 65 text
65
Parser
require 'parser/current'
node = Parser::CurrentRuby.parse(File.read(file_path))
constants_defined = recursively_map_children(node) do |child|
child.to_a.last.to_s if %i[const casgn].include?(child.type)
end
yield example
constants_called = used_files.flat_map do |file_path|
node = Parser::CurrentRuby.parse(File.read(file_path))
recursively_map_children(node) do |child|
if child.type == :send && child.children.detect { |c| c.type == :const }
child.to_a.first.to_a.last.to_s
end
end
end
Slide 66
Slide 66 text
66
Parser
require 'parser/current'
node = Parser::CurrentRuby.parse(File.read(file_path))
constants_defined = recursively_map_children(node) do |child|
child.to_a.last.to_s if %i[const casgn].include?(child.type)
end
yield example
constants_called = used_files.flat_map do |file_path|
node = Parser::CurrentRuby.parse(File.read(file_path))
recursively_map_children(node) do |child|
if child.type == :send && child.children.detect { |c| c.type == :const }
child.to_a.first.to_a.last.to_s
end
end
end
Slide 67
Slide 67 text
67
Parser
require 'parser/current'
node = Parser::CurrentRuby.parse(File.read(file_path))
constants_defined = recursively_map_children(node) do |child|
child.to_a.last.to_s if %i[const casgn].include?(child.type)
end
yield example
constants_called = used_files.flat_map do |file_path|
node = Parser::CurrentRuby.parse(File.read(file_path))
recursively_map_children(node) do |child|
if child.type == :send && child.children.detect { |c| c.type == :const }
child.to_a.first.to_a.last.to_s
end
end
end
Slide 68
Slide 68 text
68
Parser
require 'parser/current'
node = Parser::CurrentRuby.parse(File.read(file_path))
constants_defined = recursively_map_children(node) do |child|
child.to_a.last.to_s if %i[const casgn].include?(child.type)
end
yield example
constants_called = used_files.flat_map do |file_path|
node = Parser::CurrentRuby.parse(File.read(file_path))
recursively_map_children(node) do |child|
if child.type == :send && child.children.detect { |c| c.type == :const }
child.to_a.first.to_a.last.to_s
end
end
end
Slide 69
Slide 69 text
69
Parser
require 'parser/current'
node = Parser::CurrentRuby.parse(File.read(file_path))
constants_defined = recursively_map_children(node) do |child|
child.to_a.last.to_s if %i[const casgn].include?(child.type)
end
yield example
constants_called = used_files.flat_map do |file_path|
node = Parser::CurrentRuby.parse(File.read(file_path))
recursively_map_children(node) do |child|
if child.type == :send && child.children.detect { |c| c.type == :const }
child.to_a.first.to_a.last.to_s
end
end
end
Slide 70
Slide 70 text
70
Parser
require 'parser/current'
node = Parser::CurrentRuby.parse(File.read(file_path))
constants_defined = recursively_map_children(node) do |child|
child.to_a.last.to_s if %i[const casgn].include?(child.type)
end
yield example
constants_called = used_files.flat_map do |file_path|
node = Parser::CurrentRuby.parse(File.read(file_path))
recursively_map_children(node) do |child|
if child.type == :send && child.children.detect { |c| c.type == :const }
child.to_a.first.to_a.last.to_s
end
end
end
113
Associated specs
Crystalball::Predictor::AssociatedSpecs.new(
from: %r{models/(.*).rb},
to: “./spec/models/%s_spec.rb"
)
Slide 114
Slide 114 text
114
Modified schema
Slide 115
Slide 115 text
115
Modified schema
1. Get changed tables from schema diff
2. Find which models define those tables
3. Find which tests should be run for these models
4. Run tests