Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Crystalball: predicting test failures

Crystalball: predicting test failures

A talk about Crystalball - a regression test selection library for Ruby (https://github.com/toptal/crystalball) for RubyKaigi 2019 (https://rubykaigi.org/2019/presentations/p0deje.html). The video of the talk is available at YouTube (https://www.youtube.com/watch?v=q2q-9Td71kE).

Alex Rodionov

April 19, 2019
Tweet

More Decks by Alex Rodionov

Other Decks in Programming

Transcript

  1. 19 Aaron Patterson blogged "Predicting test failures" Feb 2015 It

    takes forever, and by the time they’re all done running, I forgot what I was doing.
  2. 21 Aaron Patterson blogged "Predicting test failures" Feb 2015 Pavel

    Shutsin made proof-of-concept Mar 2017 Crystalball 0.5.0 was released Apr 2018
  3. 24 # spec/spec_helper.rb if ENV['CRYSTALBALL'] == 'true' require 'crystalball' Crystalball::MapGenerator.start!

    do |config| config.register Crystalball::MapGenerator::CoverageStrategy.new end end
  4. 25 $ env CRYSTALBALL=true bundle exec rspec … $ cat

    tmp/crystalball_data.yml --- :type: Crystalball::ExecutionMap :commit: 647f096ed956bf6558b97364d8c1ee935d710109 :timestamp: 1551601518 :version: --- "./spec/controllers/home_controller_spec.rb[1:2]": - spec/controllers/home_controller_spec.rb - app/controllers/home_controller.rb - app/controllers/application_controller.rb …
  5. 26 $ env CRYSTALBALL=true bundle exec rspec … $ cat

    tmp/crystalball_data.yml --- :type: Crystalball::ExecutionMap :commit: 647f096ed956bf6558b97364d8c1ee935d710109 :timestamp: 1551601518 :version: --- "./spec/controllers/home_controller_spec.rb[1:2]": - spec/controllers/home_controller_spec.rb - app/controllers/home_controller.rb - app/controllers/application_controller.rb …
  6. 27 $ env CRYSTALBALL=true bundle exec rspec … $ cat

    tmp/crystalball_data.yml --- :type: Crystalball::ExecutionMap :commit: 647f096ed956bf6558b97364d8c1ee935d710109 :timestamp: 1551601518 :version: --- "./spec/controllers/home_controller_spec.rb[1:2]": - spec/controllers/home_controller_spec.rb - app/controllers/home_controller.rb - app/controllers/application_controller.rb …
  7. 28 --- a/app/models/user.rb +++ b/app/models/user.rb @@ -5,7 +5,6 @@ class

    User < ApplicationRecord validates :first_name, presence: true - validates :last_name, presence: true
  8. 29 $ bundle exec crystalball I, [2019-04-02T10:17:11.595696 #92687] INFO --

    : Crystalball starts to glow... I, [2019-04-02T10:17:11.626689 #92687] INFO -- : Starting RSpec. User ... should validate that :last_name cannot be empty/falsy (FAILED - 1)
  9. 39 Coverage require 'coverage' Coverage.start before = Coverage.peek_result yield example

    after = Coverage.peek_result after.reject! do |file_name, after_coverage| before[file_name] == after_coverage end
  10. 40 Coverage require 'coverage' Coverage.start before = Coverage.peek_result yield example

    after = Coverage.peek_result after.reject! do |file_name, after_coverage| before[file_name] == after_coverage end
  11. 41 Coverage require 'coverage' Coverage.start before = Coverage.peek_result yield example

    after = Coverage.peek_result after.select! do |file_name, after_coverage| before[file_name] != after_coverage end
  12. 44 Allocated Objects 1. Add tracepoint for constant definition 2.

    Load tests 3. Add tracepoint for object allocation 4. Run test 5. Get the list of objects allocated during test 6. Find which files define constants of these objects
  13. 45 Allocated Objects TracePoint.new(:class) do |tp| mod = tp.self path

    = tp.path constants_definition_paths[mod] ||= [] constants_definition_paths[mod] << path end.enable
  14. 46 Allocated Objects TracePoint.new(:class) do |tp| mod = tp.self path

    = tp.path constants_definition_paths[mod] ||= [] constants_definition_paths[mod] << path end.enable
  15. 47 Allocated Objects TracePoint.new(:class) do |tp| mod = tp.self path

    = tp.path constants_definition_paths[mod] ||= [] constants_definition_paths[mod] << path end.enable
  16. 48 Allocated Objects TracePoint.new(:c_call) do |tp| next if tp.method_id !=

    :new || tp.method_id != :allocate created_object_classes << tp.self end.enable(&example)
  17. 49 Allocated Objects TracePoint.new(:c_call) do |tp| next if tp.method_id !=

    :new || tp.method_id != :allocate created_object_classes << tp.self end.enable(&example)
  18. 50 Allocated Objects TracePoint.new(:c_call) do |tp| next if tp.method_id !=

    :new || tp.method_id != :allocate created_object_classes << tp.self end.enable(&example)
  19. 53 Described Class 1. Add tracepoint for constant definition 2.

    Load tests 3. Run test 4. Find file defining “described class” of the test
  20. 54 Described Class RSpec.describe User do # ... end yield

    example described_class = example.metadata[:described_class] constants_definition_paths[described_class]
  21. 55 Described Class RSpec.describe User do # ... end yield

    example described_class = example.metadata[:described_class] constants_definition_paths[described_class]
  22. 56 Described Class RSpec.describe User do # ... end yield

    example described_class = example.metadata[:described_class] constants_definition_paths[described_class]
  23. 57 Described Class RSpec.describe User do # ... end yield

    example described_class = example.metadata[:described_class] constants_definition_paths[described_class]
  24. 60 Parser 1. Parse source code for constant definitions
 2.

    Run test 3. Parse files used by test 4. Search for calls to constants
  25. 61 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  26. 62 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  27. 63 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  28. 64 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  29. 65 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  30. 66 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  31. 67 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  32. 68 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  33. 69 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  34. 70 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  35. 73 FactoryBot 1. Patch FactoryBot to collect all defined factories

    2. Patch FactoryBot to collect factories used during test 3. Run test
  36. 82 I18n 1. Patch I18n to collect all loaded translations

    2. Patch I18n to collect all used translations 3. Run test
  37. 87 Tables 1. Add tracepoint for constant definition 2. Run

    test 3. Collect all table names from used constants
  38. 94 Modified execution paths 1.Get modified files from Git 2.Find

    related tests in generated map 3.Run tests
  39. 95 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map

    do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact
  40. 96 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map

    do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact
  41. 97 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map

    do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact
  42. 98 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map

    do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact
  43. 99 Modified execution paths --- a/app/models/user.rb +++ b/app/models/user.rb @@ -5,7

    +5,6 @@ class User < ApplicationRecord validates :first_name, presence: true - validates :last_name, presence: true
  44. 100 Modified execution paths { "./spec/models/user_spec.rb" => [ "1:9", "1:2",

    "1:3", "1:4", "1:5", "1:6", "1:7", "1:8", "1:1" ] }
  45. 103 Modified specs --- a/spec/models/project_spec.rb +++ b/spec/models/project_spec.rb @@ -1,6 +1,7

    @@ require 'rails_helper' RSpec.describe Project, type: :model do + it { is_expected.to be_a(Project) }
  46. 106 Modified support specs 1. Get modified spec support files

    from Git 2. Find tests using them from generated map 3. Run tests
  47. 107 Modified support specs RSpec.describe TasksController do include_context "project setup"

    … RSpec.describe ProjectsController do include_context "project setup" …
  48. 108 Modified support specs RSpec.describe TasksController do include_context "project setup"

    … RSpec.describe ProjectsController do include_context "project setup" …
  49. 109 Modified support specs --- a/spec/support/contexts/project_setup.rb +++ b/spec/support/contexts/project_setup.rb @@ -1,5

    +1,4 @@ RSpec.shared_context "project setup" do let(:user) { FactoryBot.create(:user) } let(:project) { FactoryBot.create(:project, owner: user) } - let(:task) { project.tasks.create!(name: "Test task") } end
  50. 115 Modified schema 1. Get changed tables from schema diff

    2. Find which models define those tables 3. Find which tests should be run for these models 4. Run tests
  51. 116 Modified schema --- a/db/schema.rb +++ b/db/schema.rb create_table "tasks", force:

    :cascade do |t| t.string "name" t.integer "project_id" t.boolean "completed" + t.string "title" end
  52. 117 Modified schema { "./spec/controllers/tasks_controller_spec.rb" => [ "1:1:1", "1:2:1", "1:2:2",

    ], "./spec/models/task_spec.rb" => [ "1:1", "1:2", "1:3" ], "./spec/system/tasks_spec.rb" => ["1:1"] }