Crystalball: predicting test failures

Crystalball: predicting test failures

A talk about Crystalball - a regression test selection library for Ruby (https://github.com/toptal/crystalball) for RubyKaigi 2019 (https://rubykaigi.org/2019/presentations/p0deje.html). The video of the talk is available at YouTube (https://www.youtube.com/watch?v=q2q-9Td71kE).

D049f551aa71e6326c74002ac8e6788a?s=128

Alex Rodionov

April 19, 2019
Tweet

Transcript

  1. CRYSTALBALL Predicting test failures

  2. my name is Alex 2

  3. my name is Alex 3

  4. my name is Alex @p0deje 4

  5. my name is Alex @p0deje 5

  6. my name is Alex @p0deje 6

  7. CRYSTALBALL Predicting test failures

  8. 1. Problems of regression testing 2. Crystalball 3. Live demo

  9. 1. Problems of regression testing 2. Crystalball 3. Live demo

  10. 10 Tests are

  11. 11 Tests are vital

  12. 12 Tests are slow

  13. 13 Tests are integrated

  14. 14 Run all the tests on every change

  15. 15 Matz hates tests

  16. 16 Aaron Patterson blogged "Predicting test failures" Feb 2015

  17. 17 Aaron Patterson blogged "Predicting test failures" Feb 2015 Running

    tests is the worst.
  18. 18 Aaron Patterson blogged "Predicting test failures" Feb 2015 Seriously.

  19. 19 Aaron Patterson blogged "Predicting test failures" Feb 2015 It

    takes forever, and by the time they’re all done running, I forgot what I was doing.
  20. 20 Aaron Patterson blogged "Predicting test failures" Feb 2015 Pavel

    Shutsin made proof-of-concept Mar 2017
  21. 21 Aaron Patterson blogged "Predicting test failures" Feb 2015 Pavel

    Shutsin made proof-of-concept Mar 2017 Crystalball 0.5.0 was released Apr 2018
  22. 1. Problems of regression testing 2. Crystalball 3. Live demo

  23. 23 Crystalball is a regression test selection library.

  24. 24 # spec/spec_helper.rb if ENV['CRYSTALBALL'] == 'true' require 'crystalball' Crystalball::MapGenerator.start!

    do |config| config.register Crystalball::MapGenerator::CoverageStrategy.new end end
  25. 25 $ env CRYSTALBALL=true bundle exec rspec … $ cat

    tmp/crystalball_data.yml --- :type: Crystalball::ExecutionMap :commit: 647f096ed956bf6558b97364d8c1ee935d710109 :timestamp: 1551601518 :version: --- "./spec/controllers/home_controller_spec.rb[1:2]": - spec/controllers/home_controller_spec.rb - app/controllers/home_controller.rb - app/controllers/application_controller.rb …
  26. 26 $ env CRYSTALBALL=true bundle exec rspec … $ cat

    tmp/crystalball_data.yml --- :type: Crystalball::ExecutionMap :commit: 647f096ed956bf6558b97364d8c1ee935d710109 :timestamp: 1551601518 :version: --- "./spec/controllers/home_controller_spec.rb[1:2]": - spec/controllers/home_controller_spec.rb - app/controllers/home_controller.rb - app/controllers/application_controller.rb …
  27. 27 $ env CRYSTALBALL=true bundle exec rspec … $ cat

    tmp/crystalball_data.yml --- :type: Crystalball::ExecutionMap :commit: 647f096ed956bf6558b97364d8c1ee935d710109 :timestamp: 1551601518 :version: --- "./spec/controllers/home_controller_spec.rb[1:2]": - spec/controllers/home_controller_spec.rb - app/controllers/home_controller.rb - app/controllers/application_controller.rb …
  28. 28 --- a/app/models/user.rb +++ b/app/models/user.rb @@ -5,7 +5,6 @@ class

    User < ApplicationRecord validates :first_name, presence: true - validates :last_name, presence: true
  29. 29 $ bundle exec crystalball I, [2019-04-02T10:17:11.595696 #92687] INFO --

    : Crystalball starts to glow... I, [2019-04-02T10:17:11.626689 #92687] INFO -- : Starting RSpec. User ... should validate that :last_name cannot be empty/falsy (FAILED - 1)
  30. 30 Generates a test-to-code map

  31. 31 Generates a test-to-code map Predicts which tests should be

    run
  32. 32 Generates a test-to-code map Predicts which tests should be

    run Runs those tests
  33. 33 MapGenerator Predicts which tests should be run Runs those

    tests
  34. 34 MapGenerator Predictor Runs those tests

  35. 35 MapGenerator Predictor Runner

  36. 36 MapGenerator Predictor Runner

  37. 37 Coverage

  38. 38 1.Get coverage before test 2.Run test 3.Get coverage after

    test 4.Compare Coverage
  39. 39 Coverage require 'coverage' Coverage.start before = Coverage.peek_result yield example

    after = Coverage.peek_result after.reject! do |file_name, after_coverage| before[file_name] == after_coverage end
  40. 40 Coverage require 'coverage' Coverage.start before = Coverage.peek_result yield example

    after = Coverage.peek_result after.reject! do |file_name, after_coverage| before[file_name] == after_coverage end
  41. 41 Coverage require 'coverage' Coverage.start before = Coverage.peek_result yield example

    after = Coverage.peek_result after.select! do |file_name, after_coverage| before[file_name] != after_coverage end
  42. 42 Coverage "./spec/models/user_spec.rb[1:6]": - spec/models/user_spec.rb - app/models/user.rb - app/mailers/user_mailer.rb -

    app/mailers/application_mailer.rb
  43. 43 Allocated Objects

  44. 44 Allocated Objects 1. Add tracepoint for constant definition 2.

    Load tests 3. Add tracepoint for object allocation 4. Run test 5. Get the list of objects allocated during test 6. Find which files define constants of these objects
  45. 45 Allocated Objects TracePoint.new(:class) do |tp| mod = tp.self path

    = tp.path constants_definition_paths[mod] ||= [] constants_definition_paths[mod] << path end.enable
  46. 46 Allocated Objects TracePoint.new(:class) do |tp| mod = tp.self path

    = tp.path constants_definition_paths[mod] ||= [] constants_definition_paths[mod] << path end.enable
  47. 47 Allocated Objects TracePoint.new(:class) do |tp| mod = tp.self path

    = tp.path constants_definition_paths[mod] ||= [] constants_definition_paths[mod] << path end.enable
  48. 48 Allocated Objects TracePoint.new(:c_call) do |tp| next if tp.method_id !=

    :new || tp.method_id != :allocate created_object_classes << tp.self end.enable(&example)
  49. 49 Allocated Objects TracePoint.new(:c_call) do |tp| next if tp.method_id !=

    :new || tp.method_id != :allocate created_object_classes << tp.self end.enable(&example)
  50. 50 Allocated Objects TracePoint.new(:c_call) do |tp| next if tp.method_id !=

    :new || tp.method_id != :allocate created_object_classes << tp.self end.enable(&example)
  51. 51 Allocated Objects "./spec/models/user_spec.rb[1:6]": - app/models/user.rb - app/models/application_record.rb

  52. 52 Described Class

  53. 53 Described Class 1. Add tracepoint for constant definition 2.

    Load tests 3. Run test 4. Find file defining “described class” of the test
  54. 54 Described Class RSpec.describe User do # ... end yield

    example described_class = example.metadata[:described_class] constants_definition_paths[described_class]
  55. 55 Described Class RSpec.describe User do # ... end yield

    example described_class = example.metadata[:described_class] constants_definition_paths[described_class]
  56. 56 Described Class RSpec.describe User do # ... end yield

    example described_class = example.metadata[:described_class] constants_definition_paths[described_class]
  57. 57 Described Class RSpec.describe User do # ... end yield

    example described_class = example.metadata[:described_class] constants_definition_paths[described_class]
  58. 58 Described Class "./spec/models/user_spec.rb[1:6]": - app/models/user.rb

  59. 59 Parser

  60. 60 Parser 1. Parse source code for constant definitions
 2.

    Run test 3. Parse files used by test 4. Search for calls to constants
  61. 61 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  62. 62 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  63. 63 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  64. 64 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  65. 65 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  66. 66 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  67. 67 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  68. 68 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  69. 69 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  70. 70 Parser require 'parser/current' node = Parser::CurrentRuby.parse(File.read(file_path)) constants_defined = recursively_map_children(node)

    do |child| child.to_a.last.to_s if %i[const casgn].include?(child.type) end yield example constants_called = used_files.flat_map do |file_path| node = Parser::CurrentRuby.parse(File.read(file_path)) recursively_map_children(node) do |child| if child.type == :send && child.children.detect { |c| c.type == :const } child.to_a.first.to_a.last.to_s end end end
  71. 71 Parser "./spec/models/user_spec.rb[1:6]": - app/models/user.rb - app/models/application_record.rb - app/mailers/user_mailer.rb -

    app/jobs/geocode_user_job.rb
  72. 72 FactoryBot

  73. 73 FactoryBot 1. Patch FactoryBot to collect all defined factories

    2. Patch FactoryBot to collect factories used during test 3. Run test
  74. 74 FactoryBot

  75. 75 FactoryBot "./spec/models/user_spec.rb[1:1]": - spec/factories/users.rb "./spec/models/user_spec.rb[1:6]": []

  76. 76 ActionView

  77. 77 ActionView 1. Patch ActionView to collect all compiled views

    2. Run test
  78. 78 ActionView

  79. 79 ActionView "./spec/models/user_spec.rb[1:6]": []

  80. 80 ActionView "./spec/controllers/home_controller_spec.rb[1:2]": - app/views/home/index.html.erb - app/views/layouts/application.html.erb

  81. 81 I18n

  82. 82 I18n 1. Patch I18n to collect all loaded translations

    2. Patch I18n to collect all used translations 3. Run test
  83. 83 I18n

  84. 84 I18n "./spec/models/user_spec.rb[1:6]": []

  85. 85 I18n "./spec/system/sign_ins_spec.rb[1:1]": - config/locales/devise.en.yml

  86. 86 Tables

  87. 87 Tables 1. Add tracepoint for constant definition 2. Run

    test 3. Collect all table names from used constants
  88. 88 Tables yield example ActiveRecord::Base.descendants.map do |constant| if constants_definition_paths[constant] constant.table_name

    end end.compact
  89. 89 Tables yield example ActiveRecord::Base.descendants.map do |constant| if constants_definition_paths[constant] constant.table_name

    end end.compact
  90. 90 Tables yield example ActiveRecord::Base.descendants.map do |constant| if constants_definition_paths[constant] constant.table_name

    end end.compact
  91. 91 Tables users: - app/models/user.rb - app/models/application_record.rb

  92. 92 MapGenerator Predictor Runner

  93. 93 Modified execution paths

  94. 94 Modified execution paths 1.Get modified files from Git 2.Find

    related tests in generated map 3.Run tests
  95. 95 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map

    do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact
  96. 96 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map

    do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact
  97. 97 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map

    do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact
  98. 98 Modified execution paths require 'git' diff = Git.repo(Dir.pwd).diff('HEAD') diff.map

    do |d| d.path if d.type == 'modified' || d.type == 'new' end.compact
  99. 99 Modified execution paths --- a/app/models/user.rb +++ b/app/models/user.rb @@ -5,7

    +5,6 @@ class User < ApplicationRecord validates :first_name, presence: true - validates :last_name, presence: true
  100. 100 Modified execution paths { "./spec/models/user_spec.rb" => [ "1:9", "1:2",

    "1:3", "1:4", "1:5", "1:6", "1:7", "1:8", "1:1" ] }
  101. 101 Modified specs

  102. 102 Modified specs 1.Get modified spec files from Git 2.Run

    all of them
  103. 103 Modified specs --- a/spec/models/project_spec.rb +++ b/spec/models/project_spec.rb @@ -1,6 +1,7

    @@ require 'rails_helper' RSpec.describe Project, type: :model do + it { is_expected.to be_a(Project) }
  104. 104 Modified specs ["spec/models/project_spec.rb"]

  105. 105 Modified support specs

  106. 106 Modified support specs 1. Get modified spec support files

    from Git 2. Find tests using them from generated map 3. Run tests
  107. 107 Modified support specs RSpec.describe TasksController do include_context "project setup"

    … RSpec.describe ProjectsController do include_context "project setup" …
  108. 108 Modified support specs RSpec.describe TasksController do include_context "project setup"

    … RSpec.describe ProjectsController do include_context "project setup" …
  109. 109 Modified support specs --- a/spec/support/contexts/project_setup.rb +++ b/spec/support/contexts/project_setup.rb @@ -1,5

    +1,4 @@ RSpec.shared_context "project setup" do let(:user) { FactoryBot.create(:user) } let(:project) { FactoryBot.create(:project, owner: user) } - let(:task) { project.tasks.create!(name: "Test task") } end
  110. 110 Modified support specs { "./spec/controllers/projects_controller_spec.rb" => ["1:1"], "./spec/controllers/tasks_controller_spec.rb" =>

    [ "1:1:1", "1:2:1", "1:2:2", "1:2:3" ] }
  111. 111 Associated specs

  112. 112 Associated specs 1.Manually associate source files with tests 2.Run

    associated tests
  113. 113 Associated specs Crystalball::Predictor::AssociatedSpecs.new( from: %r{models/(.*).rb}, to: “./spec/models/%s_spec.rb" )

  114. 114 Modified schema

  115. 115 Modified schema 1. Get changed tables from schema diff

    2. Find which models define those tables 3. Find which tests should be run for these models 4. Run tests
  116. 116 Modified schema --- a/db/schema.rb +++ b/db/schema.rb create_table "tasks", force:

    :cascade do |t| t.string "name" t.integer "project_id" t.boolean "completed" + t.string "title" end
  117. 117 Modified schema { "./spec/controllers/tasks_controller_spec.rb" => [ "1:1:1", "1:2:1", "1:2:2",

    ], "./spec/models/task_spec.rb" => [ "1:1", "1:2", "1:3" ], "./spec/system/tasks_spec.rb" => ["1:1"] }
  118. 118 MapGenerator Predictor Runner

  119. 119 RSpec

  120. 120 RSpec 1.Used when you run Crystalball 2.Build predicted specs

    3.Run specs
  121. 1. Problems of regression testing 2. Crystalball 3. Live demo

  122. 122 toptal.github.io/crystalball

  123. Alex Rodionov @p0deje 123