Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Crystalball: predicting test failures

Crystalball: predicting test failures

A talk about Crystalball - a regression test selection library for Ruby (https://github.com/toptal/crystalball) for RubyKaigi 2019 (https://rubykaigi.org/2019/presentations/p0deje.html). The video of the talk is available at YouTube (https://www.youtube.com/watch?v=q2q-9Td71kE).

Alex Rodionov

April 19, 2019
Tweet

More Decks by Alex Rodionov

Other Decks in Programming

Transcript

  1. CRYSTALBALL
    Predicting test failures

    View Slide


  2. my name is Alex
    2

    View Slide


  3. my name is Alex
    3

    View Slide


  4. my name is Alex
    @p0deje
    4

    View Slide


  5. my name is Alex
    @p0deje
    5

    View Slide


  6. my name is Alex
    @p0deje
    6

    View Slide

  7. CRYSTALBALL
    Predicting test failures

    View Slide

  8. 1. Problems of regression testing
    2. Crystalball
    3. Live demo

    View Slide

  9. 1. Problems of regression testing
    2. Crystalball
    3. Live demo

    View Slide

  10. 10
    Tests are

    View Slide

  11. 11
    Tests are vital

    View Slide

  12. 12
    Tests are slow

    View Slide

  13. 13
    Tests are integrated

    View Slide

  14. 14
    Run all the tests on
    every change

    View Slide

  15. 15
    Matz hates tests

    View Slide

  16. 16
    Aaron Patterson blogged
    "Predicting test failures"
    Feb 2015

    View Slide

  17. 17
    Aaron Patterson blogged
    "Predicting test failures"
    Feb 2015
    Running tests is
    the worst.

    View Slide

  18. 18
    Aaron Patterson blogged
    "Predicting test failures"
    Feb 2015
    Seriously.

    View Slide

  19. 19
    Aaron Patterson blogged
    "Predicting test failures"
    Feb 2015
    It takes forever, and
    by the time they’re all done
    running, I forgot what I was
    doing.

    View Slide

  20. 20
    Aaron Patterson blogged
    "Predicting test failures"
    Feb 2015
    Pavel Shutsin made
    proof-of-concept
    Mar 2017

    View Slide

  21. 21
    Aaron Patterson blogged
    "Predicting test failures"
    Feb 2015
    Pavel Shutsin made
    proof-of-concept
    Mar 2017
    Crystalball 0.5.0
    was released
    Apr 2018

    View Slide

  22. 1. Problems of regression testing
    2. Crystalball
    3. Live demo

    View Slide

  23. 23
    Crystalball is a regression
    test selection library.

    View Slide

  24. 24
    # spec/spec_helper.rb
    if ENV['CRYSTALBALL'] == 'true'
    require 'crystalball'
    Crystalball::MapGenerator.start! do |config|
    config.register Crystalball::MapGenerator::CoverageStrategy.new
    end
    end

    View Slide

  25. 25
    $ env CRYSTALBALL=true bundle exec rspec

    $ cat tmp/crystalball_data.yml
    ---
    :type: Crystalball::ExecutionMap
    :commit: 647f096ed956bf6558b97364d8c1ee935d710109
    :timestamp: 1551601518
    :version:
    ---
    "./spec/controllers/home_controller_spec.rb[1:2]":
    - spec/controllers/home_controller_spec.rb
    - app/controllers/home_controller.rb
    - app/controllers/application_controller.rb

    View Slide

  26. 26
    $ env CRYSTALBALL=true bundle exec rspec

    $ cat tmp/crystalball_data.yml
    ---
    :type: Crystalball::ExecutionMap
    :commit: 647f096ed956bf6558b97364d8c1ee935d710109
    :timestamp: 1551601518
    :version:
    ---
    "./spec/controllers/home_controller_spec.rb[1:2]":
    - spec/controllers/home_controller_spec.rb
    - app/controllers/home_controller.rb
    - app/controllers/application_controller.rb

    View Slide

  27. 27
    $ env CRYSTALBALL=true bundle exec rspec

    $ cat tmp/crystalball_data.yml
    ---
    :type: Crystalball::ExecutionMap
    :commit: 647f096ed956bf6558b97364d8c1ee935d710109
    :timestamp: 1551601518
    :version:
    ---
    "./spec/controllers/home_controller_spec.rb[1:2]":
    - spec/controllers/home_controller_spec.rb
    - app/controllers/home_controller.rb
    - app/controllers/application_controller.rb

    View Slide

  28. 28
    --- a/app/models/user.rb
    +++ b/app/models/user.rb
    @@ -5,7 +5,6 @@ class User < ApplicationRecord
    validates :first_name, presence: true
    - validates :last_name, presence: true

    View Slide

  29. 29
    $ bundle exec crystalball
    I, [2019-04-02T10:17:11.595696 #92687] INFO -- : Crystalball starts to glow...
    I, [2019-04-02T10:17:11.626689 #92687] INFO -- : Starting RSpec.
    User
    ...
    should validate that :last_name cannot be empty/falsy (FAILED - 1)

    View Slide

  30. 30
    Generates a test-to-code map

    View Slide

  31. 31
    Generates a test-to-code map
    Predicts which tests should be run

    View Slide

  32. 32
    Generates a test-to-code map
    Predicts which tests should be run
    Runs those tests


    View Slide

  33. 33
    MapGenerator
    Predicts which tests should be run
    Runs those tests


    View Slide

  34. 34
    MapGenerator
    Predictor
    Runs those tests


    View Slide

  35. 35
    MapGenerator
    Predictor
    Runner


    View Slide

  36. 36
    MapGenerator
    Predictor
    Runner


    View Slide

  37. 37
    Coverage

    View Slide

  38. 38
    1.Get coverage before test
    2.Run test
    3.Get coverage after test
    4.Compare
    Coverage

    View Slide

  39. 39
    Coverage
    require 'coverage'
    Coverage.start
    before = Coverage.peek_result
    yield example
    after = Coverage.peek_result
    after.reject! do |file_name, after_coverage|
    before[file_name] == after_coverage
    end

    View Slide

  40. 40
    Coverage
    require 'coverage'
    Coverage.start
    before = Coverage.peek_result
    yield example
    after = Coverage.peek_result
    after.reject! do |file_name, after_coverage|
    before[file_name] == after_coverage
    end

    View Slide

  41. 41
    Coverage
    require 'coverage'
    Coverage.start
    before = Coverage.peek_result
    yield example
    after = Coverage.peek_result
    after.select! do |file_name, after_coverage|
    before[file_name] != after_coverage
    end

    View Slide

  42. 42
    Coverage
    "./spec/models/user_spec.rb[1:6]":
    - spec/models/user_spec.rb
    - app/models/user.rb
    - app/mailers/user_mailer.rb
    - app/mailers/application_mailer.rb

    View Slide

  43. 43
    Allocated Objects

    View Slide

  44. 44
    Allocated Objects
    1. Add tracepoint for constant definition
    2. Load tests
    3. Add tracepoint for object allocation
    4. Run test
    5. Get the list of objects allocated during test
    6. Find which files define constants of these objects

    View Slide

  45. 45
    Allocated Objects
    TracePoint.new(:class) do |tp|
    mod = tp.self
    path = tp.path
    constants_definition_paths[mod] ||= []
    constants_definition_paths[mod] << path
    end.enable

    View Slide

  46. 46
    Allocated Objects
    TracePoint.new(:class) do |tp|
    mod = tp.self
    path = tp.path
    constants_definition_paths[mod] ||= []
    constants_definition_paths[mod] << path
    end.enable

    View Slide

  47. 47
    Allocated Objects
    TracePoint.new(:class) do |tp|
    mod = tp.self
    path = tp.path
    constants_definition_paths[mod] ||= []
    constants_definition_paths[mod] << path
    end.enable

    View Slide

  48. 48
    Allocated Objects
    TracePoint.new(:c_call) do |tp|
    next if tp.method_id != :new ||
    tp.method_id != :allocate
    created_object_classes << tp.self
    end.enable(&example)

    View Slide

  49. 49
    Allocated Objects
    TracePoint.new(:c_call) do |tp|
    next if tp.method_id != :new ||
    tp.method_id != :allocate
    created_object_classes << tp.self
    end.enable(&example)

    View Slide

  50. 50
    Allocated Objects
    TracePoint.new(:c_call) do |tp|
    next if tp.method_id != :new ||
    tp.method_id != :allocate
    created_object_classes << tp.self
    end.enable(&example)

    View Slide

  51. 51
    Allocated Objects
    "./spec/models/user_spec.rb[1:6]":
    - app/models/user.rb
    - app/models/application_record.rb

    View Slide

  52. 52
    Described Class

    View Slide

  53. 53
    Described Class
    1. Add tracepoint for constant definition
    2. Load tests
    3. Run test
    4. Find file defining “described class” of the test

    View Slide

  54. 54
    Described Class
    RSpec.describe User do
    # ...
    end
    yield example
    described_class = example.metadata[:described_class]
    constants_definition_paths[described_class]

    View Slide

  55. 55
    Described Class
    RSpec.describe User do
    # ...
    end
    yield example
    described_class = example.metadata[:described_class]
    constants_definition_paths[described_class]

    View Slide

  56. 56
    Described Class
    RSpec.describe User do
    # ...
    end
    yield example
    described_class = example.metadata[:described_class]
    constants_definition_paths[described_class]

    View Slide

  57. 57
    Described Class
    RSpec.describe User do
    # ...
    end
    yield example
    described_class = example.metadata[:described_class]
    constants_definition_paths[described_class]

    View Slide

  58. 58
    Described Class
    "./spec/models/user_spec.rb[1:6]":
    - app/models/user.rb

    View Slide

  59. 59
    Parser

    View Slide

  60. 60
    Parser
    1. Parse source code for constant definitions

    2. Run test
    3. Parse files used by test
    4. Search for calls to constants

    View Slide

  61. 61
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end

    View Slide

  62. 62
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end

    View Slide

  63. 63
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end

    View Slide

  64. 64
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end

    View Slide

  65. 65
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end

    View Slide

  66. 66
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end

    View Slide

  67. 67
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end

    View Slide

  68. 68
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end

    View Slide

  69. 69
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end

    View Slide

  70. 70
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end

    View Slide

  71. 71
    Parser
    "./spec/models/user_spec.rb[1:6]":
    - app/models/user.rb
    - app/models/application_record.rb
    - app/mailers/user_mailer.rb
    - app/jobs/geocode_user_job.rb

    View Slide

  72. 72
    FactoryBot

    View Slide

  73. 73
    FactoryBot
    1. Patch FactoryBot to collect all defined factories
    2. Patch FactoryBot to collect factories used during test
    3. Run test

    View Slide

  74. 74
    FactoryBot

    View Slide

  75. 75
    FactoryBot
    "./spec/models/user_spec.rb[1:1]":
    - spec/factories/users.rb
    "./spec/models/user_spec.rb[1:6]": []

    View Slide

  76. 76
    ActionView

    View Slide

  77. 77
    ActionView
    1. Patch ActionView to collect all compiled views
    2. Run test

    View Slide

  78. 78
    ActionView

    View Slide

  79. 79
    ActionView
    "./spec/models/user_spec.rb[1:6]": []

    View Slide

  80. 80
    ActionView
    "./spec/controllers/home_controller_spec.rb[1:2]":
    - app/views/home/index.html.erb
    - app/views/layouts/application.html.erb

    View Slide

  81. 81
    I18n

    View Slide

  82. 82
    I18n
    1. Patch I18n to collect all loaded translations
    2. Patch I18n to collect all used translations
    3. Run test

    View Slide

  83. 83
    I18n

    View Slide

  84. 84
    I18n
    "./spec/models/user_spec.rb[1:6]": []

    View Slide

  85. 85
    I18n
    "./spec/system/sign_ins_spec.rb[1:1]":
    - config/locales/devise.en.yml

    View Slide

  86. 86
    Tables

    View Slide

  87. 87
    Tables
    1. Add tracepoint for constant definition
    2. Run test
    3. Collect all table names from used constants

    View Slide

  88. 88
    Tables
    yield example
    ActiveRecord::Base.descendants.map do |constant|
    if constants_definition_paths[constant]
    constant.table_name
    end
    end.compact

    View Slide

  89. 89
    Tables
    yield example
    ActiveRecord::Base.descendants.map do |constant|
    if constants_definition_paths[constant]
    constant.table_name
    end
    end.compact

    View Slide

  90. 90
    Tables
    yield example
    ActiveRecord::Base.descendants.map do |constant|
    if constants_definition_paths[constant]
    constant.table_name
    end
    end.compact

    View Slide

  91. 91
    Tables
    users:
    - app/models/user.rb
    - app/models/application_record.rb

    View Slide

  92. 92
    MapGenerator
    Predictor
    Runner


    View Slide

  93. 93
    Modified execution paths

    View Slide

  94. 94
    Modified execution paths
    1.Get modified files from Git
    2.Find related tests in generated map
    3.Run tests

    View Slide

  95. 95
    Modified execution paths
    require 'git'
    diff = Git.repo(Dir.pwd).diff('HEAD')
    diff.map do |d|
    d.path if d.type == 'modified' || d.type == 'new'
    end.compact

    View Slide

  96. 96
    Modified execution paths
    require 'git'
    diff = Git.repo(Dir.pwd).diff('HEAD')
    diff.map do |d|
    d.path if d.type == 'modified' || d.type == 'new'
    end.compact

    View Slide

  97. 97
    Modified execution paths
    require 'git'
    diff = Git.repo(Dir.pwd).diff('HEAD')
    diff.map do |d|
    d.path if d.type == 'modified' || d.type == 'new'
    end.compact

    View Slide

  98. 98
    Modified execution paths
    require 'git'
    diff = Git.repo(Dir.pwd).diff('HEAD')
    diff.map do |d|
    d.path if d.type == 'modified' || d.type == 'new'
    end.compact

    View Slide

  99. 99
    Modified execution paths
    --- a/app/models/user.rb
    +++ b/app/models/user.rb
    @@ -5,7 +5,6 @@ class User < ApplicationRecord
    validates :first_name, presence: true
    - validates :last_name, presence: true

    View Slide

  100. 100
    Modified execution paths
    {
    "./spec/models/user_spec.rb" =>
    [
    "1:9",
    "1:2",
    "1:3",
    "1:4",
    "1:5",
    "1:6",
    "1:7",
    "1:8",
    "1:1"
    ]
    }

    View Slide

  101. 101
    Modified specs

    View Slide

  102. 102
    Modified specs
    1.Get modified spec files from Git
    2.Run all of them

    View Slide

  103. 103
    Modified specs
    --- a/spec/models/project_spec.rb
    +++ b/spec/models/project_spec.rb
    @@ -1,6 +1,7 @@
    require 'rails_helper'
    RSpec.describe Project, type: :model do
    + it { is_expected.to be_a(Project) }

    View Slide

  104. 104
    Modified specs
    ["spec/models/project_spec.rb"]

    View Slide

  105. 105
    Modified support specs

    View Slide

  106. 106
    Modified support specs
    1. Get modified spec support files from Git
    2. Find tests using them from generated map
    3. Run tests

    View Slide

  107. 107
    Modified support specs
    RSpec.describe TasksController do
    include_context "project setup"

    RSpec.describe ProjectsController do
    include_context "project setup"

    View Slide

  108. 108
    Modified support specs
    RSpec.describe TasksController do
    include_context "project setup"

    RSpec.describe ProjectsController do
    include_context "project setup"

    View Slide

  109. 109
    Modified support specs
    --- a/spec/support/contexts/project_setup.rb
    +++ b/spec/support/contexts/project_setup.rb
    @@ -1,5 +1,4 @@
    RSpec.shared_context "project setup" do
    let(:user) { FactoryBot.create(:user) }
    let(:project) { FactoryBot.create(:project, owner: user) }
    - let(:task) { project.tasks.create!(name: "Test task") }
    end

    View Slide

  110. 110
    Modified support specs
    {
    "./spec/controllers/projects_controller_spec.rb" => ["1:1"],
    "./spec/controllers/tasks_controller_spec.rb" =>
    [
    "1:1:1",
    "1:2:1",
    "1:2:2",
    "1:2:3"
    ]
    }

    View Slide

  111. 111
    Associated specs

    View Slide

  112. 112
    Associated specs
    1.Manually associate source files with tests
    2.Run associated tests

    View Slide

  113. 113
    Associated specs
    Crystalball::Predictor::AssociatedSpecs.new(
    from: %r{models/(.*).rb},
    to: “./spec/models/%s_spec.rb"
    )

    View Slide

  114. 114
    Modified schema

    View Slide

  115. 115
    Modified schema
    1. Get changed tables from schema diff
    2. Find which models define those tables
    3. Find which tests should be run for these models
    4. Run tests

    View Slide

  116. 116
    Modified schema
    --- a/db/schema.rb
    +++ b/db/schema.rb
    create_table "tasks", force: :cascade do |t|
    t.string "name"
    t.integer "project_id"
    t.boolean "completed"
    + t.string "title"
    end

    View Slide

  117. 117
    Modified schema
    {
    "./spec/controllers/tasks_controller_spec.rb" =>
    [
    "1:1:1",
    "1:2:1",
    "1:2:2",
    ],
    "./spec/models/task_spec.rb" =>
    [
    "1:1",
    "1:2",
    "1:3"
    ],
    "./spec/system/tasks_spec.rb" => ["1:1"]
    }

    View Slide

  118. 118
    MapGenerator
    Predictor
    Runner


    View Slide

  119. 119
    RSpec

    View Slide

  120. 120
    RSpec
    1.Used when you run Crystalball
    2.Build predicted specs
    3.Run specs

    View Slide

  121. 1. Problems of regression testing
    2. Crystalball
    3. Live demo

    View Slide

  122. 122
    toptal.github.io/crystalball

    View Slide


  123. Alex Rodionov
    @p0deje

    123

    View Slide