Crystalball: predicting test failures

A talk about Crystalball - a regression test selection library for Ruby (https://github.com/toptal/crystalball) for RubyKaigi 2019 (https://rubykaigi.org/2019/presentations/p0deje.html). The video of the talk is available at YouTube (https://www.youtube.com/watch?v=q2q-9Td71kE).

Transcript

1. CRYSTALBALL
   Predicting test failures
   

   View Slide

2. 
   my name is Alex
   2
   

   View Slide

3. 
   my name is Alex
   3
   

   View Slide

4. 
   my name is Alex
   @p0deje
   4
   

   View Slide

5. 
   my name is Alex
   @p0deje
   5
   

   View Slide

6. 
   my name is Alex
   @p0deje
   6
   

   View Slide

7. CRYSTALBALL
   Predicting test failures
   

   View Slide

8. 1. Problems of regression testing
   2. Crystalball
   3. Live demo
   

   View Slide

9. 1. Problems of regression testing
   2. Crystalball
   3. Live demo
   

   View Slide

10. 10
    Tests are
    

    View Slide

11. 11
    Tests are vital
    

    View Slide

12. 12
    Tests are slow
    

    View Slide

13. 13
    Tests are integrated
    

    View Slide

14. 14
    Run all the tests on
    every change
    

    View Slide

15. 15
    Matz hates tests
    

    View Slide

16. 16
    Aaron Patterson blogged
    "Predicting test failures"
    Feb 2015
    

    View Slide

17. 17
    Aaron Patterson blogged
    "Predicting test failures"
    Feb 2015
    Running tests is
    the worst.
    

    View Slide

18. 18
    Aaron Patterson blogged
    "Predicting test failures"
    Feb 2015
    Seriously.
    

    View Slide

19. 19
    Aaron Patterson blogged
    "Predicting test failures"
    Feb 2015
    It takes forever, and
    by the time they’re all done
    running, I forgot what I was
    doing.
    

    View Slide

20. 20
    Aaron Patterson blogged
    "Predicting test failures"
    Feb 2015
    Pavel Shutsin made
    proof-of-concept
    Mar 2017
    

    View Slide

21. 21
    Aaron Patterson blogged
    "Predicting test failures"
    Feb 2015
    Pavel Shutsin made
    proof-of-concept
    Mar 2017
    Crystalball 0.5.0
    was released
    Apr 2018
    
    

    View Slide

22. 1. Problems of regression testing
    2. Crystalball
    3. Live demo
    

    View Slide

23. 23
    Crystalball is a regression
    test selection library.
    

    View Slide

24. 24
    # spec/spec_helper.rb
    if ENV['CRYSTALBALL'] == 'true'
    require 'crystalball'
    Crystalball::MapGenerator.start! do |config|
    config.register Crystalball::MapGenerator::CoverageStrategy.new
    end
    end
    

    View Slide

25. 25
    $ env CRYSTALBALL=true bundle exec rspec
    …
    $ cat tmp/crystalball_data.yml
    ---
    :type: Crystalball::ExecutionMap
    :commit: 647f096ed956bf6558b97364d8c1ee935d710109
    :timestamp: 1551601518
    :version:
    ---
    "./spec/controllers/home_controller_spec.rb[1:2]":
    - spec/controllers/home_controller_spec.rb
    - app/controllers/home_controller.rb
    - app/controllers/application_controller.rb
    …
    

    View Slide

26. 26
    $ env CRYSTALBALL=true bundle exec rspec
    …
    $ cat tmp/crystalball_data.yml
    ---
    :type: Crystalball::ExecutionMap
    :commit: 647f096ed956bf6558b97364d8c1ee935d710109
    :timestamp: 1551601518
    :version:
    ---
    "./spec/controllers/home_controller_spec.rb[1:2]":
    - spec/controllers/home_controller_spec.rb
    - app/controllers/home_controller.rb
    - app/controllers/application_controller.rb
    …
    

    View Slide

27. 27
    $ env CRYSTALBALL=true bundle exec rspec
    …
    $ cat tmp/crystalball_data.yml
    ---
    :type: Crystalball::ExecutionMap
    :commit: 647f096ed956bf6558b97364d8c1ee935d710109
    :timestamp: 1551601518
    :version:
    ---
    "./spec/controllers/home_controller_spec.rb[1:2]":
    - spec/controllers/home_controller_spec.rb
    - app/controllers/home_controller.rb
    - app/controllers/application_controller.rb
    …
    

    View Slide

28. 28
    --- a/app/models/user.rb
    +++ b/app/models/user.rb
    @@ -5,7 +5,6 @@ class User < ApplicationRecord
    validates :first_name, presence: true
    - validates :last_name, presence: true
    

    View Slide

29. 29
    $ bundle exec crystalball
    I, [2019-04-02T10:17:11.595696 #92687] INFO -- : Crystalball starts to glow...
    I, [2019-04-02T10:17:11.626689 #92687] INFO -- : Starting RSpec.
    User
    ...
    should validate that :last_name cannot be empty/falsy (FAILED - 1)
    

    View Slide

30. 30
    Generates a test-to-code map
    

    View Slide

31. 31
    Generates a test-to-code map
    Predicts which tests should be run
    
    

    View Slide

32. 32
    Generates a test-to-code map
    Predicts which tests should be run
    Runs those tests
    
    
    

    View Slide

33. 33
    MapGenerator
    Predicts which tests should be run
    Runs those tests
    
    
    

    View Slide

34. 34
    MapGenerator
    Predictor
    Runs those tests
    
    
    

    View Slide

35. 35
    MapGenerator
    Predictor
    Runner
    
    
    

    View Slide

36. 36
    MapGenerator
    Predictor
    Runner
    
    
    

    View Slide

37. 37
    Coverage
    

    View Slide

38. 38
    1.Get coverage before test
    2.Run test
    3.Get coverage after test
    4.Compare
    Coverage
    

    View Slide

39. 39
    Coverage
    require 'coverage'
    Coverage.start
    before = Coverage.peek_result
    yield example
    after = Coverage.peek_result
    after.reject! do |file_name, after_coverage|
    before[file_name] == after_coverage
    end
    

    View Slide

40. 40
    Coverage
    require 'coverage'
    Coverage.start
    before = Coverage.peek_result
    yield example
    after = Coverage.peek_result
    after.reject! do |file_name, after_coverage|
    before[file_name] == after_coverage
    end
    

    View Slide

41. 41
    Coverage
    require 'coverage'
    Coverage.start
    before = Coverage.peek_result
    yield example
    after = Coverage.peek_result
    after.select! do |file_name, after_coverage|
    before[file_name] != after_coverage
    end
    

    View Slide

42. 42
    Coverage
    "./spec/models/user_spec.rb[1:6]":
    - spec/models/user_spec.rb
    - app/models/user.rb
    - app/mailers/user_mailer.rb
    - app/mailers/application_mailer.rb
    

    View Slide

43. 43
    Allocated Objects
    

    View Slide

44. 44
    Allocated Objects
    1. Add tracepoint for constant definition
    2. Load tests
    3. Add tracepoint for object allocation
    4. Run test
    5. Get the list of objects allocated during test
    6. Find which files define constants of these objects
    

    View Slide

45. 45
    Allocated Objects
    TracePoint.new(:class) do |tp|
    mod = tp.self
    path = tp.path
    constants_definition_paths[mod] ||= []
    constants_definition_paths[mod] << path
    end.enable
    

    View Slide

46. 46
    Allocated Objects
    TracePoint.new(:class) do |tp|
    mod = tp.self
    path = tp.path
    constants_definition_paths[mod] ||= []
    constants_definition_paths[mod] << path
    end.enable
    

    View Slide

47. 47
    Allocated Objects
    TracePoint.new(:class) do |tp|
    mod = tp.self
    path = tp.path
    constants_definition_paths[mod] ||= []
    constants_definition_paths[mod] << path
    end.enable
    

    View Slide

48. 48
    Allocated Objects
    TracePoint.new(:c_call) do |tp|
    next if tp.method_id != :new ||
    tp.method_id != :allocate
    created_object_classes << tp.self
    end.enable(&example)
    

    View Slide

49. 49
    Allocated Objects
    TracePoint.new(:c_call) do |tp|
    next if tp.method_id != :new ||
    tp.method_id != :allocate
    created_object_classes << tp.self
    end.enable(&example)
    

    View Slide

50. 50
    Allocated Objects
    TracePoint.new(:c_call) do |tp|
    next if tp.method_id != :new ||
    tp.method_id != :allocate
    created_object_classes << tp.self
    end.enable(&example)
    

    View Slide

51. 51
    Allocated Objects
    "./spec/models/user_spec.rb[1:6]":
    - app/models/user.rb
    - app/models/application_record.rb
    

    View Slide

52. 52
    Described Class
    

    View Slide

53. 53
    Described Class
    1. Add tracepoint for constant definition
    2. Load tests
    3. Run test
    4. Find file defining “described class” of the test
    

    View Slide

54. 54
    Described Class
    RSpec.describe User do
    # ...
    end
    yield example
    described_class = example.metadata[:described_class]
    constants_definition_paths[described_class]
    

    View Slide

55. 55
    Described Class
    RSpec.describe User do
    # ...
    end
    yield example
    described_class = example.metadata[:described_class]
    constants_definition_paths[described_class]
    

    View Slide

56. 56
    Described Class
    RSpec.describe User do
    # ...
    end
    yield example
    described_class = example.metadata[:described_class]
    constants_definition_paths[described_class]
    

    View Slide

57. 57
    Described Class
    RSpec.describe User do
    # ...
    end
    yield example
    described_class = example.metadata[:described_class]
    constants_definition_paths[described_class]
    

    View Slide

58. 58
    Described Class
    "./spec/models/user_spec.rb[1:6]":
    - app/models/user.rb
    

    View Slide

59. 59
    Parser
    

    View Slide

60. 60
    Parser
    1. Parse source code for constant definitions

    2. Run test
    3. Parse files used by test
    4. Search for calls to constants
    

    View Slide

61. 61
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end
    

    View Slide

62. 62
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end
    

    View Slide

63. 63
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end
    

    View Slide

64. 64
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end
    

    View Slide

65. 65
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end
    

    View Slide

66. 66
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end
    

    View Slide

67. 67
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end
    

    View Slide

68. 68
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end
    

    View Slide

69. 69
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end
    

    View Slide

70. 70
    Parser
    require 'parser/current'
    node = Parser::CurrentRuby.parse(File.read(file_path))
    constants_defined = recursively_map_children(node) do |child|
    child.to_a.last.to_s if %i[const casgn].include?(child.type)
    end
    yield example
    constants_called = used_files.flat_map do |file_path|
    node = Parser::CurrentRuby.parse(File.read(file_path))
    recursively_map_children(node) do |child|
    if child.type == :send && child.children.detect { |c| c.type == :const }
    child.to_a.first.to_a.last.to_s
    end
    end
    end
    

    View Slide

71. 71
    Parser
    "./spec/models/user_spec.rb[1:6]":
    - app/models/user.rb
    - app/models/application_record.rb
    - app/mailers/user_mailer.rb
    - app/jobs/geocode_user_job.rb
    

    View Slide

72. 72
    FactoryBot
    

    View Slide

73. 73
    FactoryBot
    1. Patch FactoryBot to collect all defined factories
    2. Patch FactoryBot to collect factories used during test
    3. Run test
    

    View Slide

74. 74
    FactoryBot
    
    

    View Slide

75. 75
    FactoryBot
    "./spec/models/user_spec.rb[1:1]":
    - spec/factories/users.rb
    "./spec/models/user_spec.rb[1:6]": []
    

    View Slide

76. 76
    ActionView
    

    View Slide

77. 77
    ActionView
    1. Patch ActionView to collect all compiled views
    2. Run test
    

    View Slide

78. 78
    ActionView
    
    

    View Slide

79. 79
    ActionView
    "./spec/models/user_spec.rb[1:6]": []
    

    View Slide

80. 80
    ActionView
    "./spec/controllers/home_controller_spec.rb[1:2]":
    - app/views/home/index.html.erb
    - app/views/layouts/application.html.erb
    

    View Slide

81. 81
    I18n
    

    View Slide

82. 82
    I18n
    1. Patch I18n to collect all loaded translations
    2. Patch I18n to collect all used translations
    3. Run test
    

    View Slide

83. 83
    I18n
    
    

    View Slide

84. 84
    I18n
    "./spec/models/user_spec.rb[1:6]": []
    

    View Slide

85. 85
    I18n
    "./spec/system/sign_ins_spec.rb[1:1]":
    - config/locales/devise.en.yml
    

    View Slide

86. 86
    Tables
    

    View Slide

87. 87
    Tables
    1. Add tracepoint for constant definition
    2. Run test
    3. Collect all table names from used constants
    

    View Slide

88. 88
    Tables
    yield example
    ActiveRecord::Base.descendants.map do |constant|
    if constants_definition_paths[constant]
    constant.table_name
    end
    end.compact
    

    View Slide

89. 89
    Tables
    yield example
    ActiveRecord::Base.descendants.map do |constant|
    if constants_definition_paths[constant]
    constant.table_name
    end
    end.compact
    

    View Slide

90. 90
    Tables
    yield example
    ActiveRecord::Base.descendants.map do |constant|
    if constants_definition_paths[constant]
    constant.table_name
    end
    end.compact
    

    View Slide

91. 91
    Tables
    users:
    - app/models/user.rb
    - app/models/application_record.rb
    

    View Slide

92. 92
    MapGenerator
    Predictor
    Runner
    
    
    

    View Slide

93. 93
    Modified execution paths
    

    View Slide

94. 94
    Modified execution paths
    1.Get modified files from Git
    2.Find related tests in generated map
    3.Run tests
    

    View Slide

95. 95
    Modified execution paths
    require 'git'
    diff = Git.repo(Dir.pwd).diff('HEAD')
    diff.map do |d|
    d.path if d.type == 'modified' || d.type == 'new'
    end.compact
    

    View Slide

96. 96
    Modified execution paths
    require 'git'
    diff = Git.repo(Dir.pwd).diff('HEAD')
    diff.map do |d|
    d.path if d.type == 'modified' || d.type == 'new'
    end.compact
    

    View Slide

97. 97
    Modified execution paths
    require 'git'
    diff = Git.repo(Dir.pwd).diff('HEAD')
    diff.map do |d|
    d.path if d.type == 'modified' || d.type == 'new'
    end.compact
    

    View Slide

98. 98
    Modified execution paths
    require 'git'
    diff = Git.repo(Dir.pwd).diff('HEAD')
    diff.map do |d|
    d.path if d.type == 'modified' || d.type == 'new'
    end.compact
    

    View Slide

99. 99
    Modified execution paths
    --- a/app/models/user.rb
    +++ b/app/models/user.rb
    @@ -5,7 +5,6 @@ class User < ApplicationRecord
    validates :first_name, presence: true
    - validates :last_name, presence: true
    

    View Slide

100. 100
     Modified execution paths
     {
     "./spec/models/user_spec.rb" =>
     [
     "1:9",
     "1:2",
     "1:3",
     "1:4",
     "1:5",
     "1:6",
     "1:7",
     "1:8",
     "1:1"
     ]
     }
     

     View Slide

101. 101
     Modified specs
     

     View Slide

102. 102
     Modified specs
     1.Get modified spec files from Git
     2.Run all of them
     

     View Slide

103. 103
     Modified specs
     --- a/spec/models/project_spec.rb
     +++ b/spec/models/project_spec.rb
     @@ -1,6 +1,7 @@
     require 'rails_helper'
     RSpec.describe Project, type: :model do
     + it { is_expected.to be_a(Project) }
     

     View Slide

104. 104
     Modified specs
     ["spec/models/project_spec.rb"]
     

     View Slide

105. 105
     Modified support specs
     

     View Slide

106. 106
     Modified support specs
     1. Get modified spec support files from Git
     2. Find tests using them from generated map
     3. Run tests
     

     View Slide

107. 107
     Modified support specs
     RSpec.describe TasksController do
     include_context "project setup"
     …
     RSpec.describe ProjectsController do
     include_context "project setup"
     …
     

     View Slide

108. 108
     Modified support specs
     RSpec.describe TasksController do
     include_context "project setup"
     …
     RSpec.describe ProjectsController do
     include_context "project setup"
     …
     

     View Slide

109. 109
     Modified support specs
     --- a/spec/support/contexts/project_setup.rb
     +++ b/spec/support/contexts/project_setup.rb
     @@ -1,5 +1,4 @@
     RSpec.shared_context "project setup" do
     let(:user) { FactoryBot.create(:user) }
     let(:project) { FactoryBot.create(:project, owner: user) }
     - let(:task) { project.tasks.create!(name: "Test task") }
     end
     

     View Slide

110. 110
     Modified support specs
     {
     "./spec/controllers/projects_controller_spec.rb" => ["1:1"],
     "./spec/controllers/tasks_controller_spec.rb" =>
     [
     "1:1:1",
     "1:2:1",
     "1:2:2",
     "1:2:3"
     ]
     }
     

     View Slide

111. 111
     Associated specs
     

     View Slide

112. 112
     Associated specs
     1.Manually associate source files with tests
     2.Run associated tests
     

     View Slide

113. 113
     Associated specs
     Crystalball::Predictor::AssociatedSpecs.new(
     from: %r{models/(.*).rb},
     to: “./spec/models/%s_spec.rb"
     )
     

     View Slide

114. 114
     Modified schema
     

     View Slide

115. 115
     Modified schema
     1. Get changed tables from schema diff
     2. Find which models define those tables
     3. Find which tests should be run for these models
     4. Run tests
     

     View Slide

116. 116
     Modified schema
     --- a/db/schema.rb
     +++ b/db/schema.rb
     create_table "tasks", force: :cascade do |t|
     t.string "name"
     t.integer "project_id"
     t.boolean "completed"
     + t.string "title"
     end
     

     View Slide

117. 117
     Modified schema
     {
     "./spec/controllers/tasks_controller_spec.rb" =>
     [
     "1:1:1",
     "1:2:1",
     "1:2:2",
     ],
     "./spec/models/task_spec.rb" =>
     [
     "1:1",
     "1:2",
     "1:3"
     ],
     "./spec/system/tasks_spec.rb" => ["1:1"]
     }
     

     View Slide

118. 118
     MapGenerator
     Predictor
     Runner
     
     
     

     View Slide

119. 119
     RSpec
     

     View Slide

120. 120
     RSpec
     1.Used when you run Crystalball
     2.Build predicted specs
     3.Run specs
     

     View Slide

121. 1. Problems of regression testing
     2. Crystalball
     3. Live demo
     

     View Slide

122. 122
     toptal.github.io/crystalball
     

     View Slide

123. 
     Alex Rodionov
     @p0deje
     
     123
     

     View Slide