Slide 1
Slide 1 text
RUN TEST RUN
Vladimir Dementyev, Evil Martians
RubyConfBy, 2017
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
EVIL OPEN SOURCE
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
THIS TALK
Slide 7
Slide 7 text
TESTING
* slap shit together and deploy
Slide 8
Slide 8 text
PROFILING
* Gorbypuff
Slide 9
Slide 9 text
AUTOMATION
Slide 10
Slide 10 text
LEGACY
* But it's not by foot
Slide 11
Slide 11 text
THE QUESTION
Slide 12
Slide 12 text
THE SURVEY
Q: Do you write tests?
Yes
No
94%
6%
* Conducted on Feb, 2017, about 200 respondents
Slide 13
Slide 13 text
THE SURVEY
Q: How long does your whole suite run?
Slide 14
Slide 14 text
THE SURVEY
Q: How long does your whole suite run?
5–10 min 10–20 min 20–60 min > 60 min
< 5 min
12%
10%
5%
26%
47%
Slide 15
Slide 15 text
THE SURVEY
Q: How long does your whole suite run?
Having more than 1000 examples
5–10 min 10–20 min 20–60 min > 60 min
< 5 min
22%
19%
8%
28%
23%
Slide 16
Slide 16 text
THE SURVEY
Q: How long does your whole suite run?
Having more than 1000 examples
5–10 min 10–20 min 20–60 min > 60 min
< 5 min
22%
19%
8%
28%
23%
~30% suites runs about 20 minutes
Slide 17
Slide 17 text
No content
Slide 18
Slide 18 text
THE STORY
OnboardIQ (Oct, 2016)
— ~3700 tests
— 20-22 minutes* on TravisCI (no
parallelism)
— Sometimes build spent hours in a queue
* only RSpec
Slide 19
Slide 19 text
No content
Slide 20
Slide 20 text
THE STORY
OnboardIQ (Oct, 2016)
— ~3700 tests
— 20-22 minutes on TravisCI (no
parallelism)
Migrated to CircleCI:
— 10 minutes (2x parallelism*)
— 4 minutes (5x parallelism)
* the same price as TravisCI
Slide 21
Slide 21 text
THE STORY
OnboardIQ (Mar, 2017)
— ~6100 tests
— 2.5 minutes on CircleCI (5x parallelism)
Slide 22
Slide 22 text
THE STORY
OnboardIQ (Mar, 2017)
— ~6100 tests
— 2.5 minutes on CircleCI (5x parallelism)
1.6x more tests – 1.6x faster
Slide 23
Slide 23 text
THE SETUP
Slide 24
Slide 24 text
THE SETUP
— RSpec http://rspec.info
Slide 25
Slide 25 text
No content
Slide 26
Slide 26 text
THE SURVEY
Q: Which testing framework do you use?
RSpec
Other
80%
2%
MiniT
est
18%
Slide 27
Slide 27 text
RSPEC VS. MINITEST
Synthetic benchmark: MiniTest
class SuperTest < MT Unit TestCase
1.upto(1_000).each do |i|
define_method "test_truth_ i}" do
assert_equal i, i
end
end
end
Finished in 0.040523s
Slide 28
Slide 28 text
RSPEC VS. MINITEST
Synthetic benchmark: RSpec
describe "Test" do
1.upto(1_000).each do |i|
specify { expect(i).to eq i }
end
end
Finished in 0.21693 seconds
Slide 29
Slide 29 text
RSPEC VS. MINITEST
Synthetic benchmark: RSpec
describe "Test" do
1.upto(1_000).each do |i|
specify { expect(i).to eq i }
end
end
Finished in 0.21693 seconds
5x slower
Slide 30
Slide 30 text
RSPEC VS. MINITEST
Real-life benchmark:
— ~4k examples*
— RSpec: 9 minutes 30 seconds
— MiniTest: 8 minutes 40 seconds
* Rails request tests
Slide 31
Slide 31 text
RSPEC VS. MINITEST
Real-life benchmark:
— ~4k examples*
— RSpec: 9 minutes 30 seconds
— MiniTest: 8 minutes 40 seconds
only 10% slower
* Rails request tests
Slide 32
Slide 32 text
No content
Slide 33
Slide 33 text
THE SETUP
— RSpec http://rspec.info
— FactoryGirl
Slide 34
Slide 34 text
THE SETUP
— RSpec http://rspec.info
— FactoryGirl (for both factories and
fixtures )
Slide 35
Slide 35 text
STEP #0
The worst case
— Sub-suite (500 examples)
— Run-time – 7 minutes
Slide 36
Slide 36 text
PROFILING
Slide 37
Slide 37 text
PROFILING
— General profilers (ruby prof, stackprof)
Slide 38
Slide 38 text
RUBYPROF
%self calls name
2.51 4690 *ActiveSupport Notifications Instrumenter#instrument
1.31 211554 String#to_s
1.31 47489 Arel Visitors Visitor#dispatch
1.15 117241 *Class#new
1.07 74655 ActiveSupport PerThreadRegistry#instance
1.01 531385 Module#
0.99 45291 ActiveRecord AttributeMethods#column_for_attribute
0.97 117482 #current
0.96 16038 *ActiveRecord AttributeMethods #primary_key
Slide 39
Slide 39 text
ENCRYPTION STORY
%self calls name
20.85 721 # bc_crypt
2.31 4690 *ActiveSupport Notifications Instrumenter#instrument
1.12 47489 Arel Visitors Visitor#dispatch
1.04 205208 String#to_s
0.87 531377 Module#
0.87 117109 *Class#new
Slide 40
Slide 40 text
ENCRYPTION STORY
%self calls name
20.85 721 # bc_crypt
2.31 4690 *ActiveSupport Notifications Instrumenter#instrument
1.12 47489 Arel Visitors Visitor#dispatch
1.04 205208 String#to_s
0.87 531377 Module#
0.87 117109 *Class#new
gem "sorcery"
Slide 41
Slide 41 text
ENCRYPTION STORY
# Sorcery config
config.user_config do |config|
# default is 10 for bcrypt!
config.stretches = 1 if Rails.env.test?
end
Slide 42
Slide 42 text
ENCRYPTION STORY
# Sorcery config
config.user_config do |config|
# default is 10 for bcrypt!
config.stretches = 1 if Rails.env.test?
end
Great impact – about 2x faster!
Slide 43
Slide 43 text
ENCRYPTION STORY
# Sorcery config
config.user_config do |config|
# default is 10 for bcrypt!
config.stretches = 1 if Rails.env.test?
end
Great impact – about 2x faster!
# Devise has the same in initializer
config.stretches = Rails.env.test? ? 1
: 10
Slide 44
Slide 44 text
* script example can be found here
Slide 45
Slide 45 text
* script example can be found here
Slide 46
Slide 46 text
* script example can be found here
Slide 47
Slide 47 text
STACKPROF
* http://www.brendangregg.com/flamegraphs.html
Slide 48
Slide 48 text
STACKPROF
* http://www.brendangregg.com/flamegraphs.html
Slide 49
Slide 49 text
STACKPROF
* http://www.brendangregg.com/flamegraphs.html
Slide 50
Slide 50 text
PROFILING
— General profilers (ruby prof, stackprof)
— ActiveSupport instrumentation
Slide 51
Slide 51 text
EVENT PROFILER
PROF_EVENT='sql.active_record' rspec
Slide 52
Slide 52 text
EVENT PROFILER
PROF_EVENT='sql.active_record' rspec
Profiling sql.active_record
Total time: 03 09.059
Total events: 162454
Top 5 slowest suites:
ApplicantsController – 00 34.050 (28637 / 102)
Api: V2 ApplicantsController – 00 23.019 (19275 / 30)
Slide 53
Slide 53 text
EVENT PROFILER
RSpec.configure do |config|
listener = RSpec: EventProfiler.new
config.reporter.register_listener(listener, )
config.after(:suite) { listener.print }
end
class RSpec EventProfiler
def initialize
ActiveSupport: Notifications.subscribe( )
end
end
* complete example can be found here
Slide 54
Slide 54 text
EVENT PROFILER
PROF_EVENT='factory_girl.run_factory' rspec
Slide 55
Slide 55 text
EVENT PROFILER
PROF_EVENT='factory_girl.run_factory' rspec
Caveat! Factory events are not exclusive
Profiling factory_girl.run_factory
Total time: 04 49.064
But total time is 03:52!
Slide 56
Slide 56 text
STEP #0
The worst case
— Sub-suite (500 examples)
— Run-time – 7 minutes
— DB* time – 3 minutes
* raw DB time, without ActiveRecord overhead
Slide 57
Slide 57 text
DB CLEANER STORY
Slide 58
Slide 58 text
DB CLEANER STORY
config.before(:each) do |example|
DatabaseCleaner.strategy = :deletion
DatabaseCleaner.start
end
config.after(:each) do
DatabaseCleaner.clean
end
Yep, this is real
Slide 59
Slide 59 text
No content
Slide 60
Slide 60 text
No content
Slide 61
Slide 61 text
gem "test_after_commit" # or Rails 5
Slide 62
Slide 62 text
PROFILING
— Take a look at your code
— General profilers (ruby prof, stackprof)
— ActiveSupport instrumentation
Slide 63
Slide 63 text
No content
Slide 64
Slide 64 text
STEP #1
Transactional tests (baseline)
— Sub-suite (500 examples)
— Run-time – 4 minutes 40 seconds
— DB time – 1 minute 10 seconds
1.5x faster
Slide 65
Slide 65 text
DATA GENERATION
Slide 66
Slide 66 text
DATA GENERATION
Slow Parts
— Excess Data
Slide 67
Slide 67 text
EXCESS DATA
context "test pagination" do
# page size is 40
before { create_list(:post, 41) }
it "shows second page" do
end
end
Slide 68
Slide 68 text
EXCESS DATA
# rails_helper.rb
Kaminari.configure do |conf|
conf.default_per_page = 2
end
WillPaginate.per_page = 2
context "test pagination" do
before { create_list(:post, 3) }
it "shows second page" do
end
end
Slide 69
Slide 69 text
EXCESS DATA
it "validates name presence" do
user = create(:user)
expect(user).not_to be_valid
end
Slide 70
Slide 70 text
EXCESS DATA
it "validates name presence" do
user = create(:user)
expect(user).not_to be_valid
end
it "validates name presence" do
user = build_stubbed(:user)
expect(user).not_to be_valid
end
Slide 71
Slide 71 text
EXCESS DATA
it "validates name presence" do
user = create(:user)
expect(user).not_to be_valid
end
it "validates name presence" do
user = build_stubbed(:user)
expect(user).not_to be_valid
end
Wouldn't it be nice to detect such bad examples
automatically
Slide 72
Slide 72 text
FACTORY DOCTOR
Slide 73
Slide 73 text
FACTORY DOCTOR
FDOC=1 rspec
FactoryDoctor found useless data generation:
User (./spec/models/user_spec.rb:3)
./spec/user_spec.rb:8 – 1 created objects, 00 00.114
Total wasted time: 00 00.165
Finished in 0.83153 seconds
* Complete example can be found here
Slide 74
Slide 74 text
FACTORY DOCTOR
When 200 examples:
FDOC=1 rspec
Total wasted time: 00 01.636
Finished in 2.87 seconds
* Complete example can be found here
Slide 75
Slide 75 text
DATA GENERATION
Slow Parts
— Excess Data
— Factory Cascade
Slide 76
Slide 76 text
factory :comment do
answer
author
end
factory :answer do
question
author
end
factory :question do
author
end
Slide 77
Slide 77 text
factory :comment do
answer
author
end
factory :answer do
question
author
end
factory :question do
author
end
create(:comment) # creates 5 records
Slide 78
Slide 78 text
FACTORY PROF
Slide 79
Slide 79 text
FACTORY PROF
Factory flame graphs
Slide 80
Slide 80 text
FACTORY STACK
create(:comment)
# stack = [
# :comment,
# :answer,
# :question,
# :user,
# :user,
# :user
# ]
Slide 81
Slide 81 text
FACTORY PROF
FPROF=flamegraph rspec spec/
Flamegraph written to tmp/factory flame.html
Total factories time: 00 31 06.012
* Complete example can be found here
Slide 82
Slide 82 text
FACTORY PROF
FPROF=event PROF_EVENT='factory.create' rspec spec/
Profiling factory.create
# Exclusive time
Total time: 03 21.170
Total events: 3056
Top 5 slowest suites:
* Complete example can be found here
Slide 83
Slide 83 text
CASCADE FIX #1
factory :stage do
funnel
end
create(:stage, funnel: funnel)
# raise ActiveRecord Invalid
# account can't be blank
Slide 84
Slide 84 text
CASCADE FIX #2
factory :stage do
funnel
account do
funnel&.account build(:account)
end
end
create(:stage, funnel: funnel)
# OK
Slide 85
Slide 85 text
CASCADE FIX #3
user = create_default(:user)
question = create_default(:question)
answer = create(:answer)
answer.question question
answer.user question.user user
Slide 86
Slide 86 text
FACTORY DEFAULT
module FactoryDefault
module CreateDefaultMethod
def create_default(name, args, &block)
res = create(name, args, &block)
FactoryDefault.register(name, res)
res
end
end
module StrategyExt
def association(runner)
return super unless FactoryDefault.exists?(runner.name)
FactoryDefault.get(runner.name)
end
end
# main stuff here
end
FactoryGirl: Syntax: Methods.include CreateDefaultMethod
FactoryGirl: Strategy: Create.prepend StrategyExt
* Complete example can be found here
Slide 87
Slide 87 text
W/O FACTORY
DEFAULT
Slide 88
Slide 88 text
WITH FACTORY
DEFAULT
Slide 89
Slide 89 text
No content
Slide 90
Slide 90 text
PROFILING
— Take a look at your code
— General profilers (ruby prof, stackprof)
— ActiveSupport instrumentation
— Factory profilers (FactoryProf,
FactoryDoctor)
Slide 91
Slide 91 text
DATA GENERATION
Slow Parts
— Excess Data
— Factory Cascade
— Repeatable Data / Actions
Slide 92
Slide 92 text
REPEATABLE DATA
before(:each) do
# some heavy setup
end
it "one" { }
it "ten" { }
Slide 93
Slide 93 text
REPEATABLE DATA
describe BeatleWeightedSearchQuery do
before(:each) do
@paul = create(:beatle, name: 'Paul')
@ringo = create(:beatle, name: 'Ringo')
@george = create(:beatle, name: 'George')
@john = create(:beatle, name: 'John')
end
# and about 15 examples here
end
Slide 94
Slide 94 text
REPEATABLE DATA
describe BeatleWeightedSearchQuery do
before_all do
@paul = create(:beatle, name: 'Paul')
@ringo = create(:beatle, name: 'Richard')
@george = create(:beatle, name: 'George')
@john = create(:beatle, name: 'John')
end
# and about 15 examples here
end
before_all = before(:all) + transaction
Slide 95
Slide 95 text
module BeforeAll
def before_all(&block)
raise "Block is required!" unless block_given?
before(:all) do
ActiveRecord: Base.connection
.begin_transaction(joinable: false)
instance_eval(&block)
end
after(:all) do
ActiveRecord: Base.connection
.rollback_transaction
end
end
end
Slide 96
Slide 96 text
REPEATABLE ACTIONS
subject { get "/api/v2/users", params; response }
it { is_expected.to be_success }
it { is_expected.to have_header('X-TOTAL-PAGES', 10) }
it { is_expected.to have_header('X-NEXT-PAGE', 2) }
Slide 97
Slide 97 text
REPEATABLE ACTIONS
aggregate_failures
it "returns the second page", :aggregate_failures do
is_expected.to be_success
is_expected.to have_header('X-TOTAL-PAGES', 10)
is_expected.to have_header('X-NEXT-PAGE', 2)
end
Slide 98
Slide 98 text
REPEATABLE ACTIONS
aggregate_failures
it "returns the second page", :aggregate_failures do
is_expected.to be_success
is_expected.to have_header('X-TOTAL-PAGES', 10)
is_expected.to have_header('X-NEXT-PAGE', 2)
end
Detect and auto-correct?
Slide 99
Slide 99 text
No content
Slide 100
Slide 100 text
No content
Slide 101
Slide 101 text
RUBOCOP
rubocop only RSpec/AggregateFailures
Offenses:
create_spec.rb:277 7 C RSpec/AggregateFailures:
Use :aggregate_failures instead of one liners.
context 'not valid secure fields' do
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
18 files inspected, 1 offense detected
* cop can be found here
Slide 102
Slide 102 text
RUBOCOP
rubocop only RSpec/AggregateFailures auto correct
1 offense detected, 1 offense corrected
* cop can be found here
Slide 103
Slide 103 text
STEP #2
Optimized factories and hooks
— Sub-suite (500 examples)
— Run-time – 3 minutes 10 seconds
— DB time – 55 seconds
1.5x faster than the baseline
2.1x faster than the worst case
Slide 104
Slide 104 text
FIXTURES
Slide 105
Slide 105 text
No content
Slide 106
Slide 106 text
Almost every test needs account (tenant) and funnel
(top-level model within a tenant)
Slide 107
Slide 107 text
SHARED CONTEXT
shared_context "account with funnel", account: true do
let(:account) { create(:account, :with_funnel) }
let(:funnel) { account.funnels.first }
end
describe "smth", :account do
end
Use the context everywhere you need both account and funnel
Slide 108
Slide 108 text
ANY FIXTURE
shared_context "account with funnel", account: true do
before(:all) do
@account = AnyFixture.add(:account) do
create(:account)
end
@funnel = AnyFixture.add(:funnel) do
create(:funnel, :with_stages, account: @account)
end
end
# Use `find` to have fresh AR object every time
let(:account) { Account.find(@account.id) }
let(:funnel) { account.funnels.first }
end
Slide 109
Slide 109 text
module AnyFixture
class self
def add(id)
# cache stores the result for the whole run
# (or unless cleaned up)
cache.fetch(id) do
# Track all queries made during the block call
ActiveSupport: Notifications.subscribed(
method(:subscriber),
"sql.active_record") do
yield
end
end
end
end
end
* complete example can be found here
Slide 110
Slide 110 text
module AnyFixture
INSERT_RXP = /^INSERT INTO ([\S]+)/
class self
def subscriber(_event, _start, _finish, _id, data)
matches = data.fetch(:sql).match(INSERT_RXP)
# Store every affected table
tables_cache[matches[1]] = true if matches
end
end
end
* complete example can be found here
Slide 111
Slide 111 text
module AnyFixture
class self
# Called within `after(:suite)` hook
def clean
# Delete all records from the affected tables
tables_cache.keys.each do |table|
ActiveRecord: Base.connection.execute %(
"DELETE FROM table}"
)
end
tables_cache.clear
cache.clear
end
end
end
* complete example can be found here
Slide 112
Slide 112 text
STEP #3
Adding fixtures
— Sub-suite (500 examples)
— Run-time – 2 minutes 10 seconds
— DB time – 45 seconds
2.2x faster than the baseline
3.1x faster than the worst case
Slide 113
Slide 113 text
FUTURE
— Sub-suite (500 examples)
— Run-time – 1 minutes 25 seconds
— DB time – 30 seconds
3x faster than the baseline
Slide 114
Slide 114 text
THE SETUP
— MRI
— RSpec http://rspec.info
— FactoryGirl
Slide 115
Slide 115 text
No content
Slide 116
Slide 116 text
MINITEST HELL
require "minitest/hell"
class UserControllerTest < MT U Test
# or
# parallelize_me!
end
Slide 117
Slide 117 text
MINITEST HELL
require "minitest/hell"
class UserControllerTest < MT U Test
# or
# parallelize_me!
end
bin/rails test test/users_controller_test.rb
20 runs, 23 assertions, 11 failures, 1 errors, 0 skips
Slide 118
Slide 118 text
RSPEC HELL
HELL=4 rspec ./spec/users_controller_spec.rb
Using RSpecHell
20 examples, 0 failures
Sometimes fails too
* complete example can be found here
Slide 119
Slide 119 text
MULTITHREADED
— Frameworks are not designed for
concurrent tests
— Libraries are not thread-safe (e.g.
Timecop, test adapters for ActiveJob,
ActionMailer)
— Only 20-25% faster*
* maybe, I don't know how to cook JRuby
Slide 120
Slide 120 text
DATABASE TRICKS
— Use in-memory FS for database
— Disable durability
NOTE: haven't noticed any effect with
Docker tmpfs
* read more in Nate Berkopec book
Slide 121
Slide 121 text
BOOT TIME
— Split helpers (rails_helper.rb,
spec_helper.rb, features_helper.rb,
etc)
— Try bootsnap (released 2 days ago)
— Use code preloaders
Slide 122
Slide 122 text
THE SURVEY
Q: Do you use code preloaders such as
Spring, Zeus?
Spring
Zeus
57%
3%
Nothing
40%
Slide 123
Slide 123 text
THE COST OF SPRING
— Save you 15-20 seconds every time you
run a command
— Once in a week you spend a couple of
minutes investigating the problem and
end with "F**king Spring! "
Slide 124
Slide 124 text
THANKS? QUESTIONS?
Vladimir Dementyev, Evil Martians
GitHub: @palkan
Twitter: @evilmartians, @palkan_tula