Tweet
Tweet

Slide 1

Slide 1 text

RUN TEST RUN Vladimir Dementyev, Evil Martians RubyConfLT, 2017

Slide 2

Slide 2 text

RUN TEST RUN Vladimir Dementyev, Evil Martians RubyConfLT, 2017, 10th anniversary

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

EVIL OPEN SOURCE

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

THIS TALK

Slide 8

Slide 8 text

TESTING * slap shit together and deploy

Slide 9

Slide 9 text

PROFILING * Gorbypuff

Slide 10

Slide 10 text

AUTOMATION

Slide 11

Slide 11 text

LEGACY * But it's not by foot

Slide 12

Slide 12 text

THE QUESTION

Slide 13

Slide 13 text

THE SURVEY Q: Do you write tests? Yes No 94% 6% * Conducted on Feb, 2017, about 200 respondents

Slide 14

Slide 14 text

THE SURVEY Q: How long does your whole suite run?

Slide 15

Slide 15 text

THE SURVEY Q: How long does your whole suite run? 5–10 min 10–20 min 20–60 min > 60 min < 5 min 12% 10% 5% 26% 47%

Slide 16

Slide 16 text

THE SURVEY Q: How long does your whole suite run? Having more than 1000 examples 5–10 min 10–20 min 20–60 min > 60 min < 5 min 22% 19% 8% 28% 23%

Slide 17

Slide 17 text

THE SURVEY Q: How long does your whole suite run? Having more than 1000 examples 5–10 min 10–20 min 20–60 min > 60 min < 5 min 22% 19% 8% 28% 23% ~30% suites runs about 20 minutes

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

THE STORY OnboardIQ (Oct, 2016) — ~3700 tests — 20-22 minutes* on TravisCI (no parallelism) — Sometimes build spent hours in a queue * only RSpec

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

THE STORY OnboardIQ (Oct, 2016) — ~3700 tests — 20-22 minutes on TravisCI (no parallelism) Migrated to CircleCI: — 10 minutes (2x parallelism*) — 4 minutes (5x parallelism) * the same price as TravisCI

Slide 22

Slide 22 text

THE STORY OnboardIQ (Mar, 2017) — ~6100 tests — 2.5 minutes on CircleCI (5x parallelism)

Slide 23

Slide 23 text

THE STORY OnboardIQ (Mar, 2017) — ~6100 tests — 2.5 minutes on CircleCI (5x parallelism) 2.5k more tests – 40% faster

Slide 24

Slide 24 text

THE SETUP

Slide 25

Slide 25 text

THE SETUP — RSpec http://rspec.info

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

THE SURVEY Q: Which testing framework do you use? RSpec Other 80% 2% MiniT est 18%

Slide 28

Slide 28 text

RSPEC VS. MINITEST Synthetic benchmark: MiniTest class SuperTest < MT Unit TestCase 1.upto(1_000).each do |i| define_method "test_truth_ i}" do assert_equal i, i end end end Finished in 0.040523s

Slide 29

Slide 29 text

RSPEC VS. MINITEST Synthetic benchmark: RSpec describe "Test" do 1.upto(1_000).each do |i| specify { expect(i).to eq i } end end Finished in 0.21693 seconds

Slide 30

Slide 30 text

RSPEC VS. MINITEST Synthetic benchmark: RSpec describe "Test" do 1.upto(1_000).each do |i| specify { expect(i).to eq i } end end Finished in 0.21693 seconds 5x slower

Slide 31

Slide 31 text

RSPEC VS. MINITEST Real-life benchmark: — ~4k examples* — RSpec: 9 minutes 30 seconds — MiniTest: 8 minutes 40 seconds * Rails request tests

Slide 32

Slide 32 text

RSPEC VS. MINITEST Real-life benchmark: — ~4k examples* — RSpec: 9 minutes 30 seconds — MiniTest: 8 minutes 40 seconds only 10% slower * Rails request tests

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

THE SETUP — RSpec http://rspec.info — FactoryGirl

Slide 35

Slide 35 text

THE SETUP — RSpec http://rspec.info — FactoryGirl (for both factories and fixtures )

Slide 36

Slide 36 text

STEP #0 — Sub-suite (500 examples) — Run-time – 7 minutes

Slide 37

Slide 37 text

PROFILING — General profilers (ruby prof, stackprof)

Slide 38

Slide 38 text

RUBYPROF %self calls name 2.51 4690 *ActiveSupport Notifications Instrumenter#instrument 1.31 211554 String#to_s 1.31 47489 Arel Visitors Visitor#dispatch 1.15 117241 *Class#new 1.07 74655 ActiveSupport PerThreadRegistry#instance 1.01 531385 Module# 0.99 45291 ActiveRecord AttributeMethods#column_for_attribute 0.97 117482 #current 0.96 16038 *ActiveRecord AttributeMethods #primary_key

Slide 39

Slide 39 text

ENCRYPTION STORY %self calls name 20.85 721 # bc_crypt 2.31 4690 *ActiveSupport Notifications Instrumenter#instrument 1.12 47489 Arel Visitors Visitor#dispatch 1.04 205208 String#to_s 0.87 531377 Module# 0.87 117109 *Class#new

Slide 40

Slide 40 text

ENCRYPTION STORY %self calls name 20.85 721 # bc_crypt 2.31 4690 *ActiveSupport Notifications Instrumenter#instrument 1.12 47489 Arel Visitors Visitor#dispatch 1.04 205208 String#to_s 0.87 531377 Module# 0.87 117109 *Class#new gem "sorcery"

Slide 41

Slide 41 text

ENCRYPTION STORY # Sorcery config config.user_config do |config| # default is 10 for bcrypt! config.stretches = 1 if Rails.env.test? end

Slide 42

Slide 42 text

ENCRYPTION STORY # Sorcery config config.user_config do |config| # default is 10 for bcrypt! config.stretches = 1 if Rails.env.test? end Great impact – about 60-70%!

Slide 43

Slide 43 text

ENCRYPTION STORY # Sorcery config config.user_config do |config| # default is 10 for bcrypt! config.stretches = 1 if Rails.env.test? end Great impact – about 60-70%! # Devise has the same in initializer config.stretches = Rails.env.test? ? 1 : 10

Slide 44

Slide 44 text

* script example can be found here

Slide 45

Slide 45 text

* script example can be found here

Slide 46

Slide 46 text

STACKPROF * http://www.brendangregg.com/flamegraphs.html

Slide 47

Slide 47 text

STACKPROF * http://www.brendangregg.com/flamegraphs.html

Slide 48

Slide 48 text

PROFILING — General profilers (ruby prof, stackprof) — ActiveSupport instrumentation

Slide 49

Slide 49 text

EVENT PROFILER PROF_EVENT='sql.active_record' rspec

Slide 50

Slide 50 text

EVENT PROFILER PROF_EVENT='sql.active_record' rspec Profiling sql.active_record Total time: 03 09.059 Total events: 162454 Top 5 slowest suites: ApplicantsController – 00 34.050 (28637 / 102) Api: V2 ApplicantsController – 00 23.019 (19275 / 30)

Slide 51

Slide 51 text

EVENT PROFILER RSpec.configure do |config| listener = RSpec: EventProfiler.new config.reporter.register_listener(listener, ) config.after(:suite) { listener.print } end class RSpec EventProfiler def initialize ActiveSupport: Notifications.subscribe( ) end end * complete example can be found here

Slide 52

Slide 52 text

EVENT PROFILER PROF_EVENT='factory_girl.run_factory' rspec

Slide 53

Slide 53 text

EVENT PROFILER PROF_EVENT='factory_girl.run_factory' rspec Caveat! Factory events are not exclusive Profiling factory_girl.run_factory Total time: 04 49.064 But real time is 03:52* * How did I find it out? Continue reading

Slide 54

Slide 54 text

STEP #0 — Sub-suite (500 examples) — Run-time – 7 minutes — DB* time – 3 minutes * raw DB time, without ActiveRecord overhead

Slide 55

Slide 55 text

DB CLEANER STORY

Slide 56

Slide 56 text

DB CLEANER STORY config.before(:each) do |example| DatabaseCleaner.strategy = :deletion DatabaseCleaner.start end config.after(:each) do DatabaseCleaner.clean end Yep, this is real

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

gem "test_after_commit" # or Rails 5

Slide 60

Slide 60 text

PROFILING — Take a look at your code — General profilers (ruby prof, stackprof) — ActiveSupport instrumentation

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

STEP #1 — Sub-suite (500 examples) — Run-time – 4 minutes 40 seconds — DB time – 1 minute 10 seconds 30% faster

Slide 63

Slide 63 text

DATA GENERATION

Slide 64

Slide 64 text

DATA GENERATION Slow Parts — Excess Data

Slide 65

Slide 65 text

EXCESS DATA context "test pagination" do # page size is 40 before { create_list(:post, 41) } it "shows second page" do end end

Slide 66

Slide 66 text

EXCESS DATA # rails_helper.rb Kaminari.configure do |conf| conf.default_per_page = 2 end WillPaginate.per_page = 2 context "test pagination" do before { create_list(:post, 3) } it "shows second page" do end end

Slide 67

Slide 67 text

EXCESS DATA it "validates name presence" do user = create(:user) expect(user).not_to be_valid end

Slide 68

Slide 68 text

EXCESS DATA it "validates name presence" do user = create(:user) expect(user).not_to be_valid end it "validates name presence" do user = build_stubbed(:user) expect(user).not_to be_valid end

Slide 69

Slide 69 text

EXCESS DATA it "validates name presence" do user = create(:user) expect(user).not_to be_valid end it "validates name presence" do user = build_stubbed(:user) expect(user).not_to be_valid end Wouldn't it be nice to detect such bad examples automatically

Slide 70

Slide 70 text

FACTORY DOCTOR FDOC=1 rspec FactoryDoctor found useless data generation: User (./spec/models/user_spec.rb:3) ./spec/user_spec.rb:8 – 1 created objects, 00 00.114 Total wasted time: 00 00.165 Finished in 0.83153 seconds * Complete example can be found here

Slide 71

Slide 71 text

FACTORY DOCTOR When 200 examples: FDOC=1 rspec Total wasted time: 00 01.636 Finished in 2.87 seconds * Complete example can be found here

Slide 72

Slide 72 text

DATA GENERATION Slow Parts — Excess Data — Factory Cascade

Slide 73

Slide 73 text

factory :comment do answer author end factory :answer do question author end factory :question do author end

Slide 74

Slide 74 text

factory :comment do answer author end factory :answer do question author end factory :question do author end create(:comment) # creates 5 records

Slide 75

Slide 75 text

FACTORY CASCADE — How to fix?

Slide 76

Slide 76 text

FACTORY CASCADE — How to fix? — How to detect?

Slide 77

Slide 77 text

FACTORY PROF

Slide 78

Slide 78 text

FACTORY PROF Factory flame graphs

Slide 79

Slide 79 text

FACTORY STACK create(:comment) # stack = [ # :comment, # :answer, # :question, # :user, # :user, # :user # ]

Slide 80

Slide 80 text

FACTORY PROF FPROF=flamegraph rspec spec/ Flamegraph written to tmp/factory flame.html Total factories time: 00 31 06.012 * Complete example can be found here

Slide 81

Slide 81 text

FACTORY PROF FPROF=event PROF_EVENT='factory.create' rspec spec/ Profiling factory.create # Exclusive time Total time: 03 21.170 Total events: 3056 Top 5 slowest suites: * Complete example can be found here

Slide 82

Slide 82 text

PROFILING — Take a look at your code — General profilers (ruby prof, stackprof) — ActiveSupport instrumentation — Factory profilers (FactoryProf, FactoryDoctor)

Slide 83

Slide 83 text

CASCADE FIX #1 factory :stage do funnel end create(:stage, funnel: funnel) # raise ActiveRecord Invalid # account can't be blank

Slide 84

Slide 84 text

CASCADE FIX #2 factory :stage do funnel account do funnel&.account build(:account) end end create(:stage, funnel: funnel) # OK

Slide 85

Slide 85 text

CASCADE FIX #3 user = create_default(:user) question = create_default(:question) answer = create(:answer) answer.question question answer.user question.user user

Slide 86

Slide 86 text

FACTORY DEFAULT module FactoryDefault module CreateDefaultMethod def create_default(name, args, &block) res = create(name, args, &block) FactoryDefault.register(name, res) res end end module StrategyExt def association(runner) return super unless FactoryDefault.exists?(runner.name) FactoryDefault.get(runner.name) end end # main stuff here end FactoryGirl: Syntax: Methods.include CreateDefaultMethod FactoryGirl: Strategy: Create.prepend StrategyExt * Complete example can be found here

Slide 87

Slide 87 text

W/O FACTORY DEFAULT

Slide 88

Slide 88 text

WITH FACTORY DEFAULT

Slide 89

Slide 89 text

No content

Slide 90

Slide 90 text

DATA GENERATION Slow Parts — Excess Data — Factory Cascade — Repeatable Data / Actions

Slide 91

Slide 91 text

REPEATABLE DATA before(:each) do # some heavy setup end it "one" { } it "ten" { }

Slide 92

Slide 92 text

REPEATABLE DATA describe BeatleWeightedSearchQuery do before(:each) do @paul = create(:beatle, name: 'Paul') @ringo = create(:beatle, name: 'Ringo') @george = create(:beatle, name: 'George') @john = create(:beatle, name: 'John') end # and about 15 examples here end

Slide 93

Slide 93 text

REPEATABLE DATA describe BeatleWeightedSearchQuery do before_all do @paul = create(:beatle, name: 'Paul') @ringo = create(:beatle, name: 'Richard') @george = create(:beatle, name: 'George') @john = create(:beatle, name: 'John') end # and about 15 examples here end before_all = before(:all) + transaction

Slide 94

Slide 94 text

module BeforeAll def before_all(&block) raise "Block is required!" unless block_given? before(:all) do ActiveRecord: Base.connection .begin_transaction(joinable: false) instance_eval(&block) end after(:all) do ActiveRecord: Base.connection .rollback_transaction end end end

Slide 95

Slide 95 text

REPEATABLE ACTIONS subject { get "/api/v2/users", params; response } it { is_expected.to be_success } it { is_expected.to have_header('X-TOTAL-PAGES', 10) } it { is_expected.to have_header('X-NEXT-PAGE', 2) }

Slide 96

Slide 96 text

REPEATABLE ACTIONS aggregate_failures it "returns the second page", :aggregate_failures do is_expected.to be_success is_expected.to have_header('X-TOTAL-PAGES', 10) is_expected.to have_header('X-NEXT-PAGE', 2) end

Slide 97

Slide 97 text

REPEATABLE ACTIONS aggregate_failures it "returns the second page", :aggregate_failures do is_expected.to be_success is_expected.to have_header('X-TOTAL-PAGES', 10) is_expected.to have_header('X-NEXT-PAGE', 2) end Detect and auto-correct?

Slide 98

Slide 98 text

RUBOCOP rubocop only RSpec/AggregateFailures Offenses: create_spec.rb:277 7 C RSpec/AggregateFailures: Use :aggregate_failures instead of one liners. context 'not valid secure fields' do ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 18 files inspected, 1 offense detected

Slide 99

Slide 99 text

RUBOCOP rubocop only RSpec/AggregateFailures auto correct 1 offense detected, 1 offense corrected * cop can be found here

Slide 100

Slide 100 text

STEP #2 — Sub-suite (500 examples) — Run-time – 3 minutes 10 seconds — DB time – 55 seconds 30% faster than Step #1

Slide 101

Slide 101 text

FIXTURES

Slide 102

Slide 102 text

No content

Slide 103

Slide 103 text

Almost every test needs account (tenant) and funnel (top-level model within a tenant)

Slide 104

Slide 104 text

SHARED CONTEXT shared_context "account with funnel", account: true do let(:account) { create(:account, :with_funnel) } let(:funnel) { account.funnels.first } end describe "smth", :account do end Use the context everywhere you need both account and funnel

Slide 105

Slide 105 text

ANY FIXTURE shared_context "account with funnel", account: true do before(:all) do @account = AnyFixture.add(:account) do create(:account) end @funnel = AnyFixture.add(:funnel) do create(:funnel, :with_stages, account: @account) end end # Use `find` to have fresh AR object every time let(:account) { Account.find(@account.id) } let(:funnel) { account.funnels.first } end

Slide 106

Slide 106 text

module AnyFixture class self def add(id) # cache stores the result for the whole run # (or unless cleaned up) cache.fetch(id) do # Track all queries made during the block call ActiveSupport: Notifications.subscribed( method(:subscriber), "sql.active_record") do yield end end end end end * complete example can be found here

Slide 107

Slide 107 text

module AnyFixture INSERT_RXP = /^INSERT INTO ([\S]+)/ class self def subscriber(_event, _start, _finish, _id, data) matches = data.fetch(:sql).match(INSERT_RXP) # Store every affected table tables_cache[matches[1]] = true if matches end end end * complete example can be found here

Slide 108

Slide 108 text

module AnyFixture class self # Called within `after(:suite)` hook def clean # Delete all records from the affected tables tables_cache.keys.each do |table| ActiveRecord: Base.connection.execute %( "DELETE FROM table}" ) end tables_cache.clear cache.clear end end end * complete example can be found here

Slide 109

Slide 109 text

STEP #3 — Sub-suite (500 examples) — Run-time – 2 minutes 10 seconds — DB time – 45 seconds 50% faster than Step #2 110% faster than Step #1

Slide 110

Slide 110 text

FUTURE — Sub-suite (500 examples) — Run-time – 1 minutes 35 seconds — DB time – 30 seconds 3x faster that Step #1

Slide 111

Slide 111 text

THE SETUP — MRI — RSpec http://rspec.info — FactoryGirl

Slide 112

Slide 112 text

No content

Slide 113

Slide 113 text

MINITEST HELL require "minitest/hell" class UserControllerTest < MT U Test # or # parallelize_me! end

Slide 114

Slide 114 text

MINITEST HELL require "minitest/hell" class UserControllerTest < MT U Test # or # parallelize_me! end bin/rails test test/users_controller_test.rb 20 runs, 23 assertions, 11 failures, 1 errors, 0 skips

Slide 115

Slide 115 text

RSPEC HELL HELL=4 rspec ./spec/users_controller_spec.rb Using RSpecHell 20 examples, 0 failures Sometimes fails too * complete example can be found here

Slide 116

Slide 116 text

MULTITHREADED — Frameworks are not designed for concurrent tests — Libraries are not thread-safe (e.g. Timecop, test adapters for ActiveJob, ActionMailer) — Only 20-25% faster* * maybe, I don't know how to cook JRuby

Slide 117

Slide 117 text

DB CLEANER STORY #2 Alice: Why do we need it?

Slide 118

Slide 118 text

DB CLEANER STORY #2 Alice: Why do we need it? Bob: Multithreaded tests (e.g. Capybara)

Slide 119

Slide 119 text

DB CLEANER STORY #2 Alice: Why do we need it? Bob: Multithreaded tests (e.g. Capybara) Batman: Even with Capybara you don't need it!

Slide 120

Slide 120 text

DB CLEANER STORY #2 Alice: Why do we need it? Bob: Multithreaded tests (e.g. Capybara) Batman: Even with Capybara you don't need it! Alice: ???? Bob: ????

Slide 121

Slide 121 text

CAPYBARA STORY Classic setup with DatabaseCleaner — 230 examples — 14 minutes 20 seconds

Slide 122

Slide 122 text

CAPYBARA STORY Classic setup with DatabaseCleaner — 230 examples — 14 minutes 20 seconds What if could use the same connection in all threads

Slide 123

Slide 123 text

module ActiveRecord OneLove class self def connection=(conn) @connection = conn connection.singleton_class.prepend Connection connection end end module Ext def connection OneLove.connection super end end end * Complete example can be found here

Slide 124

Slide 124 text

module ActiveRecord OneLove # synchonize DB queries module Connection @@mutex = Mutex.new def execute(*) @@mutex.synchronize { super } end end end * Complete example can be found here

Slide 125

Slide 125 text

CAPYBARA STORY With ActiveRecord::OneLove — 230 examples — 9 minutes 30 seconds 40% faster

Slide 126

Slide 126 text

CAPYBARA STORY With ActiveRecord::OneLove — 230 examples — 9 minutes 30 seconds 40% faster NOTE: Experimental

Slide 127

Slide 127 text

SYSTEM TESTS — Selenium vs. PhantomJS, CapybaraWebkit — gem 'rack_session_access' — Proxy for static assets (e.g. NGINX)

Slide 128

Slide 128 text

SYSTEM TESTS module AcceptanceHelper def sign_in(user) visit login_path fill_in 'email', with: user.email fill_In 'passowrd', with: 'beerbeer' click_on 'Log in' end end

Slide 129

Slide 129 text

SYSTEM TESTS module AcceptanceHelper def sign_in(user) page.set_rack_session( 'warden.user.user.key' User.serialize_into_session(user) ) end end 10% faster

Slide 130

Slide 130 text

DATABASE TRICKS — Use in-memory FS for database — Disable durability NOTE: haven't noticed any effect with Docker tmpfs * read more in Nate Berkopec book

Slide 131

Slide 131 text

BOOT TIME — Split helpers (rails_helper.rb, spec_helper.rb, features_helper.rb, etc) — Try bootscale — Use Spring (or similar)

Slide 132

Slide 132 text

THE SURVEY Q: Do you use code preloaders such as Spring, Zeus? Spring Zeus 57% 3% Nothing 40%

Slide 133

Slide 133 text

THE COST OF SPRING — Save you 15-20 seconds every time you run a command — Once in a week you spend a couple of minutes investigating the problem and end with "F**king Spring! "

Slide 134

Slide 134 text

THANKS! QUESTIONS? Vladimir Dementyev, Evil Martians GitHub: @palkan Twitter: @evilmartians, @palkan_tula