Testing in Joe Rafaniello and Chris Arcand 

Chris Arcand
 @chrisarcand Joe Rafaniello @jrafanie Remote - Washington, D.C. Minneapolis, MN Mahwah office 

Application Core Team 

✨Platform Team✨ 

@chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns • Troubleshooting (“WAT!?”) • Spec performance 

@chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns • Troubleshooting (“WAT!?”) • Spec performance 

@chrisarcand @jrafanie factory_girl Technologies RSpec VCR Ruby JavaScript Jasmine * Also, tiny sprinkles of MiniTest in gems/pending, but… 

@chrisarcand @jrafanie RSpec http://rspec.info/ http://rspec.info/documentation/ https://relishapp.com/rspec 

@chrisarcand @jrafanie Test Suites, Metadata, and ManageIQ helpers RSpec ManageIQisms 

@chrisarcand @jrafanie Sweet, sweet suites • Automation • Brakeman (security scanner) • Javascript (Jasmine) • ManageIQ Providers - Amazon • Migrations • Replication • VMDB 

@chrisarcand @jrafanie Sweet, sweet suites rake test rake test:automation rake test:automation:setup rake test:brakeman rake test:javascript rake test:javascript:setup rake test:manageiq-providers-amazon rake test:manageiq-providers-amazon:setup rake test:migrations rake test:migrations:down rake test:migrations:setup rake test:migrations:up rake test:replication rake test:replication:setup rake test:replication:teardown rake test:replication_util rake test:vmdb rake test:vmdb:setup rake test::setup rake test: 

@chrisarcand @jrafanie Sweet, sweet suites • Automation • Brakeman (security scanner) • Javascript (Jasmine) • ManageIQ Providers - Amazon • Migrations • Replication • VMDB 

@chrisarcand @jrafanie VMDB Migrations Replication Automation Controllers Helpers Initializers Mailers Models Requests Routing Services Views Presenters Rake Tasks REST API Helpers, helpers everywhere: AuthHelper ViewSpecHelper UiConstants ControllerSpecHelper AutomationSpecHelper MigrationSpecHelper ApiSpecHelper PresenterSpecHelper RakeTaskExampleGroup and more… Up migrations Down migrations REST API environment setup Replication setup Special migration management 

@chrisarcand @jrafanie VMDB Migrations Replication Automation Controllers Helpers Initializers Mailers Models Requests Routing Services Views Presenters Rake Tasks REST API Helpers, helpers everywhere: AuthHelper ViewSpecHelper UiConstants ControllerSpecHelper AutomationSpecHelper MigrationSpecHelper ApiSpecHelper PresenterSpecHelper RakeTaskExampleGroup and more… Up migrations Down migrations REST API environment setup Replication setup Special migration management WAT 

@chrisarcand @jrafanie RSpec knows what to do through metadata called ✨tags✨ describe "group with tagged specs" do it "slow example", :slow => true do; end it "ordinary example" do; end end ./some_spec.rb $ rspec ./some_spec.rb --tag ~slow 

@chrisarcand @jrafanie describe User, :type => :model do # Model-specific helpers, env end …
 config.include ModelHelper, :type => :model … user_spec.rb spec_helper.rb 

@chrisarcand @jrafanie Example: All the REST API specs live in ./spec/requests/api config.define_derived_metadata( :file_path => /spec\/requests\/api/) do |metadata| metadata[:rest_api] = true end config.include ApiSpecHelper, :rest_api => true config.before(:each, :rest_api => true) do init_api_spec_env end 

@chrisarcand @jrafanie Takeaways: 1. Spec location matters a lot. (Location matters in all Rails applications, but especially ManageIQ) 2. We have custom RSpec matchers and helpers for different spec types (And you should go check them out. See ./spec/spec_helper.rb) 

@chrisarcand @jrafanie Migration Specs RSpec ManageIQisms 

@chrisarcand @jrafanie Who tests migrations? • Manually testing is hard • Setup database • Migrate • Test conversion • Repeat up and down • Migration specs automate this 

@chrisarcand @jrafanie Migration spec tips • Don’t mix schema changes and data changes • Test data migrations • Migrations are snapshots in time • Don't use application models (app/models/*) • Use model stubs • good_migrations gem (soon™) • Libor 

@chrisarcand @jrafanie # db/migrate/20150817213409_clear_tenant_seed.rb class ClearTenantSeed < ActiveRecord::Migration class Tenant < ActiveRecord::Base; end def up say_with_time("...") do Tenant.update_all(:name => nil) end end end Model stub 

@chrisarcand @jrafanie # spec/migrations/20150817213409_clear_tenant_seed_spec.rb require_migration describe ClearTenantSeed do migration_context :up do it "works with one tenant" do tenant = migration_stub(:Tenant).create(…) migrate expect(tenant.reload.name).to be_nil end end end ✨ Magic 

@chrisarcand @jrafanie Who tests migrations? • FYI, Rails 5.0 generates versioned migrations class AddNameToUsers < ActiveRecord::Migration[5.0] ... end • Existing migrations use 4.2 migration code 

@chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns • Troubleshooting (“WAT!?”) • Spec performance 

@chrisarcand @jrafanie OH: “Don’t nest contexts EVARRRRRR!” RSpec structure 

@chrisarcand @jrafanie RSpec structure OH: “Don’t nest contexts EVARRRRRR!” Using context and describe blocks correctly will make you happier 

@chrisarcand @jrafanie 

@chrisarcand @jrafanie BAD: Contextual tests at the top level! In the future: • Harder to understand context • Promotes inefficiencies • Therefore, harder to add tests 

@chrisarcand @jrafanie 

@chrisarcand @jrafanie Harder to understand context
 Imagine seeing just this snippet buried in other examples as this file grows. WAT? 

@chrisarcand @jrafanie Promotes inefficiencies
 Creating an unused second record here! 

@chrisarcand @jrafanie Harder to add tests
 I now need to edit the setup here just to add a new test to avoid these issues 

@chrisarcand @jrafanie Harder to add tests
 I now need to edit the setup here just to add a new test to avoid these issues WAT 

@chrisarcand @jrafanie Separate context from expectation… 

@chrisarcand @jrafanie …and create a self-contained, portable set of circumstances 

@chrisarcand @jrafanie Easy to add different cases for #generation Easy to add different cases for totally different things at top level 

@chrisarcand @jrafanie 

@chrisarcand @jrafanie $ rspec ./spec/some_spec.rb --format=documentation 

@chrisarcand @jrafanie Your mileage may vary. Especially in a legacy codebase. 

@chrisarcand @jrafanie RSpec Anti-Patterns • Complex setup • Hard-to-test code • Random test failures 

@chrisarcand @jrafanie “I don’t understand what this test is doing.” “I have no idea why this test exists.” Complex setup 

@chrisarcand @jrafanie before(:each) do _guid_2, _server_2, @zone_2 = EvmSpecHelper.create_guid_miq_ guid, server, @zone = EvmSpecHelper.create_guid_miq_server_z @ems = FactoryGirl.create(:ems_vmware_with_auth…, :zone => @ other_ems = FactoryGirl.create(:ems_…_with_auth…, :zone => @ @worker_guid = MiqUUID.new_guid @worker_record = FactoryGirl.create(:miq_vim_broker_worker, :guid => …, :mi @drb_uri = "drb://127.0.0.1:12345" allow(DRb).to receive(:uri).and_return(@drb_uri) allow_any_instance_of(described_class).to receive(:sync_acti allow_any_instance_of(described_class).to receive(:sync_conf allow_any_instance_of(described_class).to receive(:set_conne allow_any_instance_of(ManageIQ::Providers::Vmware::InfraMana .to receive(:authentication_check).and_return([true, ""]) allow_any_instance_of(ManageIQ::Providers::Vmware::InfraMana .to receive(:authentication_status_ok?).and_return(true) end 

@chrisarcand @jrafanie • Find what you really need for setup • Create only what's needed • Figure out where it belongs - Abstractions can hide relevant details - Make setup methods explicit and flexible Beware of setup complexity 

@chrisarcand @jrafanie describe User do before do special_magic_setup_method @user = User.first end it “has write access when authenticated” do expect(@user).to have_write_access end it “can’t write when unauthenticated” do @user.update_attributes(…) expect(@user).to_not have_write_access end end 

@chrisarcand @jrafanie • instance_variable_get • .send • Many mocks, stubs, doubles • and the best one of all… Test smells: Write testable code 

@chrisarcand @jrafanie RSpec anti-patterns *_any_instance_of expect_any_instance_of(SomeClass).to receive(:some_method) allow_any_instance_of(SomeClass).to receive(:some_method) { … } 

@chrisarcand @jrafanie *_any_instance_of • Added ruby 2.3 support • rspec-mocks PR 1043 ☹ • rspec-mocks PR 1060 (for real) 

@chrisarcand @jrafanie PRs that remove these are welcome! git grep "any_instance_of" spec |wc -l 534 ✂✂✂ 

@chrisarcand @jrafanie • Add public APIs • Reduce state mutation Write testable code! 

@chrisarcand @jrafanie RSpec anti-patterns:
 Sporadic Failures This will randomly fail. Why? vm1 = Vm.create(…) vm2 = Vm.create(…) expect(Vm.pluck(:id)) .to eq([vm1.id, vm2.id]) No explicit order
 Use testing constructs like match_array! 

@chrisarcand @jrafanie Fix sporadic failures by… • Limiting the problem scope - Automated bisecting - Bisecting by comments! • Ask for help in Gitter! 

@chrisarcand @jrafanie Caching “There are only two hard things in Computer Science: cache invalidation and naming things.” -- Phil Karlton http://martinfowler.com/bliki/TwoHardThings.html • Add caching only if absolutely required • Clear caches with tests - See spec/support/evm_spec_helper.rb 

@chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns • Troubleshooting (“WAT!?”) • Spec performance 

@chrisarcand @jrafanie Travis failures • Look at the travis build log • Read the error • Now, read the backtrace • No, really, did you read the error? 

@chrisarcand @jrafanie Travis failures • Check other builds on travis • Master could be broken! • See build history @ github 

@chrisarcand @jrafanie Travis failures • Look at bundle install section for NEW gems • Share your findings: • https://gitter.im/ManageIQ/manageiq 

@chrisarcand @jrafanie Travis failures • Enable travis on your fork • Test without waiting behind ManageIQ org • Useful for debugging Travis errors https://travis-ci.org/jrafanie/manageiq/settings 

@chrisarcand @jrafanie Useful RSpec CLI switches -t ~slow (--tag)filter by tags (~ excludes tag) -f d (--format) [p]rogress(dots), [d]ocumentation, etc. --seed 1234 Run examples with a specific test order --bisect Find the examples to recreate a sporadic bug --only-failures Run the examples that just failed --fail-fast Abort a test run on the first error 

@chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns • Troubleshooting (“WAT!?”) • Spec performance 

@chrisarcand @jrafanie 1. Boot time “I can’t TDD, too slow!” 

@chrisarcand @jrafanie Boot time improvements • Ongoing: Huge improvements to Rails boot time • 2.5 seconds - lazy loading message catalogs, #8525 • bundler 1.12.1+, rubygems 2.5.0+ • Future: • Ruby 2.3.x • Sprockets #211 (lazy stat assets for size) • Bundler bugfix to prevent re-resolve (#4618) 

@chrisarcand @jrafanie Rails boot time • Spring • https://github.com/rails/spring • Preloads application boot for faster development • Rake tasks • Tests • Migrations • Boot speed must still be fast 

@chrisarcand @jrafanie 2. Total suite run time “The tests take forever.” “Let’s just push it to Travis lolzzzzz” 

@chrisarcand @jrafanie 

@chrisarcand @jrafanie Why? *Besides the fact that it’s a large, successful codebase originally written over a decade ago and we have over 12,000 individual tests 

@chrisarcand @jrafanie % of specs by individual test length
 (VMDB suite) 

@chrisarcand @jrafanie % of total suite time by individual test length 

@chrisarcand @jrafanie % of total suite time vs. % of total specs 70.3% of the test suite accounts for 11.45% of total time 25.3% of the test suite accounts for 37.43% of total time 

@chrisarcand @jrafanie % of total suite time vs. % of total specs 70.3% of the test suite accounts for 11.45% of total time 25.3% of the test suite accounts for 37.43% of total time WAT 

@chrisarcand @jrafanie Promotes inefficiencies
 Creating a unused second record here! Remember this? …and this example doesn’t even include FG associations 

@chrisarcand @jrafanie Analysis helps us improve… …but there’s no silver bullet. 

@chrisarcand @jrafanie However… 

@chrisarcand @jrafanie Good news! We’re already making improvements • Fall 2015: Moved to RSpec 3.x 
 (core performance ++) • In the future: 
 Pluggable providers split? Flag slow specs? 

@chrisarcand @jrafanie AND… *drumroll* 

@chrisarcand @jrafanie ✨Run your tests in parallel!✨ $ PARALLEL=true bundle exec rake test PR #8868 merged over the weekend (June 5th) Paired effort over the last few weeks First championed by Joe last year! Thanks Jason for reviewing/merge! 

@chrisarcand @jrafanie All benchmarks on Mid-2015 Macbook Pro 2.5 GHz Intel Core i7, ManageIQ commit #1689382 (May 18th, 2016) Time includes full load time Local vmdb test run before: ~7.5 minutes ~25 minutes Local vmdb test run after (8 cores): 

@chrisarcand @jrafanie Travis CI before: ~30 minutes ~45 minutes Travis CI after (2 cores): 

Thank you Questions? chrisarcand chrisarcand www.chrisarcand.com jrafanie jrafanie Slides available here