Testing in ManageIQ

Testing in Joe Rafaniello and Chris Arcand

Chris Arcand  @chrisarcand Joe Rafaniello @jrafanie Remote - Washington, D.C.
Minneapolis, MN Mahwah ofﬁce

Application Core Team

✨Platform Team✨

@chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns
• Troubleshooting (“WAT!?”) • Spec performance

@chrisarcand @jrafanie factory_girl Technologies RSpec VCR Ruby JavaScript Jasmine *
Also, tiny sprinkles of MiniTest in gems/pending, but…

@chrisarcand @jrafanie RSpec http://rspec.info/ http://rspec.info/documentation/ https://relishapp.com/rspec

@chrisarcand @jrafanie Test Suites, Metadata, and ManageIQ helpers RSpec ManageIQisms

@chrisarcand @jrafanie Sweet, sweet suites • Automation • Brakeman (security
scanner) • Javascript (Jasmine) • ManageIQ Providers - Amazon • Migrations • Replication • VMDB

@chrisarcand @jrafanie Sweet, sweet suites rake test rake test:automation rake
test:automation:setup rake test:brakeman rake test:javascript rake test:javascript:setup rake test:manageiq-providers-amazon rake test:manageiq-providers-amazon:setup rake test:migrations rake test:migrations:down rake test:migrations:setup rake test:migrations:up rake test:replication rake test:replication:setup rake test:replication:teardown rake test:replication_util rake test:vmdb rake test:vmdb:setup rake test:<suite>:setup rake test:<suite>

@chrisarcand @jrafanie Sweet, sweet suites • Automation • Brakeman (security
scanner) • Javascript (Jasmine) • ManageIQ Providers - Amazon • Migrations • Replication • VMDB

@chrisarcand @jrafanie VMDB Migrations Replication Automation Controllers Helpers Initializers Mailers
Models Requests Routing Services Views Presenters Rake Tasks REST API Helpers, helpers everywhere: AuthHelper ViewSpecHelper UiConstants ControllerSpecHelper AutomationSpecHelper MigrationSpecHelper ApiSpecHelper PresenterSpecHelper RakeTaskExampleGroup and more… Up migrations Down migrations REST API environment setup Replication setup Special migration management

@chrisarcand @jrafanie VMDB Migrations Replication Automation Controllers Helpers Initializers Mailers
Models Requests Routing Services Views Presenters Rake Tasks REST API Helpers, helpers everywhere: AuthHelper ViewSpecHelper UiConstants ControllerSpecHelper AutomationSpecHelper MigrationSpecHelper ApiSpecHelper PresenterSpecHelper RakeTaskExampleGroup and more… Up migrations Down migrations REST API environment setup Replication setup Special migration management WAT

@chrisarcand @jrafanie RSpec knows what to do through metadata called
✨tags✨ describe "group with tagged specs" do it "slow example", :slow => true do; end it "ordinary example" do; end end ./some_spec.rb $ rspec ./some_spec.rb --tag ~slow

@chrisarcand @jrafanie describe User, :type => :model do # Model-specific
helpers, env end …  config.include ModelHelper, :type => :model … user_spec.rb spec_helper.rb

@chrisarcand @jrafanie Example: All the REST API specs live in
./spec/requests/api config.define_derived_metadata( :file_path => /spec\/requests\/api/) do |metadata| metadata[:rest_api] = true end config.include ApiSpecHelper, :rest_api => true config.before(:each, :rest_api => true) do init_api_spec_env end

@chrisarcand @jrafanie Takeaways: 1. Spec location matters a lot. (Location
matters in all Rails applications, but especially ManageIQ) 2. We have custom RSpec matchers and helpers for different spec types (And you should go check them out. See ./spec/spec_helper.rb)

@chrisarcand @jrafanie Migration Specs RSpec ManageIQisms

@chrisarcand @jrafanie Who tests migrations? • Manually testing is hard
• Setup database • Migrate • Test conversion • Repeat up and down • Migration specs automate this

@chrisarcand @jrafanie Migration spec tips • Don’t mix schema changes
and data changes • Test data migrations • Migrations are snapshots in time • Don't use application models (app/models/*) • Use model stubs • good_migrations gem (soon™) • Libor

@chrisarcand @jrafanie # db/migrate/20150817213409_clear_tenant_seed.rb class ClearTenantSeed < ActiveRecord::Migration class Tenant
< ActiveRecord::Base; end def up say_with_time("...") do Tenant.update_all(:name => nil) end end end Model stub

@chrisarcand @jrafanie # spec/migrations/20150817213409_clear_tenant_seed_spec.rb require_migration describe ClearTenantSeed do migration_context :up
do it "works with one tenant" do tenant = migration_stub(:Tenant).create(…) migrate expect(tenant.reload.name).to be_nil end end end ✨ Magic

@chrisarcand @jrafanie Who tests migrations? • FYI, Rails 5.0 generates
versioned migrations class AddNameToUsers < ActiveRecord::Migration[5.0] ... end • Existing migrations use 4.2 migration code

@chrisarcand @jrafanie OH: “Don’t nest contexts EVARRRRRR!” RSpec structure

@chrisarcand @jrafanie RSpec structure OH: “Don’t nest contexts EVARRRRRR!” Using
context and describe blocks correctly will make you happier

@chrisarcand @jrafanie

@chrisarcand @jrafanie BAD: Contextual tests at the top level! In
the future: • Harder to understand context • Promotes inefﬁciencies • Therefore, harder to add tests

@chrisarcand @jrafanie Harder to understand context  Imagine seeing just this
snippet buried in other examples as this ﬁle grows. WAT?

@chrisarcand @jrafanie Promotes inefﬁciencies  Creating an unused second record here!

@chrisarcand @jrafanie Harder to add tests  I now need to
edit the setup here just to add a new test to avoid these issues

@chrisarcand @jrafanie Harder to add tests  I now need to
edit the setup here just to add a new test to avoid these issues WAT

@chrisarcand @jrafanie Separate context from expectation…

@chrisarcand @jrafanie …and create a self-contained, portable set of circumstances

@chrisarcand @jrafanie Easy to add different cases for #generation Easy
to add different cases for totally different things at top level

@chrisarcand @jrafanie $ rspec ./spec/some_spec.rb --format=documentation

@chrisarcand @jrafanie Your mileage may vary. Especially in a legacy
codebase.

@chrisarcand @jrafanie RSpec Anti-Patterns • Complex setup • Hard-to-test code
• Random test failures

@chrisarcand @jrafanie “I don’t understand what this test is doing.”
“I have no idea why this test exists.” Complex setup

@chrisarcand @jrafanie before(:each) do _guid_2, _server_2, @zone_2 = EvmSpecHelper.create_guid_miq_ guid,
server, @zone = EvmSpecHelper.create_guid_miq_server_z @ems = FactoryGirl.create(:ems_vmware_with_auth…, :zone => @ other_ems = FactoryGirl.create(:ems_…_with_auth…, :zone => @ @worker_guid = MiqUUID.new_guid @worker_record = FactoryGirl.create(:miq_vim_broker_worker, :guid => …, :mi @drb_uri = "drb://127.0.0.1:12345" allow(DRb).to receive(:uri).and_return(@drb_uri) allow_any_instance_of(described_class).to receive(:sync_acti allow_any_instance_of(described_class).to receive(:sync_conf allow_any_instance_of(described_class).to receive(:set_conne allow_any_instance_of(ManageIQ::Providers::Vmware::InfraMana .to receive(:authentication_check).and_return([true, ""]) allow_any_instance_of(ManageIQ::Providers::Vmware::InfraMana .to receive(:authentication_status_ok?).and_return(true) end

@chrisarcand @jrafanie • Find what you really need for setup
• Create only what's needed • Figure out where it belongs - Abstractions can hide relevant details - Make setup methods explicit and ﬂexible Beware of setup complexity

@chrisarcand @jrafanie describe User do before do special_magic_setup_method @user =
User.first end it “has write access when authenticated” do expect(@user).to have_write_access end it “can’t write when unauthenticated” do @user.update_attributes(…) expect(@user).to_not have_write_access end end

@chrisarcand @jrafanie • instance_variable_get • .send • Many mocks, stubs,
doubles • and the best one of all… Test smells: Write testable code

@chrisarcand @jrafanie RSpec anti-patterns *_any_instance_of expect_any_instance_of(SomeClass).to receive(:some_method) allow_any_instance_of(SomeClass).to receive(:some_method) {
… }

@chrisarcand @jrafanie *_any_instance_of • Added ruby 2.3 support • rspec-mocks
PR 1043 ☹ • rspec-mocks PR 1060 (for real)

@chrisarcand @jrafanie PRs that remove these are welcome! git grep
"any_instance_of" spec |wc -l 534 ✂✂✂

@chrisarcand @jrafanie • Add public APIs • Reduce state mutation
Write testable code!

@chrisarcand @jrafanie RSpec anti-patterns:  Sporadic Failures This will randomly fail.
Why? vm1 = Vm.create(…) vm2 = Vm.create(…) expect(Vm.pluck(:id)) .to eq([vm1.id, vm2.id]) No explicit order  Use testing constructs like match_array!

@chrisarcand @jrafanie Fix sporadic failures by… • Limiting the problem
scope - Automated bisecting - Bisecting by comments! • Ask for help in Gitter!

@chrisarcand @jrafanie Caching “There are only two hard things in
Computer Science: cache invalidation and naming things.” -- Phil Karlton http://martinfowler.com/bliki/TwoHardThings.html • Add caching only if absolutely required • Clear caches with tests - See spec/support/evm_spec_helper.rb

@chrisarcand @jrafanie Travis failures • Look at the travis build
log • Read the error • Now, read the backtrace • No, really, did you read the error?

@chrisarcand @jrafanie Travis failures • Check other builds on travis
• Master could be broken! • See build history @ github

@chrisarcand @jrafanie Travis failures • Look at bundle install section
for NEW gems • Share your ﬁndings: • https://gitter.im/ManageIQ/manageiq

@chrisarcand @jrafanie Travis failures • Enable travis on your fork
• Test without waiting behind ManageIQ org • Useful for debugging Travis errors https://travis-ci.org/jrafanie/manageiq/settings

@chrisarcand @jrafanie Useful RSpec CLI switches -t ~slow (--tag)filter by
tags (~ excludes tag) -f d (--format) [p]rogress(dots), [d]ocumentation, etc. --seed 1234 Run examples with a specific test order --bisect Find the examples to recreate a sporadic bug --only-failures Run the examples that just failed --fail-fast Abort a test run on the first error

@chrisarcand @jrafanie 1. Boot time “I can’t TDD, too slow!”

@chrisarcand @jrafanie Boot time improvements • Ongoing: Huge improvements to
Rails boot time • 2.5 seconds - lazy loading message catalogs, #8525 • bundler 1.12.1+, rubygems 2.5.0+ • Future: • Ruby 2.3.x • Sprockets #211 (lazy stat assets for size) • Bundler bugﬁx to prevent re-resolve (#4618)

@chrisarcand @jrafanie Rails boot time • Spring • https://github.com/rails/spring •
Preloads application boot for faster development • Rake tasks • Tests • Migrations • Boot speed must still be fast

@chrisarcand @jrafanie 2. Total suite run time “The tests take
forever.” “Let’s just push it to Travis lolzzzzz”

@chrisarcand @jrafanie Why? *Besides the fact that it’s a large,
successful codebase originally written over a decade ago and we have over 12,000 individual tests

@chrisarcand @jrafanie % of specs by individual test length  (VMDB
suite)

@chrisarcand @jrafanie % of total suite time by individual test
length

@chrisarcand @jrafanie % of total suite time vs. % of
total specs 70.3% of the test suite accounts for 11.45% of total time 25.3% of the test suite accounts for 37.43% of total time

@chrisarcand @jrafanie % of total suite time vs. % of
total specs 70.3% of the test suite accounts for 11.45% of total time 25.3% of the test suite accounts for 37.43% of total time WAT

@chrisarcand @jrafanie Promotes inefﬁciencies  Creating a unused second record here!
Remember this? …and this example doesn’t even include FG associations

@chrisarcand @jrafanie Analysis helps us improve… …but there’s no silver
bullet.

@chrisarcand @jrafanie However…

@chrisarcand @jrafanie Good news! We’re already making improvements • Fall
2015: Moved to RSpec 3.x   (core performance ++) • In the future:   Pluggable providers split? Flag slow specs?

@chrisarcand @jrafanie AND… *drumroll*

@chrisarcand @jrafanie ✨Run your tests in parallel!✨ $ PARALLEL=true bundle
exec rake test PR #8868 merged over the weekend (June 5th) Paired effort over the last few weeks First championed by Joe last year! Thanks Jason for reviewing/merge!

@chrisarcand @jrafanie All benchmarks on Mid-2015 Macbook Pro 2.5 GHz
Intel Core i7, ManageIQ commit #1689382 (May 18th, 2016) Time includes full load time Local vmdb test run before: ~7.5 minutes ~25 minutes Local vmdb test run after (8 cores):

@chrisarcand @jrafanie Travis CI before: ~30 minutes ~45 minutes Travis
CI after (2 cores):

Thank you Questions? chrisarcand chrisarcand www.chrisarcand.com jrafanie jrafanie Slides available
here

Testing in ManageIQ

Testing in ManageIQ

More Decks by Chris Arcand

Other Decks in Programming

Featured

Transcript