Share
Embed
Testing in ManageIQ
A talk about testing with RSpec in the ManageIQ cloud management platform. A short tour of some RSpec ManageIQisms, good spec structure and anti-patterns, troubleshooting test failures, and an overview of test runtime performance.
Chris Arcand
June 09, 2016
Transcript
79 slides
-
Chris Arcand @chrisarcand Joe Rafaniello @jrafanie Remote - Washington, D.C.
Minneapolis, MN Mahwah office -
@chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns
• Troubleshooting (“WAT!?”) • Spec performance -
@chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns
• Troubleshooting (“WAT!?”) • Spec performance -
@chrisarcand @jrafanie factory_girl Technologies RSpec VCR Ruby JavaScript Jasmine *
Also, tiny sprinkles of MiniTest in gems/pending, but… -
@chrisarcand @jrafanie Sweet, sweet suites • Automation • Brakeman (security
scanner) • Javascript (Jasmine) • ManageIQ Providers - Amazon • Migrations • Replication • VMDB -
@chrisarcand @jrafanie Sweet, sweet suites rake test rake test:automation rake
test:automation:setup rake test:brakeman rake test:javascript rake test:javascript:setup rake test:manageiq-providers-amazon rake test:manageiq-providers-amazon:setup rake test:migrations rake test:migrations:down rake test:migrations:setup rake test:migrations:up rake test:replication rake test:replication:setup rake test:replication:teardown rake test:replication_util rake test:vmdb rake test:vmdb:setup rake test:<suite>:setup rake test:<suite> -
@chrisarcand @jrafanie Sweet, sweet suites • Automation • Brakeman (security
scanner) • Javascript (Jasmine) • ManageIQ Providers - Amazon • Migrations • Replication • VMDB -
@chrisarcand @jrafanie VMDB Migrations Replication Automation Controllers Helpers Initializers Mailers
Models Requests Routing Services Views Presenters Rake Tasks REST API Helpers, helpers everywhere: AuthHelper ViewSpecHelper UiConstants ControllerSpecHelper AutomationSpecHelper MigrationSpecHelper ApiSpecHelper PresenterSpecHelper RakeTaskExampleGroup and more… Up migrations Down migrations REST API environment setup Replication setup Special migration management -
@chrisarcand @jrafanie VMDB Migrations Replication Automation Controllers Helpers Initializers Mailers
Models Requests Routing Services Views Presenters Rake Tasks REST API Helpers, helpers everywhere: AuthHelper ViewSpecHelper UiConstants ControllerSpecHelper AutomationSpecHelper MigrationSpecHelper ApiSpecHelper PresenterSpecHelper RakeTaskExampleGroup and more… Up migrations Down migrations REST API environment setup Replication setup Special migration management WAT -
@chrisarcand @jrafanie RSpec knows what to do through metadata called
✨tags✨ describe "group with tagged specs" do it "slow example", :slow => true do; end it "ordinary example" do; end end ./some_spec.rb $ rspec ./some_spec.rb --tag ~slow -
@chrisarcand @jrafanie describe User, :type => :model do # Model-specific
helpers, env end … config.include ModelHelper, :type => :model … user_spec.rb spec_helper.rb -
@chrisarcand @jrafanie Example: All the REST API specs live in
./spec/requests/api config.define_derived_metadata( :file_path => /spec\/requests\/api/) do |metadata| metadata[:rest_api] = true end config.include ApiSpecHelper, :rest_api => true config.before(:each, :rest_api => true) do init_api_spec_env end -
@chrisarcand @jrafanie Takeaways: 1. Spec location matters a lot. (Location
matters in all Rails applications, but especially ManageIQ) 2. We have custom RSpec matchers and helpers for different spec types (And you should go check them out. See ./spec/spec_helper.rb) -
@chrisarcand @jrafanie Who tests migrations? • Manually testing is hard
• Setup database • Migrate • Test conversion • Repeat up and down • Migration specs automate this -
@chrisarcand @jrafanie Migration spec tips • Don’t mix schema changes
and data changes • Test data migrations • Migrations are snapshots in time • Don't use application models (app/models/*) • Use model stubs • good_migrations gem (soon™) • Libor -
@chrisarcand @jrafanie # db/migrate/20150817213409_clear_tenant_seed.rb class ClearTenantSeed < ActiveRecord::Migration class Tenant
< ActiveRecord::Base; end def up say_with_time("...") do Tenant.update_all(:name => nil) end end end Model stub -
@chrisarcand @jrafanie # spec/migrations/20150817213409_clear_tenant_seed_spec.rb require_migration describe ClearTenantSeed do migration_context :up
do it "works with one tenant" do tenant = migration_stub(:Tenant).create(…) migrate expect(tenant.reload.name).to be_nil end end end ✨ Magic -
@chrisarcand @jrafanie Who tests migrations? • FYI, Rails 5.0 generates
versioned migrations class AddNameToUsers < ActiveRecord::Migration[5.0] ... end • Existing migrations use 4.2 migration code -
@chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns
• Troubleshooting (“WAT!?”) • Spec performance -
@chrisarcand @jrafanie RSpec structure OH: “Don’t nest contexts EVARRRRRR!” Using
context and describe blocks correctly will make you happier -
@chrisarcand @jrafanie BAD: Contextual tests at the top level! In
the future: • Harder to understand context • Promotes inefficiencies • Therefore, harder to add tests -
@chrisarcand @jrafanie Harder to understand context Imagine seeing just this
snippet buried in other examples as this file grows. WAT? -
@chrisarcand @jrafanie Harder to add tests I now need to
edit the setup here just to add a new test to avoid these issues -
@chrisarcand @jrafanie Harder to add tests I now need to
edit the setup here just to add a new test to avoid these issues WAT -
@chrisarcand @jrafanie Easy to add different cases for #generation Easy
to add different cases for totally different things at top level -
@chrisarcand @jrafanie RSpec Anti-Patterns • Complex setup • Hard-to-test code
• Random test failures -
@chrisarcand @jrafanie “I don’t understand what this test is doing.”
“I have no idea why this test exists.” Complex setup -
@chrisarcand @jrafanie before(:each) do _guid_2, _server_2, @zone_2 = EvmSpecHelper.create_guid_miq_ guid,
server, @zone = EvmSpecHelper.create_guid_miq_server_z @ems = FactoryGirl.create(:ems_vmware_with_auth…, :zone => @ other_ems = FactoryGirl.create(:ems_…_with_auth…, :zone => @ @worker_guid = MiqUUID.new_guid @worker_record = FactoryGirl.create(:miq_vim_broker_worker, :guid => …, :mi @drb_uri = "drb://127.0.0.1:12345" allow(DRb).to receive(:uri).and_return(@drb_uri) allow_any_instance_of(described_class).to receive(:sync_acti allow_any_instance_of(described_class).to receive(:sync_conf allow_any_instance_of(described_class).to receive(:set_conne allow_any_instance_of(ManageIQ::Providers::Vmware::InfraMana .to receive(:authentication_check).and_return([true, ""]) allow_any_instance_of(ManageIQ::Providers::Vmware::InfraMana .to receive(:authentication_status_ok?).and_return(true) end -
@chrisarcand @jrafanie • Find what you really need for setup
• Create only what's needed • Figure out where it belongs - Abstractions can hide relevant details - Make setup methods explicit and flexible Beware of setup complexity -
@chrisarcand @jrafanie describe User do before do special_magic_setup_method @user =
User.first end it “has write access when authenticated” do expect(@user).to have_write_access end it “can’t write when unauthenticated” do @user.update_attributes(…) expect(@user).to_not have_write_access end end -
@chrisarcand @jrafanie • instance_variable_get • .send • Many mocks, stubs,
doubles • and the best one of all… Test smells: Write testable code -
@chrisarcand @jrafanie *_any_instance_of • Added ruby 2.3 support • rspec-mocks
PR 1043 ☹ • rspec-mocks PR 1060 (for real) -
@chrisarcand @jrafanie PRs that remove these are welcome! git grep
"any_instance_of" spec |wc -l 534 ✂✂✂ -
@chrisarcand @jrafanie • Add public APIs • Reduce state mutation
Write testable code! -
@chrisarcand @jrafanie RSpec anti-patterns: Sporadic Failures This will randomly fail.
Why? vm1 = Vm.create(…) vm2 = Vm.create(…) expect(Vm.pluck(:id)) .to eq([vm1.id, vm2.id]) No explicit order Use testing constructs like match_array! -
@chrisarcand @jrafanie Fix sporadic failures by… • Limiting the problem
scope - Automated bisecting - Bisecting by comments! • Ask for help in Gitter! -
@chrisarcand @jrafanie Caching “There are only two hard things in
Computer Science: cache invalidation and naming things.” -- Phil Karlton http://martinfowler.com/bliki/TwoHardThings.html • Add caching only if absolutely required • Clear caches with tests - See spec/support/evm_spec_helper.rb -
@chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns
• Troubleshooting (“WAT!?”) • Spec performance -
@chrisarcand @jrafanie Travis failures • Look at the travis build
log • Read the error • Now, read the backtrace • No, really, did you read the error? -
@chrisarcand @jrafanie Travis failures • Check other builds on travis
• Master could be broken! • See build history @ github -
@chrisarcand @jrafanie Travis failures • Look at bundle install section
for NEW gems • Share your findings: • https://gitter.im/ManageIQ/manageiq -
@chrisarcand @jrafanie Travis failures • Enable travis on your fork
• Test without waiting behind ManageIQ org • Useful for debugging Travis errors https://travis-ci.org/jrafanie/manageiq/settings -
@chrisarcand @jrafanie Useful RSpec CLI switches -t ~slow (--tag)filter by
tags (~ excludes tag) -f d (--format) [p]rogress(dots), [d]ocumentation, etc. --seed 1234 Run examples with a specific test order --bisect Find the examples to recreate a sporadic bug --only-failures Run the examples that just failed --fail-fast Abort a test run on the first error -
@chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns
• Troubleshooting (“WAT!?”) • Spec performance -
@chrisarcand @jrafanie Boot time improvements • Ongoing: Huge improvements to
Rails boot time • 2.5 seconds - lazy loading message catalogs, #8525 • bundler 1.12.1+, rubygems 2.5.0+ • Future: • Ruby 2.3.x • Sprockets #211 (lazy stat assets for size) • Bundler bugfix to prevent re-resolve (#4618) -
@chrisarcand @jrafanie Rails boot time • Spring • https://github.com/rails/spring •
Preloads application boot for faster development • Rake tasks • Tests • Migrations • Boot speed must still be fast -
@chrisarcand @jrafanie 2. Total suite run time “The tests take
forever.” “Let’s just push it to Travis lolzzzzz” -
@chrisarcand @jrafanie Why? *Besides the fact that it’s a large,
successful codebase originally written over a decade ago and we have over 12,000 individual tests -
@chrisarcand @jrafanie % of total suite time vs. % of
total specs 70.3% of the test suite accounts for 11.45% of total time 25.3% of the test suite accounts for 37.43% of total time -
@chrisarcand @jrafanie % of total suite time vs. % of
total specs 70.3% of the test suite accounts for 11.45% of total time 25.3% of the test suite accounts for 37.43% of total time WAT -
@chrisarcand @jrafanie Promotes inefficiencies Creating a unused second record here!
Remember this? …and this example doesn’t even include FG associations -
@chrisarcand @jrafanie Good news! We’re already making improvements • Fall
2015: Moved to RSpec 3.x (core performance ++) • In the future: Pluggable providers split? Flag slow specs? -
@chrisarcand @jrafanie ✨Run your tests in parallel!✨ $ PARALLEL=true bundle
exec rake test PR #8868 merged over the weekend (June 5th) Paired effort over the last few weeks First championed by Joe last year! Thanks Jason for reviewing/merge! -
@chrisarcand @jrafanie All benchmarks on Mid-2015 Macbook Pro 2.5 GHz
Intel Core i7, ManageIQ commit #1689382 (May 18th, 2016) Time includes full load time Local vmdb test run before: ~7.5 minutes ~25 minutes Local vmdb test run after (8 cores): -
@chrisarcand @jrafanie Travis CI before: ~30 minutes ~45 minutes Travis
CI after (2 cores):