Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing in ManageIQ

Testing in ManageIQ

A talk about testing with RSpec in the ManageIQ cloud management platform. A short tour of some RSpec ManageIQisms, good spec structure and anti-patterns, troubleshooting test failures, and an overview of test runtime performance.

Chris Arcand

June 09, 2016
Tweet

More Decks by Chris Arcand

Other Decks in Programming

Transcript

  1. Testing in Joe Rafaniello and Chris Arcand

  2. Chris Arcand
 @chrisarcand Joe Rafaniello @jrafanie Remote - Washington, D.C.

    Minneapolis, MN Mahwah office
  3. Application Core Team

  4. ✨Platform Team✨

  5. @chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns

    • Troubleshooting (“WAT!?”) • Spec performance
  6. @chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns

    • Troubleshooting (“WAT!?”) • Spec performance
  7. @chrisarcand @jrafanie factory_girl Technologies RSpec VCR Ruby JavaScript Jasmine *

    Also, tiny sprinkles of MiniTest in gems/pending, but…
  8. @chrisarcand @jrafanie RSpec http://rspec.info/ http://rspec.info/documentation/ https://relishapp.com/rspec

  9. @chrisarcand @jrafanie Test Suites, Metadata, and ManageIQ helpers RSpec ManageIQisms

  10. @chrisarcand @jrafanie Sweet, sweet suites • Automation • Brakeman (security

    scanner) • Javascript (Jasmine) • ManageIQ Providers - Amazon • Migrations • Replication • VMDB
  11. @chrisarcand @jrafanie Sweet, sweet suites rake test rake test:automation rake

    test:automation:setup rake test:brakeman rake test:javascript rake test:javascript:setup rake test:manageiq-providers-amazon rake test:manageiq-providers-amazon:setup rake test:migrations rake test:migrations:down rake test:migrations:setup rake test:migrations:up rake test:replication rake test:replication:setup rake test:replication:teardown rake test:replication_util rake test:vmdb rake test:vmdb:setup rake test:<suite>:setup rake test:<suite>
  12. @chrisarcand @jrafanie Sweet, sweet suites • Automation • Brakeman (security

    scanner) • Javascript (Jasmine) • ManageIQ Providers - Amazon • Migrations • Replication • VMDB
  13. @chrisarcand @jrafanie VMDB Migrations Replication Automation Controllers Helpers Initializers Mailers

    Models Requests Routing Services Views Presenters Rake Tasks REST API Helpers, helpers everywhere: AuthHelper ViewSpecHelper UiConstants ControllerSpecHelper AutomationSpecHelper MigrationSpecHelper ApiSpecHelper PresenterSpecHelper RakeTaskExampleGroup and more… Up migrations Down migrations REST API environment setup Replication setup Special migration management
  14. @chrisarcand @jrafanie VMDB Migrations Replication Automation Controllers Helpers Initializers Mailers

    Models Requests Routing Services Views Presenters Rake Tasks REST API Helpers, helpers everywhere: AuthHelper ViewSpecHelper UiConstants ControllerSpecHelper AutomationSpecHelper MigrationSpecHelper ApiSpecHelper PresenterSpecHelper RakeTaskExampleGroup and more… Up migrations Down migrations REST API environment setup Replication setup Special migration management WAT
  15. @chrisarcand @jrafanie RSpec knows what to do through metadata called

    ✨tags✨ describe "group with tagged specs" do it "slow example", :slow => true do; end it "ordinary example" do; end end ./some_spec.rb $ rspec ./some_spec.rb --tag ~slow
  16. @chrisarcand @jrafanie describe User, :type => :model do # Model-specific

    helpers, env end …
 config.include ModelHelper, :type => :model … user_spec.rb spec_helper.rb
  17. @chrisarcand @jrafanie Example: All the REST API specs live in

    ./spec/requests/api config.define_derived_metadata( :file_path => /spec\/requests\/api/) do |metadata| metadata[:rest_api] = true end config.include ApiSpecHelper, :rest_api => true config.before(:each, :rest_api => true) do init_api_spec_env end
  18. @chrisarcand @jrafanie Takeaways: 1. Spec location matters a lot. (Location

    matters in all Rails applications, but especially ManageIQ) 2. We have custom RSpec matchers and helpers for different spec types (And you should go check them out. See ./spec/spec_helper.rb)
  19. @chrisarcand @jrafanie Migration Specs RSpec ManageIQisms

  20. @chrisarcand @jrafanie Who tests migrations? • Manually testing is hard

    • Setup database • Migrate • Test conversion • Repeat up and down • Migration specs automate this
  21. @chrisarcand @jrafanie Migration spec tips • Don’t mix schema changes

    and data changes • Test data migrations • Migrations are snapshots in time • Don't use application models (app/models/*) • Use model stubs • good_migrations gem (soon™) • Libor
  22. @chrisarcand @jrafanie # db/migrate/20150817213409_clear_tenant_seed.rb class ClearTenantSeed < ActiveRecord::Migration class Tenant

    < ActiveRecord::Base; end def up say_with_time("...") do Tenant.update_all(:name => nil) end end end Model stub
  23. @chrisarcand @jrafanie # spec/migrations/20150817213409_clear_tenant_seed_spec.rb require_migration describe ClearTenantSeed do migration_context :up

    do it "works with one tenant" do tenant = migration_stub(:Tenant).create(…) migrate expect(tenant.reload.name).to be_nil end end end ✨ Magic
  24. @chrisarcand @jrafanie Who tests migrations? • FYI, Rails 5.0 generates

    versioned migrations class AddNameToUsers < ActiveRecord::Migration[5.0] ... end • Existing migrations use 4.2 migration code
  25. @chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns

    • Troubleshooting (“WAT!?”) • Spec performance
  26. @chrisarcand @jrafanie OH: “Don’t nest contexts EVARRRRRR!” RSpec structure

  27. @chrisarcand @jrafanie RSpec structure OH: “Don’t nest contexts EVARRRRRR!” Using

    context and describe blocks correctly will make you happier
  28. @chrisarcand @jrafanie

  29. @chrisarcand @jrafanie BAD: Contextual tests at the top level! In

    the future: • Harder to understand context • Promotes inefficiencies • Therefore, harder to add tests
  30. @chrisarcand @jrafanie

  31. @chrisarcand @jrafanie Harder to understand context
 Imagine seeing just this

    snippet buried in other examples as this file grows. WAT?
  32. @chrisarcand @jrafanie Promotes inefficiencies
 Creating an unused second record here!

  33. @chrisarcand @jrafanie Harder to add tests
 I now need to

    edit the setup here just to add a new test to avoid these issues
  34. @chrisarcand @jrafanie Harder to add tests
 I now need to

    edit the setup here just to add a new test to avoid these issues WAT
  35. @chrisarcand @jrafanie Separate context from expectation…

  36. @chrisarcand @jrafanie …and create a self-contained, portable set of circumstances

  37. @chrisarcand @jrafanie Easy to add different cases for #generation Easy

    to add different cases for totally different things at top level
  38. @chrisarcand @jrafanie

  39. @chrisarcand @jrafanie $ rspec ./spec/some_spec.rb --format=documentation

  40. @chrisarcand @jrafanie Your mileage may vary. Especially in a legacy

    codebase.
  41. @chrisarcand @jrafanie RSpec Anti-Patterns • Complex setup • Hard-to-test code

    • Random test failures
  42. @chrisarcand @jrafanie “I don’t understand what this test is doing.”

    “I have no idea why this test exists.” Complex setup
  43. @chrisarcand @jrafanie before(:each) do _guid_2, _server_2, @zone_2 = EvmSpecHelper.create_guid_miq_ guid,

    server, @zone = EvmSpecHelper.create_guid_miq_server_z @ems = FactoryGirl.create(:ems_vmware_with_auth…, :zone => @ other_ems = FactoryGirl.create(:ems_…_with_auth…, :zone => @ @worker_guid = MiqUUID.new_guid @worker_record = FactoryGirl.create(:miq_vim_broker_worker, :guid => …, :mi @drb_uri = "drb://127.0.0.1:12345" allow(DRb).to receive(:uri).and_return(@drb_uri) allow_any_instance_of(described_class).to receive(:sync_acti allow_any_instance_of(described_class).to receive(:sync_conf allow_any_instance_of(described_class).to receive(:set_conne allow_any_instance_of(ManageIQ::Providers::Vmware::InfraMana .to receive(:authentication_check).and_return([true, ""]) allow_any_instance_of(ManageIQ::Providers::Vmware::InfraMana .to receive(:authentication_status_ok?).and_return(true) end
  44. @chrisarcand @jrafanie • Find what you really need for setup

    • Create only what's needed • Figure out where it belongs - Abstractions can hide relevant details - Make setup methods explicit and flexible Beware of setup complexity
  45. @chrisarcand @jrafanie describe User do before do special_magic_setup_method @user =

    User.first end it “has write access when authenticated” do expect(@user).to have_write_access end it “can’t write when unauthenticated” do @user.update_attributes(…) expect(@user).to_not have_write_access end end
  46. @chrisarcand @jrafanie • instance_variable_get • .send • Many mocks, stubs,

    doubles • and the best one of all… Test smells: Write testable code
  47. @chrisarcand @jrafanie RSpec anti-patterns *_any_instance_of expect_any_instance_of(SomeClass).to receive(:some_method) allow_any_instance_of(SomeClass).to receive(:some_method) {

    … }
  48. @chrisarcand @jrafanie *_any_instance_of • Added ruby 2.3 support • rspec-mocks

    PR 1043 ☹ • rspec-mocks PR 1060 (for real)
  49. @chrisarcand @jrafanie PRs that remove these are welcome! git grep

    "any_instance_of" spec |wc -l 534 ✂✂✂
  50. @chrisarcand @jrafanie • Add public APIs • Reduce state mutation

    Write testable code!
  51. @chrisarcand @jrafanie RSpec anti-patterns:
 Sporadic Failures This will randomly fail.

    Why? vm1 = Vm.create(…) vm2 = Vm.create(…) expect(Vm.pluck(:id)) .to eq([vm1.id, vm2.id]) No explicit order
 Use testing constructs like match_array!
  52. @chrisarcand @jrafanie Fix sporadic failures by… • Limiting the problem

    scope - Automated bisecting - Bisecting by comments! • Ask for help in Gitter!
  53. @chrisarcand @jrafanie Caching “There are only two hard things in

    Computer Science: cache invalidation and naming things.” -- Phil Karlton http://martinfowler.com/bliki/TwoHardThings.html • Add caching only if absolutely required • Clear caches with tests - See spec/support/evm_spec_helper.rb
  54. @chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns

    • Troubleshooting (“WAT!?”) • Spec performance
  55. @chrisarcand @jrafanie Travis failures • Look at the travis build

    log • Read the error • Now, read the backtrace • No, really, did you read the error?
  56. @chrisarcand @jrafanie Travis failures • Check other builds on travis

    • Master could be broken! • See build history @ github
  57. @chrisarcand @jrafanie Travis failures • Look at bundle install section

    for NEW gems • Share your findings: • https://gitter.im/ManageIQ/manageiq
  58. @chrisarcand @jrafanie Travis failures • Enable travis on your fork

    • Test without waiting behind ManageIQ org • Useful for debugging Travis errors https://travis-ci.org/jrafanie/manageiq/settings
  59. @chrisarcand @jrafanie Useful RSpec CLI switches -t ~slow (--tag)filter by

    tags (~ excludes tag) -f d (--format) [p]rogress(dots), [d]ocumentation, etc. --seed 1234 Run examples with a specific test order --bisect Find the examples to recreate a sporadic bug --only-failures Run the examples that just failed --fail-fast Abort a test run on the first error
  60. @chrisarcand @jrafanie • RSpec ManageIQisms • Spec structure and anti-patterns

    • Troubleshooting (“WAT!?”) • Spec performance
  61. @chrisarcand @jrafanie 1. Boot time “I can’t TDD, too slow!”

  62. @chrisarcand @jrafanie Boot time improvements • Ongoing: Huge improvements to

    Rails boot time • 2.5 seconds - lazy loading message catalogs, #8525 • bundler 1.12.1+, rubygems 2.5.0+ • Future: • Ruby 2.3.x • Sprockets #211 (lazy stat assets for size) • Bundler bugfix to prevent re-resolve (#4618)
  63. @chrisarcand @jrafanie Rails boot time • Spring • https://github.com/rails/spring •

    Preloads application boot for faster development • Rake tasks • Tests • Migrations • Boot speed must still be fast
  64. @chrisarcand @jrafanie 2. Total suite run time “The tests take

    forever.” “Let’s just push it to Travis lolzzzzz”
  65. @chrisarcand @jrafanie

  66. @chrisarcand @jrafanie Why? *Besides the fact that it’s a large,

    successful codebase originally written over a decade ago and we have over 12,000 individual tests
  67. @chrisarcand @jrafanie % of specs by individual test length
 (VMDB

    suite)
  68. @chrisarcand @jrafanie % of total suite time by individual test

    length
  69. @chrisarcand @jrafanie % of total suite time vs. % of

    total specs 70.3% of the test suite accounts for 11.45% of total time 25.3% of the test suite accounts for 37.43% of total time
  70. @chrisarcand @jrafanie % of total suite time vs. % of

    total specs 70.3% of the test suite accounts for 11.45% of total time 25.3% of the test suite accounts for 37.43% of total time WAT
  71. @chrisarcand @jrafanie Promotes inefficiencies
 Creating a unused second record here!

    Remember this? …and this example doesn’t even include FG associations
  72. @chrisarcand @jrafanie Analysis helps us improve… …but there’s no silver

    bullet.
  73. @chrisarcand @jrafanie However…

  74. @chrisarcand @jrafanie Good news! We’re already making improvements • Fall

    2015: Moved to RSpec 3.x 
 (core performance ++) • In the future: 
 Pluggable providers split? Flag slow specs?
  75. @chrisarcand @jrafanie AND… *drumroll*

  76. @chrisarcand @jrafanie ✨Run your tests in parallel!✨ $ PARALLEL=true bundle

    exec rake test PR #8868 merged over the weekend (June 5th) Paired effort over the last few weeks First championed by Joe last year! Thanks Jason for reviewing/merge!
  77. @chrisarcand @jrafanie All benchmarks on Mid-2015 Macbook Pro 2.5 GHz

    Intel Core i7, ManageIQ commit #1689382 (May 18th, 2016) Time includes full load time Local vmdb test run before: ~7.5 minutes ~25 minutes Local vmdb test run after (8 cores):
  78. @chrisarcand @jrafanie Travis CI before: ~30 minutes ~45 minutes Travis

    CI after (2 cores):
  79. Thank you Questions? chrisarcand chrisarcand www.chrisarcand.com jrafanie jrafanie Slides available

    here