Pro Yearly is on sale from $80 to $50! »

Rocket Fuelled Cucumbers

3c28deaff162aeda44f2e0bcdca1dacf?s=47 Joseph Wilk
September 22, 2011

Rocket Fuelled Cucumbers

Talk given at Railsconf 2010 in Baltimore, MD. Introduces lots of ideas about how to scale your tests.

3c28deaff162aeda44f2e0bcdca1dacf?s=128

Joseph Wilk

September 22, 2011
Tweet

Transcript

  1. Rocket Fuelled Cucumbers Joseph Wilk

  2. Dealing with an Outbreak of Cucumbers Joseph Wilk

  3. No Cucumbers where harmed in the making of this presentation.

    Just mildly shaken up
  4. Scenario: Happy Railsconf attendees Given you have a vague idea

    what Cucumber is When Joseph completes his presentation Then you should understand the scaling choices And you should feel better prepared to face them
  5. Cucumber • BDD framework • Plaintext • Promotes communication Given

    /^I have trained monkeys$/ do @test_pilots << Monkeys.new end Feature: Filling Cucumbers with rocket fuel Scenario: Rocket fuel Given I have trained monkeys When I click the launch button Then the monkeys should not die
  6. Your code makes it slow Cucumber provides the venue Shame

    on you
  7. What are Acceptance Tests? • Customer-facing • Cut through the

    whole stack • Examples of how the system is intended to be used
  8. None
  9. You made me Cuke

  10. None
  11. 1 minute ~ build

  12. 1 minute ~ build Local builds

  13. 1 minute ~ build Autotest Local builds

  14. 1 minute ~ build Autotest Local builds Growl

  15. None
  16. None
  17. Don’t Panic

  18. Slow testing problems Development is slowed down by the time

    to run focused Cukes Running all the Cukes takes too long!
  19. Development Test Build Types of Feedback Speed Confidence - +

    Speed Confidence + -
  20. Happiness is a 10 min build time

  21. Slow builds are the enemy of continuous integration

  22. Development Feedback Its too slow to run the cukes. Just

    push the code and run away
  23. Reuse - Spork $ spork cucumber $ cucumber --drb require

    'rubygems' require 'spork' Spork.prefork do puts "I'm loading all the heavy stuff..." end Spork.each_run do puts "I'm loading the stuff just for this run..." # Cucumber hooks must go here end git://github.com/timcharper/spork.git
  24. Slow services • Search • Solr • Sphinx • Databases

    • Mongo • Redis • Mysql • Message Queues • RabbitMQ Before('@solr') do Solr.boot unless Solr.running? end Feature: Searching @solr Scenario: Indexed search Given ... When ... Then ...
  25. Run Just Enough Tests # * Test files must be

    stored in test/ # * Test files names must start with test_ # * Test class names must start with Test # * Implementation files must be stored in lib/ # * Implementation files must match up with a # test file named # test_.*implementation.rb @analyzer = Rcov::CodeCoverageAnalyzer.new Cucover Autotest
  26. http://vimeo.com/12500864

  27. Slicing Features Tagging Filenames $ cucumber --tags @media Subsystems $

    cucumber visitor_* $ cucumber features/admin/* Profiles $ cucumber --profile admin @media, @public Feature: Visitor views artist’s media @feed, @admin Feature: Logged in user views views their activity feed
  28. Just enough Database

  29. Just enough Database city data venue data SELECT * FROM

    `cities` WHERE (`cities`.`id` = 105838) INSERT INTO `cities` (`name`) VALUES ('Test name') INSERT INTO `venues` ... SELECT * FROM `venues` WHERE ... Cache Cache
  30. Test Build Feedback Commit code Have a 1 hour nap

  31. Options •Hardware •Intelligent test selection •Tighter focused tests •Divide and

    Conquer
  32. Testjour Songkick.com 254 Features, 1257 scenarios, 10807 steps 1 build

    server ~ 4 hours 5 build servers ~ 2 hours
  33. 120 sad minutes of my life...

  34. $ git clone git://github.com/brynary/testjour.git $ rake gem install $ testjour

    --help
  35. $ git clone git://github.com/brynary/testjour.git $ rake gem install $ testjour

    --help testjour help:
  36. Testjour Master Work Queue Slave Redis Slave Slave Worker Worker

    Worker Worker Worker Worker
  37. Testjour Master Work Queue Slave Redis Slave Slave rsync -

    code rsync - code Worker Worker Worker Worker Worker Worker
  38. Testjour Master Work Queue Slave Redis Slave Slave rsync -

    code rsync - code Worker Worker Worker Worker Worker Worker
  39. Testjour Master Work Queue Slave Redis Slave Slave rsync -

    code rsync - code Worker Worker Worker Worker Worker Worker
  40. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure

  41. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1

    build server ~ 4 hours
  42. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1

    build server ~ 4 hours 20 build servers ~ 11:40 minutes
  43. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1

    build server ~ 4 hours 20 build servers ~ 11:40 minutes $2068.99 / $3000
  44. Ec2 AMI slim-sumo Disc image

  45. Ec2 AMI Launch slim-sumo Disc image

  46. Ec2 AMI Launch ec2 node ec2 node ec2 node ec2

    node slim-sumo Disc image
  47. Ec2 AMI Launch ec2 node ec2 node ec2 node ec2

    node slim-sumo Build ENV['EC2_URL'] = ec2_url require 'slim-sumo' Sumo.new.running.map{|ec2_node| ec2_node[:hostname]} Disc image
  48. http://vimeo.com/12467100

  49. require 'rubygems' require 'hydra' require 'hydra/tasks' Hydra::TestTask.new('hydra') do |t| t.add_files

    'features/**/*.feature' t.verbose = false t.autosort = false end
  50. None
  51. “Buy lots of hardware”

  52. “Buy lots of hardware” “more hardware please”

  53. “Buy lots of hardware” “more hardware please” “Just a little

    bit more hardware”
  54. “Buy lots of hardware” “more hardware please” “Just a little

    bit more hardware”
  55. Just run the tests that matter...

  56. Failure probability matters European Computing manufacturer

  57. Failure probability matters European Computing manufacturer Automated test suite ~

    18 hours
  58. Failure probability matters European Computing manufacturer Tests that were most

    ~ 2 hours likely to fail Automated test suite ~ 18 hours
  59. http://vimeo.com/12467122

  60. None
  61. Tests that never fail

  62. Tests that never fail

  63. Tests that never fail Tests which regularly fail

  64. Daily Build Tests that never fail Tests which regularly fail

  65. Daily Build Nightly Build Tests that never fail Tests which

    regularly fail
  66. Daily Build Nightly Build Tests that never fail Tests which

    regularly fail $cucumber --tags ~@nightly $cucumber --tags @nightly
  67. Flickering Confidence

  68. Flickering Confidence FAIL

  69. Flickering Confidence FAIL FAIL

  70. Flickering Confidence FAIL PASS FAIL

  71. Flickering Confidence FAIL PASS FAIL Main Flicker @flicker

  72. Divide and conquer • Rails - Engines • Service Orientated

    Architecture 1 hour 7 7 7 7 7 7 7 7 7
  73. Divide and Concuquer

  74. Divide and Concuquer

  75. Divide and Concuquer

  76. Divide and Concuquer

  77. Divide and Concuquer

  78. Divide and Concuquer Scenario: Related artists Given an artist “SYGC”

    And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  79. Divide and conquer Rails Scenario: Related artists Given an artist

    “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  80. Divide and conquer Related Artists Service Rails Scenario: Related artists

    Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  81. Divide and conquer related(artist) Related Artists Service Rails Scenario: Related

    artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  82. Divide and conquer related(artist) Related Artists Service HTML Rails Scenario:

    Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  83. Divide and conquer related(artist) Related Artists Service HTML Rails Scenario:

    Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  84. Divide and conquer related(artist) Related Artists Service HTML Rails Scenario:

    Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison” unit test unit test
  85. Don’t use Acceptance tests http://jamesshore.com/Blog/Alternatives- to-Acceptance-Testing.html Heresy!

  86. Pairwise Scenario: Testing Cucumber against different environments Given I have

    a rails app <Rails version> And I’m using Ruby <Ruby version> And I am using the <Test Framework> And I am using the driver <Driver> Then Cucumber should install and play nicely Examples: | Rails version | Ruby version | test framework | Driver | | 3.0 | 1.8.7 | Rspec | Webrat | | 3.0 | 1.8.7 | Testunit | Capybara | | 3.0 | 1.8.7 | Rspec | Capybara | | 3.0 | 1.8.7 | Testunit | Webrat | | 3.0 | 1.9 | Rspec | Webrat | | 3.0 | 1.9 | Testunit | Webrat | | 2.3.1 | 1.9 | Rspec | Webrat | ....
  87. | Rails version | Ruby version | test framework |

    Driver | | 2.3.1 | 1.8.7 | Rspec | Webrat | | 2.3.1 | 1.8.7 | Rspec | Capybara | | 2.3.1 | 1.8.7 | Testunit | Webrat | | 2.3.1 | 1.8.7 | Testunit | Capybara | | 2.3.1 | 1.9 | Rspec | Capybara | | 2.3.1 | 1.9 | Rspec | Webrat | | 2.3.1 | 1.9 | Testunit | Capybara | | 2.3.1 | 1.9 | Testunit | Webrat | .... “most faults are caused by interactions of at most two factors”
  88. Pairwise http://www.screencast.com/t/NGI0NjVk

  89. We should add some funky Ajax features Just when you

    thought things were going well
  90. JavaScript In-memory Browser based Selenium, Watir Celerity

  91. Need for speed Webrat Celerity Watir Selenium 0 3.75 7.5

    11.25 15 12.9 12 12 0.8 start-up time (seconds)
  92. Capybara Like Webrat but bigger envjs Johnson spidermonkey gem install

    johnson --prerelease gem install envjs sudo gem install capybara
  93. http://vimeo.com/12467133

  94. Capybara-envjs

  95. None
  96. Find the right compromise for you

  97. Find the right compromise for you Slow test builds produce

    slow release cycles
  98. Find the right compromise for you Slow test builds produce

    slow release cycles Your test builds are a goldmine of data
  99. Thanks Joseph Wilk http://blog.josephwilk.net @josephwilk Its been emotional