Rocket Fuelled Cucumbers

Talk given at Railsconf 2010 in Baltimore, MD. Introduces lots of ideas about how to scale your tests.

Transcript

1. Rocket Fuelled Cucumbers Joseph Wilk
2. Dealing with an Outbreak of Cucumbers Joseph Wilk
3. No Cucumbers where harmed in the making of this presentation. Just mildly shaken up
4. Scenario: Happy Railsconf attendees Given you have a vague idea what Cucumber is When Joseph completes his presentation Then you should understand the scaling choices And you should feel better prepared to face them
5. Cucumber • BDD framework • Plaintext • Promotes communication Given /^I have trained monkeys$/ do @test_pilots << Monkeys.new end Feature: Filling Cucumbers with rocket fuel Scenario: Rocket fuel Given I have trained monkeys When I click the launch button Then the monkeys should not die
6. Your code makes it slow Cucumber provides the venue Shame on you
7. What are Acceptance Tests? • Customer-facing • Cut through the whole stack • Examples of how the system is intended to be used
9. You made me Cuke
11. 1 minute ~ build
12. 1 minute ~ build Local builds
13. 1 minute ~ build Autotest Local builds
14. 1 minute ~ build Autotest Local builds Growl
17. Don’t Panic
18. Slow testing problems Development is slowed down by the time to run focused Cukes Running all the Cukes takes too long!
19. Development Test Build Types of Feedback Speed Confidence - + Speed Confidence + -
20. Happiness is a 10 min build time
21. Slow builds are the enemy of continuous integration
22. Development Feedback Its too slow to run the cukes. Just push the code and run away
23. Reuse - Spork $ spork cucumber $ cucumber --drb require 'rubygems' require 'spork' Spork.prefork do puts "I'm loading all the heavy stuff..." end Spork.each_run do puts "I'm loading the stuff just for this run..." # Cucumber hooks must go here end git://github.com/timcharper/spork.git
24. Slow services • Search • Solr • Sphinx • Databases • Mongo • Redis • Mysql • Message Queues • RabbitMQ Before('@solr') do Solr.boot unless Solr.running? end Feature: Searching @solr Scenario: Indexed search Given ... When ... Then ...
25. Run Just Enough Tests # * Test files must be stored in test/ # * Test files names must start with test_ # * Test class names must start with Test # * Implementation files must be stored in lib/ # * Implementation files must match up with a # test file named # test_.*implementation.rb @analyzer = Rcov::CodeCoverageAnalyzer.new Cucover Autotest
26. http://vimeo.com/12500864
27. Slicing Features Tagging Filenames $ cucumber --tags @media Subsystems $ cucumber visitor_* $ cucumber features/admin/* Profiles $ cucumber --profile admin @media, @public Feature: Visitor views artist’s media @feed, @admin Feature: Logged in user views views their activity feed
28. Just enough Database
29. Just enough Database city data venue data SELECT * FROM `cities` WHERE (`cities`.`id` = 105838) INSERT INTO `cities` (`name`) VALUES ('Test name') INSERT INTO `venues` ... SELECT * FROM `venues` WHERE ... Cache Cache
30. Test Build Feedback Commit code Have a 1 hour nap
31. Options •Hardware •Intelligent test selection •Tighter focused tests •Divide and Conquer
32. Testjour Songkick.com 254 Features, 1257 scenarios, 10807 steps 1 build server ~ 4 hours 5 build servers ~ 2 hours
33. 120 sad minutes of my life...
34. $ git clone git://github.com/brynary/testjour.git $ rake gem install $ testjour --help
35. $ git clone git://github.com/brynary/testjour.git $ rake gem install $ testjour --help testjour help:
36. Testjour Master Work Queue Slave Redis Slave Slave Worker Worker Worker Worker Worker Worker
37. Testjour Master Work Queue Slave Redis Slave Slave rsync - code rsync - code Worker Worker Worker Worker Worker Worker
38. Testjour Master Work Queue Slave Redis Slave Slave rsync - code rsync - code Worker Worker Worker Worker Worker Worker
39. Testjour Master Work Queue Slave Redis Slave Slave rsync - code rsync - code Worker Worker Worker Worker Worker Worker
40. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure
41. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours
42. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours 20 build servers ~ 11:40 minutes
43. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours 20 build servers ~ 11:40 minutes $2068.99 / $3000
44. Ec2 AMI slim-sumo Disc image
45. Ec2 AMI Launch slim-sumo Disc image
46. Ec2 AMI Launch ec2 node ec2 node ec2 node ec2 node slim-sumo Disc image
47. Ec2 AMI Launch ec2 node ec2 node ec2 node ec2 node slim-sumo Build ENV['EC2_URL'] = ec2_url require 'slim-sumo' Sumo.new.running.map{|ec2_node| ec2_node[:hostname]} Disc image
48. http://vimeo.com/12467100
49. require 'rubygems' require 'hydra' require 'hydra/tasks' Hydra::TestTask.new('hydra') do |t| t.add_files 'features/**/*.feature' t.verbose = false t.autosort = false end
51. “Buy lots of hardware”
52. “Buy lots of hardware” “more hardware please”
53. “Buy lots of hardware” “more hardware please” “Just a little bit more hardware”
54. “Buy lots of hardware” “more hardware please” “Just a little bit more hardware”
55. Just run the tests that matter...
56. Failure probability matters European Computing manufacturer
57. Failure probability matters European Computing manufacturer Automated test suite ~ 18 hours
58. Failure probability matters European Computing manufacturer Tests that were most ~ 2 hours likely to fail Automated test suite ~ 18 hours
59. http://vimeo.com/12467122
61. Tests that never fail
62. Tests that never fail
63. Tests that never fail Tests which regularly fail
64. Daily Build Tests that never fail Tests which regularly fail
65. Daily Build Nightly Build Tests that never fail Tests which regularly fail
66. Daily Build Nightly Build Tests that never fail Tests which regularly fail $cucumber --tags ~@nightly $cucumber --tags @nightly
67. Flickering Confidence
68. Flickering Confidence FAIL
69. Flickering Confidence FAIL FAIL
70. Flickering Confidence FAIL PASS FAIL
71. Flickering Confidence FAIL PASS FAIL Main Flicker @flicker
72. Divide and conquer • Rails - Engines • Service Orientated Architecture 1 hour 7 7 7 7 7 7 7 7 7
73. Divide and Concuquer
74. Divide and Concuquer
75. Divide and Concuquer
76. Divide and Concuquer
77. Divide and Concuquer
78. Divide and Concuquer Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
79. Divide and conquer Rails Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
80. Divide and conquer Related Artists Service Rails Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
81. Divide and conquer related(artist) Related Artists Service Rails Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
82. Divide and conquer related(artist) Related Artists Service HTML Rails Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
83. Divide and conquer related(artist) Related Artists Service HTML Rails Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
84. Divide and conquer related(artist) Related Artists Service HTML Rails Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison” unit test unit test
85. Don’t use Acceptance tests http://jamesshore.com/Blog/Alternatives- to-Acceptance-Testing.html Heresy!
86. Pairwise Scenario: Testing Cucumber against different environments Given I have a rails app <Rails version> And I’m using Ruby <Ruby version> And I am using the <Test Framework> And I am using the driver <Driver> Then Cucumber should install and play nicely Examples: | Rails version | Ruby version | test framework | Driver | | 3.0 | 1.8.7 | Rspec | Webrat | | 3.0 | 1.8.7 | Testunit | Capybara | | 3.0 | 1.8.7 | Rspec | Capybara | | 3.0 | 1.8.7 | Testunit | Webrat | | 3.0 | 1.9 | Rspec | Webrat | | 3.0 | 1.9 | Testunit | Webrat | | 2.3.1 | 1.9 | Rspec | Webrat | ....
87. | Rails version | Ruby version | test framework | Driver | | 2.3.1 | 1.8.7 | Rspec | Webrat | | 2.3.1 | 1.8.7 | Rspec | Capybara | | 2.3.1 | 1.8.7 | Testunit | Webrat | | 2.3.1 | 1.8.7 | Testunit | Capybara | | 2.3.1 | 1.9 | Rspec | Capybara | | 2.3.1 | 1.9 | Rspec | Webrat | | 2.3.1 | 1.9 | Testunit | Capybara | | 2.3.1 | 1.9 | Testunit | Webrat | .... “most faults are caused by interactions of at most two factors”
88. Pairwise http://www.screencast.com/t/NGI0NjVk
89. We should add some funky Ajax features Just when you thought things were going well
90. JavaScript In-memory Browser based Selenium, Watir Celerity
91. Need for speed Webrat Celerity Watir Selenium 0 3.75 7.5 11.25 15 12.9 12 12 0.8 start-up time (seconds)
92. Capybara Like Webrat but bigger envjs Johnson spidermonkey gem install johnson --prerelease gem install envjs sudo gem install capybara
93. http://vimeo.com/12467133
94. Capybara-envjs
96. Find the right compromise for you
97. Find the right compromise for you Slow test builds produce slow release cycles
98. Find the right compromise for you Slow test builds produce slow release cycles Your test builds are a goldmine of data
99. Thanks Joseph Wilk http://blog.josephwilk.net @josephwilk Its been emotional