Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Rocket Fuelled Cucumbers

Joseph Wilk
September 22, 2011

Rocket Fuelled Cucumbers

Talk given at Railsconf 2010 in Baltimore, MD. Introduces lots of ideas about how to scale your tests.

Joseph Wilk

September 22, 2011
Tweet

More Decks by Joseph Wilk

Other Decks in Programming

Transcript

  1. Rocket Fuelled
    Cucumbers
    Joseph Wilk

    View Slide

  2. Dealing with an
    Outbreak of Cucumbers
    Joseph Wilk

    View Slide

  3. No Cucumbers where
    harmed in the making of
    this presentation.
    Just mildly shaken up

    View Slide

  4. Scenario: Happy Railsconf attendees
    Given you have a vague idea what Cucumber is
    When Joseph completes his presentation
    Then you should understand the scaling choices
    And you should feel better prepared to face them

    View Slide

  5. Cucumber
    • BDD framework
    • Plaintext
    • Promotes
    communication
    Given /^I have trained monkeys$/ do
    @test_pilots << Monkeys.new
    end
    Feature: Filling Cucumbers with
    rocket fuel
    Scenario: Rocket fuel
    Given I have trained monkeys
    When I click the launch button
    Then the monkeys should not die

    View Slide

  6. Your code makes it slow
    Cucumber provides the venue
    Shame on you

    View Slide

  7. What are Acceptance Tests?
    • Customer-facing
    • Cut through the whole stack
    • Examples of how the system is intended
    to be used

    View Slide

  8. View Slide

  9. You made
    me Cuke

    View Slide

  10. View Slide

  11. 1 minute ~ build

    View Slide

  12. 1 minute ~ build
    Local builds

    View Slide

  13. 1 minute ~ build
    Autotest
    Local builds

    View Slide

  14. 1 minute ~ build
    Autotest
    Local builds
    Growl

    View Slide

  15. View Slide

  16. View Slide

  17. Don’t
    Panic

    View Slide

  18. Slow testing problems
    Development is slowed
    down by the time to run
    focused Cukes
    Running all the Cukes
    takes too long!

    View Slide

  19. Development
    Test Build
    Types of Feedback
    Speed
    Confidence
    -
    +
    Speed
    Confidence
    +
    -

    View Slide

  20. Happiness is a
    10 min build time

    View Slide

  21. Slow builds are
    the enemy of
    continuous integration

    View Slide

  22. Development Feedback
    Its too slow to run the cukes.
    Just push the code and run away

    View Slide

  23. Reuse - Spork
    $ spork cucumber
    $ cucumber --drb
    require 'rubygems'
    require 'spork'
    Spork.prefork do
    puts "I'm loading all the heavy stuff..."
    end
    Spork.each_run do
    puts "I'm loading the stuff just for this run..."
    # Cucumber hooks must go here
    end
    git://github.com/timcharper/spork.git

    View Slide

  24. Slow services
    • Search
    • Solr
    • Sphinx
    • Databases
    • Mongo
    • Redis
    • Mysql
    • Message Queues
    • RabbitMQ
    Before('@solr') do
    Solr.boot unless Solr.running?
    end
    Feature: Searching
    @solr
    Scenario: Indexed search
    Given ...
    When ...
    Then ...

    View Slide

  25. Run Just Enough Tests
    # * Test files must be stored in test/
    # * Test files names must start with test_
    # * Test class names must start with Test
    # * Implementation files must be stored in lib/
    # * Implementation files must match up with a
    # test file named
    # test_.*implementation.rb
    @analyzer = Rcov::CodeCoverageAnalyzer.new
    Cucover
    Autotest

    View Slide

  26. http://vimeo.com/12500864

    View Slide

  27. Slicing Features
    Tagging
    Filenames
    $ cucumber --tags @media
    Subsystems
    $ cucumber visitor_*
    $ cucumber features/admin/*
    Profiles $ cucumber --profile admin
    @media, @public
    Feature: Visitor views artist’s media
    @feed, @admin
    Feature: Logged in user views views their activity feed

    View Slide

  28. Just enough Database

    View Slide

  29. Just enough Database
    city data
    venue data
    SELECT * FROM `cities`
    WHERE (`cities`.`id` = 105838)
    INSERT INTO `cities`
    (`name`) VALUES ('Test name')
    INSERT INTO
    `venues` ...
    SELECT * FROM
    `venues`
    WHERE ...
    Cache
    Cache

    View Slide

  30. Test Build Feedback
    Commit code
    Have a 1 hour nap

    View Slide

  31. Options
    •Hardware
    •Intelligent test selection
    •Tighter focused tests
    •Divide and Conquer

    View Slide

  32. Testjour
    Songkick.com
    254 Features, 1257 scenarios, 10807 steps
    1 build server ~ 4 hours
    5 build servers ~ 2 hours

    View Slide

  33. 120 sad minutes of my life...

    View Slide

  34. $ git clone git://github.com/brynary/testjour.git
    $ rake gem install
    $ testjour --help

    View Slide

  35. $ git clone git://github.com/brynary/testjour.git
    $ rake gem install
    $ testjour --help
    testjour help:

    View Slide

  36. Testjour
    Master
    Work
    Queue
    Slave
    Redis
    Slave
    Slave
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker

    View Slide

  37. Testjour
    Master
    Work
    Queue
    Slave
    Redis
    Slave
    Slave
    rsync - code
    rsync - code
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker

    View Slide

  38. Testjour
    Master
    Work
    Queue
    Slave
    Redis
    Slave
    Slave
    rsync - code
    rsync - code
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker

    View Slide

  39. Testjour
    Master
    Work
    Queue
    Slave
    Redis
    Slave
    Slave
    rsync - code
    rsync - code
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker

    View Slide

  40. Looking to the Cloud
    EC2, Rackspace Cloud, Windows Azure

    View Slide

  41. Looking to the Cloud
    EC2, Rackspace Cloud, Windows Azure
    1 build server ~ 4 hours

    View Slide

  42. Looking to the Cloud
    EC2, Rackspace Cloud, Windows Azure
    1 build server ~ 4 hours
    20 build servers ~ 11:40 minutes

    View Slide

  43. Looking to the Cloud
    EC2, Rackspace Cloud, Windows Azure
    1 build server ~ 4 hours
    20 build servers ~ 11:40 minutes
    $2068.99 / $3000

    View Slide

  44. Ec2
    AMI
    slim-sumo
    Disc image

    View Slide

  45. Ec2
    AMI
    Launch
    slim-sumo
    Disc image

    View Slide

  46. Ec2
    AMI
    Launch
    ec2
    node
    ec2
    node
    ec2
    node
    ec2
    node
    slim-sumo
    Disc image

    View Slide

  47. Ec2
    AMI
    Launch
    ec2
    node
    ec2
    node
    ec2
    node
    ec2
    node
    slim-sumo
    Build
    ENV['EC2_URL'] = ec2_url
    require 'slim-sumo'
    Sumo.new.running.map{|ec2_node|
    ec2_node[:hostname]}
    Disc image

    View Slide

  48. http://vimeo.com/12467100

    View Slide

  49. require 'rubygems'
    require 'hydra'
    require 'hydra/tasks'
    Hydra::TestTask.new('hydra') do |t|
    t.add_files 'features/**/*.feature'
    t.verbose = false
    t.autosort = false
    end

    View Slide

  50. View Slide

  51. “Buy lots of hardware”

    View Slide

  52. “Buy lots of hardware”
    “more hardware please”

    View Slide

  53. “Buy lots of hardware”
    “more hardware please”
    “Just a little bit more hardware”

    View Slide

  54. “Buy lots of hardware”
    “more hardware please”
    “Just a little bit more hardware”

    View Slide

  55. Just run the tests
    that matter...

    View Slide

  56. Failure probability
    matters
    European Computing manufacturer

    View Slide

  57. Failure probability
    matters
    European Computing manufacturer
    Automated test suite ~ 18 hours

    View Slide

  58. Failure probability
    matters
    European Computing manufacturer
    Tests that were most ~ 2 hours
    likely to fail
    Automated test suite ~ 18 hours

    View Slide

  59. http://vimeo.com/12467122

    View Slide

  60. View Slide

  61. Tests that
    never fail

    View Slide

  62. Tests that
    never fail

    View Slide

  63. Tests that
    never fail
    Tests which
    regularly fail

    View Slide

  64. Daily Build
    Tests that
    never fail
    Tests which
    regularly fail

    View Slide

  65. Daily Build Nightly Build
    Tests that
    never fail
    Tests which
    regularly fail

    View Slide

  66. Daily Build Nightly Build
    Tests that
    never fail
    Tests which
    regularly fail
    $cucumber --tags [email protected] $cucumber --tags @nightly

    View Slide

  67. Flickering Confidence

    View Slide

  68. Flickering Confidence
    FAIL

    View Slide

  69. Flickering Confidence
    FAIL
    FAIL

    View Slide

  70. Flickering Confidence
    FAIL
    PASS
    FAIL

    View Slide

  71. Flickering Confidence
    FAIL
    PASS
    FAIL
    Main Flicker
    @flicker

    View Slide

  72. Divide and conquer
    • Rails - Engines
    • Service Orientated Architecture
    1 hour
    7 7 7
    7 7 7
    7 7 7

    View Slide

  73. Divide and Concuquer

    View Slide

  74. Divide and Concuquer

    View Slide

  75. Divide and Concuquer

    View Slide

  76. Divide and Concuquer

    View Slide

  77. Divide and Concuquer

    View Slide

  78. Divide and Concuquer
    Scenario: Related artists
    Given an artist “SYGC”
    And “M.Bison” is related to “SYGC”
    When I visit the artist
    Then I should see within Similar
    Artists a link to “M.Bison”

    View Slide

  79. Divide and conquer
    Rails
    Scenario: Related artists
    Given an artist “SYGC”
    And “M.Bison” is related to “SYGC”
    When I visit the artist
    Then I should see within Similar
    Artists a link to “M.Bison”

    View Slide

  80. Divide and conquer
    Related
    Artists
    Service
    Rails
    Scenario: Related artists
    Given an artist “SYGC”
    And “M.Bison” is related to “SYGC”
    When I visit the artist
    Then I should see within Similar
    Artists a link to “M.Bison”

    View Slide

  81. Divide and conquer
    related(artist)
    Related
    Artists
    Service
    Rails
    Scenario: Related artists
    Given an artist “SYGC”
    And “M.Bison” is related to “SYGC”
    When I visit the artist
    Then I should see within Similar
    Artists a link to “M.Bison”

    View Slide

  82. Divide and conquer
    related(artist)
    Related
    Artists
    Service
    HTML
    Rails
    Scenario: Related artists
    Given an artist “SYGC”
    And “M.Bison” is related to “SYGC”
    When I visit the artist
    Then I should see within Similar
    Artists a link to “M.Bison”

    View Slide

  83. Divide and conquer
    related(artist)
    Related
    Artists
    Service
    HTML
    Rails
    Scenario: Related artists
    Given an artist “SYGC”
    And “M.Bison” is related to “SYGC”
    When I visit the artist
    Then I should see within Similar
    Artists a link to “M.Bison”

    View Slide

  84. Divide and conquer
    related(artist)
    Related
    Artists
    Service
    HTML
    Rails
    Scenario: Related artists
    Given an artist “SYGC”
    And “M.Bison” is related to “SYGC”
    When I visit the artist
    Then I should see within Similar
    Artists a link to “M.Bison”
    unit test
    unit test

    View Slide

  85. Don’t use Acceptance
    tests
    http://jamesshore.com/Blog/Alternatives-
    to-Acceptance-Testing.html
    Heresy!

    View Slide

  86. Pairwise
    Scenario: Testing Cucumber against different environments
    Given I have a rails app
    And I’m using Ruby
    And I am using the
    And I am using the driver
    Then Cucumber should install and play nicely
    Examples:
    | Rails version | Ruby version | test framework | Driver |
    | 3.0 | 1.8.7 | Rspec | Webrat |
    | 3.0 | 1.8.7 | Testunit | Capybara |
    | 3.0 | 1.8.7 | Rspec | Capybara |
    | 3.0 | 1.8.7 | Testunit | Webrat |
    | 3.0 | 1.9 | Rspec | Webrat |
    | 3.0 | 1.9 | Testunit | Webrat |
    | 2.3.1 | 1.9 | Rspec | Webrat |
    ....

    View Slide

  87. | Rails version | Ruby version | test framework | Driver |
    | 2.3.1 | 1.8.7 | Rspec | Webrat |
    | 2.3.1 | 1.8.7 | Rspec | Capybara |
    | 2.3.1 | 1.8.7 | Testunit | Webrat |
    | 2.3.1 | 1.8.7 | Testunit | Capybara |
    | 2.3.1 | 1.9 | Rspec | Capybara |
    | 2.3.1 | 1.9 | Rspec | Webrat |
    | 2.3.1 | 1.9 | Testunit | Capybara |
    | 2.3.1 | 1.9 | Testunit | Webrat |
    ....
    “most faults are caused by interactions of at most two
    factors”

    View Slide

  88. Pairwise
    http://www.screencast.com/t/NGI0NjVk

    View Slide

  89. We should add some
    funky Ajax features
    Just when you thought
    things were going well

    View Slide

  90. JavaScript
    In-memory
    Browser based
    Selenium, Watir Celerity

    View Slide

  91. Need for speed
    Webrat
    Celerity
    Watir
    Selenium
    0 3.75 7.5 11.25 15
    12.9
    12
    12
    0.8
    start-up time (seconds)

    View Slide

  92. Capybara
    Like Webrat but bigger
    envjs
    Johnson
    spidermonkey
    gem install johnson --prerelease
    gem install envjs
    sudo gem install capybara

    View Slide

  93. http://vimeo.com/12467133

    View Slide

  94. Capybara-envjs

    View Slide

  95. View Slide

  96. Find the right
    compromise for you

    View Slide

  97. Find the right
    compromise for you
    Slow test builds produce
    slow release cycles

    View Slide

  98. Find the right
    compromise for you
    Slow test builds produce
    slow release cycles
    Your test builds are a
    goldmine of data

    View Slide

  99. Thanks
    Joseph Wilk
    http://blog.josephwilk.net
    @josephwilk
    Its been
    emotional

    View Slide