Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Sustainable BDD

Sustainable BDD

The BDD hype cycle is over. Recently, there’s been a lot of backlash against popular BDD libraries like Cucumber. Some developers blame their test frameworks for brittle test suites and long build times. Others go so far as to claim that acceptance testing is simply not sustainable, period. In this talk, we’ll do some root cause analysis of this phenomenon with shocking results - it’s not the test framework, it’s not the methodology, it’s you. You’ve abused your test framework, you’ve cargo-culted the methodology, and now you’re feeling the pain. We’ll show you a way out of the mess you’ve made. We’ll discuss the main problems BDD was intended to solve. We’ll show you how to groom your test suite into journey, functional, integration, and unit tests in order to address build times. We’ll teach how to mitigate against brittleness and flickers, and how to let your tests reveal the intent of the application and actually become the executable documentation we’ve been waiting for.

1668868370ee5829339e06031ad0b145?s=128

Robbie Clutton

February 23, 2013
Tweet

Transcript

  1. SUSTAINABLE BDD Matt Parker Robbie Clutton Software Engineers, Pivotal Labs

  2. HYPE CYCLE

  3. WTF BDD 1. Brittle Tests 2. Dev-written Acceptance Criteria 3.

    Unreadable Tests 4. Flickering Tests 5. Slow Tests
  4. CARGO CULTING

  5. HISTORY In the beginning... there was the unit test. And

    lo it was good.
  6. HISTORY sUnit -> xUnit

  7. HISTORY And then there was Dan North.

  8. SPECIFICATION BY EXAMPLE

  9. FEATURE: SIGN UP

  10. FEATURE: SIGN UP Give me an example.

  11. FEATURE: SIGN UP Give me an example. "User opens app

    for the first time, provides their email, desired username, and desired password, and is immediately logged in and allowed to start using the application."
  12. FEATURE: SIGN UP Give me an example. • Valid username,

    email, password
  13. FEATURE: SIGN UP Give me an example. • Valid username,

    email, password • Blank username, email, or password
  14. FEATURE: SIGN UP Give me an example. • Valid username,

    email, password • Blank username, email, or password • Non-matching Password
  15. FEATURE: SIGN UP Give me an example. • Valid username,

    email, password • Blank username, email, or password • Non-matching Password • Password too weak
  16. FEATURE: SIGN UP Give me an example. • Valid username,

    email, password • Blank username, email, or password • Non-matching Password • Password too weak • Email address invalid
  17. FEATURE: SIGN UP Give me an example. • Valid username,

    email, password • Blank username, email, or password • Non-matching Password • Password too weak • Email address invalid • Username unavailable
  18. FEATURE: ACCOUNT RECOVERY Background: User forgot password • User remembers

    email • User remembers username • User forgot everything
  19. FEATURE: SIGN UP Give me an example. • Valid username,

    email, password • Blank username, email, or password • Non-matching Password • Password too weak • Email address invalid • Username unavailable • User provides someone else’s email
  20. FEATURE: EMAIL VERIFICATION Give me an example. • Verification link

    followed • Fraud link followed • No links ever followed
  21. FEATURE: SIGN IN Give me an example. • Existing username

    / password • Unknown username • Wrong password
  22. HIDDEN COMPLEXITY 1 sentence => 4 features => 17 scenarios

  23. HISTORY JBehave

  24. HISTORY Business Value Feature Scenario

  25. HISTORY RBehave

  26. HISTORY Story Runner / RSpec

  27. HISTORY Cucumber

  28. WHY GHERKIN? 1. State Diagrams for the Masses

  29. APP AS STATE MACHINE Every feature of your application is

    a state transition. GIVEN (start state) WHEN (event) THEN (state transition)
  30. WHY GHERKIN? 1. State Diagrams for the Masses 2. Copy/Paste

    from Tracker to editor
  31. WHY GHERKIN? 1. State Diagrams for the Masses 2. Copy/Paste

    from Tracker to editor 3. Living / Executable Documentation
  32. HISTORY Spinach, Turnip, Steak, Bacon, Filet, Lemon, ......................

  33. HISTORY 10 build shiny new tool 20 make same mistakes

    30 goto 10
  34. BRITTLE A single change in the application causes many tests

    to break
  35. BRITTLE A single change in the application causes many tests

    to break, and you are forced to visit each and every test and fix each one individually.
  36. Brittle tests typically don’t reveal intent.

  37. Feature: Tweet Scenario: Valid Tweet #.... Scenario: Overlong Tweet #....

    Scenario: Duplicate tweet #....
  38. Scenario: Valid Tweet Given there is a user "bob" with

    password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see "Your tweet was submitted" When I visit "/bob" Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." When I click "Log out" And I visit "/" And I fill in "username" with "alice" And I fill in "password" with "password" And I click "Log in" Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters."
  39. Scenario: Valid Tweet Given there is a user "bob" with

    password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see "Your tweet was submitted" When I visit "/bob" Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." When I click "Log out" And I visit "/" And I fill in "username" with "alice" And I fill in "password" with "password" And I click "Log in" Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." Login
  40. Scenario: Valid Tweet Given there is a user "bob" with

    password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see "Your tweet was submitted" When I visit "/bob" Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." When I click "Log out" And I visit "/" And I fill in "username" with "alice" And I fill in "password" with "password" And I click "Log in" Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." Tweet
  41. Scenario: Overlong Tweet Given there is a user "bob" with

    password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise OH NOEZ IM TOO LONG NOOOOOOOOOOOOOOOOOOOOOO" And I click "Submit" Then I should see "Tweet too long"
  42. Scenario: Overlong Tweet Given there is a user "bob" with

    password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise OH NOEZ IM TOO LONG NOOOOOOOOOOOOOOOOOOOOOO" And I click "Submit" Then I should see "Tweet too long" Tweet
  43. Scenario: Overlong Tweet Given there is a user "bob" with

    password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise OH NOEZ IM TOO LONG NOOOOOOOOOOOOOOOOOOOOOO" And I click "Submit" Then I should see "Tweet too long" Login
  44. Scenario: Duplicate Tweet Given there is a user "bob" with

    password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see "Your tweet was submitted" When I visit "/" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see “Your tweet is a duplicate” When I visit "/bob" Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise OH NOEZ IM TOO LONG NOOOOOOOOOOOOOOOOOOOOOO" When I visit "/bob" Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters."
  45. Scenario: Duplicate Tweet Given there is a user "bob" with

    password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see "Your tweet was submitted" When I visit "/" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see “Your tweet is a duplicate” When I visit "/bob" Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise OH NOEZ IM TOO LONG NOOOOOOOOOOOOOOOOOOOOOO" When I visit "/bob" Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." Login
  46. Scenario: Duplicate Tweet Given there is a user "bob" with

    password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see "Your tweet was submitted" When I visit "/" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click “Submit” Then I should see “Your tweet is a duplicate” When I visit "/bob" Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise OH NOEZ IM TOO LONG NOOOOOOOOOOOOOOOOOOOOOO" When I visit "/bob" Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." Tweet
  47. feature “Tweet” do scenario "valid tweet" do create :user, username:

    "bob", password: "password" create :user, username: "alice", password: "password" visit "/" fill_in "Username", with: "bob" fill_in "Password", with: "password" click_button "Log In" fill_in "Tweet", with: "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." click_button "Submit" page.should have_content "Your tweet was submitted" visit "/bob" page.should have_content "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." click_link "Log out" visit "/" fill_in "Username", with: "alice" fill_in "Password", with: "password" click_button "Log in" page.should have_content "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." end end
  48. BIG RATS LEAVE BIG POOS

  49. BIG RATS LEAVE BIG POOS click_button "Tweet"

  50. BIG RATS LEAVE BIG POOS click_button "Tweet" click_link_or_button "Tweet"

  51. BIG RATS LEAVE BIG POOS click_button "Tweet" click_link_or_button "Tweet" click_on

    "Tweet"
  52. BIG RATS LEAVE BIG POOS click_button "Tweet" click_link_or_button "Tweet" click_on

    "Tweet" click_on "#tweet"
  53. BIG RATS LEAVE BIG POOS click_button "Tweet" click_link_or_button "Tweet" click_on

    "Tweet" click_on "#tweet" find("#tweet").click
  54. BIG RATS LEAVE BIG POOS click_button "Tweet" click_link_or_button "Tweet" click_on

    "Tweet" click_on "#tweet" find("#tweet").click page.execute_script("$('#tweet').click()")
  55. BIG RATS LEAVE BIG POOS “Now, whenever you tweet, pop

    up a modal that forces the user to enter a CAPTCHA.”
  56. BIG RATS LEAVE BIG POOS FFFFF...FFF....F....FFFFFFFFFFFFF.........

  57. BIG RATS LEAVE BIG POOS module WebHelpers def tweet(message) fill_in

    “Message”, with: message click_on “Tweet” end end
  58. BIG RATS LEAVE BIG POOS #rspec RSpec.configure do |c| c.include

    WebHelpers, type: :request end #cucumber World WebHelpers
  59. Feature: Tweet Scenario: Valid Tweet Given Bob has authenticated When

    Bob submits a valid tweet Then Alice can see Bob’s tweet And Bob should see that tweet in his timeline Scenario: Overlong Tweet #.... Scenario: Duplicate tweet #....
  60. BIG RATS LEAVE BIG POOS Given /^Bob has authenticated$/ do

    visit “/” fill_in “Username”, with: “Bob” fill_in “Password”, with: “password” click_on “Sign In” end
  61. BIG RATS LEAVE BIG POOS Given /^Bob has authenticated$/ do

    login bob end
  62. BIG RATS LEAVE BIG POOS module WebHelpers #... def bob

    @bob ||= create :user, name: “Bob” end #... end
  63. BIG RATS LEAVE BIG POOS When /^Bob submits a valid

    Tweet$/ do click_on “Tweet” within(“#submit_tweet_modal”) do fill_in “Captcha”, with: captcha click_on “Tweet” end end
  64. BIG RATS LEAVE BIG POOS When /^Bob submits a valid

    Tweet$/ do tweet “Hello, world” end
  65. BIG RATS LEAVE BIG POOS Then /^Alice can see Bob’s

    Tweet$/ do click_on “Sign Out” fill_in “Username”, with: “Alice” fill_in “Password”, with: “password” click_on “Sign In” visit “/alice” page.should have_content “Hello, world” end
  66. BIG RATS LEAVE BIG POOS Then /^Alice can see Bob’s

    Tweet$/ do login alice alice.should see_tweet “Hello, world” end
  67. BIG RATS LEAVE BIG POOS feature “Tweet” do scenario “Valid

    Tweet” do login bob tweet “hello, world” login alice alice.should see “hello, world” end end
  68. PRODUCT OWNERS DON’T COLLABORATE ON ACCEPTANCE CRITERIA

  69. PRODUCT OWNERS DON’T COLLABORATE ON ACCEPTANCE CRITERIA 1. CARGO CULT

  70. PRODUCT OWNERS DON’T COLLABORATE ON ACCEPTANCE CRITERIA 1. CARGO CULT

    2. It’s hard
  71. PRODUCT OWNERS DON’T COLLABORATE ON ACCEPTANCE CRITERIA 1. CARGO CULT

    2. It’s hard 3. You’re annoying them
  72. Given I click on “Sign In” And I fill in

    “Username” with “Bob” And I fill in “Password” with “password” And I click on “Submit” #....
  73. Given Bob logged in

  74. NO ONE READS THE CUKES

  75. NO ONE READS THE CUKES 1. They’re unreadable

  76. NO ONE READS THE CUKES 1. They’re unreadable 2. They’re

    not exposed anywhere
  77. $ cucumber -f html

  78. $ relish push myproj

  79. $ wally push \ $PROJ features/

  80. NO ONE READS THE CUKES 1. They’re unreadable 2. They’re

    not exposed anywhere 3. You haven’t needed documentation
  81. $ time rake

  82. $ time rake real 18m0.926s

  83. $ time rake real 18m0.926s imagined ETERNITY

  84. Slow tests find a way of getting run less often.

  85. $ time rake real 18m0.926s imagined ETERNITY visit root_path click_on

    “Some Link” sleep 10 page.should have_content “FOO”
  86. $ rake ..........F......... $ rake ................... $ rake ....F...........F.... $

    rake ...................
  87. $ time rake real 18m0.926s imagined ETERNITY visit root_path click_on

    “Some Link” wait_for { page.has_selector? (“.foo_container”) } page.should have_content “Foo”
  88. $ time rake real 18m0.926s imagined ETERNITY visit root_path click_on

    “Some Link” wait_for { page.should have_selector(“.foo_container”) } page.should have_content “Foo” Whoops
  89. $ time rake real 18m0.926s imagined ETERNITY visit root_path click_on

    “Some Link” page.should have_content “Foo”
  90. Q IS FOR QUARANTINE @quarantine Scenario: Something flakey #....

  91. Q IS FOR QUARANTINE #regular ci build #!/bin/bash cucumber --tags

    ~@quarantine
  92. Q IS FOR QUARANTINE #quarantine ci build #!/bin/bash cucumber --tags

    @quarantine
  93. BUILD NANNY 1. Timebox 2. Rethink 3. Delete

  94. Photo by Kris Hicks

  95. $ time rake “You should not be afraid to delete

    tests that are no longer providing value, no matter whether you originally planned to keep them or not. We tend to treat tests as these holy creatures that live blameless, irreproachable lives once they have sprung into existence. Not so. The maintenance required to keep a test running weighs against its value in further development. Sometimes these lines cross, and the test simply becomes a burden on the project. Having the skill and experience to recognize a burdensome test is something we should be bringing to our clients, as well as the fortitude to rewrite it, rethink it, or delete it." - ADAM MILLIGAN
  96. $ time rake

  97. $ time rake real 9m0.926s

  98. $ time rake real 9m0.926s imagined ETERNITY

  99. Why are you testing the login functionality 50 times?

  100. 1. Journey 2. Functional

  101. $ time rake real 9m0.926s imagined ETERNITY @authenticated Feature: Publishing

    Widgets Scenario: ....
  102. $ time rake real 9m0.926s imagined ETERNITY Before(“@authenticated”) do stub_authentication

    end
  103. $ time rake

  104. $ time rake real 4m0.926s

  105. $ time rake real 4m0.926s imagined ETERNITY

  106. $ time rake Feature: Sign Up Scenario: Valid username, email,

    password Scenario: Blank username, email, or password Scenario: Non-matching Password Scenario: Password too weak Scenario: Email address invalid Scenario: Username unavailable Scenario: User provides someone else’s email
  107. $ time rake Feature: Sign Up Scenario: Valid username, email,

    password Scenario: Blank username, email, password Scenario: Non-matching Password Scenario: Password too weak Scenario: Email address invalid Scenario: Username unavailable Scenario: User provides someone else’s email
  108. $ time rake Feature: Sign Up Scenario: Valid username, email,

    password Scenario: Blank username, email, password Scenario: Non-matching Password Scenario: Password too weak Scenario: Email address invalid Scenario: Username unavailable Scenario: User provides someone else’s email
  109. $ time rake Feature: Sign Up Scenario: Valid username, email,

    password Scenario: Blank username, email, password @ci Scenario: Non-matching Password ....
  110. $ time rake #.cucumber.yml default: --tags ~@ci

  111. $ time rake Feature: Sign Up Scenario: Valid username, email,

    password Scenario: Blank username, email, password @no-ui Scenario: Non-matching Password ....
  112. $ time rake Before “@no-ui” do extend DomainHelpers end

  113. $ time rake module DomainHelpers attr_reader :current_user def authenticate @current_user

    = create :user end def tweet(message) current_user.tweet message end #... end
  114. $ time rake

  115. $ time rake real 30.926s

  116. 1. FIGHT PAIN 2. REVEAL INTENT 3. BDD LIKE YOU

    MEAN IT
  117. Q/A