Slide 1
Slide 1 text
SUSTAINABLE
BDD
Matt Parker
Robbie Clutton
Software Engineers, Pivotal Labs
Slide 2
Slide 2 text
HYPE CYCLE
Slide 3
Slide 3 text
WTF BDD
1. Brittle Tests
2. Dev-written Acceptance Criteria
3. Unreadable Tests
4. Flickering Tests
5. Slow Tests
Slide 4
Slide 4 text
CARGO CULTING
Slide 5
Slide 5 text
HISTORY
In the beginning... there
was the unit test.
And lo it was good.
Slide 6
Slide 6 text
HISTORY
sUnit -> xUnit
Slide 7
Slide 7 text
HISTORY
And then there was Dan
North.
Slide 8
Slide 8 text
SPECIFICATION
BY EXAMPLE
Slide 9
Slide 9 text
FEATURE: SIGN UP
Slide 10
Slide 10 text
FEATURE: SIGN UP
Give me an example.
Slide 11
Slide 11 text
FEATURE: SIGN UP
Give me an example.
"User opens app for the first time,
provides their email, desired
username, and desired password, and
is immediately logged in and allowed
to start using the application."
Slide 12
Slide 12 text
FEATURE: SIGN UP
Give me an example.
• Valid username, email, password
Slide 13
Slide 13 text
FEATURE: SIGN UP
Give me an example.
• Valid username, email, password
• Blank username, email, or password
Slide 14
Slide 14 text
FEATURE: SIGN UP
Give me an example.
• Valid username, email, password
• Blank username, email, or password
• Non-matching Password
Slide 15
Slide 15 text
FEATURE: SIGN UP
Give me an example.
• Valid username, email, password
• Blank username, email, or password
• Non-matching Password
• Password too weak
Slide 16
Slide 16 text
FEATURE: SIGN UP
Give me an example.
• Valid username, email, password
• Blank username, email, or password
• Non-matching Password
• Password too weak
• Email address invalid
Slide 17
Slide 17 text
FEATURE: SIGN UP
Give me an example.
• Valid username, email, password
• Blank username, email, or password
• Non-matching Password
• Password too weak
• Email address invalid
• Username unavailable
Slide 18
Slide 18 text
FEATURE: ACCOUNT
RECOVERY
Background: User forgot password
• User remembers email
• User remembers username
• User forgot everything
Slide 19
Slide 19 text
FEATURE: SIGN UP
Give me an example.
• Valid username, email, password
• Blank username, email, or password
• Non-matching Password
• Password too weak
• Email address invalid
• Username unavailable
• User provides someone else’s email
Slide 20
Slide 20 text
FEATURE: EMAIL
VERIFICATION
Give me an example.
• Verification link followed
• Fraud link followed
• No links ever followed
Slide 21
Slide 21 text
FEATURE: SIGN IN
Give me an example.
• Existing username / password
• Unknown username
• Wrong password
Slide 22
Slide 22 text
HIDDEN
COMPLEXITY
1 sentence =>
4 features =>
17 scenarios
Slide 23
Slide 23 text
HISTORY
JBehave
Slide 24
Slide 24 text
HISTORY
Business Value
Feature
Scenario
Slide 25
Slide 25 text
HISTORY
RBehave
Slide 26
Slide 26 text
HISTORY
Story Runner / RSpec
Slide 27
Slide 27 text
HISTORY
Cucumber
Slide 28
Slide 28 text
WHY GHERKIN?
1. State Diagrams for the Masses
Slide 29
Slide 29 text
APP AS STATE MACHINE
Every feature of your application is a
state transition.
GIVEN (start state)
WHEN (event)
THEN (state transition)
Slide 30
Slide 30 text
WHY GHERKIN?
1. State Diagrams for the Masses
2. Copy/Paste from Tracker to editor
Slide 31
Slide 31 text
WHY GHERKIN?
1. State Diagrams for the Masses
2. Copy/Paste from Tracker to editor
3. Living / Executable Documentation
Slide 32
Slide 32 text
HISTORY
Spinach, Turnip, Steak,
Bacon, Filet,
Lemon, ......................
Slide 33
Slide 33 text
HISTORY
10 build shiny new tool
20 make same mistakes
30 goto 10
Slide 34
Slide 34 text
BRITTLE
A single change in the application
causes many tests to break
Slide 35
Slide 35 text
BRITTLE
A single change in the application
causes many tests to break, and you
are forced to visit each and every test
and fix each one individually.
Slide 36
Slide 36 text
Brittle tests
typically don’t
reveal intent.
Slide 37
Slide 37 text
Feature: Tweet
Scenario: Valid Tweet
#....
Scenario: Overlong Tweet
#....
Scenario: Duplicate tweet
#....
Slide 38
Slide 38 text
Scenario: Valid Tweet
Given there is a user "bob" with password "password"
And there is a user "alice" with password "password" that follows "bob"
And I visit "/"
And I fill in "username" with "bob"
And I fill in "password" with "password"
And I click "Log in"
When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise
noise noise noise noise noise noise noise noise haha I'm less than 140 characters."
And I click "Submit"
Then I should see "Your tweet was submitted"
When I visit "/bob"
Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise
noise noise noise noise noise noise haha I'm less than 140 characters."
When I click "Log out"
And I visit "/"
And I fill in "username" with "alice"
And I fill in "password" with "password"
And I click "Log in"
Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise
noise noise noise noise noise noise haha I'm less than 140 characters."
Slide 39
Slide 39 text
Scenario: Valid Tweet
Given there is a user "bob" with password "password"
And there is a user "alice" with password "password" that follows "bob"
And I visit "/"
And I fill in "username" with "bob"
And I fill in "password" with "password"
And I click "Log in"
When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise
noise noise noise noise noise noise noise noise haha I'm less than 140 characters."
And I click "Submit"
Then I should see "Your tweet was submitted"
When I visit "/bob"
Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise
noise noise noise noise noise noise haha I'm less than 140 characters."
When I click "Log out"
And I visit "/"
And I fill in "username" with "alice"
And I fill in "password" with "password"
And I click "Log in"
Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise
noise noise noise noise noise noise haha I'm less than 140 characters."
Login
Slide 40
Slide 40 text
Scenario: Valid Tweet
Given there is a user "bob" with password "password"
And there is a user "alice" with password "password" that follows "bob"
And I visit "/"
And I fill in "username" with "bob"
And I fill in "password" with "password"
And I click "Log in"
When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise
noise noise noise noise noise noise noise noise haha I'm less than 140 characters."
And I click "Submit"
Then I should see "Your tweet was submitted"
When I visit "/bob"
Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise
noise noise noise noise noise noise haha I'm less than 140 characters."
When I click "Log out"
And I visit "/"
And I fill in "username" with "alice"
And I fill in "password" with "password"
And I click "Log in"
Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise
noise noise noise noise noise noise haha I'm less than 140 characters."
Tweet
Slide 41
Slide 41 text
Scenario: Overlong Tweet
Given there is a user "bob" with password "password"
And there is a user "alice" with password "password" that
follows "bob"
And I visit "/"
And I fill in "username" with "bob"
And I fill in "password" with "password"
And I click "Log in"
When I fill in "Tweet" with "this is a test tweet noise noise
noise noise noise noise noise noise noise noise noise noise
noise noise noise noise OH NOEZ IM TOO LONG
NOOOOOOOOOOOOOOOOOOOOOO"
And I click "Submit"
Then I should see "Tweet too long"
Slide 42
Slide 42 text
Scenario: Overlong Tweet
Given there is a user "bob" with password "password"
And there is a user "alice" with password "password" that
follows "bob"
And I visit "/"
And I fill in "username" with "bob"
And I fill in "password" with "password"
And I click "Log in"
When I fill in "Tweet" with "this is a test tweet noise noise
noise noise noise noise noise noise noise noise noise noise
noise noise noise noise OH NOEZ IM TOO LONG
NOOOOOOOOOOOOOOOOOOOOOO"
And I click "Submit"
Then I should see "Tweet too long"
Tweet
Slide 43
Slide 43 text
Scenario: Overlong Tweet
Given there is a user "bob" with password "password"
And there is a user "alice" with password "password" that
follows "bob"
And I visit "/"
And I fill in "username" with "bob"
And I fill in "password" with "password"
And I click "Log in"
When I fill in "Tweet" with "this is a test tweet noise noise
noise noise noise noise noise noise noise noise noise noise
noise noise noise noise OH NOEZ IM TOO LONG
NOOOOOOOOOOOOOOOOOOOOOO"
And I click "Submit"
Then I should see "Tweet too long"
Login
Slide 44
Slide 44 text
Scenario: Duplicate Tweet
Given there is a user "bob" with password "password"
And there is a user "alice" with password "password" that follows "bob"
And I visit "/"
And I fill in "username" with "bob"
And I fill in "password" with "password"
And I click "Log in"
When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise
noise noise noise noise noise noise noise noise haha I'm less than 140 characters."
And I click "Submit"
Then I should see "Your tweet was submitted"
When I visit "/"
When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise
noise noise noise noise noise noise noise noise haha I'm less than 140 characters."
And I click "Submit"
Then I should see “Your tweet is a duplicate”
When I visit "/bob"
Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise
noise noise noise noise noise noise noise OH NOEZ IM TOO LONG NOOOOOOOOOOOOOOOOOOOOOO"
When I visit "/bob"
Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise
noise noise noise noise noise noise noise haha I'm less than 140 characters."
Slide 45
Slide 45 text
Scenario: Duplicate Tweet
Given there is a user "bob" with password "password"
And there is a user "alice" with password "password" that follows "bob"
And I visit "/"
And I fill in "username" with "bob"
And I fill in "password" with "password"
And I click "Log in"
When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise
noise noise noise noise noise noise noise noise haha I'm less than 140 characters."
And I click "Submit"
Then I should see "Your tweet was submitted"
When I visit "/"
When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise
noise noise noise noise noise noise noise noise haha I'm less than 140 characters."
And I click "Submit"
Then I should see “Your tweet is a duplicate”
When I visit "/bob"
Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise
noise noise noise noise noise noise noise OH NOEZ IM TOO LONG NOOOOOOOOOOOOOOOOOOOOOO"
When I visit "/bob"
Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise
noise noise noise noise noise noise noise haha I'm less than 140 characters."
Login
Slide 46
Slide 46 text
Scenario: Duplicate Tweet
Given there is a user "bob" with password "password"
And there is a user "alice" with password "password" that follows "bob"
And I visit "/"
And I fill in "username" with "bob"
And I fill in "password" with "password"
And I click "Log in"
When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise
noise noise noise noise noise noise noise noise haha I'm less than 140 characters."
And I click "Submit"
Then I should see "Your tweet was submitted"
When I visit "/"
When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise
noise noise noise noise noise noise noise noise haha I'm less than 140 characters."
And I click “Submit”
Then I should see “Your tweet is a duplicate”
When I visit "/bob"
Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise
noise noise noise noise noise noise noise OH NOEZ IM TOO LONG NOOOOOOOOOOOOOOOOOOOOOO"
When I visit "/bob"
Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise
noise noise noise noise noise noise noise haha I'm less than 140 characters."
Tweet
Slide 47
Slide 47 text
feature “Tweet” do
scenario "valid tweet" do
create :user, username: "bob", password: "password"
create :user, username: "alice", password: "password"
visit "/"
fill_in "Username", with: "bob"
fill_in "Password", with: "password"
click_button "Log In"
fill_in "Tweet", with: "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise
noise noise noise noise haha I'm less than 140 characters."
click_button "Submit"
page.should have_content "Your tweet was submitted"
visit "/bob"
page.should have_content "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise
noise noise noise noise noise haha I'm less than 140 characters."
click_link "Log out"
visit "/"
fill_in "Username", with: "alice"
fill_in "Password", with: "password"
click_button "Log in"
page.should have_content "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise
noise noise noise noise noise haha I'm less than 140 characters."
end
end
Slide 48
Slide 48 text
BIG RATS LEAVE BIG
POOS
Slide 49
Slide 49 text
BIG RATS LEAVE BIG
POOS
click_button "Tweet"
Slide 50
Slide 50 text
BIG RATS LEAVE BIG
POOS
click_button "Tweet"
click_link_or_button "Tweet"
Slide 51
Slide 51 text
BIG RATS LEAVE BIG
POOS
click_button "Tweet"
click_link_or_button "Tweet"
click_on "Tweet"
Slide 52
Slide 52 text
BIG RATS LEAVE BIG
POOS
click_button "Tweet"
click_link_or_button "Tweet"
click_on "Tweet"
click_on "#tweet"
Slide 53
Slide 53 text
BIG RATS LEAVE BIG
POOS
click_button "Tweet"
click_link_or_button "Tweet"
click_on "Tweet"
click_on "#tweet"
find("#tweet").click
Slide 54
Slide 54 text
BIG RATS LEAVE BIG
POOS
click_button "Tweet"
click_link_or_button "Tweet"
click_on "Tweet"
click_on "#tweet"
find("#tweet").click
page.execute_script("$('#tweet').click()")
Slide 55
Slide 55 text
BIG RATS LEAVE BIG
POOS
“Now, whenever you tweet, pop up a modal
that forces the user to enter a CAPTCHA.”
Slide 56
Slide 56 text
BIG RATS LEAVE BIG
POOS
FFFFF...FFF....F....FFFFFFFFFFFFF.........
Slide 57
Slide 57 text
BIG RATS LEAVE BIG
POOS
module WebHelpers
def tweet(message)
fill_in “Message”, with: message
click_on “Tweet”
end
end
Slide 58
Slide 58 text
BIG RATS LEAVE BIG
POOS
#rspec
RSpec.configure do |c|
c.include WebHelpers, type: :request
end
#cucumber
World WebHelpers
Slide 59
Slide 59 text
Feature: Tweet
Scenario: Valid Tweet
Given Bob has authenticated
When Bob submits a valid tweet
Then Alice can see Bob’s tweet
And Bob should see that tweet in his timeline
Scenario: Overlong Tweet
#....
Scenario: Duplicate tweet
#....
Slide 60
Slide 60 text
BIG RATS LEAVE BIG
POOS
Given /^Bob has authenticated$/ do
visit “/”
fill_in “Username”, with: “Bob”
fill_in “Password”, with: “password”
click_on “Sign In”
end
Slide 61
Slide 61 text
BIG RATS LEAVE BIG
POOS
Given /^Bob has authenticated$/ do
login bob
end
Slide 62
Slide 62 text
BIG RATS LEAVE BIG
POOS
module WebHelpers
#...
def bob
@bob ||= create :user, name: “Bob”
end
#...
end
Slide 63
Slide 63 text
BIG RATS LEAVE BIG
POOS
When /^Bob submits a valid Tweet$/ do
click_on “Tweet”
within(“#submit_tweet_modal”) do
fill_in “Captcha”, with: captcha
click_on “Tweet”
end
end
Slide 64
Slide 64 text
BIG RATS LEAVE BIG
POOS
When /^Bob submits a valid Tweet$/ do
tweet “Hello, world”
end
Slide 65
Slide 65 text
BIG RATS LEAVE BIG
POOS
Then /^Alice can see Bob’s Tweet$/ do
click_on “Sign Out”
fill_in “Username”, with: “Alice”
fill_in “Password”, with: “password”
click_on “Sign In”
visit “/alice”
page.should have_content “Hello, world”
end
Slide 66
Slide 66 text
BIG RATS LEAVE BIG
POOS
Then /^Alice can see Bob’s Tweet$/ do
login alice
alice.should see_tweet “Hello, world”
end
Slide 67
Slide 67 text
BIG RATS LEAVE BIG
POOS
feature “Tweet” do
scenario “Valid Tweet” do
login bob
tweet “hello, world”
login alice
alice.should see “hello, world”
end
end
Slide 68
Slide 68 text
PRODUCT OWNERS
DON’T COLLABORATE
ON ACCEPTANCE
CRITERIA
Slide 69
Slide 69 text
PRODUCT OWNERS
DON’T COLLABORATE
ON ACCEPTANCE
CRITERIA
1. CARGO CULT
Slide 70
Slide 70 text
PRODUCT OWNERS
DON’T COLLABORATE
ON ACCEPTANCE
CRITERIA
1. CARGO CULT
2. It’s hard
Slide 71
Slide 71 text
PRODUCT OWNERS
DON’T COLLABORATE
ON ACCEPTANCE
CRITERIA
1. CARGO CULT
2. It’s hard
3. You’re annoying them
Slide 72
Slide 72 text
Given I click on “Sign In”
And I fill in “Username” with “Bob”
And I fill in “Password” with “password”
And I click on “Submit”
#....
Slide 73
Slide 73 text
Given Bob logged in
Slide 74
Slide 74 text
NO ONE READS THE
CUKES
Slide 75
Slide 75 text
NO ONE READS THE
CUKES
1. They’re unreadable
Slide 76
Slide 76 text
NO ONE READS THE
CUKES
1. They’re unreadable
2. They’re not exposed anywhere
Slide 77
Slide 77 text
$ cucumber -f html
Slide 78
Slide 78 text
$ relish push myproj
Slide 79
Slide 79 text
$ wally push \
$PROJ features/
Slide 80
Slide 80 text
NO ONE READS THE
CUKES
1. They’re unreadable
2. They’re not exposed anywhere
3. You haven’t needed documentation
Slide 81
Slide 81 text
$ time rake
Slide 82
Slide 82 text
$ time rake
real 18m0.926s
Slide 83
Slide 83 text
$ time rake
real 18m0.926s
imagined ETERNITY
Slide 84
Slide 84 text
Slow tests find
a way of getting
run less often.
Slide 85
Slide 85 text
$ time rake
real 18m0.926s
imagined ETERNITY
visit root_path
click_on “Some Link”
sleep 10
page.should have_content “FOO”
Slide 86
Slide 86 text
$ rake
..........F.........
$ rake
...................
$ rake
....F...........F....
$ rake
...................
Slide 87
Slide 87 text
$ time rake
real 18m0.926s
imagined ETERNITY
visit root_path
click_on “Some Link”
wait_for { page.has_selector?
(“.foo_container”) }
page.should have_content “Foo”
Slide 88
Slide 88 text
$ time rake
real 18m0.926s
imagined ETERNITY
visit root_path
click_on “Some Link”
wait_for { page.should
have_selector(“.foo_container”) }
page.should have_content “Foo”
Whoops
Slide 89
Slide 89 text
$ time rake
real 18m0.926s
imagined ETERNITY
visit root_path
click_on “Some Link”
page.should have_content “Foo”
Slide 90
Slide 90 text
Q IS FOR QUARANTINE
@quarantine
Scenario: Something flakey
#....
Slide 91
Slide 91 text
Q IS FOR QUARANTINE
#regular ci build
#!/bin/bash
cucumber --tags ~@quarantine
Slide 92
Slide 92 text
Q IS FOR QUARANTINE
#quarantine ci build
#!/bin/bash
cucumber --tags @quarantine
Slide 93
Slide 93 text
BUILD NANNY
1. Timebox
2. Rethink
3. Delete
Slide 94
Slide 94 text
Photo by Kris Hicks
Slide 95
Slide 95 text
$ time rake
“You should not be afraid to delete tests that are no
longer providing value, no matter whether you originally
planned to keep them or not. We tend to treat tests as
these holy creatures that live blameless, irreproachable
lives once they have sprung into existence. Not so. The
maintenance required to keep a test running weighs
against its value in further development. Sometimes
these lines cross, and the test simply becomes a burden
on the project. Having the skill and experience to
recognize a burdensome test is something we should be
bringing to our clients, as well as the fortitude to
rewrite it, rethink it, or delete it." - ADAM MILLIGAN
Slide 96
Slide 96 text
$ time rake
Slide 97
Slide 97 text
$ time rake
real 9m0.926s
Slide 98
Slide 98 text
$ time rake
real 9m0.926s
imagined ETERNITY
Slide 99
Slide 99 text
Why are you
testing the
login
functionality 50
times?
Slide 100
Slide 100 text
1. Journey
2. Functional
Slide 101
Slide 101 text
$ time rake
real 9m0.926s
imagined ETERNITY
@authenticated
Feature: Publishing Widgets
Scenario: ....
Slide 102
Slide 102 text
$ time rake
real 9m0.926s
imagined ETERNITY
Before(“@authenticated”) do
stub_authentication
end
Slide 103
Slide 103 text
$ time rake
Slide 104
Slide 104 text
$ time rake
real 4m0.926s
Slide 105
Slide 105 text
$ time rake
real 4m0.926s
imagined ETERNITY
Slide 106
Slide 106 text
$ time rake
Feature: Sign Up
Scenario: Valid username, email, password
Scenario: Blank username, email, or password
Scenario: Non-matching Password
Scenario: Password too weak
Scenario: Email address invalid
Scenario: Username unavailable
Scenario: User provides someone else’s email
Slide 107
Slide 107 text
$ time rake
Feature: Sign Up
Scenario: Valid username, email, password
Scenario: Blank username, email, password
Scenario: Non-matching Password
Scenario: Password too weak
Scenario: Email address invalid
Scenario: Username unavailable
Scenario: User provides someone else’s email
Slide 108
Slide 108 text
$ time rake
Feature: Sign Up
Scenario: Valid username, email, password
Scenario: Blank username, email, password
Scenario: Non-matching Password
Scenario: Password too weak
Scenario: Email address invalid
Scenario: Username unavailable
Scenario: User provides someone else’s email
Slide 109
Slide 109 text
$ time rake
Feature: Sign Up
Scenario: Valid username, email, password
Scenario: Blank username, email, password
@ci
Scenario: Non-matching Password
....
Slide 110
Slide 110 text
$ time rake
#.cucumber.yml
default: --tags ~@ci
Slide 111
Slide 111 text
$ time rake
Feature: Sign Up
Scenario: Valid username, email, password
Scenario: Blank username, email, password
@no-ui
Scenario: Non-matching Password
....
Slide 112
Slide 112 text
$ time rake
Before “@no-ui” do
extend DomainHelpers
end
Slide 113
Slide 113 text
$ time rake
module DomainHelpers
attr_reader :current_user
def authenticate
@current_user = create :user
end
def tweet(message)
current_user.tweet message
end
#...
end
Slide 114
Slide 114 text
$ time rake
Slide 115
Slide 115 text
$ time rake
real 30.926s
Slide 116
Slide 116 text
1.
FIGHT PAIN
2.
REVEAL INTENT
3.
BDD LIKE YOU
MEAN IT
Slide 117
Slide 117 text
Q/A