Share
Embed
Slide 1
Slide 1 text
YOUR TESTS ARE
LYING TO YOU
Slide 2
Slide 2 text
RUBYCONF
ALCOHOL
Slide 3
Slide 3 text
WHEN TESTS LIE
~ Joe Ruby ~
Slide 4
Slide 4 text
The Problem
describe Customer do
describe “payment” do
it “processes the payment” do
credit_card = mock_model(CreditCard)
result = mock(CreditCardGatewayResult)
result.should_receive(:success?).
and_return(true)
gateway = mock(:gateway)
Gateway.stub(:new => gateway)
gateway.should_receive(:process).twice.
and_return(result)
credit_card.should_receive(:gateway).
exactly(2).times.and_return(result)
Subscription.stub_chain(:new, :add_data,
:make_live, :check).and_return(true)
customer = Customer.new
customer.pay(credit_card)
end
it “fails when the gateway refuses” do
...
end
end
end
Slide 5
Slide 5 text
The Problem
describe Customer do
describe “payment” do
it “processes the payment” do
credit_card = mock_model(CreditCard)
result = mock(CreditCardGatewayResult)
result.should_receive(:success?).
and_return(true)
gateway = mock(:gateway)
Gateway.stub(:new => gateway)
gateway.should_receive(:process).twice.
and_return(result)
credit_card.should_receive(:gateway).
exactly(2).times.and_return(result)
Subscription.stub_chain(:new, :add_data,
:make_live, :check).and_return(true)
customer = Customer.new
customer.pay(credit_card)
end
it “fails when the gateway refuses” do
...
end
end
end
The Verdict
Slide 6
Slide 6 text
The Problem
describe Customer do
describe “payment” do
it “processes the payment” do
credit_card = mock_model(CreditCard)
result = mock(CreditCardGatewayResult)
result.should_receive(:success?).
and_return(true)
gateway = mock(:gateway)
Gateway.stub(:new => gateway)
gateway.should_receive(:process).twice.
and_return(result)
credit_card.should_receive(:gateway).
exactly(2).times.and_return(result)
Subscription.stub_chain(:new, :add_data,
:make_live, :check).and_return(true)
customer = Customer.new
customer.pay(credit_card)
end
it “fails when the gateway refuses” do
...
end
end
end
1. Look at all this setup
code.
2. What’s going on with
that stub_chain?
3. This kind of thing is
duplicated all over this
file, and the whole
codebase.
4. Mocks cause lots of
trouble. I need to start
this test from scratch.
The Verdict
Slide 7
Slide 7 text
The Rewrite
describe Customer do
describe “payment” do
it “processes the payment” do
Gateway.stub(:process => true)
card = CreditCard.new
result = Customer.new.pay(card)
result.should == SUCCESS
end
it “fails when the gateway refuses” do
Gateway.stub(:process => false)
...
end
end
end
Slide 8
Slide 8 text
The Rewrite
describe Customer do
describe “payment” do
it “processes the payment” do
Gateway.stub(:process => true)
card = CreditCard.new
result = Customer.new.pay(card)
result.should == SUCCESS
end
it “fails when the gateway refuses” do
Gateway.stub(:process => false)
...
end
end
end SOLVED
Slide 9
Slide 9 text
The Rewrite
describe Customer do
describe “payment” do
it “processes the payment” do
Gateway.stub(:process => true)
card = CreditCard.new
result = Customer.new.pay(card)
result.should == SUCCESS
end
it “fails when the gateway refuses” do
Gateway.stub(:process => false)
...
end
end
end
601 Bourbon St
New Orleans
LA 30492
29th Sep 2011
Dear Mr Ruby,
Just a brief note to say thanks so
much for sorting out my website.
I’ll certainly be recommending
you next time I hear of someone
in a fix.
Best wishes
Jane
SOLVED
Slide 10
Slide 10 text
The Callback
Jane called. Can you call her
back urgently? 1-516-403-2039.
-- Tracey
Slide 11
Slide 11 text
The Callback
Jane called. Can you call her
back urgently? 1-516-403-2039.
-- Tracey
Customer
CreditCard
Gateway
Subscription
Slide 12
Slide 12 text
The Callback
Jane called. Can you call her
back urgently? 1-516-403-2039.
-- Tracey
The Verdict
Customer
CreditCard
Gateway
Subscription
Slide 13
Slide 13 text
The Callback
Jane called. Can you call her
back urgently? 1-516-403-2039.
-- Tracey
1. Mocks and stubs can
be a pain to set up.
2. Newer code is simpler
and cleaner, but relies on
everything working
together correctly, and
doesn’t catch all edge
cases.
3. I must write more
edge cases!
The Verdict
Customer
CreditCard
Gateway
Subscription
Slide 14
Slide 14 text
The Rewrite #2
describe Customer do
describe “payment” do
it “proceses the payment” do
...
end
it “fails when the gateway refuses” do
...
end
it “fails when the gateway times out” do
...
end
it “fails if the subscription isn’t created” do
...
end
...
end
end
Slide 15
Slide 15 text
The Rewrite #2
describe Customer do
describe “payment” do
it “proceses the payment” do
...
end
it “fails when the gateway refuses” do
...
end
it “fails when the gateway times out” do
...
end
it “fails if the subscription isn’t created” do
...
end
...
end
end
SOLVED
Slide 16
Slide 16 text
The Callback #2
Jane called again. She really
didn’t sound happy. You have
her number. --Tracey
Slide 17
Slide 17 text
The Callback #2
Jane called again. She really
didn’t sound happy. You have
her number. --Tracey
Customer
CreditCard
Gateway
Annual
Monthly
Subscription
Slide 18
Slide 18 text
The Callback #2
Jane called again. She really
didn’t sound happy. You have
her number. --Tracey
The Verdict
Customer
CreditCard
Gateway
Annual
Monthly
Subscription
Slide 19
Slide 19 text
The Callback #2
Jane called again. She really
didn’t sound happy. You have
her number. --Tracey
1. When testing a series
of objects from just the
outside point of view, you
have millions of
combinations.
2. You can’t keep up with
the edge cases.
3. Tests are getting
slower.
4. Therefore: ???
The Verdict
Customer
CreditCard
Gateway
Annual
Monthly
Subscription
Slide 20
Slide 20 text
The Autopsy
The Autopsy
Slide 21
Slide 21 text
The Autopsy
Pathologist: J.B. Rainsberger
Testimony: http://infoq.com/
presentations/integration-tests-scam
The Autopsy
Slide 22
Slide 22 text
The Autopsy
Pathologist: J.B. Rainsberger
Testimony: http://infoq.com/
presentations/integration-tests-scam
The Autopsy
In cases like these, someone is usually
lying.
The Witness
Slide 23
Slide 23 text
The Autopsy
Pathologist: J.B. Rainsberger
Testimony: http://infoq.com/
presentations/integration-tests-scam
The Autopsy
In cases like these, someone is usually
lying.
The Witness
describe Customer do
describe “payment” do
it “proceses the payment” do
Gateway.stub(:process => true)
card = CreditCard.new
result = Customer.new.pay(card)
result.should == SUCCESS
end
it “fails when the gateway refuses” do
Gateway.stub(:process => false)
...
end
end
end
Slide 24
Slide 24 text
The Autopsy
testimony
invalid
Pathologist: J.B. Rainsberger
Testimony: http://infoq.com/
presentations/integration-tests-scam
The Autopsy
In cases like these, someone is usually
lying.
The Witness
describe Customer do
describe “payment” do
it “proceses the payment” do
Gateway.stub(:process => true)
card = CreditCard.new
result = Customer.new.pay(card)
result.should == SUCCESS
end
it “fails when the gateway refuses” do
Gateway.stub(:process => false)
...
end
end
end
Slide 25
Slide 25 text
The Investigation
describe Customer do
it “takes payment” do
c = Customer.new(details)
c.pay.should == true
end
end
Customer
Slide 26
Slide 26 text
The Investigation
describe Customer do
it “takes payment” do
c = Customer.new(details)
c.pay.should == true
end
end
Customer
Gateway
Slide 27
Slide 27 text
The Investigation
describe Customer do
it “takes payment” do
c = Customer.new(details)
c.pay.should == true
end
end
Customer
Gateway
Subscription
describe Subscription do
it “can be created” do
c = Customer.new(details)
s = Subscription.new(c)
s.add_data(data)
s.make_live
s.verify.should == true
end
end
Slide 28
Slide 28 text
The Dilemma
describe Customer do
describe “payment” do
it “proceses the payment” do
credit_card = mock_model(CreditCard)
result = mock(CreditCardGatewayResult)
result.should_receive(:success?).
and_return(true)
gateway = mock(:gateway)
Gateway.stub(:new => gateway)
gateway.should_receive(:process).twice.
and_return(result)
credit_card.should_receive(:gateway).
exactly(2).times.and_return(result)
Subscription.stub_chain(:new, :add_data,
:make_live, :check).and_return(true)
customer = Customer.new
customer.pay(credit_card)
end
it “fails when the gateway refuses” do
...
end
end
end
Slide 29
Slide 29 text
The Dilemma
describe Customer do
describe “payment” do
it “proceses the payment” do
credit_card = mock_model(CreditCard)
result = mock(CreditCardGatewayResult)
result.should_receive(:success?).
and_return(true)
gateway = mock(:gateway)
Gateway.stub(:new => gateway)
gateway.should_receive(:process).twice.
and_return(result)
credit_card.should_receive(:gateway).
exactly(2).times.and_return(result)
Subscription.stub_chain(:new, :add_data,
:make_live, :check).and_return(true)
customer = Customer.new
customer.pay(credit_card)
end
it “fails when the gateway refuses” do
...
end
end
end
“Well designed code is easy to
test.” -- Corey Haines
Slide 30
Slide 30 text
The Dilemma
describe Customer do
describe “payment” do
it “proceses the payment” do
credit_card = mock_model(CreditCard)
result = mock(CreditCardGatewayResult)
result.should_receive(:success?).
and_return(true)
gateway = mock(:gateway)
Gateway.stub(:new => gateway)
gateway.should_receive(:process).twice.
and_return(result)
credit_card.should_receive(:gateway).
exactly(2).times.and_return(result)
Subscription.stub_chain(:new, :add_data,
:make_live, :check).and_return(true)
customer = Customer.new
customer.pay(credit_card)
end
it “fails when the gateway refuses” do
...
end
end
end
Customer
CreditCard
Gateway
Annual
Monthly
Subscription
“Well designed code is easy to
test.” -- Corey Haines
Slide 31
Slide 31 text
The Dilemma
describe Customer do
describe “payment” do
it “proceses the payment” do
credit_card = mock_model(CreditCard)
result = mock(CreditCardGatewayResult)
result.should_receive(:success?).
and_return(true)
gateway = mock(:gateway)
Gateway.stub(:new => gateway)
gateway.should_receive(:process).twice.
and_return(result)
credit_card.should_receive(:gateway).
exactly(2).times.and_return(result)
Subscription.stub_chain(:new, :add_data,
:make_live, :check).and_return(true)
customer = Customer.new
customer.pay(credit_card)
end
it “fails when the gateway refuses” do
...
end
end
end
Customer
CreditCard
Gateway
Annual
Monthly
Subscription
“Well designed code is easy to
test.” -- Corey Haines
Slide 32
Slide 32 text
The Jail
Slide 33
Slide 33 text
The Jail
Ruby on Rails
Slide 34
Slide 34 text
The Jail
Customer
MySQL
Ruby on Rails
Slide 35
Slide 35 text
The Jail
Customer
MySQL
CustomerBilling
CustomerDetails
CustomerSearch
Ruby on Rails
Slide 36
Slide 36 text
The Jail
Customer
MySQL
CustomerBilling
CustomerDetails
CustomerSearch
Destroy All Software
http://destroyallsoftware.com
http://confreaks.net/videos/
641-gogaruco2011-fast-rails-tests
Ruby on Rails
Slide 37
Slide 37 text
The Gang Feud
Slide 38
Slide 38 text
The Gang Feud
Mockists vs Classisists
Slide 39
Slide 39 text
The Gang Feud
Mockists vs Classisists
Mocks vs Stubs is not the
same feud.
Slide 40
Slide 40 text
The Gang Feud
Mockists vs Classisists
Mocks vs Stubs is not the
same feud.
Looks like I can use
either mocks or stubs
when it suits me.
Slide 41
Slide 41 text
The Gang Feud
Mockists vs Classisists
Mocks vs Stubs is not the
same feud.
Looks like I can use
either mocks or stubs
when it suits me.
http://martinfowler.com/articles/
mocksArentStubs.html
Slide 42
Slide 42 text
The Gang Feud
Mockists vs Classisists
Mocks vs Stubs is not the
same feud.
Looks like I can use
either mocks or stubs
when it suits me.
http://martinfowler.com/articles/
mocksArentStubs.html
Pros and Cons
Slide 43
Slide 43 text
The Gang Feud
Mockists vs Classisists
Mocks vs Stubs is not the
same feud.
Looks like I can use
either mocks or stubs
when it suits me.
http://martinfowler.com/articles/
mocksArentStubs.html
Pros and Cons
1. Isolation
Slide 44
Slide 44 text
The Gang Feud
Mockists vs Classisists
Mocks vs Stubs is not the
same feud.
Looks like I can use
either mocks or stubs
when it suits me.
http://martinfowler.com/articles/
mocksArentStubs.html
Pros and Cons
1. Isolation
2. Flexibility
Slide 45
Slide 45 text
The Gang Feud
Mockists vs Classisists
Mocks vs Stubs is not the
same feud.
Looks like I can use
either mocks or stubs
when it suits me.
http://martinfowler.com/articles/
mocksArentStubs.html
Pros and Cons
1. Isolation
2. Flexibility
3. Setup
Slide 46
Slide 46 text
The Gang Feud
Mockists vs Classisists
Mocks vs Stubs is not the
same feud.
Looks like I can use
either mocks or stubs
when it suits me.
http://martinfowler.com/articles/
mocksArentStubs.html
Pros and Cons
1. Isolation
2. Flexibility
3. Setup
4. Design
Slide 47
Slide 47 text
The Gang Feud
Mockists vs Classisists
Mocks vs Stubs is not the
same feud.
Looks like I can use
either mocks or stubs
when it suits me.
http://martinfowler.com/articles/
mocksArentStubs.html
Pros and Cons
1. Isolation
2. Flexibility
3. Setup
4. Design
5. Speed
Slide 48
Slide 48 text
Integration Tests
Slide 49
Slide 49 text
Integration Tests
You still need some
integration tests to
ensure the interactions
are valid.
Slide 50
Slide 50 text
Integration Tests
You still need some
integration tests to
ensure the interactions
are valid.
Acceptance Tests
Slide 51
Slide 51 text
Integration Tests
You still need some
integration tests to
ensure the interactions
are valid.
Acceptance Tests
The customer still needs
to know if the software
does what they asked for.
Slide 52
Slide 52 text
The Key Questions
Slide 53
Slide 53 text
The Key Questions
1. Am I being lied to?
Slide 54
Slide 54 text
The Key Questions
1. Am I being lied to?
2. Is my test code telling
me something?
Slide 55
Slide 55 text
The Key Questions
1. Am I being lied to?
2. Is my test code telling
me something?
3. Which gang do I belong
to? Do I understand the
pros and cons?
Slide 56
Slide 56 text
The Solution
describe Customer do
describe “payment” do
it “proceses the payment” do
credit_card = mock_model(CreditCard)
result = mock(CreditCardGatewayResult)
result.should_receive(:success?).
and_return(true)
gateway = mock(:gateway)
Gateway.stub(:new => gateway)
gateway.should_receive(:process).twice.
and_return(result)
credit_card.should_receive(:gateway).
exactly(2).times.and_return(result)
Subscription.stub_chain(:new, :add_data,
:make_live, :check).and_return(true)
customer = Customer.new
customer.pay(credit_card)
end
it “fails when the gateway refuses” do
...
end
end
end
Customer
CreditCard
Gateway
Annual
Monthly
Subscription
Slide 57
Slide 57 text
The Solution
describe Customer do
describe “payment” do
before do
PaymentGateway.stub(
:take_payment => true)
SubscriptionManager.stub(
:create_for => true)
it “creates the subscription” do
SubscriptionManager.should_receive(
:create_for(customer).and_return(true)
customer = Customer.new
customer.pay(credit_card)
end
it “takes payment” do
PaymentGateway.should_receive(
:take_payment).with(:credit_card).
and_return(true)
customer = Customer.new
customer.pay(credit_card)
end
end
end
Customer
PaymentService
SubscriptionManager
Slide 58
Slide 58 text
The Solution
describe Customer do
describe “payment” do
before do
PaymentGateway.stub(
:take_payment => true)
SubscriptionManager.stub(
:create_for => true)
it “creates the subscription” do
SubscriptionManager.should_receive(
:create_for(customer).and_return(true)
customer = Customer.new
customer.pay(credit_card)
end
it “takes payment” do
PaymentGateway.should_receive(
:take_payment).with(:credit_card).
and_return(true)
customer = Customer.new
customer.pay(credit_card)
end
end
end
Customer
PaymentService
SubscriptionManager
SOLVED
Slide 59
Slide 59 text
No content
Slide 60
Slide 60 text
- THE END -
Slide 61
Slide 61 text
About the author
Name: Chris Parsons
Age: 33
Occupation: programmer/coach
Location: near London, UK
Notes:
- writes at http://pa.rsons.org
- twitter is @chrismdp
- available to hire
- THE END -