Your tests are lying to you

Slides for talk given at #rubyconf 30th September 2011.

Transcript

1. YOUR TESTS ARE LYING TO YOU
2. RUBYCONF ALCOHOL
3. WHEN TESTS LIE ~ Joe Ruby ~
4. The Problem describe Customer do describe “payment” do it “processes the payment” do credit_card = mock_model(CreditCard) result = mock(CreditCardGatewayResult) result.should_receive(:success?). and_return(true) gateway = mock(:gateway) Gateway.stub(:new => gateway) gateway.should_receive(:process).twice. and_return(result) credit_card.should_receive(:gateway). exactly(2).times.and_return(result) Subscription.stub_chain(:new, :add_data, :make_live, :check).and_return(true) customer = Customer.new customer.pay(credit_card) end it “fails when the gateway refuses” do ... end end end
5. The Problem describe Customer do describe “payment” do it “processes the payment” do credit_card = mock_model(CreditCard) result = mock(CreditCardGatewayResult) result.should_receive(:success?). and_return(true) gateway = mock(:gateway) Gateway.stub(:new => gateway) gateway.should_receive(:process).twice. and_return(result) credit_card.should_receive(:gateway). exactly(2).times.and_return(result) Subscription.stub_chain(:new, :add_data, :make_live, :check).and_return(true) customer = Customer.new customer.pay(credit_card) end it “fails when the gateway refuses” do ... end end end The Verdict
6. The Problem describe Customer do describe “payment” do it “processes the payment” do credit_card = mock_model(CreditCard) result = mock(CreditCardGatewayResult) result.should_receive(:success?). and_return(true) gateway = mock(:gateway) Gateway.stub(:new => gateway) gateway.should_receive(:process).twice. and_return(result) credit_card.should_receive(:gateway). exactly(2).times.and_return(result) Subscription.stub_chain(:new, :add_data, :make_live, :check).and_return(true) customer = Customer.new customer.pay(credit_card) end it “fails when the gateway refuses” do ... end end end 1. Look at all this setup code. 2. What’s going on with that stub_chain? 3. This kind of thing is duplicated all over this file, and the whole codebase. 4. Mocks cause lots of trouble. I need to start this test from scratch. The Verdict
7. The Rewrite describe Customer do describe “payment” do it “processes the payment” do Gateway.stub(:process => true) card = CreditCard.new result = Customer.new.pay(card) result.should == SUCCESS end it “fails when the gateway refuses” do Gateway.stub(:process => false) ... end end end
8. The Rewrite describe Customer do describe “payment” do it “processes the payment” do Gateway.stub(:process => true) card = CreditCard.new result = Customer.new.pay(card) result.should == SUCCESS end it “fails when the gateway refuses” do Gateway.stub(:process => false) ... end end end SOLVED
9. The Rewrite describe Customer do describe “payment” do it “processes the payment” do Gateway.stub(:process => true) card = CreditCard.new result = Customer.new.pay(card) result.should == SUCCESS end it “fails when the gateway refuses” do Gateway.stub(:process => false) ... end end end 601 Bourbon St New Orleans LA 30492 29th Sep 2011 Dear Mr Ruby, Just a brief note to say thanks so much for sorting out my website. I’ll certainly be recommending you next time I hear of someone in a fix. Best wishes Jane SOLVED
10. The Callback Jane called. Can you call her back urgently? 1-516-403-2039. -- Tracey
11. The Callback Jane called. Can you call her back urgently? 1-516-403-2039. -- Tracey Customer CreditCard Gateway Subscription
12. The Callback Jane called. Can you call her back urgently? 1-516-403-2039. -- Tracey The Verdict Customer CreditCard Gateway Subscription
13. The Callback Jane called. Can you call her back urgently? 1-516-403-2039. -- Tracey 1. Mocks and stubs can be a pain to set up. 2. Newer code is simpler and cleaner, but relies on everything working together correctly, and doesn’t catch all edge cases. 3. I must write more edge cases! The Verdict Customer CreditCard Gateway Subscription
14. The Rewrite #2 describe Customer do describe “payment” do it “proceses the payment” do ... end it “fails when the gateway refuses” do ... end it “fails when the gateway times out” do ... end it “fails if the subscription isn’t created” do ... end ... end end
15. The Rewrite #2 describe Customer do describe “payment” do it “proceses the payment” do ... end it “fails when the gateway refuses” do ... end it “fails when the gateway times out” do ... end it “fails if the subscription isn’t created” do ... end ... end end SOLVED
16. The Callback #2 Jane called again. She really didn’t sound happy. You have her number. --Tracey
17. The Callback #2 Jane called again. She really didn’t sound happy. You have her number. --Tracey Customer CreditCard Gateway Annual Monthly Subscription
18. The Callback #2 Jane called again. She really didn’t sound happy. You have her number. --Tracey The Verdict Customer CreditCard Gateway Annual Monthly Subscription
19. The Callback #2 Jane called again. She really didn’t sound happy. You have her number. --Tracey 1. When testing a series of objects from just the outside point of view, you have millions of combinations. 2. You can’t keep up with the edge cases. 3. Tests are getting slower. 4. Therefore: ??? The Verdict Customer CreditCard Gateway Annual Monthly Subscription
20. The Autopsy The Autopsy
21. The Autopsy Pathologist: J.B. Rainsberger Testimony: http://infoq.com/ presentations/integration-tests-scam The Autopsy
22. The Autopsy Pathologist: J.B. Rainsberger Testimony: http://infoq.com/ presentations/integration-tests-scam The Autopsy In cases like these, someone is usually lying. The Witness
23. The Autopsy Pathologist: J.B. Rainsberger Testimony: http://infoq.com/ presentations/integration-tests-scam The Autopsy In cases like these, someone is usually lying. The Witness describe Customer do describe “payment” do it “proceses the payment” do Gateway.stub(:process => true) card = CreditCard.new result = Customer.new.pay(card) result.should == SUCCESS end it “fails when the gateway refuses” do Gateway.stub(:process => false) ... end end end
24. The Autopsy testimony invalid Pathologist: J.B. Rainsberger Testimony: http://infoq.com/ presentations/integration-tests-scam The Autopsy In cases like these, someone is usually lying. The Witness describe Customer do describe “payment” do it “proceses the payment” do Gateway.stub(:process => true) card = CreditCard.new result = Customer.new.pay(card) result.should == SUCCESS end it “fails when the gateway refuses” do Gateway.stub(:process => false) ... end end end
25. The Investigation describe Customer do it “takes payment” do c = Customer.new(details) c.pay.should == true end end Customer
26. The Investigation describe Customer do it “takes payment” do c = Customer.new(details) c.pay.should == true end end Customer Gateway
27. The Investigation describe Customer do it “takes payment” do c = Customer.new(details) c.pay.should == true end end Customer Gateway Subscription describe Subscription do it “can be created” do c = Customer.new(details) s = Subscription.new(c) s.add_data(data) s.make_live s.verify.should == true end end
28. The Dilemma describe Customer do describe “payment” do it “proceses the payment” do credit_card = mock_model(CreditCard) result = mock(CreditCardGatewayResult) result.should_receive(:success?). and_return(true) gateway = mock(:gateway) Gateway.stub(:new => gateway) gateway.should_receive(:process).twice. and_return(result) credit_card.should_receive(:gateway). exactly(2).times.and_return(result) Subscription.stub_chain(:new, :add_data, :make_live, :check).and_return(true) customer = Customer.new customer.pay(credit_card) end it “fails when the gateway refuses” do ... end end end
29. The Dilemma describe Customer do describe “payment” do it “proceses the payment” do credit_card = mock_model(CreditCard) result = mock(CreditCardGatewayResult) result.should_receive(:success?). and_return(true) gateway = mock(:gateway) Gateway.stub(:new => gateway) gateway.should_receive(:process).twice. and_return(result) credit_card.should_receive(:gateway). exactly(2).times.and_return(result) Subscription.stub_chain(:new, :add_data, :make_live, :check).and_return(true) customer = Customer.new customer.pay(credit_card) end it “fails when the gateway refuses” do ... end end end “Well designed code is easy to test.” -- Corey Haines
30. The Dilemma describe Customer do describe “payment” do it “proceses the payment” do credit_card = mock_model(CreditCard) result = mock(CreditCardGatewayResult) result.should_receive(:success?). and_return(true) gateway = mock(:gateway) Gateway.stub(:new => gateway) gateway.should_receive(:process).twice. and_return(result) credit_card.should_receive(:gateway). exactly(2).times.and_return(result) Subscription.stub_chain(:new, :add_data, :make_live, :check).and_return(true) customer = Customer.new customer.pay(credit_card) end it “fails when the gateway refuses” do ... end end end Customer CreditCard Gateway Annual Monthly Subscription “Well designed code is easy to test.” -- Corey Haines
31. The Dilemma describe Customer do describe “payment” do it “proceses the payment” do credit_card = mock_model(CreditCard) result = mock(CreditCardGatewayResult) result.should_receive(:success?). and_return(true) gateway = mock(:gateway) Gateway.stub(:new => gateway) gateway.should_receive(:process).twice. and_return(result) credit_card.should_receive(:gateway). exactly(2).times.and_return(result) Subscription.stub_chain(:new, :add_data, :make_live, :check).and_return(true) customer = Customer.new customer.pay(credit_card) end it “fails when the gateway refuses” do ... end end end Customer CreditCard Gateway Annual Monthly Subscription “Well designed code is easy to test.” -- Corey Haines
32. The Jail
33. The Jail Ruby on Rails
34. The Jail Customer MySQL Ruby on Rails
35. The Jail Customer MySQL CustomerBilling CustomerDetails CustomerSearch Ruby on Rails
36. The Jail Customer MySQL CustomerBilling CustomerDetails CustomerSearch Destroy All Software http://destroyallsoftware.com http://confreaks.net/videos/ 641-gogaruco2011-fast-rails-tests Ruby on Rails
37. The Gang Feud
38. The Gang Feud Mockists vs Classisists
39. The Gang Feud Mockists vs Classisists Mocks vs Stubs is not the same feud.
40. The Gang Feud Mockists vs Classisists Mocks vs Stubs is not the same feud. Looks like I can use either mocks or stubs when it suits me.
41. The Gang Feud Mockists vs Classisists Mocks vs Stubs is not the same feud. Looks like I can use either mocks or stubs when it suits me. http://martinfowler.com/articles/ mocksArentStubs.html
42. The Gang Feud Mockists vs Classisists Mocks vs Stubs is not the same feud. Looks like I can use either mocks or stubs when it suits me. http://martinfowler.com/articles/ mocksArentStubs.html Pros and Cons
43. The Gang Feud Mockists vs Classisists Mocks vs Stubs is not the same feud. Looks like I can use either mocks or stubs when it suits me. http://martinfowler.com/articles/ mocksArentStubs.html Pros and Cons 1. Isolation
44. The Gang Feud Mockists vs Classisists Mocks vs Stubs is not the same feud. Looks like I can use either mocks or stubs when it suits me. http://martinfowler.com/articles/ mocksArentStubs.html Pros and Cons 1. Isolation 2. Flexibility
45. The Gang Feud Mockists vs Classisists Mocks vs Stubs is not the same feud. Looks like I can use either mocks or stubs when it suits me. http://martinfowler.com/articles/ mocksArentStubs.html Pros and Cons 1. Isolation 2. Flexibility 3. Setup
46. The Gang Feud Mockists vs Classisists Mocks vs Stubs is not the same feud. Looks like I can use either mocks or stubs when it suits me. http://martinfowler.com/articles/ mocksArentStubs.html Pros and Cons 1. Isolation 2. Flexibility 3. Setup 4. Design
47. The Gang Feud Mockists vs Classisists Mocks vs Stubs is not the same feud. Looks like I can use either mocks or stubs when it suits me. http://martinfowler.com/articles/ mocksArentStubs.html Pros and Cons 1. Isolation 2. Flexibility 3. Setup 4. Design 5. Speed
48. Integration Tests
49. Integration Tests You still need some integration tests to ensure the interactions are valid.
50. Integration Tests You still need some integration tests to ensure the interactions are valid. Acceptance Tests
51. Integration Tests You still need some integration tests to ensure the interactions are valid. Acceptance Tests The customer still needs to know if the software does what they asked for.
52. The Key Questions
53. The Key Questions 1. Am I being lied to?
54. The Key Questions 1. Am I being lied to? 2. Is my test code telling me something?
55. The Key Questions 1. Am I being lied to? 2. Is my test code telling me something? 3. Which gang do I belong to? Do I understand the pros and cons?
56. The Solution describe Customer do describe “payment” do it “proceses the payment” do credit_card = mock_model(CreditCard) result = mock(CreditCardGatewayResult) result.should_receive(:success?). and_return(true) gateway = mock(:gateway) Gateway.stub(:new => gateway) gateway.should_receive(:process).twice. and_return(result) credit_card.should_receive(:gateway). exactly(2).times.and_return(result) Subscription.stub_chain(:new, :add_data, :make_live, :check).and_return(true) customer = Customer.new customer.pay(credit_card) end it “fails when the gateway refuses” do ... end end end Customer CreditCard Gateway Annual Monthly Subscription
57. The Solution describe Customer do describe “payment” do before do PaymentGateway.stub( :take_payment => true) SubscriptionManager.stub( :create_for => true) it “creates the subscription” do SubscriptionManager.should_receive( :create_for(customer).and_return(true) customer = Customer.new customer.pay(credit_card) end it “takes payment” do PaymentGateway.should_receive( :take_payment).with(:credit_card). and_return(true) customer = Customer.new customer.pay(credit_card) end end end Customer PaymentService SubscriptionManager
58. The Solution describe Customer do describe “payment” do before do PaymentGateway.stub( :take_payment => true) SubscriptionManager.stub( :create_for => true) it “creates the subscription” do SubscriptionManager.should_receive( :create_for(customer).and_return(true) customer = Customer.new customer.pay(credit_card) end it “takes payment” do PaymentGateway.should_receive( :take_payment).with(:credit_card). and_return(true) customer = Customer.new customer.pay(credit_card) end end end Customer PaymentService SubscriptionManager SOLVED
60. - THE END -
61. About the author Name: Chris Parsons Age: 33 Occupation: programmer/coach Location: near London, UK Notes: - writes at http://pa.rsons.org - twitter is @chrismdp - available to hire - THE END -