iOS BDD Beatdown

Figuring out how to do BDD for iOS isn't exactly clear cut. There are a number of contenders for framework of choice for both the integration and unit layers. Even when you've figured out which framework, it's not always easy to figure out what parts of your application to test and why. You're in luck! I've spent the last 1.5 years doing all the hard work for you! I'll offer a comparison of the popular BDD frameworks so you can figure out what will work best for you. I'll also be going over how to effectively BDD iOS applications as well as looking at how to retrofit your application with tests to cover the application's high value paths.

Transcript

1. Stephen Caudill
   I’ll be your announcer for today’s
   

   View Slide

2. iOS BDD Beatdown
   Stephen Caudill
   Ask the audience what their level of experience with BDD is
   Show of hands: if you know what BDD is. If you practice BDD. If you do it on iOS.
   BDD is an evolution on Test Driven Development that provides a specific shared vocabulary
   that bridges the divide between Business and Technology
   

   View Slide

3. 2 Years of BDD on iOS
   Stephen Caudill
   I’m here to help you leverage my experience in your selection of a BDD framework
   

   View Slide

4. Evaluation Criteria
   Stephen Caudill
   

   View Slide

5. Ubiquitous Language
   Stephen Caudill
   This is our shared vocabulary and the thing that separates BDD from TDD
   

   View Slide

6. • Run our specs atomically
   • Pend specs
   • Set expectations
   • Mocks/Doubles
   • Run our suite on devices
   • With a minimal amount of pain
   Stephen Caudill
   as developers we also need a suite of other tools
   

   View Slide

7. TDD with BDD Language?
   Stephen Caudill
   Seems like an acceptable shim, let’s look at some of the offerings
   

   View Slide

8. OCUnit
   Stephen Caudill
   http://www.sente.ch/software/ocunit
   

   View Slide

9. Ease of Installation
   Stephen Caudill
   A+
   

   View Slide

10. Ubiquitous Language
    Stephen Caudill
    Not so much...
    Note about “Ubiquitous Language” at the unit level...
    Terse external assertions lead to thinking at a low level... and I don’t know about you, but I
    constantly forget which order the expected and actual results should go in.
    

    View Slide

11. Stephen Caudill
    

    View Slide

12. Focused Test Runs
    Stephen Caudill
    No
    The closest you’ll come is having a duplicate OCUnit target and adding only the class you
    want to run to that
    

    View Slide

13. Expectations
    Stephen Caudill
    Not so much
    What is an expectation?
    

    View Slide

14. Mocks/Doubles
    Stephen Caudill
    BYOF
    OCMock is a popular solution for this... also provides expectations.
    

    View Slide

15. Pending Tests
    Stephen Caudill
    Nope
    Common workaround is to use #warning imperatives
    

    View Slide

16. Run tests on Device?
    Stephen Caudill
    No
    This is disappointing, given the “official” nature of the framework
    

    View Slide

17. Documentation
    Stephen Caudill
    NO OFFICIAL DOCS!
    The closest you’ll get is the “sentesting macro reference” from apple. Googling specific cases
    is easy due to the sheer amount of people using it.
    

    View Slide

18. Verdict
    Stephen Caudill
    Serviceable
    But far from ideal
    

    View Slide

19. GHUnit
    Stephen Caudill
    http://github.com/gabriel/gh-unit
    

    View Slide

20. Ease of Installation
    Stephen Caudill
    C-
    Readme says to download the framework (presumably prebuilt)... It doesn’t say where from
    though and googling didn’t help. You can make it yourself by cloning the repo, running make
    from the Project-iOS directory, then running a script in the Scripts directory... there is no
    documentation around this.
    

    View Slide

21. Ubiquitous Language
    Stephen Caudill
    TDD, so no.
    

    View Slide

22. Focused Runs
    Stephen Caudill
    Yes!
    

    View Slide

23. And it looks like this
    

    View Slide

24. Expectations
    Stephen Caudill
    Same issue here as with OCUnit...
    

    View Slide

25. Mocks/Doubles
    Stephen Caudill
    Bring Your Own
    Again, OCMock is the goto solution here.
    

    View Slide

26. Pending Tests
    Stephen Caudill
    Nope
    #warning imperatives
    

    View Slide

27. Run tests on Device?
    Stephen Caudill
    Yes!
    

    View Slide

28. Documentation
    Stephen Caudill
    Hit or miss...
    Often outdated or incorrect, with respect to installation
    

    View Slide

29. Verdict
    Stephen Caudill
    Better than OCUnit
    Transitioning to this from OCUnit is a breeze! And lo, there was a pretty test runner.
    

    View Slide

30. OCHamcrest
    Stephen Caudill
    http://github.com/hamcrest/OCHamcrest
    

    View Slide

31. Ease of Installation
    Stephen Caudill
    A
    it's initially unclear what target to install to… this augments SenTestingKit though, so just add it to a new or
    existing SenTestingKit target
    

    View Slide

32. Ubiquitous Language
    Stephen Caudill
    Getting Closer
    

    View Slide

33. Stephen Caudill
    We’re starting to disambiguate between our expectation and our actual result, at least...
    

    View Slide

34. Focused Runs
    Stephen Caudill
    Sadly, no.
    

    View Slide

35. Expectations
    Stephen Caudill
    No, but...
    http://github.com/jonreid/OCMockito
    Jon Reid (author of OCHamcrest) also wrote OCMockito, which handles expectations.
    

    View Slide

36. Mocks/Doubles
    Stephen Caudill
    Bring Your Own
    OCMockito is our huckleberry here as well.
    OCMock is another option here.
    

    View Slide

37. Pending Tests
    Stephen Caudill
    Uh-uh.
    our old friend the #warning imperative
    

    View Slide

38. Run tests on Device?
    Stephen Caudill
    Yes, indeed.
    

    View Slide

39. Documentation
    Stephen Caudill
    Good!
    Readme calls out the important stuff, there are lots of tutorials and the API has full
    documentation.
    

    View Slide

40. Verdict
    Stephen Caudill
    Best of the TDD lot
    

    View Slide

41. Let’s talk BDD!
    Stephen Caudill
    

    View Slide

42. Specta
    Stephen Caudill
    http://github.com/petejkim/specta
    

    View Slide

43. Ease of Installation
    Stephen Caudill
    A
    Cocoapods is the recommended manner of installation and it makes it a breeze!
    

    View Slide

44. Cocoapods
    http://cocoapods.org/
    Stephen Caudill
    quick aside...
    

    View Slide

45. Ubiquitous Language
    Stephen Caudill
    And how!
    

    View Slide

46. Stephen Caudill
    

    View Slide

47. Focused Runs
    Stephen Caudill
    Negative
    

    View Slide

48. Expectations
    Stephen Caudill
    Bring Your Own
    http://github.com/petejkim/expecta
    Sister project called Expecta, from the same developer.
    Worth noting that Specta isn’t really useful without Expecta.
    

    View Slide

49. Stephen Caudill
    

    View Slide

50. Mocks/Doubles
    Stephen Caudill
    No
    

    View Slide

51. Pending Tests
    Stephen Caudill
    Yay!
    

    View Slide

52. Run tests on Device?
    Stephen Caudill
    :sadtrombone:
    

    View Slide

53. Nice Xcode Integration
    Stephen Caudill
    Success and Failure feedback is integrated nicely with Xcode.
    

    View Slide

54. Documentation
    Stephen Caudill
    Very little...
    It’s pretty much just the readme, but it identifies the most important aspects of the API.
    

    View Slide

55. Verdict
    Stephen Caudill
    Pretty good (with Expecta)
    

    View Slide

56. Kiwi
    Stephen Caudill
    http://github.com/allending/Kiwi
    

    View Slide

57. Ease of Installation
    Stephen Caudill
    B-
    - Using Kiwi 2, which is young
    - Install documentation is sparse
    - Cocoapods (pay attention to the output of pod install)
    

    View Slide

58. Ubiquitous Language
    Stephen Caudill
    Definitely.
    

    View Slide

59. Stephen Caudill
    - similar to Specta
    - expectations are idiomatic Obj-C
    

    View Slide

60. Focused Runs
    Stephen Caudill
    Still No
    

    View Slide

61. Expectations
    Stephen Caudill
    Yes!
    As we just saw...
    

    View Slide

62. Mocks/Doubles
    Stephen Caudill
    Yes, 1000 times! Yes!
    

    View Slide

63. Mocks
    Protocol Mocks
    Stubs
    Oh my!
    Stephen Caudill
    

    View Slide

64. Pending Tests
    Stephen Caudill
    Yay!
    

    View Slide

65. Run tests on Device?
    Stephen Caudill
    Yes!
    

    View Slide

66. Documentation
    Stephen Caudill
    As long as you don’t need to
    install it...
    http://kiwi-lib.info
    Kiwi 2 is new and documentation for it is pretty sparse
    Luckily most documentation for Kiwi 1 is still valid.
    

    View Slide

67. Verdict
    Stephen Caudill
    Ya das is gut!
    

    View Slide

68. Cedar
    Stephen Caudill
    http://github.com/pivotal/cedar
    

    View Slide

69. Ease of Installation
    Stephen Caudill
    A+
    Super easy! Between install script for the templates and the templates themselves, it's really painless.
    

    View Slide

70. Stephen Caudill
    

    View Slide

71. Ubiquitous Language
    Stephen Caudill
    Real BDD
    

    View Slide

72. Stephen Caudill
    C++ ?!?!
    - So as we can see syntax is standardizing...
    - BUT! C++!?!?
    - Just because you can do something...
    

    View Slide

73. Focused Runs
    Stephen Caudill
    http://github.com/cppforlife/CedarShortcuts
    For sure!
    let’s you do things like ctrl+alt+u to run the spec under the cursor
    

    View Slide

74. Expectations
    Stephen Caudill
    Yepper
    

    View Slide

75. Mocks/Doubles
    Stephen Caudill
    Yessiree Bob!
    

    View Slide

76. Spies
    Mocks
    Stubs
    Stephen Caudill
    

    View Slide

77. Pending Tests
    Stephen Caudill
    Yes!
    

    View Slide

78. Run tests on Device?
    Stephen Caudill
    Yes!
    

    View Slide

79. Documentation
    Stephen Caudill
    In the form of a README
    in it’s defense, it’s a very good readme
    

    View Slide

80. Verdict
    Stephen Caudill
    It does all of the things.
    

    View Slide

81. Acceptance Tests!
    Stephen Caudill
    

    View Slide

82. Calabash-iOS
    Stephen Caudill
    http://github.com/calabash
    Android developers take heed!
    

    View Slide

83. Ease of Installation
    Stephen Caudill
    A++ Would install again.
    Installation was very straightforward
    scripts work well and readme is accurate
    

    View Slide

84. Ubiquitous Language
    Stephen Caudill
    Yes. Sir.
    - Ubiquitous language in the context of acceptance tests
    - Acts as an interpretation layer for Cucumber
    

    View Slide

85. Stephen Caudill
    You may have seen this before...
    Feature implementation is in Ruby
    

    View Slide

86. Focused Runs
    Stephen Caudill
    By the power of cucumber
    

    View Slide

87. Pending Features
    Stephen Caudill
    Yessiree
    

    View Slide

88. Run on Device?
    Stephen Caudill
    Yes!
    $ NO_LAUNCH=1 cucumber
    

    View Slide

89. Documentation
    Stephen Caudill
    Very Good
    Extensive wiki
    

    View Slide

90. Verdict
    Stephen Caudill
    Very good.
    Autostarting the simulator is broken - passing `NO_LAUNCH=1 cucumber` will get around it, but you have to manually
    start/stop the simulator
    

    View Slide

91. Frank
    Stephen Caudill
    http://testingwithfrank.com
    

    View Slide

92. Ease of Installation
    Stephen Caudill
    A-
    If you have an OCUnit target, it’ll try to copy your octest socket (using Ruby) and fail as it’s an abnormal file type.
    Outside of that, the frank CLI and it’s subcommands make it easy to set things up
    

    View Slide

93. Ubiquitous Language
    Stephen Caudill
    Customer read/write? Check!
    We’re sitting on top of Cucumber again here, so yeah...
    

    View Slide

94. Stephen Caudill
    This probably looks familiar (again)
    Tangent about web steps and the steps frank ships with
    - they’re difficult to remove
    

    View Slide

95. Focused Runs
    Stephen Caudill
    Yes! Thank you cucumber.
    

    View Slide

96. Pending Features
    Stephen Caudill
    Yeah
    

    View Slide

97. Run on Device?
    Stephen Caudill
    NO! WHY, SCIENCE? WHY??
    

    View Slide

98. Documentation
    Stephen Caudill
    Mediocre
    writing custom steps is a PITA, Ruby API isn’t documented
    

    View Slide

99. Verdict
    Stephen Caudill
    Serviceable
    requiring the objective-c dev to know Ruby to implement steps is a big drawback
    

    View Slide

100. KIF
     Stephen Caudill
     http://github.com/square/KIF
     Keep It Functional
     

     View Slide

101. Ease of Installation
     Stephen Caudill
     D
     KIF is a PITA to install.
     Cocoapods helps, but only if you cobble together the incantations from github issues and selectively ignore the
     README
     

     View Slide

102. Stephen Caudill
     * only head works currently, due to changes in Xcode 4.5 (current version is 4.6)
     * ensure you specify the target for the pod
     

     View Slide

103. Ubiquitous Language
     Stephen Caudill
     Well...
     

     View Slide

104. Stephen Caudill
     Can your customer write that?
     

     View Slide

105. Stephen Caudill
     How do they feel about giant walls of text for output?
     

     View Slide

106. Focused Runs
     Stephen Caudill
     No
     

     View Slide

107. Pending Features
     Stephen Caudill
     No
     

     View Slide

108. Run on Device?
     Stephen Caudill
     Yes
     

     View Slide

109. Documentation
     Stephen Caudill
     Poor
     very fragmented
     

     View Slide

110. Verdict
     Stephen Caudill
     Meh.
     Hard to set up
     Hard to learn
     No use of ubiquitous language
     But hey, it’s in objective C
     May be unfair to judge as an “acceptance” framework
     

     View Slide

111. Recap
     Stephen Caudill
     

     View Slide

112. Installation
     Ubiquitous
     Language
     Focused
     Tests
     Pending
     Tests
     Expectations Mocking
     Runs on
     Device
     Docs
     OCUnit
     GHUnit
     OCHamcrest
     Kiwi
     Cedar
     Specta/Expecta
     KIF
     Calabash
     Frank
     A+    N/A   D-
     C    N/A   C
     A    N/A   B
     B       C
     A+       B-
     A       B-
     D     N/A  D
     A+     N/A  A
     A-     N/A  C
     

     View Slide

113. Installation
     Ubiquitous
     Language
     Focused
     Tests
     Pending
     Tests
     Expectations Mocking
     Runs on
     Device
     Docs
     OCUnit
     GHUnit
     OCHamcrest
     Kiwi
     Cedar
     Specta/Expecta
     KIF
     Calabash
     Frank
     A+    N/A   D-
     C    N/A   C
     A    N/A   B
     B       C
     A+       B-
     A       B-
     D     N/A  D
     A+     N/A  A
     A-     N/A  C
     Only Use the TDD frameworks if you *have* to
     

     View Slide

114. Installation
     Ubiquitous
     Language
     Focused
     Tests
     Pending
     Tests
     Expectations Mocking
     Runs on
     Device
     Docs
     OCUnit
     GHUnit
     OCHamcrest
     Kiwi
     Cedar
     Specta/Expecta
     KIF
     Calabash
     Frank
     A+    N/A   D-
     C    N/A   C
     A    N/A   B
     B       C
     A+       B-
     A       B-
     D     N/A  D
     A+     N/A  A
     A-     N/A  C
     

     View Slide

115. Installation
     Ubiquitous
     Language
     Focused
     Tests
     Pending
     Tests
     Expectations Mocking
     Runs on
     Device
     Docs
     OCUnit
     GHUnit
     OCHamcrest
     Kiwi
     Cedar
     Specta/Expecta
     KIF
     Calabash
     Frank
     A+    N/A   D-
     C    N/A   C
     A    N/A   B
     B       C
     A+       B-
     A       B-
     D     N/A  D
     A+     N/A  A
     A-     N/A  C
     Because C++ is groooooss.
     

     View Slide

116. Installation
     Ubiquitous
     Language
     Focused
     Tests
     Pending
     Tests
     Expectations Mocking
     Runs on
     Device
     Docs
     OCUnit
     GHUnit
     OCHamcrest
     Kiwi
     Cedar
     Specta/Expecta
     KIF
     Calabash
     Frank
     A+    N/A   D-
     C    N/A   C
     A    N/A   B
     B       C
     A+       B-
     A       B-
     D     N/A  D
     A+     N/A  A
     A-     N/A  C
     Only Use the TDD frameworks if you *have* to
     

     View Slide

117. Installation
     Ubiquitous
     Language
     Focused
     Tests
     Pending
     Tests
     Expectations Mocking
     Runs on
     Device
     Docs
     OCUnit
     GHUnit
     OCHamcrest
     Kiwi
     Cedar
     Specta/Expecta
     KIF
     Calabash
     Frank
     A+    N/A   D-
     C    N/A   C
     A    N/A   B
     B       C
     A+       B-
     A       B-
     D     N/A  D
     A+     N/A  A
     A-     N/A  C
     

     View Slide

118. Installation
     Ubiquitous
     Language
     Focused
     Tests
     Pending
     Tests
     Expectations Mocking
     Runs on
     Device
     Docs
     OCUnit
     GHUnit
     OCHamcrest
     Kiwi
     Cedar
     Specta/Expecta
     KIF
     Calabash
     Frank
     A+    N/A   D-
     C    N/A   C
     A    N/A   B
     B       C
     A+       B-
     A       B-
     D     N/A  D
     A+     N/A  A
     A-     N/A  C
     Automates with actual touch events
     

     View Slide

119. Honorable Mentions
     Stephen Caudill
     

     View Slide

120. UIAutomation
     Wrappers
     Stephen Caudill
     

     View Slide

121. Zucchini
     http://www.zucchiniframework.org/
     Stephen Caudill
     Cucumber-like stories
     Maps to Coffeescript definitions
     Looks promising, avoided because it’s javascript
     

     View Slide

122. Bwoken
     http://bendyworks.github.com/bwoken/
     Stephen Caudill
     Very nice tooling built in Ruby
     Implement in Coffeescript or Javascript
     

     View Slide

123. Other Honorable
     Mentions
     Stephen Caudill
     

     View Slide

124. UISpec/UISpecRunner
     http://code.google.com/p/uispec/
     Stephen Caudill
     Idiomatic Objective-C
     Code several other frameworks were originally based on (e.g. KIF)
     

     View Slide

125. Simon
     http://drekka.github.com/simon/
     Stephen Caudill
     Looks promising
     Feature files
     Native Objective-C mappings
     Just couldn’t get it to work
     

     View Slide

126. Go forth and Specify!
     Stephen Caudill
     

     View Slide

127. Stephen Caudill
     @voxdolo
     @voxdolo
     http://hashrocket.com
     So you know where to deliver your hatemail
     

     View Slide