iOS BDD Beatdown

Figuring out how to do BDD for iOS isn't exactly clear cut. There are a number of contenders for framework of choice for both the integration and unit layers. Even when you've figured out which framework, it's not always easy to figure out what parts of your application to test and why. You're in luck! I've spent the last 1.5 years doing all the hard work for you! I'll offer a comparison of the popular BDD frameworks so you can figure out what will work best for you. I'll also be going over how to effectively BDD iOS applications as well as looking at how to retrofit your application with tests to cover the application's high value paths.

Transcript

1. Stephen Caudill I’ll be your announcer for today’s
2. iOS BDD Beatdown Stephen Caudill Ask the audience what their level of experience with BDD is Show of hands: if you know what BDD is. If you practice BDD. If you do it on iOS. BDD is an evolution on Test Driven Development that provides a specific shared vocabulary that bridges the divide between Business and Technology
3. 2 Years of BDD on iOS Stephen Caudill I’m here to help you leverage my experience in your selection of a BDD framework
4. Evaluation Criteria Stephen Caudill
5. Ubiquitous Language Stephen Caudill This is our shared vocabulary and the thing that separates BDD from TDD
6. • Run our specs atomically • Pend specs • Set expectations • Mocks/Doubles • Run our suite on devices • With a minimal amount of pain Stephen Caudill as developers we also need a suite of other tools
7. TDD with BDD Language? Stephen Caudill Seems like an acceptable shim, let’s look at some of the offerings
8. OCUnit Stephen Caudill http://www.sente.ch/software/ocunit
9. Ease of Installation Stephen Caudill A+
10. Ubiquitous Language Stephen Caudill Not so much... Note about “Ubiquitous Language” at the unit level... Terse external assertions lead to thinking at a low level... and I don’t know about you, but I constantly forget which order the expected and actual results should go in.
11. Stephen Caudill
12. Focused Test Runs Stephen Caudill No The closest you’ll come is having a duplicate OCUnit target and adding only the class you want to run to that
13. Expectations Stephen Caudill Not so much What is an expectation?
14. Mocks/Doubles Stephen Caudill BYOF OCMock is a popular solution for this... also provides expectations.
15. Pending Tests Stephen Caudill Nope Common workaround is to use #warning imperatives
16. Run tests on Device? Stephen Caudill No This is disappointing, given the “official” nature of the framework
17. Documentation Stephen Caudill NO OFFICIAL DOCS! The closest you’ll get is the “sentesting macro reference” from apple. Googling specific cases is easy due to the sheer amount of people using it.
18. Verdict Stephen Caudill Serviceable But far from ideal
19. GHUnit Stephen Caudill http://github.com/gabriel/gh-unit
20. Ease of Installation Stephen Caudill C- Readme says to download the framework (presumably prebuilt)... It doesn’t say where from though and googling didn’t help. You can make it yourself by cloning the repo, running make from the Project-iOS directory, then running a script in the Scripts directory... there is no documentation around this.
21. Ubiquitous Language Stephen Caudill TDD, so no.
22. Focused Runs Stephen Caudill Yes!
23. And it looks like this
24. Expectations Stephen Caudill Same issue here as with OCUnit...
25. Mocks/Doubles Stephen Caudill Bring Your Own Again, OCMock is the goto solution here.
26. Pending Tests Stephen Caudill Nope #warning imperatives
27. Run tests on Device? Stephen Caudill Yes!
28. Documentation Stephen Caudill Hit or miss... Often outdated or incorrect, with respect to installation
29. Verdict Stephen Caudill Better than OCUnit Transitioning to this from OCUnit is a breeze! And lo, there was a pretty test runner.
30. OCHamcrest Stephen Caudill http://github.com/hamcrest/OCHamcrest
31. Ease of Installation Stephen Caudill A it's initially unclear what target to install to… this augments SenTestingKit though, so just add it to a new or existing SenTestingKit target
32. Ubiquitous Language Stephen Caudill Getting Closer
33. Stephen Caudill We’re starting to disambiguate between our expectation and our actual result, at least...
34. Focused Runs Stephen Caudill Sadly, no.
35. Expectations Stephen Caudill No, but... http://github.com/jonreid/OCMockito Jon Reid (author of OCHamcrest) also wrote OCMockito, which handles expectations.
36. Mocks/Doubles Stephen Caudill Bring Your Own OCMockito is our huckleberry here as well. OCMock is another option here.
37. Pending Tests Stephen Caudill Uh-uh. our old friend the #warning imperative
38. Run tests on Device? Stephen Caudill Yes, indeed.
39. Documentation Stephen Caudill Good! Readme calls out the important stuff, there are lots of tutorials and the API has full documentation.
40. Verdict Stephen Caudill Best of the TDD lot
41. Let’s talk BDD! Stephen Caudill
42. Specta Stephen Caudill http://github.com/petejkim/specta
43. Ease of Installation Stephen Caudill A Cocoapods is the recommended manner of installation and it makes it a breeze!
44. Cocoapods http://cocoapods.org/ Stephen Caudill quick aside...
45. Ubiquitous Language Stephen Caudill And how!
46. Stephen Caudill
47. Focused Runs Stephen Caudill Negative
48. Expectations Stephen Caudill Bring Your Own http://github.com/petejkim/expecta Sister project called Expecta, from the same developer. Worth noting that Specta isn’t really useful without Expecta.
49. Stephen Caudill
50. Mocks/Doubles Stephen Caudill No
51. Pending Tests Stephen Caudill Yay!
52. Run tests on Device? Stephen Caudill :sadtrombone:
53. Nice Xcode Integration Stephen Caudill Success and Failure feedback is integrated nicely with Xcode.
54. Documentation Stephen Caudill Very little... It’s pretty much just the readme, but it identifies the most important aspects of the API.
55. Verdict Stephen Caudill Pretty good (with Expecta)
56. Kiwi Stephen Caudill http://github.com/allending/Kiwi
57. Ease of Installation Stephen Caudill B- - Using Kiwi 2, which is young - Install documentation is sparse - Cocoapods (pay attention to the output of pod install)
58. Ubiquitous Language Stephen Caudill Definitely.
59. Stephen Caudill - similar to Specta - expectations are idiomatic Obj-C
60. Focused Runs Stephen Caudill Still No
61. Expectations Stephen Caudill Yes! As we just saw...
62. Mocks/Doubles Stephen Caudill Yes, 1000 times! Yes!
63. Mocks Protocol Mocks Stubs Oh my! Stephen Caudill
64. Pending Tests Stephen Caudill Yay!
65. Run tests on Device? Stephen Caudill Yes!
66. Documentation Stephen Caudill As long as you don’t need to install it... http://kiwi-lib.info Kiwi 2 is new and documentation for it is pretty sparse Luckily most documentation for Kiwi 1 is still valid.
67. Verdict Stephen Caudill Ya das is gut!
68. Cedar Stephen Caudill http://github.com/pivotal/cedar
69. Ease of Installation Stephen Caudill A+ Super easy! Between install script for the templates and the templates themselves, it's really painless.
70. Stephen Caudill
71. Ubiquitous Language Stephen Caudill Real BDD
72. Stephen Caudill C++ ?!?! - So as we can see syntax is standardizing... - BUT! C++!?!? - Just because you can do something...
73. Focused Runs Stephen Caudill http://github.com/cppforlife/CedarShortcuts For sure! let’s you do things like ctrl+alt+u to run the spec under the cursor
74. Expectations Stephen Caudill Yepper
75. Mocks/Doubles Stephen Caudill Yessiree Bob!
76. Spies Mocks Stubs Stephen Caudill
77. Pending Tests Stephen Caudill Yes!
78. Run tests on Device? Stephen Caudill Yes!
79. Documentation Stephen Caudill In the form of a README in it’s defense, it’s a very good readme
80. Verdict Stephen Caudill It does all of the things.
81. Acceptance Tests! Stephen Caudill
82. Calabash-iOS Stephen Caudill http://github.com/calabash Android developers take heed!
83. Ease of Installation Stephen Caudill A++ Would install again. Installation was very straightforward scripts work well and readme is accurate
84. Ubiquitous Language Stephen Caudill Yes. Sir. - Ubiquitous language in the context of acceptance tests - Acts as an interpretation layer for Cucumber
85. Stephen Caudill You may have seen this before... Feature implementation is in Ruby
86. Focused Runs Stephen Caudill By the power of cucumber
87. Pending Features Stephen Caudill Yessiree
88. Run on Device? Stephen Caudill Yes! $ NO_LAUNCH=1 cucumber
89. Documentation Stephen Caudill Very Good Extensive wiki
90. Verdict Stephen Caudill Very good. Autostarting the simulator is broken - passing `NO_LAUNCH=1 cucumber` will get around it, but you have to manually start/stop the simulator
91. Frank Stephen Caudill http://testingwithfrank.com
92. Ease of Installation Stephen Caudill A- If you have an OCUnit target, it’ll try to copy your octest socket (using Ruby) and fail as it’s an abnormal file type. Outside of that, the frank CLI and it’s subcommands make it easy to set things up
93. Ubiquitous Language Stephen Caudill Customer read/write? Check! We’re sitting on top of Cucumber again here, so yeah...
94. Stephen Caudill This probably looks familiar (again) Tangent about web steps and the steps frank ships with - they’re difficult to remove
95. Focused Runs Stephen Caudill Yes! Thank you cucumber.
96. Pending Features Stephen Caudill Yeah
97. Run on Device? Stephen Caudill NO! WHY, SCIENCE? WHY??
98. Documentation Stephen Caudill Mediocre writing custom steps is a PITA, Ruby API isn’t documented
99. Verdict Stephen Caudill Serviceable requiring the objective-c dev to know Ruby to implement steps is a big drawback
100. KIF Stephen Caudill http://github.com/square/KIF Keep It Functional
101. Ease of Installation Stephen Caudill D KIF is a PITA to install. Cocoapods helps, but only if you cobble together the incantations from github issues and selectively ignore the README
102. Stephen Caudill * only head works currently, due to changes in Xcode 4.5 (current version is 4.6) * ensure you specify the target for the pod
103. Ubiquitous Language Stephen Caudill Well...
104. Stephen Caudill Can your customer write that?
105. Stephen Caudill How do they feel about giant walls of text for output?
106. Focused Runs Stephen Caudill No
107. Pending Features Stephen Caudill No
108. Run on Device? Stephen Caudill Yes
109. Documentation Stephen Caudill Poor very fragmented
110. Verdict Stephen Caudill Meh. Hard to set up Hard to learn No use of ubiquitous language But hey, it’s in objective C May be unfair to judge as an “acceptance” framework
111. Recap Stephen Caudill
112. Installation Ubiquitous Language Focused Tests Pending Tests Expectations Mocking Runs on Device Docs OCUnit GHUnit OCHamcrest Kiwi Cedar Specta/Expecta KIF Calabash Frank A+    N/A   D- C    N/A   C A    N/A   B B       C A+       B- A       B- D     N/A  D A+     N/A  A A-     N/A  C
113. Installation Ubiquitous Language Focused Tests Pending Tests Expectations Mocking Runs on Device Docs OCUnit GHUnit OCHamcrest Kiwi Cedar Specta/Expecta KIF Calabash Frank A+    N/A   D- C    N/A   C A    N/A   B B       C A+       B- A       B- D     N/A  D A+     N/A  A A-     N/A  C Only Use the TDD frameworks if you *have* to
114. Installation Ubiquitous Language Focused Tests Pending Tests Expectations Mocking Runs on Device Docs OCUnit GHUnit OCHamcrest Kiwi Cedar Specta/Expecta KIF Calabash Frank A+    N/A   D- C    N/A   C A    N/A   B B       C A+       B- A       B- D     N/A  D A+     N/A  A A-     N/A  C
115. Installation Ubiquitous Language Focused Tests Pending Tests Expectations Mocking Runs on Device Docs OCUnit GHUnit OCHamcrest Kiwi Cedar Specta/Expecta KIF Calabash Frank A+    N/A   D- C    N/A   C A    N/A   B B       C A+       B- A       B- D     N/A  D A+     N/A  A A-     N/A  C Because C++ is groooooss.
116. Installation Ubiquitous Language Focused Tests Pending Tests Expectations Mocking Runs on Device Docs OCUnit GHUnit OCHamcrest Kiwi Cedar Specta/Expecta KIF Calabash Frank A+    N/A   D- C    N/A   C A    N/A   B B       C A+       B- A       B- D     N/A  D A+     N/A  A A-     N/A  C Only Use the TDD frameworks if you *have* to
117. Installation Ubiquitous Language Focused Tests Pending Tests Expectations Mocking Runs on Device Docs OCUnit GHUnit OCHamcrest Kiwi Cedar Specta/Expecta KIF Calabash Frank A+    N/A   D- C    N/A   C A    N/A   B B       C A+       B- A       B- D     N/A  D A+     N/A  A A-     N/A  C
118. Installation Ubiquitous Language Focused Tests Pending Tests Expectations Mocking Runs on Device Docs OCUnit GHUnit OCHamcrest Kiwi Cedar Specta/Expecta KIF Calabash Frank A+    N/A   D- C    N/A   C A    N/A   B B       C A+       B- A       B- D     N/A  D A+     N/A  A A-     N/A  C Automates with actual touch events
119. Honorable Mentions Stephen Caudill
120. UIAutomation Wrappers Stephen Caudill
121. Zucchini http://www.zucchiniframework.org/ Stephen Caudill Cucumber-like stories Maps to Coffeescript definitions Looks promising, avoided because it’s javascript
122. Bwoken http://bendyworks.github.com/bwoken/ Stephen Caudill Very nice tooling built in Ruby Implement in Coffeescript or Javascript
123. Other Honorable Mentions Stephen Caudill
124. UISpec/UISpecRunner http://code.google.com/p/uispec/ Stephen Caudill Idiomatic Objective-C Code several other frameworks were originally based on (e.g. KIF)
125. Simon http://drekka.github.com/simon/ Stephen Caudill Looks promising Feature files Native Objective-C mappings Just couldn’t get it to work
126. Go forth and Specify! Stephen Caudill
127. Stephen Caudill @voxdolo @voxdolo http://hashrocket.com So you know where to deliver your hatemail