$30 off During Our Annual Pro Sale. View Details »

iOS BDD Beatdown

iOS BDD Beatdown

Figuring out how to do BDD for iOS isn't exactly clear cut. There are a number of contenders for framework of choice for both the integration and unit layers. Even when you've figured out which framework, it's not always easy to figure out what parts of your application to test and why. You're in luck! I've spent the last 1.5 years doing all the hard work for you! I'll offer a comparison of the popular BDD frameworks so you can figure out what will work best for you. I'll also be going over how to effectively BDD iOS applications as well as looking at how to retrofit your application with tests to cover the application's high value paths.

Stephen Caudill

February 16, 2013
Tweet

More Decks by Stephen Caudill

Other Decks in Technology

Transcript

  1. Stephen Caudill
    I’ll be your announcer for today’s

    View Slide

  2. iOS BDD Beatdown
    Stephen Caudill
    Ask the audience what their level of experience with BDD is
    Show of hands: if you know what BDD is. If you practice BDD. If you do it on iOS.
    BDD is an evolution on Test Driven Development that provides a specific shared vocabulary
    that bridges the divide between Business and Technology

    View Slide

  3. 2 Years of BDD on iOS
    Stephen Caudill
    I’m here to help you leverage my experience in your selection of a BDD framework

    View Slide

  4. Evaluation Criteria
    Stephen Caudill

    View Slide

  5. Ubiquitous Language
    Stephen Caudill
    This is our shared vocabulary and the thing that separates BDD from TDD

    View Slide

  6. • Run our specs atomically
    • Pend specs
    • Set expectations
    • Mocks/Doubles
    • Run our suite on devices
    • With a minimal amount of pain
    Stephen Caudill
    as developers we also need a suite of other tools

    View Slide

  7. TDD with BDD Language?
    Stephen Caudill
    Seems like an acceptable shim, let’s look at some of the offerings

    View Slide

  8. OCUnit
    Stephen Caudill
    http://www.sente.ch/software/ocunit

    View Slide

  9. Ease of Installation
    Stephen Caudill
    A+

    View Slide

  10. Ubiquitous Language
    Stephen Caudill
    Not so much...
    Note about “Ubiquitous Language” at the unit level...
    Terse external assertions lead to thinking at a low level... and I don’t know about you, but I
    constantly forget which order the expected and actual results should go in.

    View Slide

  11. Stephen Caudill

    View Slide

  12. Focused Test Runs
    Stephen Caudill
    No
    The closest you’ll come is having a duplicate OCUnit target and adding only the class you
    want to run to that

    View Slide

  13. Expectations
    Stephen Caudill
    Not so much
    What is an expectation?

    View Slide

  14. Mocks/Doubles
    Stephen Caudill
    BYOF
    OCMock is a popular solution for this... also provides expectations.

    View Slide

  15. Pending Tests
    Stephen Caudill
    Nope
    Common workaround is to use #warning imperatives

    View Slide

  16. Run tests on Device?
    Stephen Caudill
    No
    This is disappointing, given the “official” nature of the framework

    View Slide

  17. Documentation
    Stephen Caudill
    NO OFFICIAL DOCS!
    The closest you’ll get is the “sentesting macro reference” from apple. Googling specific cases
    is easy due to the sheer amount of people using it.

    View Slide

  18. Verdict
    Stephen Caudill
    Serviceable
    But far from ideal

    View Slide

  19. GHUnit
    Stephen Caudill
    http://github.com/gabriel/gh-unit

    View Slide

  20. Ease of Installation
    Stephen Caudill
    C-
    Readme says to download the framework (presumably prebuilt)... It doesn’t say where from
    though and googling didn’t help. You can make it yourself by cloning the repo, running make
    from the Project-iOS directory, then running a script in the Scripts directory... there is no
    documentation around this.

    View Slide

  21. Ubiquitous Language
    Stephen Caudill
    TDD, so no.

    View Slide

  22. Focused Runs
    Stephen Caudill
    Yes!

    View Slide

  23. And it looks like this

    View Slide

  24. Expectations
    Stephen Caudill
    Same issue here as with OCUnit...

    View Slide

  25. Mocks/Doubles
    Stephen Caudill
    Bring Your Own
    Again, OCMock is the goto solution here.

    View Slide

  26. Pending Tests
    Stephen Caudill
    Nope
    #warning imperatives

    View Slide

  27. Run tests on Device?
    Stephen Caudill
    Yes!

    View Slide

  28. Documentation
    Stephen Caudill
    Hit or miss...
    Often outdated or incorrect, with respect to installation

    View Slide

  29. Verdict
    Stephen Caudill
    Better than OCUnit
    Transitioning to this from OCUnit is a breeze! And lo, there was a pretty test runner.

    View Slide

  30. OCHamcrest
    Stephen Caudill
    http://github.com/hamcrest/OCHamcrest

    View Slide

  31. Ease of Installation
    Stephen Caudill
    A
    it's initially unclear what target to install to… this augments SenTestingKit though, so just add it to a new or
    existing SenTestingKit target

    View Slide

  32. Ubiquitous Language
    Stephen Caudill
    Getting Closer

    View Slide

  33. Stephen Caudill
    We’re starting to disambiguate between our expectation and our actual result, at least...

    View Slide

  34. Focused Runs
    Stephen Caudill
    Sadly, no.

    View Slide

  35. Expectations
    Stephen Caudill
    No, but...
    http://github.com/jonreid/OCMockito
    Jon Reid (author of OCHamcrest) also wrote OCMockito, which handles expectations.

    View Slide

  36. Mocks/Doubles
    Stephen Caudill
    Bring Your Own
    OCMockito is our huckleberry here as well.
    OCMock is another option here.

    View Slide

  37. Pending Tests
    Stephen Caudill
    Uh-uh.
    our old friend the #warning imperative

    View Slide

  38. Run tests on Device?
    Stephen Caudill
    Yes, indeed.

    View Slide

  39. Documentation
    Stephen Caudill
    Good!
    Readme calls out the important stuff, there are lots of tutorials and the API has full
    documentation.

    View Slide

  40. Verdict
    Stephen Caudill
    Best of the TDD lot

    View Slide

  41. Let’s talk BDD!
    Stephen Caudill

    View Slide

  42. Specta
    Stephen Caudill
    http://github.com/petejkim/specta

    View Slide

  43. Ease of Installation
    Stephen Caudill
    A
    Cocoapods is the recommended manner of installation and it makes it a breeze!

    View Slide

  44. Cocoapods
    http://cocoapods.org/
    Stephen Caudill
    quick aside...

    View Slide

  45. Ubiquitous Language
    Stephen Caudill
    And how!

    View Slide

  46. Stephen Caudill

    View Slide

  47. Focused Runs
    Stephen Caudill
    Negative

    View Slide

  48. Expectations
    Stephen Caudill
    Bring Your Own
    http://github.com/petejkim/expecta
    Sister project called Expecta, from the same developer.
    Worth noting that Specta isn’t really useful without Expecta.

    View Slide

  49. Stephen Caudill

    View Slide

  50. Mocks/Doubles
    Stephen Caudill
    No

    View Slide

  51. Pending Tests
    Stephen Caudill
    Yay!

    View Slide

  52. Run tests on Device?
    Stephen Caudill
    :sadtrombone:

    View Slide

  53. Nice Xcode Integration
    Stephen Caudill
    Success and Failure feedback is integrated nicely with Xcode.

    View Slide

  54. Documentation
    Stephen Caudill
    Very little...
    It’s pretty much just the readme, but it identifies the most important aspects of the API.

    View Slide

  55. Verdict
    Stephen Caudill
    Pretty good (with Expecta)

    View Slide

  56. Kiwi
    Stephen Caudill
    http://github.com/allending/Kiwi

    View Slide

  57. Ease of Installation
    Stephen Caudill
    B-
    - Using Kiwi 2, which is young
    - Install documentation is sparse
    - Cocoapods (pay attention to the output of pod install)

    View Slide

  58. Ubiquitous Language
    Stephen Caudill
    Definitely.

    View Slide

  59. Stephen Caudill
    - similar to Specta
    - expectations are idiomatic Obj-C

    View Slide

  60. Focused Runs
    Stephen Caudill
    Still No

    View Slide

  61. Expectations
    Stephen Caudill
    Yes!
    As we just saw...

    View Slide

  62. Mocks/Doubles
    Stephen Caudill
    Yes, 1000 times! Yes!

    View Slide

  63. Mocks
    Protocol Mocks
    Stubs
    Oh my!
    Stephen Caudill

    View Slide

  64. Pending Tests
    Stephen Caudill
    Yay!

    View Slide

  65. Run tests on Device?
    Stephen Caudill
    Yes!

    View Slide

  66. Documentation
    Stephen Caudill
    As long as you don’t need to
    install it...
    http://kiwi-lib.info
    Kiwi 2 is new and documentation for it is pretty sparse
    Luckily most documentation for Kiwi 1 is still valid.

    View Slide

  67. Verdict
    Stephen Caudill
    Ya das is gut!

    View Slide

  68. Cedar
    Stephen Caudill
    http://github.com/pivotal/cedar

    View Slide

  69. Ease of Installation
    Stephen Caudill
    A+
    Super easy! Between install script for the templates and the templates themselves, it's really painless.

    View Slide

  70. Stephen Caudill

    View Slide

  71. Ubiquitous Language
    Stephen Caudill
    Real BDD

    View Slide

  72. Stephen Caudill
    C++ ?!?!
    - So as we can see syntax is standardizing...
    - BUT! C++!?!?
    - Just because you can do something...

    View Slide

  73. Focused Runs
    Stephen Caudill
    http://github.com/cppforlife/CedarShortcuts
    For sure!
    let’s you do things like ctrl+alt+u to run the spec under the cursor

    View Slide

  74. Expectations
    Stephen Caudill
    Yepper

    View Slide

  75. Mocks/Doubles
    Stephen Caudill
    Yessiree Bob!

    View Slide

  76. Spies
    Mocks
    Stubs
    Stephen Caudill

    View Slide

  77. Pending Tests
    Stephen Caudill
    Yes!

    View Slide

  78. Run tests on Device?
    Stephen Caudill
    Yes!

    View Slide

  79. Documentation
    Stephen Caudill
    In the form of a README
    in it’s defense, it’s a very good readme

    View Slide

  80. Verdict
    Stephen Caudill
    It does all of the things.

    View Slide

  81. Acceptance Tests!
    Stephen Caudill

    View Slide

  82. Calabash-iOS
    Stephen Caudill
    http://github.com/calabash
    Android developers take heed!

    View Slide

  83. Ease of Installation
    Stephen Caudill
    A++ Would install again.
    Installation was very straightforward
    scripts work well and readme is accurate

    View Slide

  84. Ubiquitous Language
    Stephen Caudill
    Yes. Sir.
    - Ubiquitous language in the context of acceptance tests
    - Acts as an interpretation layer for Cucumber

    View Slide

  85. Stephen Caudill
    You may have seen this before...
    Feature implementation is in Ruby

    View Slide

  86. Focused Runs
    Stephen Caudill
    By the power of cucumber

    View Slide

  87. Pending Features
    Stephen Caudill
    Yessiree

    View Slide

  88. Run on Device?
    Stephen Caudill
    Yes!
    $ NO_LAUNCH=1 cucumber

    View Slide

  89. Documentation
    Stephen Caudill
    Very Good
    Extensive wiki

    View Slide

  90. Verdict
    Stephen Caudill
    Very good.
    Autostarting the simulator is broken - passing `NO_LAUNCH=1 cucumber` will get around it, but you have to manually
    start/stop the simulator

    View Slide

  91. Frank
    Stephen Caudill
    http://testingwithfrank.com

    View Slide

  92. Ease of Installation
    Stephen Caudill
    A-
    If you have an OCUnit target, it’ll try to copy your octest socket (using Ruby) and fail as it’s an abnormal file type.
    Outside of that, the frank CLI and it’s subcommands make it easy to set things up

    View Slide

  93. Ubiquitous Language
    Stephen Caudill
    Customer read/write? Check!
    We’re sitting on top of Cucumber again here, so yeah...

    View Slide

  94. Stephen Caudill
    This probably looks familiar (again)
    Tangent about web steps and the steps frank ships with
    - they’re difficult to remove

    View Slide

  95. Focused Runs
    Stephen Caudill
    Yes! Thank you cucumber.

    View Slide

  96. Pending Features
    Stephen Caudill
    Yeah

    View Slide

  97. Run on Device?
    Stephen Caudill
    NO! WHY, SCIENCE? WHY??

    View Slide

  98. Documentation
    Stephen Caudill
    Mediocre
    writing custom steps is a PITA, Ruby API isn’t documented

    View Slide

  99. Verdict
    Stephen Caudill
    Serviceable
    requiring the objective-c dev to know Ruby to implement steps is a big drawback

    View Slide

  100. KIF
    Stephen Caudill
    http://github.com/square/KIF
    Keep It Functional

    View Slide

  101. Ease of Installation
    Stephen Caudill
    D
    KIF is a PITA to install.
    Cocoapods helps, but only if you cobble together the incantations from github issues and selectively ignore the
    README

    View Slide

  102. Stephen Caudill
    * only head works currently, due to changes in Xcode 4.5 (current version is 4.6)
    * ensure you specify the target for the pod

    View Slide

  103. Ubiquitous Language
    Stephen Caudill
    Well...

    View Slide

  104. Stephen Caudill
    Can your customer write that?

    View Slide

  105. Stephen Caudill
    How do they feel about giant walls of text for output?

    View Slide

  106. Focused Runs
    Stephen Caudill
    No

    View Slide

  107. Pending Features
    Stephen Caudill
    No

    View Slide

  108. Run on Device?
    Stephen Caudill
    Yes

    View Slide

  109. Documentation
    Stephen Caudill
    Poor
    very fragmented

    View Slide

  110. Verdict
    Stephen Caudill
    Meh.
    Hard to set up
    Hard to learn
    No use of ubiquitous language
    But hey, it’s in objective C
    May be unfair to judge as an “acceptance” framework

    View Slide

  111. Recap
    Stephen Caudill

    View Slide

  112. Installation
    Ubiquitous
    Language
    Focused
    Tests
    Pending
    Tests
    Expectations Mocking
    Runs on
    Device
    Docs
    OCUnit
    GHUnit
    OCHamcrest
    Kiwi
    Cedar
    Specta/Expecta
    KIF
    Calabash
    Frank
    A+    N/A   D-
    C    N/A   C
    A    N/A   B
    B       C
    A+       B-
    A       B-
    D     N/A  D
    A+     N/A  A
    A-     N/A  C

    View Slide

  113. Installation
    Ubiquitous
    Language
    Focused
    Tests
    Pending
    Tests
    Expectations Mocking
    Runs on
    Device
    Docs
    OCUnit
    GHUnit
    OCHamcrest
    Kiwi
    Cedar
    Specta/Expecta
    KIF
    Calabash
    Frank
    A+    N/A   D-
    C    N/A   C
    A    N/A   B
    B       C
    A+       B-
    A       B-
    D     N/A  D
    A+     N/A  A
    A-     N/A  C
    Only Use the TDD frameworks if you *have* to

    View Slide

  114. Installation
    Ubiquitous
    Language
    Focused
    Tests
    Pending
    Tests
    Expectations Mocking
    Runs on
    Device
    Docs
    OCUnit
    GHUnit
    OCHamcrest
    Kiwi
    Cedar
    Specta/Expecta
    KIF
    Calabash
    Frank
    A+    N/A   D-
    C    N/A   C
    A    N/A   B
    B       C
    A+       B-
    A       B-
    D     N/A  D
    A+     N/A  A
    A-     N/A  C

    View Slide

  115. Installation
    Ubiquitous
    Language
    Focused
    Tests
    Pending
    Tests
    Expectations Mocking
    Runs on
    Device
    Docs
    OCUnit
    GHUnit
    OCHamcrest
    Kiwi
    Cedar
    Specta/Expecta
    KIF
    Calabash
    Frank
    A+    N/A   D-
    C    N/A   C
    A    N/A   B
    B       C
    A+       B-
    A       B-
    D     N/A  D
    A+     N/A  A
    A-     N/A  C
    Because C++ is groooooss.

    View Slide

  116. Installation
    Ubiquitous
    Language
    Focused
    Tests
    Pending
    Tests
    Expectations Mocking
    Runs on
    Device
    Docs
    OCUnit
    GHUnit
    OCHamcrest
    Kiwi
    Cedar
    Specta/Expecta
    KIF
    Calabash
    Frank
    A+    N/A   D-
    C    N/A   C
    A    N/A   B
    B       C
    A+       B-
    A       B-
    D     N/A  D
    A+     N/A  A
    A-     N/A  C
    Only Use the TDD frameworks if you *have* to

    View Slide

  117. Installation
    Ubiquitous
    Language
    Focused
    Tests
    Pending
    Tests
    Expectations Mocking
    Runs on
    Device
    Docs
    OCUnit
    GHUnit
    OCHamcrest
    Kiwi
    Cedar
    Specta/Expecta
    KIF
    Calabash
    Frank
    A+    N/A   D-
    C    N/A   C
    A    N/A   B
    B       C
    A+       B-
    A       B-
    D     N/A  D
    A+     N/A  A
    A-     N/A  C

    View Slide

  118. Installation
    Ubiquitous
    Language
    Focused
    Tests
    Pending
    Tests
    Expectations Mocking
    Runs on
    Device
    Docs
    OCUnit
    GHUnit
    OCHamcrest
    Kiwi
    Cedar
    Specta/Expecta
    KIF
    Calabash
    Frank
    A+    N/A   D-
    C    N/A   C
    A    N/A   B
    B       C
    A+       B-
    A       B-
    D     N/A  D
    A+     N/A  A
    A-     N/A  C
    Automates with actual touch events

    View Slide

  119. Honorable Mentions
    Stephen Caudill

    View Slide

  120. UIAutomation
    Wrappers
    Stephen Caudill

    View Slide

  121. Zucchini
    http://www.zucchiniframework.org/
    Stephen Caudill
    Cucumber-like stories
    Maps to Coffeescript definitions
    Looks promising, avoided because it’s javascript

    View Slide

  122. Bwoken
    http://bendyworks.github.com/bwoken/
    Stephen Caudill
    Very nice tooling built in Ruby
    Implement in Coffeescript or Javascript

    View Slide

  123. Other Honorable
    Mentions
    Stephen Caudill

    View Slide

  124. UISpec/UISpecRunner
    http://code.google.com/p/uispec/
    Stephen Caudill
    Idiomatic Objective-C
    Code several other frameworks were originally based on (e.g. KIF)

    View Slide

  125. Simon
    http://drekka.github.com/simon/
    Stephen Caudill
    Looks promising
    Feature files
    Native Objective-C mappings
    Just couldn’t get it to work

    View Slide

  126. Go forth and Specify!
    Stephen Caudill

    View Slide

  127. Stephen Caudill
    @voxdolo
    @voxdolo
    http://hashrocket.com
    So you know where to deliver your hatemail

    View Slide