iOS and Android Acceptance Testing with Calabash - Xcake Dublin

Sometimes, you would be forgiven for thinking that testing your apps' user interface is not the most fun bit of the whole process of creation. But as a matter of fact, it is, especially if you use interesting tools that help with the heavy lifting and have a freaky domain specific language that makes it straightforward to specify and automate your test cases.

Roland Gröpmair, a regular face at Xcake, will present to us his latest investigations in this area, as part of his talk iOS and Android Test Automation with Calaba.sh.

http://www.meetup.com/Xcake-Mobile-app-development-with-an-Appley-flavour/events/219080531/

Transcript

1. Acceptance Testing with Calabash @roland9 roland@intercom.io Xcake Dublin, 9th Dec 2014
2. What is Calabash?
3. What is Calabash? • Automated acceptance testing of mobile apps
4. What is Calabash? • Automated acceptance testing of mobile apps • Cross-platform (Android & iOS)
5. What is Calabash? • Automated acceptance testing of mobile apps • Cross-platform (Android & iOS) • Open source - Company called Xamarin backing it
6. What is Calabash? • Automated acceptance testing of mobile apps • Cross-platform (Android & iOS) • Open source - Company called Xamarin backing it • https://github.com/calabash/calabash-ios
7. What is Calabash? • Automated acceptance testing of mobile apps • Cross-platform (Android & iOS) • Open source - Company called Xamarin backing it • https://github.com/calabash/calabash-ios • https://github.com/calabash/calabash-android
8. Others • Frank, KIF, Zucchini, UIAutomation, iCuke, …
9. Why would I do it?
10. Why would I do it?
11. Why would I do it?
12. Other Testing Methods • Unit Tests • Integration Tests • Acceptance Tests
13. Unit Tests • Tests smallest unit of functionality public void TestPhoneValidator() { string goodPhone = "(123) 555-1212"; string badPhone = "555 12" PhoneValidator validator = new PhoneValidator(); Assert.IsTrue(validator.IsValid(goodPhone)); Assert.IsFalse(validator.IsValid(badPhone)); } • Mocks and Stubs to simulate behaviour of other objects
14. Integration Tests • Verify that combined modules function correctly
15. Acceptance Tests • Performing tests on the full system (e.g. using your web page via a web browser) • Described in plain English
16. How does it work? • http://blog.lesspainful.com/2012/03/07/Calabash-iOS/
17. Install
18. Install • ruby -v -> recommended version 2.0+
19. Install • ruby -v -> recommended version 2.0+ • gem install calabash-cucumber
20. Install • ruby -v -> recommended version 2.0+ • gem install calabash-cucumber • cd path-to-my-ios-project
21. Install • ruby -v -> recommended version 2.0+ • gem install calabash-cucumber • cd path-to-my-ios-project • calabash-ios setup
22. Install • ruby -v -> recommended version 2.0+ • gem install calabash-cucumber • cd path-to-my-ios-project • calabash-ios setup • calabash-ios gen
23. Install • ruby -v -> recommended version 2.0+ • gem install calabash-cucumber • cd path-to-my-ios-project • calabash-ios setup • calabash-ios gen • in Xcode, build -cal scheme
24. Install • ruby -v -> recommended version 2.0+ • gem install calabash-cucumber • cd path-to-my-ios-project • calabash-ios setup • calabash-ios gen • in Xcode, build -cal scheme • cucumber
25. If all goes well: • LPSimpleExample[11298:13703] HTTPServer: Started HTTP server on port 37265
26. Build Directory • export APP_BUNDLE_PATH=/Users/rolandgr/ <…>/Build/Products/Debug-iphonesimulator/ <…>-cal.app
27. Gherkin • https://github.com/calabash/calabash-ios/wiki/01-Getting-started-guide
28. • Given … • When … • Then … • And …
29. Useful Stuff
30. Useful Stuff • export DEVICE_TARGET="iPhone 5s (7.1 Simulator)"
31. Useful Stuff • export DEVICE_TARGET="iPhone 5s (7.1 Simulator)" • cucumber NO_STOP=1
32. Useful Stuff • export DEVICE_TARGET="iPhone 5s (7.1 Simulator)" • cucumber NO_STOP=1 • cucumber -t @current
33. Useful Stuff • export DEVICE_TARGET="iPhone 5s (7.1 Simulator)" • cucumber NO_STOP=1 • cucumber -t @current • cucumber -t @~excludedtag
34. Useful Stuff • export DEVICE_TARGET="iPhone 5s (7.1 Simulator)" • cucumber NO_STOP=1 • cucumber -t @current • cucumber -t @~excludedtag • calabash-ios console
35. Useful Stuff • export DEVICE_TARGET="iPhone 5s (7.1 Simulator)" • cucumber NO_STOP=1 • cucumber -t @current • cucumber -t @~excludedtag • calabash-ios console • > start_test_server_in_background
36. Query Syntax • query(“button") => [ {"class"=>"UIRoundedRectButton", "frame"=>{"y"=>287, "width"=>72, "x"=>100, "height"=>37}, "UIType"=>"UIControl", "description"=>"<UIRoundedRectButton: 0x7d463d0; frame = (100 287; 72 37); opaque = NO; autoresize = RM+BM; layer = <CALayer: 0x7d46ae0>>"}, {"class"=>"UIRoundedRectButton", "frame"=>{"y"=>215, "width"=>73, "x"=>109, "height"=>37}, "UIType"=>"UIControl", "description"=>"<UIRoundedRectButton: 0x7d3a760; frame = (109 215; 73 37); opaque = NO; autoresize = RM+BM; layer = <CALayer: 0x7d3a8a0>>”} ]
37. • query("button index:0 label”) "find the first button, and then inside of that find all labels”. • query("button index:0 label", :text)
38. • touch("tabBarButton index:1")
39. accessibilityLabel accessibilityIdentifier
40. Custom Steps - Ruby API
41. Custom Steps - Ruby API • https://github.com/calabash/calabash-ios/wiki/03.5-Calabash- iOS-Ruby-API
42. Custom Steps - Ruby API • https://github.com/calabash/calabash-ios/wiki/03.5-Calabash- iOS-Ruby-API • query, check for existence of elements, wait for elements to exist
43. Custom Steps - Ruby API • https://github.com/calabash/calabash-ios/wiki/03.5-Calabash- iOS-Ruby-API • query, check for existence of elements, wait for elements to exist • keyboard
44. Custom Steps - Ruby API • https://github.com/calabash/calabash-ios/wiki/03.5-Calabash- iOS-Ruby-API • query, check for existence of elements, wait for elements to exist • keyboard • scroll
45. Custom Steps - Ruby API • https://github.com/calabash/calabash-ios/wiki/03.5-Calabash- iOS-Ruby-API • query, check for existence of elements, wait for elements to exist • keyboard • scroll • rotate
46. Custom Steps - Ruby API • https://github.com/calabash/calabash-ios/wiki/03.5-Calabash- iOS-Ruby-API • query, check for existence of elements, wait for elements to exist • keyboard • scroll • rotate • backdoor -> call method in iOS app (implement in app delegate)
47. Custom Steps - Ruby API • https://github.com/calabash/calabash-ios/wiki/03.5-Calabash- iOS-Ruby-API • query, check for existence of elements, wait for elements to exist • keyboard • scroll • rotate • backdoor -> call method in iOS app (implement in app delegate) • macro -> call other step definitions
48. http://www.slideshare.net/mobiletestsummit/ios-automated-testing-with-calabash-tips-and- tricks?qid=0afecec1-fe64-4168-bb69-a548bd01b832&v=default&b=&from_search=8
49. Tips and Tricks
50. Tips and Tricks • Resetting app data between scenarios
51. Tips and Tricks • Resetting app data between scenarios • Avoid sleep - use wait instead:
52. Tips and Tricks • Resetting app data between scenarios • Avoid sleep - use wait instead: And I wait to see "send button"
53. Tips and Tricks • Resetting app data between scenarios • Avoid sleep - use wait instead: And I wait to see "send button" • Wait before you interact with any element
54. Tips and Tricks • Resetting app data between scenarios • Avoid sleep - use wait instead: And I wait to see "send button" • Wait before you interact with any element • Wait after every action to finish
55. Tips and Tricks • Resetting app data between scenarios • Avoid sleep - use wait instead: And I wait to see "send button" • Wait before you interact with any element • Wait after every action to finish • Escape quotes…
56. Tips and Tricks • Resetting app data between scenarios • Avoid sleep - use wait instead: And I wait to see "send button" • Wait before you interact with any element • Wait after every action to finish • Escape quotes… • https://groups.google.com/forum/#!forum/calabash-ios
57. Optional: Compare Images • gem install imagemagick-binaries • check you have compare: $ which compare
58. Optional: Compare Images • gem install imagemagick-binaries • check you have compare: $ which compare
59. Continuous Integration
60. Code Coverage
61. Other Info • UIKonf talk ‘Unit Testing on iOS’ by Tim Brückmann https://speakerdeck.com/tibr/unit- testing-on-ios • http://calaba.sh • https://github.com/calabash/calabash-ios • http://developer.xamarin.com/guides/testcloud/ calabash/introduction-to-calabash/
62. Summary • Automated Testing - investment that pays off • Together with unit testing etc: make changes with confidence (still, manual QA required) • Cross-platform mature toolset • Continuous integration / code coverage
64. Acceptance Testing with Calabash @roland9 roland@intercom.io Xcake Dublin, 9th Dec 2014