iOS Test Automation The Good, the Bad and the Ugly 

KIF, Appium, Calabash, EarlGrey or Xcode7 automation? How to choose the best UI Test framework for your project. 

Test frameworks under evaluation Name: Appium Developed by: Sauce Labs Language: Ruby, Java, C#, Python, JS, PHP Setup: Works out of the box after all needed tooling is installed (brew, node.js, appium) Status: Used and developed actively Name: KIF Developed by: Square, Inc. Language: Objective C, Swift Setup: Doesn’t work out of the box Support: Poor Name: Calabash Developed by: Xamarin (currently Microsoft) Language: Ruby Setup: One-click setup when running existing tests but need some coding for seVing it up from scratch. Status: : Used and developed actively Name: XCUITest Developed by: Apple Language: Objective C, Swift Setup: Easy setup. Built-in in Xcode Status: Used and developed actively Name: EarlGrey Developer by: Google Language: Objective C, Swift Setup: Easy setup when using CocoaPods Status: Used and developed actively 

Questions to be answered before evaluation begins   Who will write UI Tests?   Do we want to have a possibility to write cross-platform tests?   Do we want to change project structure and code in order to run UI Automation? 

Questions to be answered while evaluating   How responsive is community and how fast are new features developing?   How easy will searching and defining APP’s elements be?   What are the possibilities to start tests on physical devices and build a device farm? 

#5 KIF Advantages: Supports iOS 8.1 and above (according documentation) Transitions are fast It is possible to access APP’s assets (e.g. LocalizedStrgings) Disadvantages: Poor support Cucumber is not supported: 1. It could be hard to write tests for non-devs 2. Shiny Cucumber test reports are not available. Note: it is possible to use other plugins to get test reports Accesses elements only by accessibility aVributes Has to be in the same repo as main APP Deeplinks (Universal links) are not supported Necessity to manage schemes to run tests on CI 

#5 KIF code example 

Summary To make test runs stable, coding on top of framework is needed. You can checkout example from Linkedin: link Disadvantages: ①  Poor support ②  Hard to write tests for non-devs as it is incompatible with tools like Cucumber ③  Accesses elements only by accessibility attributes 

#4 XCUITest Advantages: Developed by Apple Perform gestures fast and stable Record function Has ‘wait APP to idle’ mechanism (but it is mostly waiting for animation and not listening the APP) Disadvantages: Cucumber is not supported: 1. It could be hard to write tests for non- devs 2. Shiny Cucumber test reports are not available. Note: it is possible to use other plugins to get test reports Record function is not stable. Recorded steps could be wrong Support exists but it is really hard to get something done (e.g. bug) Limited by iOS9 Access elements mostly by accessibility aVributes Provides access to native elements via accessibility classes, not the concrete classes. Xcode could crash trying to access elements with concrete classes. Necessity to touch APP code (like add accessibility ids/labels or disable animations during the test) to make it compatible with framework requirements Has to be in the same repo as main APP Deeplinks are not supported Necessity to manage schemes to run tests on CI 

#4 XCUITest code example 

Summary XCUITest could be a tool that is really nice to work with. But lack of possibility to get in contact with Apple representatives and constantly broken features like type text take away all the pleasure. There is also an amount of limitations because Apple has their vision on how tests should look like. Disadvantages: ①  Changes in APP code is needed ②  May be hard to write tests for non-devs as it doesn’t support tools like Cucumber ③  Accesses elements mostly by accessibility attributes 

#3 EarlGrey from Google Advantages: Google support Fast and stable performance of gestures (because EarlGrey synchronizes with the app and waits for it to idle) It is possible to access APP’s assets (e.g. LocalizedStrgings) Supports iOS 8 and above Could see concrete classes of the elements Possible to test deeplinks (Universal Links) using workaround Disadvantages: Cucumber is not supported: 1. It could be hard to write tests for non-devs 2. Shiny Cucumber test reports are not available. Note: it is possible to use other plugins to get test reports May require additional coding in your APP for using EarlGrey feature like ‘wait APP to idle’ Has to be in the same repo as main APP Necessity to manage schemes to run tests on CI 

#3 EarlGrey code example 

Summary EarlGrey is young but promising tool widely used by Google to automate their APPs. Absence of own element inspector can slow you down at the start but it is worth trying. Note: It is possible to use Facebook inspector this one does not ideally pass because it uses different hierarchy then EarlGrey. Disadvantages: ①  Changes in APP code may be needed ②  Hard to write tests for non-devs as it is incompatible with tools like Cucumber 

#2 Appium Advantages: Active community Tests are stable Record function Note: You can’t rely only on record function to write your tests Easy to write tests for non-devs where it comes with Cucumber Not limited by iOS versions Good overall report where it comes with Cucumber Independent from APP code Tests are tagged, easy to run single test No additional frameworks should exist in APP to run tests Cross-platform Disadvantages: Accesses elements mostly by accessibility aVributes Provides access to native elements via UIAutomation accessibility classes, not the concrete classes Gestures performance is slow Deeplinks are not supported 

#2 Appium code example 

Summary Currently Appium is the most used test framework. Appium has UI Interface where a user can easily configure the framework. But Selenium Webdriver API is not fun to work with and was not specifically designed for Mobile UI testing. Disadvantages: ①  Provides access to native elements via UIAutomation accessibility classes, not the concrete classes. ②  Accessibility attributes should be used all over the APP ③  Impossible to access APP’s classes and methods 

#1 Calabash Advantages: Easy to write tests for non-devs as it comes with Cucumber Tests are stable Possible to use APP’s classes and methods Provides access to native elements via concrete classes Accessibility aVributes are not needed to find an element Easy to find elements (console is just awesome) Possible to identify when all the animations on the screen are done Works with physical devices Not limited by iOS versions Good overall report Independent from APP code Tests are tagged, easy to run single test Cross-platform Disadvantages: Calabash framework should be inside the APP Community is not so big Some coding is required to set up framework Gestures performance is slow Although waiting for animation methods are good and really usable, it is a pity that ‘wait APP to idle’ mechanism doesn’t exist 

#1 Calabash code example 1 2 3 

Summary Calabash tools like backdoors and direct Objective-C/Swift selector calls can make testing faster by putting your application into a state where it can be tested. It is called "Power over Purity”. It is easy to write tests for non-devs and find elements on the screen using hierarchy (child, sibling, descendant, parent). Disadvantages: ①  Calabash framework should be inside the APP ②  Community is not so big ③  Some coding is required to set up framework ④  Gestures performance is slow (should change with XCUITest support) 

iOS 10 support Apple announced dropping of UIAutomation in iOS 10. The fact that Apple will no longer support UIAutomation becomes a problem for frameworks based on it. 

Recent results   QA write tests where no programming skills are needed   As android also use it for automation, we actively share code between Android and iOS   We were able to automate Push Notifications tests   We have proof of concept for tracking tests automation   We have a good base of implemented steps. Now writing a new test take some minutes in most cases. 

for your attention! This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. Created by: Serghei Moret @xJoeSSx