Mobile test automation with Appium in real device cloud

Slides from my talk @ Ministry of Testing Meetup, Berlin Germany
Learn about challenges and how to improve your Appium tests for fast and stable execution on real devices in a cloud like Sauce Labs.
Following topics are covered:
• Basic tips on how to use Appium
• Environment: Local vs cloud
• Device selection
• Connectivity Type
• Localisation
• Device allocation
• Appium session
Useful resources:
Appium docs: http://appium.io/docs/en/about-appium/intro
J. Lipps blog about Appium: https://appiumpro.com
SauceLabs docs: https://wiki.saucelabs.com/display/DOCS
StackOverflow: https://stackoverflow.com/search?q=appium

Transcript

1. mobile test automation with Appium in real device cloud MoT Berlin @ SauceLabs
2. hello! I am Dmitry Sr. QA Automation Engineer @ 2 automateMeNow dmle
3. It’s not about: ✘ Appium vs Espresso / XCUITest ✘ Real devices vs emulators ✘ Sauce Labs vs other cloud providers 3 But: ✘ Get familiar with potential challenges ✘ Minimize test flakiness and get the most out of Appium ✘ Stabilize test execution in real device cloud
4. What works for me… 4 May not work for everyone
5. 1. Let’s start with basics 2 + 2 = 4 5
6. Keep in mind while using Appium ✘ Install app only once 6 ✘ Stop using bad locators ✘ Implicit wait = sleep = bad practice ✘ Every appium call counts ✘ Take care of keyboard ✘ Use API to generate test data / change app status ✘ Start activity / relaunch app / deep links between tests
7. Fast tests = Avoid using XPath Fast & reliable locators ✘ id ✘ Accessibility id ✘ Predicate string 7
8. “ - Don’t you ever use XPath? 8 XPath of total locators (~200): Android: 6% iOS: 4% - I do. Only when really necessary
9. 9 Starting using device cloud like... Source: http://en.espn.co.uk/f1/motorsport/image/6007.html
10. 10 After 3-4 sessions... Source: https://insurance4motorsport.blogspot.com/2013/08/formula-one-driver-insurance.html
11. 2. Local vs cloud 11
12. Be careful with version you use 12 appium-base-driver@3.12.0 appium-xcuitest-driver@2.102.0 appium-xcode@3.7.2 WebDriverAgent appium-ios-driver@2.13.0 … 40+ modules java-client@6.1.0 python-client@0.31 ruby-lib@9.16.0 wd@1.11.0 webdriverio@4.14.0 Server Client
13. Cloud is black box 13 appium-base-driver@3.12.0 appium-xcuitest-driver@2.102.0 appium-xcode@3.7.2 WebDriverAgent appium-ios-driver@2.13.0 … 40+ modules java-client@6.1.0 python-client@0.31 ruby-lib@9.16.0 wd@1.11.0 webdriverio@4.14.0 Server Client In Cloud only release version: 1.9.1 1.8.1 1.8.0 1.7.2
14. 3. Device selection 14
15. Devices have different: 15 ✘ OS version ✘ Manufacture customized OS ✘ DPI / Screen resolution ✘ CPU / RAM ✘ Default connectivity type ✘ Keyboard ✘ Preinstalled apps
16. findElementById(“highlights”) Samsung Galaxy S9 Motorola Moto G4 16 Source: https://play.google.com/store/apps/details?id=in.olx.cashmycar.dealer
17. “app_india”: [ “Huawei_Mate_Pro_real”, “HUAWEI_P20_real”, “Samsung_Galaxy_S9_real”, “Samsung_Galaxy_S8_Plus_real” “LG_G7_Thinq_real”, “OnePlus_6_real”, “ASUS_Zenfone_5_real” ] Use device matrix 17 ✘ Based on users / local market stats ✘ Different manufactures / OS versions 1. Stabilize tests for matrix 2. Remove unreliable devices 3. Sync matrix every 3+ months 4. Keep it carefully from growth
18. Define device for test run API: /available 18 Filter matrix Select random Start session Dynamic allocation filter by OS version & device type static
19. UI Automator Search for hidden elements inside scrollable View findElementByUIAutomator(“new UiScrollable(new UiSelector()...”) 19
20. 4. Connectivity Type 20
21. Check connection on session start Keep in mind: ✘ Airplane Mode: 6+ ✘ Data connection: <=4.4 ✘ Wi-Fi connection: all versions 21 ✘ getNetworkConnection() ✘ setNetworkConnection()
22. Network Connection (Android only) 22 Value Data Wifi Airplane Mode 0 (None) 0 0 0 1 (Airplane) 0 0 1 2 (Wifi only) 0 1 0 4 (Data on) 1 0 0 6 (All on) 1 1 0
23. 5. Localisation 23
24. Check language on session start ✘ Check the element text on login ✘ Change via switch button In worst case ✘ Skip tests with localisation checks 24
25. 6. Device allocation 25
26. Once in a while getting exception UnknownError - An unknown server-side error occurred while processing the command. Selenium error: Lost contact with app, please check Appium log for more information The environment you requested was unavailable. Unable to install app to the device XXX. Unable to allocate device YYY within 900000ms or 3 tries. This can happen if there were no available devices within the time or your plan's max concurrent test run is not corresponding to your usage 26
27. 27 Smart device allocation Source: http://en.espn.co.uk/f1/motorsport/image/6007.html
28. 3 iterations of custom mechanism Version 1 Pull device matrix Get available devices via API Filter matrix with available Select random one for session Dynamic allocation Version 2 Run v1 for EU device cluster Run v1 for US device cluster Dynamic allocation Version 3 Run v2 Catch session creation error Add device to “black list” Retry v2 with black list filter 28
29. Device allocation handling Select Device 29 Allocate Device Done Failed Add device to Blacklist. Retry
30. 7. Appium session 30
31. “ - Run each test separately 31 - It takes too long - Use device caching - Still too long
32. Session start ~ 40 sec IOS ~ 70 sec Running suite of 30 tests 32 Cached restart ~ 18 sec IOS ~ 25 sec
33. 33 Session restart timeout Source: https://www.mclaren.com/formula1/heritage/bruce-mclaren/
34. Custom sesion restart timeout 1. Define own session restart timeout (5 - 7 min) 2. Start session with time counter and enabled device caching 3. Starting from 2nd test check timeout - counter > max_test_time 4. Restart session before N test when timeout reached 5. Save result in SL, get report link, continue execution 34
35. Appium session handling New session 35 Close current session & start new with cached device Test 1 Check session time Test N Retry
36. 30 tests ~ 13 min Session timeout of 7 min, 1 restart 39% Time saved! 28 restarts ~ 8.4 min 18 sec / restart 36
37. Useful resources: 37 ✘ http://appium.io/docs/en/about-appium/intro ✘ https://appiumpro.com ✘ https://wiki.saucelabs.com/display/DOCS ✘ https://stackoverflow.com/search?q=appium
38. Make Appium a better tool Follow the progress: Image comparison: https://github.com/appium/appium/blob/ master/docs/en/writing-running-appium/ image-comparison.md Element find plugins: https://github.com/appium/appium/blob/ master/docs/en/advanced-concepts/ele ment-finding-plugins.md Contribute: ✘ Create issues ✘ Add/translate docs ✘ Answer questions ✘ Add new functionality ✘ Support client libraries 38
39. Let’s summarize: 39 ✘ Follow the best practices with Appium ✘ Don’t try to support tests for all the devices in cloud ✘ Make sure to set correct connection and language on device ✘ Don’t fully rely on cloud with device allocation ✘ Be critical to test time execution, try to improve
40. 40 Back on track! Source: https://www.juniqe.com/formula-1-poster-alu-frame-3445779.html
41. thanks! Any questions? You can find me at 41 automateMeNow dmle