Mobile automation in the cloud

Slides from my talk @ ComaQA Conference 2017, Minsk Belarus
This talk explores the market of real device cloud services with details on how it may work and be integrated, what are the known limitations and how it works on a project with ReactNative application.
Plan:
• Automation without a cloud: any problems?
• Difference between server-side & client-side execution
• Compare device labs, frameworks and integrations they support
• Be aware of cloud imitations
• Use case with ReactNative application

Transcript

1. mobile Automation in the cloud
2. Hello! Dmitry Lemeshko QA Automation @ Ciklum Minsk / eBuilder Sweden
3. PLAN for TODAY’s EVENING: × Automation without a cloud: any problems? × Execution in cloud: server-side vs client-side × Device labs, frameworks and integrations × What may go wrong a.k.a limitations × Tool set and Use case on real project × Your Questions
4. LOCAL RUN
5. Who took the device? UPDATE device os VERSION? APPIUM Server errors? Device cleaning? Xcode 7 or xcode 8? Who is responsible for maintenance? Parallel run?
6. WITH CLOUD
7. Who took the device? UPDATE device os version? APPIUM Server errors? DEVice Cleaning? Xcode 7 or xcode 8? Who is responsible for maintenance? Parallel run?
8. LET’S START the JOURNEY
9. POPULAR Mobile Device Clouds
10. Client-side vs Server side cloud execution
11. SERVER-side CLOUD execution PROS × Tests execution self-managed × Parallel run out-of-box × Add extra data and apps × Automatic results aggregation CONS × Limited language support × Follow code structure × No control on test execution × Limited integration
12. CLient-side cloud execution PROS × Language flexibility × Drive your own framework × Code privacy × More ways for integration CONS × Lots to manage yourself: device availability, test execution, parallel run, results
13. Device labs Android/iOS 143 / 36 357 / 42 176 / 176 Android 4.4+/6.x/7.x 67 / 31 / 7 137 / 35 / 5 150 /16 / 8 iOS 9.x/10.x 23 / 3 5 / 14 100 / 10 Manufactures 20 32 22 Tablets 18% 30% 30%
14. Framework support Android Appium Espresso Robotium Appium Calabash Espresso UI Automator Robotium iOS Appium Appium Calabash UI Automation XCTest / XCUITest
15. Integration support CLOUD API JAVA, RUBY CLIENTS JAVA, RUBY, PYTHON CLIENTS AWS API / AWS CLI PLUGINS JENkiNS, GRaDLE JENkiNS, GRaDLE App release tools Hockey App Bug Trackers JIRA, GITHUB JIRA
16. URL http://appium.testobject.com/wd/hub Desired capabilities testobject_api_key = 73A**********12 Testobject_device = Samsung_Galaxy_S7_real TEST OBJECT Appium Setup
17. testobject_appium_version = 1.6.3 testobject_app_id = 92 testobject_suite_name = Smoke Test testobject_test_name = Navigation via menu testobject_cache_device = true testobject_session_creation_timeout = 30 Return testobject_test_live_view_url testobject_test_report_url TEST OBJECT OPTIONAL CAPABILITIES
18. test execution output in cloud 1. Device logs 2. Appium logs 3. Screenshots 4. Test run recording 5. Cpu/Memory usage
19. What may go wrong a.k.a limitations 1. Appium server update frequency 2. test runs fail on device setup stage 3. No login with custom google account 4. No device settings change 5. No wifi turn on/off
20. Moved to testobject.. still have limitations 1. No device storage access 2. Sms, contacts & call history cleaning 3. No push notifications for iOS 4. No Geo location setup 5. Camera usage
21. Tool set and Use case on real project × Appium × Java/TestNG × HockeyApp API × TestObject API × Team City CI × Smoke: random in group × Regression: top 10 devices (covers 70% Android users) × Feature test (on request): 50+ devices
22. BACKUP & RESTORE FUNCTIONALity TESTING
23. Can you try it?
24. GO AHEAD! 250 min trial 5 free devices 3 free devices
25. THANKS! Any questions? You can find me at dima.inspired@gmail.com