Continuous mobile automation in Android build pipeline

Slides from my talk @ Automation Day Berlin 2018
It can be interesting if you wonder "How To":
• better organize Android functional tests
• configure environment deployment from tests
• run functional tests on each merge request
• have all the tests in the same pipeline with the Android build
• make a scalable solution with testing on the real devices
Technical stack: Appium, node.js, Gitlab CI and Sauce Labs

Transcript

1. Continuous mobile automation in Android build pipeline Automation Day Berlin 2018
2. Hello! I am Dmitry Lemeshko Mobile QA Automation @ FrontierCarGroup, Berlin 6+ years in Test Automation AutomateMeNow dmle 2
3. Frontier Car Group • 6 markets • 3 web apps • mobile apps • ~20 microservices 3
4. Let's start Mobile test automation from scratch 4
5. Mobile automation stack 5
6. Feature testing process 6 1 Build APK From feature branch and point to Staging environment Feature testing Manual testing of new functionality 2 Merge to master When passed QA Automated tests on latest master branch 3 Regression testing 4
7. 7 1 Build APK Pointing to Staging environment & upload to Google Storage Run tests Download & install apk, run tests, analyse results and report to developers 2 Gitlab CI Android pipeline Test automation pipeline
8. First result • Android/iOS tests in same repo • Regression testing on master • Staging environment • Local emulators 8
9. “ Work done. Any issues? 9
10. “ Bugs are spotted lately 10
11. “ Hard to maintain tests 11
12. “ No full control over test environment 12
13. “ Hard to scale 13
14. “ Lack of interests from developers 14
15. Let’s think What do we want to achieve? 15
16. Continuous mobile test automation It sounds good, right? 16
17. Feature testing process 17 1 Build APK From feature branch and point to isolated environment Regression testing Automated tests 2 Feature testing Manual testing of new functionality When passed QA 3 Merge to master 4 Build APK From master branch and point to Staging environment Extended regression Automated tests on latest master branch 1 2
18. 1 pipeline instead of 2 18 1 Build APK Pointing to isolated environment & uploaded to Google Storage Deploy services Specified branch for each service, services are up-and-running 2 Run tests Take recent APK and run tests on random device from market pool Upload artifacts to gitlab CI job 3 Attach artifacts 4
19. Steps to do • Move tests to Android/iOS project repos • Configure env deployment from tests • Manage API endpoints in mobile app • Scale mobile devices & Appium tools • Build in single android pipeline 19
20. Split & move: 20 Mobile tools iOS tests Android tests • page objects • tests & hooks • wd.js wrapper • setup, logging • cloud utils
21. 21 Admin Web tests Projects in AUT source code Dealer Web tests Customer Web tests Dealer iOS tests Dealer Android tests Web tools Tools (API/deploy) Mobile tools Frameworks dependencies
22. 22 Dealer iOS tests Dealer Android tests Tools (API/deploy) Mobile tools Project dependencies
23. Configure deployment from tests • List services and branches "services": [ {"name": "serviceA", "branch": "master"}, {"name": "serviceB", "branch": "B-270"} ] • Specify env prefix "prefix": "automation10" 23
24. Make custom test builds That point to API with prefix configured before 24
25. 25 Test devices infrastructure Source: Facebook In-house farm Cloud services Virtualization
26. Integrate cloud service • Upload new builds from Google Storage • Manage available devices • Run tests in a single session* • Handle cloud session time limit • Set results and pull artifacts 26
27. 27 Gitlab CI pipeline
28. Update pipeline • Update existing docker image • Add new gitlab CI runner • Add new job for tests • Configure trigger for job 28
29. “ How to add a new job in Gitlab CI? 29
30. Update .gitlab-ci.yml: deployAndUItests: stage: build script: - export branch=$(echo $CI_BUILD_REF_NAME) - bash /buildAndroid.sh ... - npm run deploy - export appiumService=testobject env=automation - npm run regression 30
31. Update .gitlab-ci.yml: artifacts: name: "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}" when: always paths: - tests/uiTests/screenshots expire_in: 4 weeks allow_failure: false only: - triggers 31
32. Trigger conditions • Target branch is master • No WIP in title • Marked with `Reviewed` label 32
33. “ How does it look in Gitlab CI? 33
36. Before • Regression after merge • Shared unreliable test env • Self support for appium & devices • Tests maintenance is on QA • Proxying test results What do we get? Now • Regression before merge • Isolated configurable test env • Cloud service • Tests maintenance is on QA/Dev • Results in build pipeline 36
37. Flexibility Any environment, any version of services Shared maintenance Developers are fixing tests :) Fast feedback Developers get results faster & directly 37
38. 38 iOS project is in progress...
39. 39 Thanks! Any questions? You can find me at AutomateMeNow dmle