Creating parallelized Android UITest (Appium) environment using Azure, Docker and Android emulator

Agile & Automation Days 2019 #aadays2019
https://aadays.pl/speakers/tadashi-nemoto/

Transcript

2. Introduction 
 • Tadashi Nemoto
 • QA Automation Engineer / Mercari, inc.
 ◦ Selenium / Appium / XCUITest
 ◦ CI / CD pipeline
 ◦ Cloud-based test environment
 tadashi0713 tadashi-nemoto
3. Agenda
 • About Mercari, inc.
 • About our automation tests and problems
 • What is Docker-Android, 
 and Docker-Android in the cloud
 • Demo
 • Challenges to be solved

4. https://tech.mercari.com/entry/2018/12/28/175147
5. About Mercari, inc.
6. About February 1st, 2013 Established Tokyo, Sendai, Fukuoka, Palo Alto, Portland, Boston Offices 1800+ Including subsidiaries Headcount 6
7. allow individuals to buy and sell items and share information through their phones. We created “Mercari,” a C2C marketplace app where users can use their smartphones to easily buy and sell secondhand goods. By linking sellers who own items to buyers who see the value in these objects, our goal is to provide an easy, fun, and safe and secure service that creates new value and redefines the cycle of consumption. In modern society, where mass production and mass consumption have become commonplace, the value of items that fall into disuse each year is expected to grow to ¥7.6 trillion in Japan alone.* Of these items, there are many that are no longer useful to one person, but could be used by someone else who values them. The Mercari Group aims to use the popularization of smartphones and social media to Create value in a global marketplace where anyone can buy & sell Mission Source: Ministry of Economy, Trade and Industry (April 2018). Estimated total value of items described as “items that became no longer needed over the past year”. Excludes aggregate value of automobiles, motorcycles and mopeds 7
8. What Is Mercari? •Service start: July 2013 •OS: Android, iOS *Can also use from web browsers •Usage fee: Free *Commission fee for sold items: 10% of the sales price •Regions/languages supported: Base specs for Japan/Japanese Many sellers enjoy having the items they no longer need purchased and used by buyers who need them, and buyers enjoy the feeling of hunting for treasure as they search through unique and diverse items for lucky finds. In addition to buying and selling, users actively communicate through the buyer/seller chat and the “Like” feature. The Mercari app is a C2C marketplace where individuals can easily sell used items. We want to provide both buyers and sellers with a service where they can enjoy safe and secure transactions. Mercari offers a unique customer experience, with a transaction environment that uses the payments Mercari holds in escrow, and simple and affordable shipping options. 8
9. By the Numbers (JP/Full Year) 232 346.8 GMV¹ 490.2 B
 In: billion JPY FY 06/2017 FY 06/2018 21.2 33.4 Net Sales 46.2 B
 In: billion JPY FY 06/2017 FY 06/2018 8.45 10.75 MAU² 13.57 M
 In: Million people FY 06/2017 FY 06/2018 Source: Internal documents, from FY2018.6 Presentation Material
 1. GMV after cancellation
 2. Monthly Active Users in June. Number of registered users that used our app in the month
 490.2 FY 06/2019 46.2 FY 06/2019 13.57 FY 06/2019 9
10. Building a new ecosystem through Merpay Merpay is a mobile payments service operated by Merpay, Inc. The company belongs to the Mercari Group, widely recognized for its service “Mercari,” the top C2C marketplace app in Japan. Money earned by selling unwanted items on the Mercari app and money charged to the app through users’ bank accounts can be used to make payments at stores and on the Mercari app itself. Our future goal is to provide a comprehensive financial service by combining primary and secondary distribution based on data, and supporting the business of merchants while utilizing users’ credit information, such as their transaction history and ratings. Primary distribution Manufactures, retail, distributors, etc. Secondary distribution Marketplace app Attract customers Marketing Transaction data Future Business Development Building an Ecosystem
11. “The Selling App.” Mercari’s mission is to “Create value in a global marketplace where anyone can buy & sell,” and the year after our founding, we began to expand overseas. We started our business in the US in April 2014, and we adapted our branding, improved our UI/UX, and created a delivery network to match local preferences and the characteristics of the US market. We recognize that succeeding in the US, with its enormous and diverse population, is an important milestone to achieve our mission, so that is where we are devoting our energy. 11 Future Business Development Strengthening Overseas Expansion To Make Selling Easier than Buying.
12. About our automation tests and problems
14. UI・E2E Tests of Android
 Build & upload apk Kick tests
17. Problems
 • Increasing test execution time
 • Maintenance cost of real devices
 • Test failures due to differences of real devices
 (e.g. display size, OS version, etc.)
 • Maintenance cost of local environments
 (e.g. Appium, Ruby, etc.)
 • These problems often made tests flaky and unreliable

18. Our headcount tripled in 3 years
 1,826 376 462 555 596 712 795 1,014 1,140 1,357 1,639 1,786 Changes in total headcount 1Q 2Q 3Q 4Q FY2016.6 1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q FY2017.6 FY2018.6 FY2019.6 from Mercari Bold Challenge
19. The number of engineers also increased greatly
 FY2018 FY2019 FY2017 FY2016 from Mercari Bold Challenge
20. We need faster and more stable QA feedback

21. Test Pyramid
 https://martinfowler.com/bliki/TestPyramid.html
22. Mobile Testing Pyramid
 http://elementalselenium.com/tips/71-mobile-testing-pyramid
23. Mobile Testing Pyramid
 http://elementalselenium.com/tips/71-mobile-testing-pyramid
24. Azure, Docker, Android Emulator

25. Azure, Docker, Android Emulator

26. Summary
 • We first created a local environment (local PC, real devices) to run Android E2E・UI Tests
 • We faced some problems, and needed faster and more stable QA feedback
 • Based on the Mobile Testing Pyramid, we tried creating parallelized test environment using Azure, Docker, and Android Emulator

27. What is Docker-Android, and Docker-Android in the cloud
28. Advantages of using
 • Docker
 • Android Emulator
 • Cloud server

29. Docker
 • Easy to scrap and build
 • Easily recreate same
 environment(local, CI)
 • Isolated environment
 • Infrastructure as Code

30. Cloud server(IaaS)
 • Reduces maintenance cost compared to on-premises server
 • Easy to scale
 • Cost determined by usage

31. Android Emulator
 • Can reduce maintenance cost
 of real devices
 • Clean state
 • Can emulate
 ◦ SMS
 ◦ Push notification

32. Docker-Android
 github.com/docker-android

33. Advantages of Docker-Android
 • Uses fast Android x86 emulator
 • Starts Appium server inside container
 • Can connect to Selenium Grid easily
 • Remote connect using noVNC
 • Can capture videos

34. Docker-Android Selenium Grid

35. How to run Docker-Android
 in the cloud 
 • Android x86 emulator needs hardware virtualization
 • Can only run limited cloud service / Instances
 ◦ Bare metal Instance(AWS)
 ◦ Nested virtualization(Azure)
 

36. AWS
 https://aws.amazon.com/jp/blogs/aws/new-amazon-ec2-bare-metal-instances-with-direct-access-to-hardware/
37. Azure (Nested Virtualization)
 https://azure.microsoft.com/en-us/blog/nested-virtualization-in-azure/
38. Run Docker-Android in Nested Virtualization
 https://stackoverflow.com/questions/49099428/is-it-possi ble-to-run-kvm-on-azure
39. Summary
 • With Docker-Android and Cloud server(Azure), we could create parallelized testing environments
 • We can also benefit from using
 ◦ Docker
 ◦ Android Emulator
 ◦ Cloud server

40. Demo
41. https://github.com/tadashi0713/webdriverio-appium-on-docker-android Demo
42. Challenges to be solved
43. Challenges to be solved
 • Test report
 • Optimise distributed tests
 • Server load on QA environment
 • Divide role of emulators and real devices

44. Test report
45. Rspec HTML Reporter
 • Problems
 ◦ Difficult to find history and trend
 ◦ Generate reports by number of parallel

46. Allure

47. Optimise distributed tests
48. https://docs.knapsackpro.com/2018/improve-circleci-para llelisation-for-rspec-minitest-cypress Optimise distributed tests

49. https://docs.knapsackpro.com/2018/improve-circleci-para llelisation-for-rspec-minitest-cypress Knapsack (Pro)

50. Server load on QA environments
51. • Solution
 ◦ Use mock server instead
 ◦ Improve QA environment infracture
 Server load to QA Environments
 API server
 (QA)

52. Divide roles of emulators and real devices
53. Mobile Testing Pyramid
 http://elementalselenium.com/tips/71-mobile-testing-pyramid
54. Real devices: real-time conditions(Headspin)
 https://tech.mercari.com/entry/2019/02/18/173236
55. Summary
 • We still need to solve the below challenges for faster and more stable QA feedback
 ◦ Test report
 ◦ Optimise distributed tests
 ◦ Server load on QA environment
 ◦ Divide roles of emulators and real devices

56. Conclusion
57. Message(Key takeaways)
 • We faced problems in testing environments, and need faster, more stable QA feedback
 • Following the Mobile Testing Pyramid, we tried creating parallelized test environment using Azure, Docker, and Android Emulator
 • There are challenges to be solved apart from the test environment for faster and more stable QA feedback