Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Creating parallelized Android UITest (Appium) environment using Azure, Docker and Android emulator

Creating parallelized Android UITest (Appium) environment using Azure, Docker and Android emulator

Agile & Automation Days 2019 #aadays2019
https://aadays.pl/speakers/tadashi-nemoto/

E8aaf6f975dda96c47412cf311089243?s=128

Tadashi Nemoto

October 29, 2019
Tweet

Transcript

  1. None
  2. Introduction 
 • Tadashi Nemoto
 • QA Automation Engineer /

    Mercari, inc.
 ◦ Selenium / Appium / XCUITest
 ◦ CI / CD pipeline
 ◦ Cloud-based test environment
 tadashi0713 tadashi-nemoto
  3. Agenda
 • About Mercari, inc.
 • About our automation tests

    and problems
 • What is Docker-Android, 
 and Docker-Android in the cloud
 • Demo
 • Challenges to be solved

  4. https://tech.mercari.com/entry/2018/12/28/175147

  5. About Mercari, inc.

  6. About February 1st, 2013 Established Tokyo, Sendai, Fukuoka, Palo Alto,

    Portland, Boston Offices 1800+ Including subsidiaries Headcount 6
  7. allow individuals to buy and sell items and share information

    through their phones. We created “Mercari,” a C2C marketplace app where users can use their smartphones to easily buy and sell secondhand goods. By linking sellers who own items to buyers who see the value in these objects, our goal is to provide an easy, fun, and safe and secure service that creates new value and redefines the cycle of consumption. In modern society, where mass production and mass consumption have become commonplace, the value of items that fall into disuse each year is expected to grow to ¥7.6 trillion in Japan alone.* Of these items, there are many that are no longer useful to one person, but could be used by someone else who values them. The Mercari Group aims to use the popularization of smartphones and social media to Create value in a global marketplace where anyone can buy & sell Mission Source: Ministry of Economy, Trade and Industry (April 2018). Estimated total value of items described as “items that became no longer needed over the past year”. Excludes aggregate value of automobiles, motorcycles and mopeds 7
  8. What Is Mercari? •Service start: July 2013 •OS: Android, iOS

    *Can also use from web browsers •Usage fee: Free *Commission fee for sold items: 10% of the sales price •Regions/languages supported: Base specs for Japan/Japanese Many sellers enjoy having the items they no longer need purchased and used by buyers who need them, and buyers enjoy the feeling of hunting for treasure as they search through unique and diverse items for lucky finds. In addition to buying and selling, users actively communicate through the buyer/seller chat and the “Like” feature. The Mercari app is a C2C marketplace where individuals can easily sell used items. We want to provide both buyers and sellers with a service where they can enjoy safe and secure transactions. Mercari offers a unique customer experience, with a transaction environment that uses the payments Mercari holds in escrow, and simple and affordable shipping options. 8
  9. By the Numbers (JP/Full Year) 232 346.8 GMV¹ 490.2 B


    In: billion JPY FY 06/2017 FY 06/2018 21.2 33.4 Net Sales 46.2 B
 In: billion JPY FY 06/2017 FY 06/2018 8.45 10.75 MAU² 13.57 M
 In: Million people FY 06/2017 FY 06/2018 Source: Internal documents, from FY2018.6 Presentation Material
 1. GMV after cancellation
 2. Monthly Active Users in June. Number of registered users that used our app in the month
 490.2 FY 06/2019 46.2 FY 06/2019 13.57 FY 06/2019 9
  10. Building a new ecosystem through Merpay Merpay is a mobile

    payments service operated by Merpay, Inc. The company belongs to the Mercari Group, widely recognized for its service “Mercari,” the top C2C marketplace app in Japan. Money earned by selling unwanted items on the Mercari app and money charged to the app through users’ bank accounts can be used to make payments at stores and on the Mercari app itself. Our future goal is to provide a comprehensive financial service by combining primary and secondary distribution based on data, and supporting the business of merchants while utilizing users’ credit information, such as their transaction history and ratings. Primary distribution Manufactures, retail, distributors, etc. Secondary distribution Marketplace app Attract customers Marketing Transaction data Future Business Development Building an Ecosystem
  11. “The Selling App.” Mercari’s mission is to “Create value in

    a global marketplace where anyone can buy & sell,” and the year after our founding, we began to expand overseas. We started our business in the US in April 2014, and we adapted our branding, improved our UI/UX, and created a delivery network to match local preferences and the characteristics of the US market. We recognize that succeeding in the US, with its enormous and diverse population, is an important milestone to achieve our mission, so that is where we are devoting our energy. 11 Future Business Development Strengthening Overseas Expansion To Make Selling Easier than Buying.
  12. About our automation tests and problems

  13. None
  14. UI・E2E Tests of Android
 Build & upload apk Kick tests

  15. None
  16. None
  17. Problems
 • Increasing test execution time
 • Maintenance cost of

    real devices
 • Test failures due to differences of real devices
 (e.g. display size, OS version, etc.)
 • Maintenance cost of local environments
 (e.g. Appium, Ruby, etc.)
 • These problems often made tests flaky and unreliable

  18. Our headcount tripled in 3 years
 1,826 376 462 555

    596 712 795 1,014 1,140 1,357 1,639 1,786 Changes in total headcount 1Q 2Q 3Q 4Q FY2016.6 1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q FY2017.6 FY2018.6 FY2019.6 from Mercari Bold Challenge
  19. The number of engineers also increased greatly
 FY2018 FY2019 FY2017

    FY2016 from Mercari Bold Challenge
  20. We need faster and more stable QA feedback


  21. Test Pyramid
 https://martinfowler.com/bliki/TestPyramid.html

  22. Mobile Testing Pyramid
 http://elementalselenium.com/tips/71-mobile-testing-pyramid

  23. Mobile Testing Pyramid
 http://elementalselenium.com/tips/71-mobile-testing-pyramid

  24. Azure, Docker, Android Emulator


  25. Azure, Docker, Android Emulator


  26. Summary
 • We first created a local environment (local PC,

    real devices) to run Android E2E・UI Tests
 • We faced some problems, and needed faster and more stable QA feedback
 • Based on the Mobile Testing Pyramid, we tried creating parallelized test environment using Azure, Docker, and Android Emulator

  27. What is Docker-Android, and Docker-Android in the cloud

  28. Advantages of using
 • Docker
 • Android Emulator
 • Cloud

    server

  29. Docker
 • Easy to scrap and build
 • Easily recreate

    same
 environment(local, CI)
 • Isolated environment
 • Infrastructure as Code

  30. Cloud server(IaaS)
 • Reduces maintenance cost compared to on-premises server


    • Easy to scale
 • Cost determined by usage

  31. Android Emulator
 • Can reduce maintenance cost
 of real devices


    • Clean state
 • Can emulate
 ◦ SMS
 ◦ Push notification

  32. Docker-Android
 github.com/docker-android


  33. Advantages of Docker-Android
 • Uses fast Android x86 emulator
 •

    Starts Appium server inside container
 • Can connect to Selenium Grid easily
 • Remote connect using noVNC
 • Can capture videos

  34. Docker-Android Selenium Grid


  35. How to run Docker-Android
 in the cloud 
 • Android

    x86 emulator needs hardware virtualization
 • Can only run limited cloud service / Instances
 ◦ Bare metal Instance(AWS)
 ◦ Nested virtualization(Azure)
 

  36. AWS
 https://aws.amazon.com/jp/blogs/aws/new-amazon-ec2-bare-metal-instances-with-direct-access-to-hardware/

  37. Azure (Nested Virtualization)
 https://azure.microsoft.com/en-us/blog/nested-virtualization-in-azure/

  38. Run Docker-Android in Nested Virtualization
 https://stackoverflow.com/questions/49099428/is-it-possi ble-to-run-kvm-on-azure

  39. Summary
 • With Docker-Android and Cloud server(Azure), we could create

    parallelized testing environments
 • We can also benefit from using
 ◦ Docker
 ◦ Android Emulator
 ◦ Cloud server

  40. Demo

  41. https://github.com/tadashi0713/webdriverio-appium-on-docker-android Demo

  42. Challenges to be solved

  43. Challenges to be solved
 • Test report
 • Optimise distributed

    tests
 • Server load on QA environment
 • Divide role of emulators and real devices

  44. Test report

  45. Rspec HTML Reporter
 • Problems
 ◦ Difficult to find history

    and trend
 ◦ Generate reports by number of parallel

  46. Allure


  47. Optimise distributed tests

  48. https://docs.knapsackpro.com/2018/improve-circleci-para llelisation-for-rspec-minitest-cypress Optimise distributed tests


  49. https://docs.knapsackpro.com/2018/improve-circleci-para llelisation-for-rspec-minitest-cypress Knapsack (Pro)


  50. Server load on QA environments

  51. • Solution
 ◦ Use mock server instead
 ◦ Improve QA

    environment infracture
 Server load to QA Environments
 API server
 (QA)

  52. Divide roles of emulators and real devices

  53. Mobile Testing Pyramid
 http://elementalselenium.com/tips/71-mobile-testing-pyramid

  54. Real devices: real-time conditions(Headspin)
 https://tech.mercari.com/entry/2019/02/18/173236

  55. Summary
 • We still need to solve the below challenges

    for faster and more stable QA feedback
 ◦ Test report
 ◦ Optimise distributed tests
 ◦ Server load on QA environment
 ◦ Divide roles of emulators and real devices

  56. Conclusion

  57. Message(Key takeaways)
 • We faced problems in testing environments, and

    need faster, more stable QA feedback
 • Following the Mobile Testing Pyramid, we tried creating parallelized test environment using Azure, Docker, and Android Emulator
 • There are challenges to be solved apart from the test environment for faster and more stable QA feedback

  58. None