Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Creating parallelized Android UITest (Appium) environment using Azure, Docker and Android emulator

Creating parallelized Android UITest (Appium) environment using Azure, Docker and Android emulator

Agile & Automation Days 2019 #aadays2019

Tadashi Nemoto

October 29, 2019

More Decks by Tadashi Nemoto

Other Decks in Technology


  1. Introduction 
 • Tadashi Nemoto
 • QA Automation Engineer /

    Mercari, inc.
 ◦ Selenium / Appium / XCUITest
 ◦ CI / CD pipeline
 ◦ Cloud-based test environment
 tadashi0713 tadashi-nemoto
  2. Agenda
 • About Mercari, inc.
 • About our automation tests

    and problems
 • What is Docker-Android, 
 and Docker-Android in the cloud
 • Demo
 • Challenges to be solved

  3. About February 1st, 2013 Established Tokyo, Sendai, Fukuoka, Palo Alto,

    Portland, Boston Offices 1800+ Including subsidiaries Headcount 6
  4. allow individuals to buy and sell items and share information

    through their phones. We created “Mercari,” a C2C marketplace app where users can use their smartphones to easily buy and sell secondhand goods. By linking sellers who own items to buyers who see the value in these objects, our goal is to provide an easy, fun, and safe and secure service that creates new value and redefines the cycle of consumption. In modern society, where mass production and mass consumption have become commonplace, the value of items that fall into disuse each year is expected to grow to ¥7.6 trillion in Japan alone.* Of these items, there are many that are no longer useful to one person, but could be used by someone else who values them. The Mercari Group aims to use the popularization of smartphones and social media to Create value in a global marketplace where anyone can buy & sell Mission Source: Ministry of Economy, Trade and Industry (April 2018). Estimated total value of items described as “items that became no longer needed over the past year”. Excludes aggregate value of automobiles, motorcycles and mopeds 7
  5. What Is Mercari? •Service start: July 2013 •OS: Android, iOS

    *Can also use from web browsers •Usage fee: Free *Commission fee for sold items: 10% of the sales price •Regions/languages supported: Base specs for Japan/Japanese Many sellers enjoy having the items they no longer need purchased and used by buyers who need them, and buyers enjoy the feeling of hunting for treasure as they search through unique and diverse items for lucky finds. In addition to buying and selling, users actively communicate through the buyer/seller chat and the “Like” feature. The Mercari app is a C2C marketplace where individuals can easily sell used items. We want to provide both buyers and sellers with a service where they can enjoy safe and secure transactions. Mercari offers a unique customer experience, with a transaction environment that uses the payments Mercari holds in escrow, and simple and affordable shipping options. 8
  6. By the Numbers (JP/Full Year) 232 346.8 GMV¹ 490.2 B

    In: billion JPY FY 06/2017 FY 06/2018 21.2 33.4 Net Sales 46.2 B
 In: billion JPY FY 06/2017 FY 06/2018 8.45 10.75 MAU² 13.57 M
 In: Million people FY 06/2017 FY 06/2018 Source: Internal documents, from FY2018.6 Presentation Material
 1. GMV after cancellation
 2. Monthly Active Users in June. Number of registered users that used our app in the month
 490.2 FY 06/2019 46.2 FY 06/2019 13.57 FY 06/2019 9
  7. Building a new ecosystem through Merpay Merpay is a mobile

    payments service operated by Merpay, Inc. The company belongs to the Mercari Group, widely recognized for its service “Mercari,” the top C2C marketplace app in Japan. Money earned by selling unwanted items on the Mercari app and money charged to the app through users’ bank accounts can be used to make payments at stores and on the Mercari app itself. Our future goal is to provide a comprehensive financial service by combining primary and secondary distribution based on data, and supporting the business of merchants while utilizing users’ credit information, such as their transaction history and ratings. Primary distribution Manufactures, retail, distributors, etc. Secondary distribution Marketplace app Attract customers Marketing Transaction data Future Business Development Building an Ecosystem
  8. “The Selling App.” Mercari’s mission is to “Create value in

    a global marketplace where anyone can buy & sell,” and the year after our founding, we began to expand overseas. We started our business in the US in April 2014, and we adapted our branding, improved our UI/UX, and created a delivery network to match local preferences and the characteristics of the US market. We recognize that succeeding in the US, with its enormous and diverse population, is an important milestone to achieve our mission, so that is where we are devoting our energy. 11 Future Business Development Strengthening Overseas Expansion To Make Selling Easier than Buying.
  9. Problems
 • Increasing test execution time
 • Maintenance cost of

    real devices
 • Test failures due to differences of real devices
 (e.g. display size, OS version, etc.)
 • Maintenance cost of local environments
 (e.g. Appium, Ruby, etc.)
 • These problems often made tests flaky and unreliable

  10. Our headcount tripled in 3 years
 1,826 376 462 555

    596 712 795 1,014 1,140 1,357 1,639 1,786 Changes in total headcount 1Q 2Q 3Q 4Q FY2016.6 1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q FY2017.6 FY2018.6 FY2019.6 from Mercari Bold Challenge
  11. Summary
 • We first created a local environment (local PC,

    real devices) to run Android E2E・UI Tests
 • We faced some problems, and needed faster and more stable QA feedback
 • Based on the Mobile Testing Pyramid, we tried creating parallelized test environment using Azure, Docker, and Android Emulator

  12. Docker
 • Easy to scrap and build
 • Easily recreate

 environment(local, CI)
 • Isolated environment
 • Infrastructure as Code

  13. Android Emulator
 • Can reduce maintenance cost
 of real devices

    • Clean state
 • Can emulate
 ◦ SMS
 ◦ Push notification

  14. Advantages of Docker-Android
 • Uses fast Android x86 emulator

    Starts Appium server inside container
 • Can connect to Selenium Grid easily
 • Remote connect using noVNC
 • Can capture videos

  15. How to run Docker-Android
 in the cloud 
 • Android

    x86 emulator needs hardware virtualization
 • Can only run limited cloud service / Instances
 ◦ Bare metal Instance(AWS)
 ◦ Nested virtualization(Azure)

  16. Summary
 • With Docker-Android and Cloud server(Azure), we could create

    parallelized testing environments
 • We can also benefit from using
 ◦ Docker
 ◦ Android Emulator
 ◦ Cloud server

  17. Challenges to be solved
 • Test report
 • Optimise distributed

 • Server load on QA environment
 • Divide role of emulators and real devices

  18. Rspec HTML Reporter
 • Problems
 ◦ Difficult to find history

    and trend
 ◦ Generate reports by number of parallel

  19. • Solution
 ◦ Use mock server instead
 ◦ Improve QA

    environment infracture
 Server load to QA Environments
 API server

  20. Summary
 • We still need to solve the below challenges

    for faster and more stable QA feedback
 ◦ Test report
 ◦ Optimise distributed tests
 ◦ Server load on QA environment
 ◦ Divide roles of emulators and real devices

  21. Message(Key takeaways)
 • We faced problems in testing environments, and

    need faster, more stable QA feedback
 • Following the Mobile Testing Pyramid, we tried creating parallelized test environment using Azure, Docker, and Android Emulator
 • There are challenges to be solved apart from the test environment for faster and more stable QA feedback