Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Introduction 
 ● Tadashi Nemoto
 ● QA Automation Engineer / Mercari, inc.
 ○ Selenium / Appium / XCUITest
 ○ CI / CD pipeline
 ○ Cloud-based test environment
 tadashi0713 tadashi-nemoto

Slide 3

Slide 3 text

Agenda
 ● About Mercari, inc.
 ● About our automation tests and problems
 ● What is Docker-Android, 
 and Docker-Android in the cloud
 ● Demo
 ● Challenges to be solved


Slide 4

Slide 4 text

https://tech.mercari.com/entry/2018/12/28/175147

Slide 5

Slide 5 text

About Mercari, inc.

Slide 6

Slide 6 text

About February 1st, 2013 Established Tokyo, Sendai, Fukuoka, Palo Alto, Portland, Boston Offices 1800+ Including subsidiaries Headcount 6

Slide 7

Slide 7 text

allow individuals to buy and sell items and share information through their phones. We created “Mercari,” a C2C marketplace app where users can use their smartphones to easily buy and sell secondhand goods. By linking sellers who own items to buyers who see the value in these objects, our goal is to provide an easy, fun, and safe and secure service that creates new value and redefines the cycle of consumption. In modern society, where mass production and mass consumption have become commonplace, the value of items that fall into disuse each year is expected to grow to ¥7.6 trillion in Japan alone.* Of these items, there are many that are no longer useful to one person, but could be used by someone else who values them. The Mercari Group aims to use the popularization of smartphones and social media to Create value in a global marketplace where anyone can buy & sell Mission Source: Ministry of Economy, Trade and Industry (April 2018). Estimated total value of items described as “items that became no longer needed over the past year”. Excludes aggregate value of automobiles, motorcycles and mopeds 7

Slide 8

Slide 8 text

What Is Mercari? ●Service start: July 2013 ●OS: Android, iOS *Can also use from web browsers ●Usage fee: Free *Commission fee for sold items: 10% of the sales price ●Regions/languages supported: Base specs for Japan/Japanese Many sellers enjoy having the items they no longer need purchased and used by buyers who need them, and buyers enjoy the feeling of hunting for treasure as they search through unique and diverse items for lucky finds. In addition to buying and selling, users actively communicate through the buyer/seller chat and the “Like” feature. The Mercari app is a C2C marketplace where individuals can easily sell used items. We want to provide both buyers and sellers with a service where they can enjoy safe and secure transactions. Mercari offers a unique customer experience, with a transaction environment that uses the payments Mercari holds in escrow, and simple and affordable shipping options. 8

Slide 9

Slide 9 text

By the Numbers (JP/Full Year) 232 346.8 GMV¹ 490.2 B
 In: billion JPY FY 06/2017 FY 06/2018 21.2 33.4 Net Sales 46.2 B
 In: billion JPY FY 06/2017 FY 06/2018 8.45 10.75 MAU² 13.57 M
 In: Million people FY 06/2017 FY 06/2018 Source: Internal documents, from FY2018.6 Presentation Material
 1. GMV after cancellation
 2. Monthly Active Users in June. Number of registered users that used our app in the month
 490.2 FY 06/2019 46.2 FY 06/2019 13.57 FY 06/2019 9

Slide 10

Slide 10 text

Building a new ecosystem through Merpay Merpay is a mobile payments service operated by Merpay, Inc. The company belongs to the Mercari Group, widely recognized for its service “Mercari,” the top C2C marketplace app in Japan. Money earned by selling unwanted items on the Mercari app and money charged to the app through users’ bank accounts can be used to make payments at stores and on the Mercari app itself. Our future goal is to provide a comprehensive financial service by combining primary and secondary distribution based on data, and supporting the business of merchants while utilizing users’ credit information, such as their transaction history and ratings. Primary distribution Manufactures, retail, distributors, etc. Secondary distribution Marketplace app Attract customers Marketing Transaction data Future Business Development Building an Ecosystem

Slide 11

Slide 11 text

“The Selling App.” Mercari’s mission is to “Create value in a global marketplace where anyone can buy & sell,” and the year after our founding, we began to expand overseas. We started our business in the US in April 2014, and we adapted our branding, improved our UI/UX, and created a delivery network to match local preferences and the characteristics of the US market. We recognize that succeeding in the US, with its enormous and diverse population, is an important milestone to achieve our mission, so that is where we are devoting our energy. 11 Future Business Development Strengthening Overseas Expansion To Make Selling Easier than Buying.

Slide 12

Slide 12 text

About our automation tests and problems

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

UI・E2E Tests of Android
 Build & upload apk Kick tests

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

Problems
 ● Increasing test execution time
 ● Maintenance cost of real devices
 ● Test failures due to differences of real devices
 (e.g. display size, OS version, etc.)
 ● Maintenance cost of local environments
 (e.g. Appium, Ruby, etc.)
 ● These problems often made tests flaky and unreliable


Slide 18

Slide 18 text

Our headcount tripled in 3 years
 1,826 376 462 555 596 712 795 1,014 1,140 1,357 1,639 1,786 Changes in total headcount 1Q 2Q 3Q 4Q FY2016.6 1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q FY2017.6 FY2018.6 FY2019.6 from Mercari Bold Challenge

Slide 19

Slide 19 text

The number of engineers also increased greatly
 FY2018 FY2019 FY2017 FY2016 from Mercari Bold Challenge

Slide 20

Slide 20 text

We need faster and more stable QA feedback


Slide 21

Slide 21 text

Test Pyramid
 https://martinfowler.com/bliki/TestPyramid.html

Slide 22

Slide 22 text

Mobile Testing Pyramid
 http://elementalselenium.com/tips/71-mobile-testing-pyramid

Slide 23

Slide 23 text

Mobile Testing Pyramid
 http://elementalselenium.com/tips/71-mobile-testing-pyramid

Slide 24

Slide 24 text

Azure, Docker, Android Emulator


Slide 25

Slide 25 text

Azure, Docker, Android Emulator


Slide 26

Slide 26 text

Summary
 ● We first created a local environment (local PC, real devices) to run Android E2E・UI Tests
 ● We faced some problems, and needed faster and more stable QA feedback
 ● Based on the Mobile Testing Pyramid, we tried creating parallelized test environment using Azure, Docker, and Android Emulator


Slide 27

Slide 27 text

What is Docker-Android, and Docker-Android in the cloud

Slide 28

Slide 28 text

Advantages of using
 ● Docker
 ● Android Emulator
 ● Cloud server


Slide 29

Slide 29 text

Docker
 ● Easy to scrap and build
 ● Easily recreate same
 environment(local, CI)
 ● Isolated environment
 ● Infrastructure as Code


Slide 30

Slide 30 text

Cloud server(IaaS)
 ● Reduces maintenance cost compared to on-premises server
 ● Easy to scale
 ● Cost determined by usage


Slide 31

Slide 31 text

Android Emulator
 ● Can reduce maintenance cost
 of real devices
 ● Clean state
 ● Can emulate
 ○ SMS
 ○ Push notification


Slide 32

Slide 32 text

Docker-Android
 github.com/docker-android


Slide 33

Slide 33 text

Advantages of Docker-Android
 ● Uses fast Android x86 emulator
 ● Starts Appium server inside container
 ● Can connect to Selenium Grid easily
 ● Remote connect using noVNC
 ● Can capture videos


Slide 34

Slide 34 text

Docker-Android Selenium Grid


Slide 35

Slide 35 text

How to run Docker-Android
 in the cloud 
 ● Android x86 emulator needs hardware virtualization
 ● Can only run limited cloud service / Instances
 ○ Bare metal Instance(AWS)
 ○ Nested virtualization(Azure)
 


Slide 36

Slide 36 text

AWS
 https://aws.amazon.com/jp/blogs/aws/new-amazon-ec2-bare-metal-instances-with-direct-access-to-hardware/

Slide 37

Slide 37 text

Azure (Nested Virtualization)
 https://azure.microsoft.com/en-us/blog/nested-virtualization-in-azure/

Slide 38

Slide 38 text

Run Docker-Android in Nested Virtualization
 https://stackoverflow.com/questions/49099428/is-it-possi ble-to-run-kvm-on-azure

Slide 39

Slide 39 text

Summary
 ● With Docker-Android and Cloud server(Azure), we could create parallelized testing environments
 ● We can also benefit from using
 ○ Docker
 ○ Android Emulator
 ○ Cloud server


Slide 40

Slide 40 text

Demo

Slide 41

Slide 41 text

https://github.com/tadashi0713/webdriverio-appium-on-docker-android Demo

Slide 42

Slide 42 text

Challenges to be solved

Slide 43

Slide 43 text

Challenges to be solved
 ● Test report
 ● Optimise distributed tests
 ● Server load on QA environment
 ● Divide role of emulators and real devices


Slide 44

Slide 44 text

Test report

Slide 45

Slide 45 text

Rspec HTML Reporter
 ● Problems
 ○ Difficult to find history and trend
 ○ Generate reports by number of parallel


Slide 46

Slide 46 text

Allure


Slide 47

Slide 47 text

Optimise distributed tests

Slide 48

Slide 48 text

https://docs.knapsackpro.com/2018/improve-circleci-para llelisation-for-rspec-minitest-cypress Optimise distributed tests


Slide 49

Slide 49 text

https://docs.knapsackpro.com/2018/improve-circleci-para llelisation-for-rspec-minitest-cypress Knapsack (Pro)


Slide 50

Slide 50 text

Server load on QA environments

Slide 51

Slide 51 text

● Solution
 ○ Use mock server instead
 ○ Improve QA environment infracture
 Server load to QA Environments
 API server
 (QA)


Slide 52

Slide 52 text

Divide roles of emulators and real devices

Slide 53

Slide 53 text

Mobile Testing Pyramid
 http://elementalselenium.com/tips/71-mobile-testing-pyramid

Slide 54

Slide 54 text

Real devices: real-time conditions(Headspin)
 https://tech.mercari.com/entry/2019/02/18/173236

Slide 55

Slide 55 text

Summary
 ● We still need to solve the below challenges for faster and more stable QA feedback
 ○ Test report
 ○ Optimise distributed tests
 ○ Server load on QA environment
 ○ Divide roles of emulators and real devices


Slide 56

Slide 56 text

Conclusion

Slide 57

Slide 57 text

Message(Key takeaways)
 ● We faced problems in testing environments, and need faster, more stable QA feedback
 ● Following the Mobile Testing Pyramid, we tried creating parallelized test environment using Azure, Docker, and Android Emulator
 ● There are challenges to be solved apart from the test environment for faster and more stable QA feedback


Slide 58

Slide 58 text

No content