$30 off During Our Annual Pro Sale. View Details »

Creating parallelized Android UITest (Appium) environment using Azure, Docker and Android emulator

Creating parallelized Android UITest (Appium) environment using Azure, Docker and Android emulator

Agile & Automation Days 2019 #aadays2019
https://aadays.pl/speakers/tadashi-nemoto/

Tadashi Nemoto

October 29, 2019
Tweet

More Decks by Tadashi Nemoto

Other Decks in Technology

Transcript

  1. View Slide

  2. Introduction 

    ● Tadashi Nemoto

    ● QA Automation Engineer / Mercari, inc.

    ○ Selenium / Appium / XCUITest

    ○ CI / CD pipeline

    ○ Cloud-based test environment

    tadashi0713 tadashi-nemoto

    View Slide

  3. Agenda

    ● About Mercari, inc.

    ● About our automation tests and problems

    ● What is Docker-Android, 

    and Docker-Android in the cloud

    ● Demo

    ● Challenges to be solved


    View Slide

  4. https://tech.mercari.com/entry/2018/12/28/175147

    View Slide

  5. About Mercari, inc.

    View Slide

  6. About
    February 1st, 2013
    Established
    Tokyo, Sendai,
    Fukuoka, Palo Alto,
    Portland, Boston
    Offices
    1800+
    Including
    subsidiaries
    Headcount
    6

    View Slide

  7. allow individuals to buy and sell items and share
    information through their phones. We created
    “Mercari,” a C2C marketplace app where users can use
    their smartphones to easily buy and sell secondhand
    goods. By linking sellers who own items to buyers who
    see the value in these objects, our goal is to provide an
    easy, fun, and safe and secure service that creates new
    value and redefines the cycle of consumption.
    In modern society, where mass production and mass
    consumption have become commonplace, the value
    of items that fall into disuse each year is expected to
    grow to ¥7.6 trillion in Japan alone.* Of these items,
    there are many that are no longer useful to one
    person, but could be used by someone else who
    values them. The Mercari Group aims to use the
    popularization of smartphones and social media to
    Create value in a global marketplace
    where anyone can buy & sell
    Mission
    Source: Ministry of Economy, Trade and Industry (April 2018). Estimated total value of items described as “items that became no longer needed over the past year”. Excludes aggregate value of automobiles,
    motorcycles and mopeds
    7

    View Slide

  8. What Is Mercari?
    ●Service start: July 2013
    ●OS: Android, iOS
    *Can also use from web browsers
    ●Usage fee: Free
    *Commission fee for sold items: 10% of the sales price
    ●Regions/languages supported: Base specs for
    Japan/Japanese
    Many sellers enjoy having the items they no longer
    need purchased and used by buyers who need them,
    and buyers enjoy the feeling of hunting for treasure as
    they search through unique and diverse items for lucky
    finds. In addition to buying and selling, users actively
    communicate through the buyer/seller chat and the
    “Like” feature.
    The Mercari app is a C2C marketplace where individuals
    can easily sell used items. We want to provide both
    buyers and sellers with a service where they can enjoy
    safe and secure transactions. Mercari offers a unique
    customer experience, with a transaction environment
    that uses the payments Mercari holds in escrow, and
    simple and affordable shipping options.
    8

    View Slide

  9. By the Numbers (JP/Full Year)
    232
    346.8
    GMV¹
    490.2 B

    In: billion JPY
    FY
    06/2017
    FY
    06/2018
    21.2
    33.4
    Net Sales
    46.2 B

    In: billion JPY
    FY
    06/2017
    FY
    06/2018
    8.45
    10.75
    MAU²
    13.57 M

    In: Million
    people
    FY
    06/2017
    FY
    06/2018
    Source: Internal documents, from FY2018.6 Presentation Material

    1. GMV after cancellation

    2. Monthly Active Users in June. Number of registered users that used our app in the month

    490.2
    FY
    06/2019
    46.2
    FY
    06/2019
    13.57
    FY
    06/2019
    9

    View Slide

  10. Building a new ecosystem through Merpay
    Merpay is a mobile payments service operated
    by Merpay, Inc. The company belongs to the
    Mercari Group, widely recognized for its
    service “Mercari,” the top C2C marketplace
    app in Japan.
    Money earned by selling unwanted items on
    the Mercari app and money charged to the
    app through users’ bank accounts can be
    used to make payments at stores and on the
    Mercari app itself.
    Our future goal is to provide a comprehensive
    financial service by combining primary and
    secondary distribution based on data, and
    supporting the business of merchants while
    utilizing users’ credit information, such as
    their transaction history and ratings.
    Primary
    distribution
    Manufactures, retail,
    distributors, etc.
    Secondary
    distribution
    Marketplace app
    Attract customers
    Marketing
    Transaction data
    Future Business Development
    Building an Ecosystem

    View Slide

  11. “The Selling App.” Mercari’s mission is to “Create value in a global
    marketplace where anyone can buy & sell,” and
    the year after our founding, we began to expand
    overseas. We started our business in the US in
    April 2014, and we adapted our branding,
    improved our UI/UX, and created a delivery
    network to match local preferences and the
    characteristics of the US market. We recognize
    that succeeding in the US, with its enormous and
    diverse population, is an important milestone to
    achieve our mission, so that is where we are
    devoting our energy.
    11
    Future Business Development
    Strengthening Overseas Expansion
    To Make Selling Easier than Buying.

    View Slide

  12. About our automation tests
    and problems

    View Slide

  13. View Slide

  14. UI・E2E Tests of Android

    Build & upload apk
    Kick tests

    View Slide

  15. View Slide

  16. View Slide

  17. Problems

    ● Increasing test execution time

    ● Maintenance cost of real devices

    ● Test failures due to differences of real devices

    (e.g. display size, OS version, etc.)

    ● Maintenance cost of local environments

    (e.g. Appium, Ruby, etc.)

    ● These problems often made tests flaky and unreliable


    View Slide

  18. Our headcount tripled in 3 years

    1,826
    376
    462
    555 596
    712
    795
    1,014
    1,140
    1,357
    1,639
    1,786
    Changes in total
    headcount
    1Q 2Q 3Q 4Q
    FY2016.6
    1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q
    FY2017.6 FY2018.6 FY2019.6
    from Mercari Bold Challenge

    View Slide

  19. The number of engineers also
    increased greatly

    FY2018 FY2019
    FY2017
    FY2016
    from Mercari Bold Challenge

    View Slide

  20. We need faster and
    more stable QA feedback


    View Slide

  21. Test Pyramid

    https://martinfowler.com/bliki/TestPyramid.html

    View Slide

  22. Mobile Testing Pyramid

    http://elementalselenium.com/tips/71-mobile-testing-pyramid

    View Slide

  23. Mobile Testing Pyramid

    http://elementalselenium.com/tips/71-mobile-testing-pyramid

    View Slide

  24. Azure, Docker, Android Emulator


    View Slide

  25. Azure, Docker, Android Emulator


    View Slide

  26. Summary

    ● We first created a local environment (local PC, real devices)
    to run Android E2E・UI Tests

    ● We faced some problems, and needed faster and more
    stable QA feedback

    ● Based on the Mobile Testing Pyramid, we tried creating
    parallelized test environment using Azure, Docker, and
    Android Emulator


    View Slide

  27. What is Docker-Android,
    and Docker-Android in the cloud

    View Slide

  28. Advantages of using

    ● Docker

    ● Android Emulator

    ● Cloud server


    View Slide

  29. Docker

    ● Easy to scrap and build

    ● Easily recreate same

    environment(local, CI)

    ● Isolated environment

    ● Infrastructure as Code


    View Slide

  30. Cloud server(IaaS)

    ● Reduces maintenance cost compared to on-premises
    server

    ● Easy to scale

    ● Cost determined by usage


    View Slide

  31. Android Emulator

    ● Can reduce maintenance cost

    of real devices

    ● Clean state

    ● Can emulate

    ○ SMS

    ○ Push notification


    View Slide

  32. Docker-Android

    github.com/docker-android


    View Slide

  33. Advantages of Docker-Android

    ● Uses fast Android x86 emulator

    ● Starts Appium server inside container

    ● Can connect to Selenium Grid easily

    ● Remote connect using noVNC

    ● Can capture videos


    View Slide

  34. Docker-Android Selenium Grid


    View Slide

  35. How to run Docker-Android

    in the cloud 

    ● Android x86 emulator needs hardware virtualization

    ● Can only run limited cloud service / Instances

    ○ Bare metal Instance(AWS)

    ○ Nested virtualization(Azure)


    View Slide

  36. AWS

    https://aws.amazon.com/jp/blogs/aws/new-amazon-ec2-bare-metal-instances-with-direct-access-to-hardware/

    View Slide

  37. Azure (Nested Virtualization)

    https://azure.microsoft.com/en-us/blog/nested-virtualization-in-azure/

    View Slide

  38. Run Docker-Android in
    Nested Virtualization

    https://stackoverflow.com/questions/49099428/is-it-possi
    ble-to-run-kvm-on-azure

    View Slide

  39. Summary

    ● With Docker-Android and Cloud server(Azure), we could create
    parallelized testing environments

    ● We can also benefit from using

    ○ Docker

    ○ Android Emulator

    ○ Cloud server


    View Slide

  40. Demo

    View Slide

  41. https://github.com/tadashi0713/webdriverio-appium-on-docker-android
    Demo

    View Slide

  42. Challenges to be solved

    View Slide

  43. Challenges to be solved

    ● Test report

    ● Optimise distributed tests

    ● Server load on QA environment

    ● Divide role of emulators and real devices


    View Slide

  44. Test report

    View Slide

  45. Rspec HTML Reporter

    ● Problems

    ○ Difficult to find history and trend

    ○ Generate reports by number of parallel


    View Slide

  46. Allure


    View Slide

  47. Optimise distributed tests

    View Slide

  48. https://docs.knapsackpro.com/2018/improve-circleci-para
    llelisation-for-rspec-minitest-cypress
    Optimise distributed tests


    View Slide

  49. https://docs.knapsackpro.com/2018/improve-circleci-para
    llelisation-for-rspec-minitest-cypress
    Knapsack (Pro)


    View Slide

  50. Server load on
    QA environments

    View Slide

  51. ● Solution

    ○ Use mock server instead

    ○ Improve QA environment infracture

    Server load to QA Environments

    API
    server

    (QA)


    View Slide

  52. Divide roles of
    emulators and real devices

    View Slide

  53. Mobile Testing Pyramid

    http://elementalselenium.com/tips/71-mobile-testing-pyramid

    View Slide

  54. Real devices: real-time conditions(Headspin)

    https://tech.mercari.com/entry/2019/02/18/173236

    View Slide

  55. Summary

    ● We still need to solve the below challenges for faster and
    more stable QA feedback

    ○ Test report

    ○ Optimise distributed tests

    ○ Server load on QA environment

    ○ Divide roles of emulators and real devices


    View Slide

  56. Conclusion

    View Slide

  57. Message(Key takeaways)

    ● We faced problems in testing environments, and need faster,
    more stable QA feedback

    ● Following the Mobile Testing Pyramid, we tried creating
    parallelized test environment using Azure, Docker, and
    Android Emulator

    ● There are challenges to be solved apart from the test
    environment for faster and more stable QA feedback


    View Slide

  58. View Slide