$30 off During Our Annual Pro Sale. View Details »

How to create an automated test suite for the Chromium based app and the Unity app

katryo
March 28, 2017

How to create an automated test suite for the Chromium based app and the Unity app

Testing, E2E test, Selenium, Appium, Selenium-Unity-Driver

katryo

March 28, 2017
Tweet

More Decks by katryo

Other Decks in Programming

Transcript

  1. How to create an

    automated test suite for the

    Chromium based app and the Unity app
    Ryo Kato
    SWET, DeNA

    View Slide

  2. Ryo Kato @katryo
    DeNA CO., Ltd.
    Software Engineer / Test Engineer
    SWET Group
    SoftWare Engineer in Test

    View Slide

  3. View Slide

  4. • DeNA’s new service (released in 2016)
    • Video game platform for PCs.
    • Runs on Windows
    • The games are made of Unity, Unreal
    Engine, etc.
    • AndApp client is made with Chromium
    What is AndApp?
    http://www.andapp.jp/

    View Slide

  5. We can assist
    you
    AndApp

    dev team
    Me
    I have never
    developed an E2E
    test for native apps

    One day…

    View Slide

  6. Our goal
    Create a software test suite about
    payment

    for AndApp
    The payment is done on the combination of
    AndApp client and the Unity app

    View Slide

  7. How users purchase an item
    I want to purchase
    this item !
    Game window
    Before the payment
    Made with
    Unity

    View Slide

  8. Game window Credit card form
    window
    appears
    Inputting my credit
    card number…
    Made with
    Chromium
    How users purchase an item

    View Slide

  9. I have purchased
    the item!
    Game window The window

    disappears
    After the payment
    How users purchase an item

    View Slide

  10. 1. The user starts the game app (Unity)
    2. He/she pushes the purchase button on the game app
    3. Credit card form window (AndApp client) opens
    4. He/she inputs credit card number on the AndApp client
    Control the Unity app
    Control the AndApp client
    How users purchase an item

    View Slide

  11. Control the Chromium based app

    and the Unity apps

    on Windows !?

    I had never done such a task

    View Slide

  12. But we finally managed to develop a

    test suite for the Chromium based app

    and the Unity app!

    View Slide

  13. Our problems
    1.Two native apps
    2.Complicated procedures
    3.Unstable tests

    View Slide

  14. Our problems
    1.Two native apps
    2.Complicated procedures
    3.Unstable tests

    View Slide

  15. Two native apps
    How can we control them at the same time?

    View Slide

  16. How to control Chromium-based apps
    Off course you can use Selenium to control
    Chromium apps.


    View Slide

  17. Controlling Unity app with Appium
    • You can control Unity apps with Appium
    • Microsoft provides WinAppDriver for Appium

    Appium

    with WinAppDriver

    View Slide

  18. Problem about controlling the Unity app
    • Appium does not offer any way to identify an
    element.
    • You can click the point at (x:100, y:200) using
    Appium, but you cannot click the button by
    specifying the ID.
    • (Appium fork, Selenium-Unity-Driver offers the
    way to identify element on Unity app, but it is
    not available for Windows apps.)
    Click the button 

    with id: “ok” !
    Click the button

    at the point of (x: 100, y: 200) !


    View Slide

  19. Confusion between two apps
    • You cannot identify an
    element on the Unity app.
    • But you can identify an
    element on the Chromium
    based app
    Controlling
    different apps is
    too confusing!

    View Slide

  20. Solution for the confusion between two apps
    We implemented the same interface for the Chromium
    based app and the Unity app using Polymorphism

    View Slide

  21. These methods have different implementation,

    but have the same name, same interface

    View Slide

  22. Developers can use these methods in the same way

    without thinking about the difference

    View Slide

  23. We managed to automate the

    Chromium based app and the Unity app


    View Slide

  24. Our problems
    1.Two native apps
    2.Complicated procedures
    3.Unstable tests

    View Slide

  25. Complicated procedures
    • We have to control multiple apps
    at the same time
    • AndApp client has multiple
    windows
    • Multiple window multiple app
    testing is more difficult than
    browser testing

    View Slide

  26. Screenplay pattern
    •A design pattern for E2E tests

    using Page, Actor, Ability, Task, Action, and
    Question.
    •Useful for complicated web app tests
    •I learned a lot from screenplay pattern!
    •We did not adopt whole the features

    View Slide

  27. We adopted some parts of screenplay pattern
    • Our framework uses Task, Action,
    and Page.
    • Our framework does not use Actor,
    Ability, and Question because they
    are not suitable for our test cases.
    • Our framework uses Operation that
    the screenplay pattern does not use.
    • We used RSpec, but other testing
    frameworks are also okay
    Task
    Action
    Page
    Task is made up of actions
    Action interacts with a page
    Operation
    Operation is made up of tasks

    View Slide

  28. Page classes have

    instance methods

    that represent elements

    on the page
    card_number_input
    confirm_button
    Page class

    View Slide

  29. We implemented

    simple

    actions in

    ACTION classes
    Action class

    View Slide

  30. You can define a task with a series of ACTIONS
    connecting actions
    with dots using
    method chain
    Task class

    View Slide

  31. Creating page objects
    Procedures are implemented with a series of TASKS
    Operation class

    View Slide

  32. Test code became easier

    View Slide

  33. Our problems
    1.Two native apps
    2.Complicated procedures
    3.Unstable tests

    View Slide

  34. Unstable tests
    When AndApp client dev team changed the client’s behavior, 

    payment dev team’s test fails.
    Payment

    dev team
    AndApp client

    dev team
    We changed the
    behaviour of AndApp
    client.
    Hey, our test failed

    because of other
    team’s

    development!
    AndApp is developed under multiple teams,

    so it is almost impossible to follow the changes made by other teams.

    View Slide

  35. When a test fails…
    We have to identify the reason and fix it.

    However, identifying the reason of the failure is not an easy task
    We started recording AndApp client’s and the Unity app’s logs,
    but it was not very useful enough to identify the reason.

    View Slide

  36. Solution for identifying the reasons for the
    failures
    We started recording
    videos that capture the
    screen of the Windows
    machine
    From the video,

    I found that

    AndApp clinent’s 

    UI has been changed

    and this is the reason
    for the failure!

    View Slide

  37. Demo

    View Slide

  38. How it works

    1. Jenkins job starts 2. It starts recording
    when each test case
    begins,

    it finishes recording
    when the test case
    finishes.
    3. After the
    tests, it uploads
    all the video files

    to the shared
    storage
    4. It notify us

    with Slack

    View Slide

  39. We solved all the problems


    View Slide

  40. Sum up
    • We Used Selenium for the Chromium based app, Appium for the
    Unity app
    • We implemented the same interface methods using polymorphism
    • We developed the test framework adopting some features from
    screenplay pattern
    • We started recording the video to identify the reasons for failures

    View Slide