How to create an automated test suite for the Chromium based app and the Unity app

Testing, E2E test, Selenium, Appium, Selenium-Unity-Driver

Transcript

1. How to create an
 automated test suite for the
 Chromium based app and the Unity app Ryo Kato SWET, DeNA
2. Ryo Kato @katryo DeNA CO., Ltd. Software Engineer / Test Engineer SWET Group SoftWare Engineer in Test
4. • DeNA’s new service (released in 2016) • Video game platform for PCs. • Runs on Windows • The games are made of Unity, Unreal Engine, etc. • AndApp client is made with Chromium What is AndApp? http://www.andapp.jp/
5. We can assist you AndApp
 dev team Me I have never developed an E2E test for native apps One day…
6. Our goal Create a software test suite about payment
 for AndApp The payment is done on the combination of AndApp client and the Unity app
7. How users purchase an item I want to purchase this item ! Game window Before the payment Made with Unity
8. Game window Credit card form window appears Inputting my credit card number… Made with Chromium How users purchase an item
9. I have purchased the item! Game window The window
 disappears After the payment How users purchase an item
10. 1. The user starts the game app (Unity) 2. He/she pushes the purchase button on the game app 3. Credit card form window (AndApp client) opens 4. He/she inputs credit card number on the AndApp client Control the Unity app Control the AndApp client How users purchase an item
11. Control the Chromium based app
 and the Unity apps
 on Windows !?
 I had never done such a task
12. But we finally managed to develop a
 test suite for the Chromium based app
 and the Unity app!
13. Our problems 1.Two native apps 2.Complicated procedures 3.Unstable tests
14. Our problems 1.Two native apps 2.Complicated procedures 3.Unstable tests
15. Two native apps How can we control them at the same time?
16. How to control Chromium-based apps Off course you can use Selenium to control Chromium apps. ⭕
17. Controlling Unity app with Appium • You can control Unity apps with Appium • Microsoft provides WinAppDriver for Appium ⭕ Appium
 with WinAppDriver
18. Problem about controlling the Unity app • Appium does not offer any way to identify an element. • You can click the point at (x:100, y:200) using Appium, but you cannot click the button by specifying the ID. • (Appium fork, Selenium-Unity-Driver offers the way to identify element on Unity app, but it is not available for Windows apps.) Click the button 
 with id: “ok” ! Click the button
 at the point of (x: 100, y: 200) ! ⭕ ❌
19. Confusion between two apps • You cannot identify an element on the Unity app. • But you can identify an element on the Chromium based app Controlling different apps is too confusing!
20. Solution for the confusion between two apps We implemented the same interface for the Chromium based app and the Unity app using Polymorphism
21. These methods have different implementation,
 but have the same name, same interface
22. Developers can use these methods in the same way
 without thinking about the difference
23. We managed to automate the
 Chromium based app and the Unity app

24. Our problems 1.Two native apps 2.Complicated procedures 3.Unstable tests
25. Complicated procedures • We have to control multiple apps at the same time • AndApp client has multiple windows • Multiple window multiple app testing is more difficult than browser testing
26. Screenplay pattern •A design pattern for E2E tests
 using Page, Actor, Ability, Task, Action, and Question. •Useful for complicated web app tests •I learned a lot from screenplay pattern! •We did not adopt whole the features
27. We adopted some parts of screenplay pattern • Our framework uses Task, Action, and Page. • Our framework does not use Actor, Ability, and Question because they are not suitable for our test cases. • Our framework uses Operation that the screenplay pattern does not use. • We used RSpec, but other testing frameworks are also okay Task Action Page Task is made up of actions Action interacts with a page Operation Operation is made up of tasks
28. Page classes have
 instance methods
 that represent elements
 on the page card_number_input confirm_button Page class
29. We implemented
 simple
 actions in
 ACTION classes Action class
30. You can define a task with a series of ACTIONS connecting actions with dots using method chain Task class
31. Creating page objects Procedures are implemented with a series of TASKS Operation class
32. Test code became easier
33. Our problems 1.Two native apps 2.Complicated procedures 3.Unstable tests
34. Unstable tests When AndApp client dev team changed the client’s behavior, 
 payment dev team’s test fails. Payment
 dev team AndApp client
 dev team We changed the behaviour of AndApp client. Hey, our test failed
 because of other team’s
 development! AndApp is developed under multiple teams,
 so it is almost impossible to follow the changes made by other teams.
35. When a test fails… We have to identify the reason and fix it.
 However, identifying the reason of the failure is not an easy task We started recording AndApp client’s and the Unity app’s logs, but it was not very useful enough to identify the reason.
36. Solution for identifying the reasons for the failures We started recording videos that capture the screen of the Windows machine From the video,
 I found that
 AndApp clinent’s 
 UI has been changed
 and this is the reason for the failure!
37. Demo
38. How it works 1. Jenkins job starts 2. It starts recording when each test case begins,
 it finishes recording when the test case finishes. 3. After the tests, it uploads all the video files
 to the shared storage 4. It notify us
 with Slack
39. We solved all the problems

40. Sum up • We Used Selenium for the Chromium based app, Appium for the Unity app • We implemented the same interface methods using polymorphism • We developed the test framework adopting some features from screenplay pattern • We started recording the video to identify the reasons for failures