This presentation focuses on the best practices for mobile game testing and presents various tips and tricks how game developers can more efficiently test their mobile games.
rights reserved. 1 2 December 2014 Ville-Veikko Helppi Technical Product Manager [email protected] Learn the Best Practices in Mobile Game Testing W E B I N A R
rights reserved. 2 Agenda • Different Types of Testing and How Game Testing Differs from Regular App Testing? • Frameworks, Tools and Technologies – What is Important in Mobile Game Testing Today? • Top Priorities for Mobile Game Testing • Infrastructure for Successful Game Testing • Hands-on Example (Image Recognition) • Testdroid Update • Q&A
rights reserved. 4 Different Types of Testing? • Black-box testing vs. White-box testing • Methods used in Mobile Game Testing – Functional – Compatibility – Performance – Localization – Regression – Load – Smoke – Acceptance
rights reserved. 5 Defining Some Ways to Test • Func8onality of the game-‐play, features Func8onal Tes8ng • Done always when new features / regressions are included Regression Tes8ng • How game runs on different configura8ons Compa8bility Tes8ng • Different languages, geo-‐focused materials Localiza8on Tes8ng • Endurance test to determine if system can handle the load Soak Tes8ng • Measures the capacity of the system Stress Tes8ng • Simplest form of performance tes8ng, measures how system handles certain loads Load Tes8ng • Isola8on of the environment (e.g. from network) to see how game works Herme8c Tes8ng Feature -based testing Performance testing End-user testing
rights reserved. 6 The Old-Fashion Thinking Deterministic systems Identifiable objects Tool support(s) User-driven actions Simple inputs/outputs Virtualization Random setups/results No handle on objects No standards User reacts to input Complex inputs/outputs Real hardware <> <> <> <> <> <> High-‐level of automaBon Low-‐level of automaBon
rights reserved. 7 Test Automation as a Solution Random setups/results -> Easy to generate random events No handle on objects -> Variety of options (e.g. image recognition) No standards -> Variety of frameworks (cross-platform etc.) User reacts to input -> Test scripts can handle random events Complex inputs/outputs -> Test scripts can generate input/content Real hardware -> Availability of real mobile hardware Test Automation brings real value with variety of options to mobile game testing
rights reserved. 8 Challenges that Test Automation Tackles • Proliferation of Devices, Brands, Operators, Platforms • Coupling the Testing and Development process (Continuous Integration/Deployment/Delivery) • Different Types of Testing (as automated) – Stress, Load, Performance, Functionality etc. • Test script & test asset usability • Cons of Manual testing – Tedious effort, Expensive human resources, Error-prone (as done by humans) & Not reusable (done once)
rights reserved. 9 OS versions Chipsets CPU + GPU Tens of OEMs Memory Displays (resolutions, physical hw) OEM mods Other hardware (connectivity, calibration) Relation to other software Significant Benefits of Using Automation
rights reserved. 10 Correct behaviour across platforms and browsers Integration with web back-ends Typically need to fully utilize HW (CPU+GPU) Resource (e.g. battery) consumption OpenGL ES 2/3 Functionality and usability Screen orientations, connectivity, user profiles Robustness Robustness and security! Brand Compliances, verification with back- ends and data Different 'Verticals' Require Different Practices
rights reserved. 12 Important Aspects in Game Testing • User Interface and Functionality – UI layouts/elements, Menus, Resolution, Orientation, Overall visuality • Graphics Performance • Usability and User Experience – Responsiveness, Real Occuring Events • Multi-player/User • Social Integrations • Security and Liabilities
rights reserved. 13 User Interface and Functional • Compared to regular app testing, mobile game testing may seem to be more challenging – e.g. OpenGL ES content not action-identifiable • Graphics and UI – Identical look&feel across all devices and variants • Usability controls in the context of game • Navigation/game-play, Progress, Delays between "game-stages" • Animations, Fonts, Other graphics assets • Screen Resolution and Orientation
rights reserved. 14 Performance Testing • Performance, Load, Stress, Spike, Soak • Measure the real user experience • Execute tests with all possible devices (SW&HW) • Discover the impact of server loads, app performance (CPU/ GPU utilization)
rights reserved. 15 • Studies: Majority of users leave game due sluggish, general slowness or other bad performance behavior • It's very important to understand how game behaves on different devices, under different circumstances Performance Makes +/- UX
rights reserved. 16 Building the User Experience Understanding How quickly gamer assimilates what the game is about, how to fully play with it, what are the functionalities. Usability The main User Interface, menus, design and how usable those are Playing Experience Intuition to follow the game and go forward with its instructions and game-play Monetization Are gamers willing to buy – invest – in game • Graphics performance capability is the most important thing for mobile games when forming UX • Too many games fail to work properly with low- end/mid-range devices due hardware performance • The real understanding of game's performance capability can be only figured out by testing on real devices
rights reserved. 17 • End-to-end between tests, AI or more simple implementations • Optimize the data communication between server and clients • Test simultaneously on several (different) devices • Check what happens if game gets interrupted (platform) • Location-based features – how to properly mock up those? • Hermetic environment – what happens? Multi-Player/User Features
rights reserved. 18 • Important part of the after- engagement (how gamers share their results) • Many of OEM branded devices provide certain SoMe apps pre-installed (version?) • Connectivity, login and data transferring are tested in context of game • Hermetic environment – what happens? Social Integrations
rights reserved. 23 Real Devices vs. Emulators • Emulators/simulators won’t help you to test… – User Experience and Usability – Hardware – Software – Infrastructure 0 % = the percentage of your app users that use emulator to run your app! Why Real Devices are Must-to-Have?
rights reserved. 27 Architecture / Infrastructure GAME Game Engine Android iOS ... API & services Server Database App Middleware Platform Connectivity/Network Back-end
rights reserved. 28 Isolated, Hermetic Testing GAME Game Engine Android iOS ... API & services Server Database Herme8c Environment App Middleware Platform Connectivity/Network Back-end
rights reserved. 29 Compatibility & Porting Testing GAME Game Engine Android iOS ... API & services Server Database Real Devices App Middleware Platform Connectivity/Network Back-end
rights reserved. 30 Smoke/Reference Testing GAME Game Engine Android iOS ... API & services Server Database Reference Environment App Middleware Platform Connectivity/Network Back-end
rights reserved. 31 Back-End Performance Testing GAME Game Engine Android iOS ... API & services Server Database Massive no. of users and significant load App Middleware Platform Connectivity/Network Back-end
rights reserved. 35 • Continuous Integration (e.g. Jenkins) • Appium framework (for Android and iOS) • No need for devices – we'll use devices on Testdroid Cloud service • Test scripts / test cases for game Setup
rights reserved. 39 Test Script Test Case Desired CapabiliBes { “device”: “Android”, “app”: “/Users/user/ApiDemos.apk” “app-package”: “com.example.android.apis” “app-activity”: “.ApiDemos” } Desired Capabilities / Android Example
rights reserved. 41 Test Script Test Case Desired CapabiliBes Appium @ Localhost WebDriver h[p://localhost_4723/wd/hub Appium Server 4723 Device Example
rights reserved. Test Script Test Case Desired CapabiliBes From Localhost to Testdroid Cloud WebDriver h[p://localhost_4723/wd/hub *Testdroid Caps h[p://appium.testdroid.com/wd/hub Example
rights reserved. Test Script Test Case Desired CapabiliBes From Localhost to Testdroid Cloud WebDriver { “testdroid_username”: “[email protected]”, “testdroid_password”: “p4s$w0rd”, “testdroid_project”: “My First Project”, “testdroid_testrun”: “Test 1”, “testdroid_device”: “iPad Mini 7.0.4 A1432”, “testdroid_app”: “http://domain.com/app_v1.ipa” . . “app”: “com.bitbar.testdroid.BitbarIOSSample” } Example
rights reserved. 59 • Test Script contains the following 'steps': 1. Identify the used platform (Android / iOS) 2. Search for 'Goldmine' graphical element (using find_image to search for screen) 3. Script runs test to go to shop 4. Script runs test to buy a cannon 5. Script runs test to place the cannon (all these three are defined in .png files) 6. Script runs test to start a battle! 7. Game is brought down and test ends. Example: Clash of Clans
rights reserved. 66 Option 1: Public Device Cloud on-demand devices (multitenant) Mobile app testing on over 400+ real Android and iOS devices hosted by Bitbar Option 2: Private Device Cloud reserved devices Hosted by Bitbar in the US and/or Europe Devices chosen by and reserved only for the Customer Option 3: On-premise Device Cloud Automated mobile app testing on real Android and iOS de- vices hosted by the customer, usually 30-500 devices Testdroid Deployment Options
rights reserved. 68 Video recording & slides of this webinar will be soon available at Vimeo and Slideshare! Please visit at www.testdroid.com for more information. THANK YOU! W E B I N A R
rights reserved. 69 We operate the largest global device cloud with over 400 devices instantly available for developers Devices from the US, Europe, China, Japan and Korea – global devices for global customers
thirion
jnunemaker
holman
codingconduct
chikuwait
morganepeng
lauravandoore
smashingmag
sarafernandez
ivargrimstad
keavy
shlominoach
