Loading time testing of web games – set-up, execution and visualization of results @ TestingStage '18

Slides from my talk at TestingStage '18 which is biggest testing conference in Ukraine.

Transcript

1. Loading time testing of web games – set-up, execution and visualization of results Andrejs Kalnačs 13th April 2018
2. who am i 2 Lead Software Developer in Test at Evolution Gaming 10 years of industry experience in QA Evolution Gaming, Nokia, Accenture, Nokia Siemens Networks Java, Groovy, Bash Riga Testing Automation Club meet-ups co- organiser @TesterFromRiga
3. Evolution Gaming 3 The World leader in live dealer gaming 3 development locations – Riga, Amsterdam, Tallinn
4. Agenda 4 Set-up and execution of loading time tests Tests results visualisation
5. Use case of Evolution Gaming 5
6. Science 101 6
7. Science 102 7
8. Science 201 8 2000 2013 12 seconds 8 seconds 9 seconds
9. Use case of Evolution Gaming 9 20+ different games 3 test environments 2 browser cache states
10. Creation and execution of web based loading time testing
11. Front end performance tests 11 Front end loading time tests Front end action timing tests Front end frame rate tests
12. Golden rule 12 Don’t call WebDriver during measurements
13. Browser automation using WebDriver API 13
14. Custom metric collection from client 14 Loading time Intermediate loading metrics Navigation Timing Metrics Proprietary API Log server
15. Interception & archiving of HTTP transactions 15 Proxy System under test HTTP HTTPS WebSocket
16. OWASP ZAP proxy 16 HTTPS support WebSocket support API Open Source Cross platform: Linux, Mac, Windows HAR
17. HTTP archive format 17 JSON Libraries Many viewers Storage solutions Diff solutions
18. Chrome 62 18 https://developers.google.com/web/updates/2017/08/devtools-release-notes#har-imports
19. Bandwidth throttling 19 Virtual or physical machine with two network interfaces Linux tc tcconfig tcset --device eth1 --rate 800k --direction outgoing tcset --device eth0 --rate 800k --direction incoming
20. Bandwidth throttling 20 Virtual or physical machine with two network interfaces Linux tc tcconfig tcset --device eth1 --rate 800k --delay 40ms \ --direction outgoing tcset --device eth0 --rate 800k --delay 40ms \ --direction incoming
21. Cloud? 21 Virtual hardware for desktop browsers Traffic throttling only at Sauce Labs & AWS No HAR produced
22. Local desktop & device farm 22 Support, maintenance & update of infrastructure, hardware & software Stability of devices Complexity of iOS testing automation
23. Execution from CI server 23 Selenium Hub Appium iOS node Appium Android node Proxy Bandwidth throttler Docker host
24. Tests results 24 + Information provided by API Calculated & aggregated metrics from HAR
25. Results visualization
26. Why? 26 Show stakeholders how good or bad user experience is Compare your products Understand how new feature is affecting performance Visualise regression
27. Ops & finance tools 27
28. 28 Point point = Point.measurement(“ENDPOINT_TEST”) .time(System.currentTimeMillis(), TimeUnit.MILLISECONDS) .tag(“endpoint”, endpoint) .addField(“load_time”, value) .build(); SELECT avg(“load_time”) FROM “ENDPOINT_TEST” WHERE endpoint = ’my_endpoint’
31. Templating 31
32. 32 Annotations
33. 33 Alerts
34. Key takeaways 34 Selenium self- instrumentation visualisation not a rocket science
35. Reference 35 https://github.com/andrejska/simplest-test-dashboard https://github.com/evolution-gaming/js-benchmark Related talks: Adding Performance Assertion to Standard Functional Testing (Marcel Verkerk, The Selenium Conf 2017) Web App Performance Testing Using WebDriver (Michael Kleptikov, The Selenium Conf 2012)
36. Thank you! Questions? andrejska Andrejs Kalnačs @TesterFromRiga