Loading time testing and results visualisation of web games @ Heisenbug Moscow 2017

Presentation from SQA Days conference about loading time testing and results visualisation of web games

Transcript

1. Loading time testing and results visualisation of web games Andrejs Kalnačs Heisenbug Moscow, 9th December 2017
2. who am i 2 Lead Software Developer in Test @ Evolution Gaming 10 years of industry experience in QA Last 6 years – testing automation Java, Groovy, Bash, Perl, Tcl Evolution Gaming, Nokia, Accenture, Nokia Siemens Networks Riga Testing Automation Club meet-ups co-organiser English, Russian and Latvian language proficiency @TesterFromRiga
3. Evolution Gaming 3 The World leader in live dealer gaming 3 development locations – Riga, Tallinn, Amsterdam B2B product company
4. Agenda 4 Web based loading time testing Test data points visualisation
5. Use case of Evolution Gaming 5
6. Use case of Evolution Gaming 6 16 different games 3 test environments 2 browser cache states
7. Creation and execution of web based loading time testing
8. Front end performance tests 8 Front end loading time tests Front end action timing tests Front end frame rate tests
9. Browser automation using WebDriver API 9
10. Custom metric collection from client 10 Loading time Intermediate loading metrics Navigation Timing Metrics Proprietary API Log server
11. Navigation Timing 11
12. Interception & archiving of HTTP transactions 12 Proxy System under test HTTP HTTPS WebSocket
13. OWASP ZAP proxy 13 HTTPS support WebSocket support API Open Source Cross platform: Linux, Mac, Windows HAR
14. HTTP archive format 14 JSON Libraries Many viewers Storage solutions Diff solutions
15. Chrome 62 15 https://developers.google.com/web/updates/2017/08/devtools-release-notes#har-imports
16. Bandwidth throttling 16 Virtual or physical machine with two network interfaces Linux tc tcconfig tcset --device eth1 --rate 800k --direction outgoing tcset --device eth0 --rate 800k --direction incoming
17. Bandwidth throttling 17 Virtual or physical machine with two network interfaces Linux tc tcconfig tcset --device eth1 --rate 800k --delay 40ms \ --direction outgoing tcset --device eth0 --rate 800k --delay 40ms \ --direction incoming
18. Cloud? 18 Virtual hardware for desktop browsers Traffic throttling only at Sauce Labs & AWS No HAR produced
19. Local desktop & device farm 19 Support, maintenance & update of infrastructure, hardware & software Stability of devices Complexity of iOS testing automation
20. Execution from CI server 20 Selenium Hub Appium iOS node Appium Android node Proxy Bandwidth throttler Docker host
21. Tests results 21 + Information provided by API Calculated & aggregated metrics from HAR
22. Test data storage 22 Jenkins (job artefacts) Attachments to report File storage (FTP, SAMBA, WebDAV, or anything else) Dedicated .har storage (harstorage) SQL or NoSQL database
23. Results visualization
24. Why? 24 Show stakeholders how good or bad user experience is Compare your products Understand how new feature is affecting performance Visualise regression
25. Ops & finance tools 25
26. Use-case of performance tests results 26 point.measurement(”performance measurement of $site”) .time(timeNow()) .addField(“loadTime”, $loadTime);
27. 27 Prometheus Prometheus Prometheus push gateway Test job push get (interval based)
28. 28 Point point = Point.measurement(“ENDPOINT_TEST”) .time(System.currentTimeMillis(), TimeUnit.MILLISECONDS) .tag(“endpoint”, endpoint) .addField(“load_time”, value) .build(); SELECT avg(“load_time”) FROM “ENDPOINT_TEST” WHERE endpoint = ’my_endpoint’
30. 30 Templating
31. 31 Annotations
32. 32 Alerts
33. simplest-test-dashboard 33 Saves web page with content to disc and calculates size on disc https://youtu.be/SzelgA2NdEM
34. Grafana dashboard version control 34 https://youtu.be/3wF0yKBQSD0
35. Grafana dashboard version control (git) 35 Edit dashboard Copy in IDE Diff & throw out unneeded things git- workflow Grafana dashboard update via API
36. Key takeaways 36 Selenium self- instrumentation visualisation not a rocket science
37. Reference 37 https://github.com/andrejska/simplest-test-dashboard https://github.com/evolution-gaming/js-benchmark Related talks: Adding Performance Assertion to Standard Functional Testing (Marcel Verkerk, The Selenium Conf 2017) Web App Performance Testing Using WebDriver (Michael Kleptikov, The Selenium Conf 2012)
38. Thank you! Questions? Andrejs Kalnačs @TesterFromRiga