Loading time testing and results visualisation of web games @ TAPOST 2017

Presentation from TAPOST conference about loading time testing and results visualisation of web games
Visualisation using InfluxDB and Grafana

Transcript

1. Loading time testing and results visualisation of web games Andrejs Kalnačs TAPOST, 12th October 2017
2. who am i 2 • Lead Software Developer in Test @ Evolution Gaming • 10 years of industry experience in QA • Many different roles • Wide business domains experience • Last 6 years – testing automation • Java, Perl, Bash, Groovy, Tcl • Evolution Gaming, Nokia, Accenture, Nokia Siemens Networks • Co-organizer of Riga Testing Automation Club meet-ups
3. Evolution Gaming 3 • The World leader in live dealer gaming • 3 development locations – Riga, Tallinn, Amsterdam • B2B product company
4. Agenda 4 Web based loading time testing Practical guide in test data points visualisation
5. Use case of Evolution Gaming 5
6. Use case of Evolution Gaming 6 16 different games 3 test environments 2 browser cache states
7. Creation and execution of web based loading time testing
8. Browser automation using WebDriver API 8
9. Custom metric collection from client 9 • Loading time • Intermediate loading metrics • Component/service loading metrics Proprietary API
10. Interception & archiving of HTTP transactions 10 .har • W3C draft • JSON format • Libraries • Many viewers • Storage solutions • Diff solutions • http://har.tech OWASP ZAP proxy • HTTP support • HTTPS support • WebSocket support • API • Open Source • Jenkins plugin • Runs on Linux, Mac OS X & Windows
11. Bandwidth throttling 11 • Virtual or physical machine with two network interfaces • Linux • tc • iptables • tcconfig
12. Local desktop & device farm 12 Why cloud testing service won’t work? • Virtual hardware for desktop browsers • Traffic throating only at Sauce Labs & AWS • No HAR produced Local solution minuses • Support, maintenance & update of infrastructure, hardware & software • Stability of devices • Complexity of iOS testing automation
13. Execution from CI server 13 Selenium Hub Appium iOS node Appium Android node Proxy Bandwidth throttler Docker host Docker host
14. Tests results 14 + Information provided by API Calculated & aggregated metrics from HAR
15. Test data storage 15 • Jenkins (job artefacts) • Attachments to report • File storage (FTP, SAMBA, WebDAV, or anything else) • Dedicated .har storage (harstorage) • SQL or NoSQL database
16. Results visualization
17. Why? 17 • Show stack holders how good or bad user experience is • Compare your products • Visualise regression • Understand how new feature is affecting performance
18. Ops & finance tools 18 • Time Series Data Base • TSDB visualisation tools • Data point vs vector
19. Time series data base 19 • Open source* • Easy to use • SQL-like query language - InfluxQL • HTTP API • Bindings (Java, Python, .NET, Ruby, Node.JS, PHP) • Data retention policy
20. Visualisation 20 • Open source • Many TSDB integrations • Different customisable dashboards • API • Alerting system • Email • Slack (Amazon S3 or WebDav needed)
21. simplest-test-dashboard 21 • Saves web page with content to disc and calculates it size • Get it here: https://github.com/andrejska/simplest-test-dashboard runner.sh http://tapost.org [comment] site_size tag: site: http://www.tapost.org field: value: 300000 comment: release 2017.10.10 date: unix_timestamp_in_ns
22. Grafana templating & annotations 22 https://youtu.be/SzelgA2NdEM
23. Riga Testing Automation Club 23 Thursday, October 26, 2017 @ 18:45 Artem Yeroshenko – New Features in Allure 2.0 https://www.meetup.com/Riga-Test-Automation-Club
24. Thank you! Any questions? andrejska Andrejs Kalnačs @TesterFromRiga