Measuring Web Performance for Wikipedia using synthetic testing tools

Presentation for the Automationeers Assemble at Mozilla.

Wikimedia performance team blog: https://phabricator.wikimedia.org/phame/blog/view/7/
Gilles article on magic performance numbers: https://www.mediawiki.org/wiki/Wikimedia_Performance_Team/Perceived_Performance
Our WebPageTest setup: https://wikitech.wikimedia.org/wiki/Performance/WebPageTest
Our Browsertime/WebPageReplay setup: https://wikitech.wikimedia.org/wiki/Performance/WebPageReplay
WebPageTest metrics: https://grafana.wikimedia.org/dashboard/db/webpagetest
WebPageReplay metrics: https://grafana.wikimedia.org/dashboard/db/webpagereplay
RUM metrics: https://grafana.wikimedia.org/dashboard/db/navigation-timing
Phabricator: https://phabricator.wikimedia.org/tag/performance-team/

Transcript

1. @soulislove Automationeers Assemble: Performance at Wikipedia 2018-10-25
2. @soulislove Actually …
3. @soulislove Measuring Web Performance for Wikipedia using synthetic testing tools
4. @soulislove Peter Hedenskog Wikimedia Performance Team sitespeed.io @soulislove
5. @soulislove Wikimedia Performance team
6. @soulislove
7. @soulislove
8. @soulislove Wikipedia
9. @soulislove Vad? Vad? Vad?
10. @soulislove Monthly average: 15,811,960,023 Per performance team member per month: 3,162,392,004 Use https://tools.wmflabs.org/siteviews/ Page views
11. The Wikipedia performance problem @soulislove •All wikis are different •All pages are different •All users are different
12. @soulislove Back to the roots
13. @soulislove What is web performance?
14. @soulislove
15. @soulislove What is RUM?
16. @soulislove
17. @soulislove What is synthetic testing?
18. @soulislove CC by 2.0, Beraldo Leal
19. @soulislove The goal
20. @soulislove Find performance regressions
21. @soulislove What do we want?
22. @soulislove Stable metrics!
23. @soulislove What do we need?
24. @soulislove A stable server A stable server CC by 2.0, Grant Hutchinson
25. @soulislove Stable connectivity Stable connectivity CC by 2.0, bobthemagicdragon
26. @soulislove A stable page
27. @soulislove A stable browser CC by 2.0, Karl Baron
28. @soulislove A stable performance tool
29. @soulislove Synthetic testing vs RUM?
30. @soulislove Synthetic testing +RUM!
31. @soulislove https://bit.ly/2CFhooB
32. @soulislove Our performance tools
33. @soulislove
34. @soulislove
35. @soulislove Browsertime + WebPageReplay
36. @soulislove How do we test?
37. @soulislove •Firefox/Chrome (Linux) •5-11 runs •Different latencies •3 (!) pages per wiki •AWS c5.large/c5.xlarge •Focus on First Visual Change and Speed Index •Alerting in Grafana •Chrome only: CPU spent
38. @soulislove Stable metrics?
39. @soulislove Getting stable metrics is surprisingly hard!
40. @soulislove Getting stable metrics is surprisingly hard!
41. @soulislove
42. @soulislove 50%
43. @soulislove 16%
44. @soulislove
45. @soulislove 7%
46. @soulislove 3%
47. @soulislove Regression example
48. @soulislove
49. @soulislove
50. @soulislove
51. @soulislove
52. @soulislove
53. @soulislove Why? Why? Why? Why? Why? Why? Why?
54. @soulislove My top 5 learnings
55. @soulislove 1. All pages and servers are different. Use stddev or something like that for all timing metrics.
56. @soulislove
57. @soulislove 2. Auto deploy performance test tools is evil.
58. @soulislove
59. @soulislove 3. One video says more than 1000 Navigation Timing metrics
60. @soulislove
61. @soulislove 4. Docker is great for web performance testing
62. @soulislove •Controlled environment •Controlled deploys •Versioning browsers •Easy rollback (by tag)
63. @soulislove 5. Performance proxies are cool for measuring frontend performance
64. @soulislove •mahimahi (search for Benedikt Wolters H2 patch) •WebPageReplay •others?
65. @soulislove 5 b. Getting stable metrics is a moving target
66. @soulislove The next step
67. @soulislove •Run performance test on commit •Run tests on real mobile phones •more …
68. @soulislove Thank you! peter@wikimedia.org phedenskog [irc #fx-test] soulislove [twitter]