Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Measuring Web Performance for Wikipedia using synthetic testing tools

Measuring Web Performance for Wikipedia using synthetic testing tools


Peter Hedenskog

October 25, 2018


  1. @soulislove Automationeers Assemble: Performance at Wikipedia 2018-10-25

  2. @soulislove Actually …

  3. @soulislove Measuring Web Performance for Wikipedia using synthetic testing tools

  4. @soulislove Peter Hedenskog Wikimedia Performance Team sitespeed.io @soulislove

  5. @soulislove Wikimedia Performance team

  6. @soulislove

  7. @soulislove

  8. @soulislove Wikipedia

  9. @soulislove Vad? Vad? Vad?

  10. @soulislove Monthly average: 15,811,960,023 Per performance team member per month:

    3,162,392,004 Use https://tools.wmflabs.org/siteviews/ Page views
  11. The Wikipedia performance problem @soulislove •All wikis are different •All

    pages are different •All users are different
  12. @soulislove Back to the roots

  13. @soulislove What is web performance?

  14. @soulislove

  15. @soulislove What is RUM?

  16. @soulislove

  17. @soulislove What is synthetic testing?

  18. @soulislove CC by 2.0, Beraldo Leal

  19. @soulislove The goal

  20. @soulislove Find performance regressions

  21. @soulislove What do we want?

  22. @soulislove Stable metrics!

  23. @soulislove What do we need?

  24. @soulislove A stable server A stable server CC by 2.0,

    Grant Hutchinson
  25. @soulislove Stable connectivity Stable connectivity CC by 2.0, bobthemagicdragon

  26. @soulislove A stable page

  27. @soulislove A stable browser CC by 2.0, Karl Baron

  28. @soulislove A stable performance tool

  29. @soulislove Synthetic testing vs RUM?

  30. @soulislove Synthetic testing +RUM!

  31. @soulislove https://bit.ly/2CFhooB

  32. @soulislove Our performance tools

  33. @soulislove

  34. @soulislove

  35. @soulislove Browsertime + WebPageReplay

  36. @soulislove How do we test?

  37. @soulislove •Firefox/Chrome (Linux) •5-11 runs •Different latencies •3 (!) pages

    per wiki •AWS c5.large/c5.xlarge •Focus on First Visual Change and Speed Index •Alerting in Grafana •Chrome only: CPU spent
  38. @soulislove Stable metrics?

  39. @soulislove Getting stable metrics is surprisingly hard!

  40. @soulislove Getting stable metrics is surprisingly hard!

  41. @soulislove

  42. @soulislove 50%

  43. @soulislove 16%

  44. @soulislove

  45. @soulislove 7%

  46. @soulislove 3%

  47. @soulislove Regression example

  48. @soulislove

  49. @soulislove

  50. @soulislove

  51. @soulislove

  52. @soulislove

  53. @soulislove Why? Why? Why? Why? Why? Why? Why?

  54. @soulislove My top 5 learnings

  55. @soulislove 1. All pages and servers are different. Use stddev

    or something like that for all timing metrics.
  56. @soulislove

  57. @soulislove 2. Auto deploy performance test tools is evil.

  58. @soulislove

  59. @soulislove 3. One video says more than 1000 Navigation Timing

  60. @soulislove

  61. @soulislove 4. Docker is great for web performance testing

  62. @soulislove •Controlled environment •Controlled deploys •Versioning browsers •Easy rollback (by

  63. @soulislove 5. Performance proxies are cool for measuring frontend performance

  64. @soulislove •mahimahi (search for Benedikt Wolters H2 patch) •WebPageReplay •others?

  65. @soulislove 5 b. Getting stable metrics is a moving target

  66. @soulislove The next step

  67. @soulislove •Run performance test on commit •Run tests on real

    mobile phones •more …
  68. @soulislove Thank you! peter@wikimedia.org phedenskog [irc #fx-test] soulislove [twitter]