Measuring Web Performance for Wikipedia using synthetic testing tools

Presentation for the Automationeers Assemble at Mozilla.

Wikimedia performance team blog: https://phabricator.wikimedia.org/phame/blog/view/7/
Gilles article on magic performance numbers: https://www.mediawiki.org/wiki/Wikimedia_Performance_Team/Perceived_Performance
Our WebPageTest setup: https://wikitech.wikimedia.org/wiki/Performance/WebPageTest
Our Browsertime/WebPageReplay setup: https://wikitech.wikimedia.org/wiki/Performance/WebPageReplay
WebPageTest metrics: https://grafana.wikimedia.org/dashboard/db/webpagetest
WebPageReplay metrics: https://grafana.wikimedia.org/dashboard/db/webpagereplay
RUM metrics: https://grafana.wikimedia.org/dashboard/db/navigation-timing
Phabricator: https://phabricator.wikimedia.org/tag/performance-team/

Transcript

1. @soulislove
   Automationeers Assemble:
   Performance
   at Wikipedia
   2018-10-25
   

   View Slide

2. @soulislove
   Actually …
   

   View Slide

3. @soulislove
   Measuring Web
   Performance for
   Wikipedia using
   synthetic testing tools
   

   View Slide

4. @soulislove
   Peter Hedenskog
   Wikimedia
   Performance Team
   sitespeed.io
   @soulislove
   

   View Slide

5. @soulislove
   Wikimedia
   Performance team
   

   View Slide

6. @soulislove
   

   View Slide

7. @soulislove
   

   View Slide

8. @soulislove
   Wikipedia
   

   View Slide

9. @soulislove
   Vad?
   Vad?
   Vad?
   

   View Slide

10. @soulislove
    Monthly average:
    15,811,960,023
    Per performance team member per month:
    3,162,392,004
    Use https://tools.wmflabs.org/siteviews/
    Page views
    

    View Slide

11. The Wikipedia
    performance
    problem
    @soulislove
    •All wikis are different
    •All pages are different
    •All users are different
    

    View Slide

12. @soulislove
    Back to the
    roots
    

    View Slide

13. @soulislove
    What is web
    performance?
    

    View Slide

14. @soulislove
    

    View Slide

15. @soulislove
    What is RUM?
    

    View Slide

16. @soulislove
    

    View Slide

17. @soulislove
    What is synthetic
    testing?
    

    View Slide

18. @soulislove
    CC by 2.0, Beraldo Leal
    

    View Slide

19. @soulislove
    The goal
    

    View Slide

20. @soulislove
    Find
    performance regressions
    

    View Slide

21. @soulislove
    What do we
    want?
    

    View Slide

22. @soulislove
    Stable
    metrics!
    

    View Slide

23. @soulislove
    What do we
    need?
    

    View Slide

24. @soulislove
    A stable
    server
    A stable server
    CC by 2.0, Grant Hutchinson
    

    View Slide

25. @soulislove
    Stable
    connectivity
    Stable connectivity
    CC by 2.0, bobthemagicdragon
    

    View Slide

26. @soulislove
    A stable page
    

    View Slide

27. @soulislove
    A stable browser
    CC by 2.0, Karl Baron
    

    View Slide

28. @soulislove
    A stable performance tool
    

    View Slide

29. @soulislove
    Synthetic testing
    vs RUM?
    

    View Slide

30. @soulislove
    Synthetic testing
    +RUM!
    

    View Slide

31. @soulislove
    https://bit.ly/2CFhooB
    

    View Slide

32. @soulislove
    Our performance
    tools
    

    View Slide

33. @soulislove
    

    View Slide

34. @soulislove
    

    View Slide

35. @soulislove
    Browsertime +
    WebPageReplay
    

    View Slide

36. @soulislove
    How do we test?
    

    View Slide

37. @soulislove
    •Firefox/Chrome (Linux)
    •5-11 runs
    •Different latencies
    •3 (!) pages per wiki
    •AWS c5.large/c5.xlarge
    •Focus on First Visual Change and Speed Index
    •Alerting in Grafana
    •Chrome only: CPU spent
    

    View Slide

38. @soulislove
    Stable metrics?
    

    View Slide

39. @soulislove
    Getting stable metrics is surprisingly hard!
    

    View Slide

40. @soulislove
    Getting stable metrics is
    surprisingly
    hard!
    

    View Slide

41. @soulislove
    

    View Slide

42. @soulislove
    50%
    

    View Slide

43. @soulislove
    16%
    

    View Slide

44. @soulislove
    

    View Slide

45. @soulislove
    7%
    

    View Slide

46. @soulislove
    3%
    

    View Slide

47. @soulislove
    Regression
    example
    

    View Slide

48. @soulislove
    

    View Slide

49. @soulislove
    

    View Slide

50. @soulislove
    

    View Slide

51. @soulislove
    

    View Slide

52. @soulislove
    

    View Slide

53. @soulislove
    Why?
    Why?
    Why?
    Why?
    Why?
    Why?
    Why?
    

    View Slide

54. @soulislove
    My top 5
    learnings
    

    View Slide

55. @soulislove
    1. All pages and servers are
    different. Use stddev or
    something like that for all
    timing metrics.
    

    View Slide

56. @soulislove
    

    View Slide

57. @soulislove
    2. Auto deploy performance
    test tools is evil.
    

    View Slide

58. @soulislove
    

    View Slide

59. @soulislove
    3. One video says more than
    1000 Navigation Timing
    metrics
    

    View Slide

60. @soulislove
    

    View Slide

61. @soulislove
    4. Docker is great for web
    performance testing
    

    View Slide

62. @soulislove
    •Controlled environment
    •Controlled deploys
    •Versioning browsers
    •Easy rollback (by tag)
    

    View Slide

63. @soulislove
    5. Performance proxies are
    cool for measuring frontend
    performance
    

    View Slide

64. @soulislove
    •mahimahi (search for Benedikt Wolters
    H2 patch)
    •WebPageReplay
    •others?
    

    View Slide

65. @soulislove
    5 b. Getting stable metrics
    is a moving target
    

    View Slide

66. @soulislove
    The next step
    

    View Slide

67. @soulislove
    •Run performance test on commit
    •Run tests on real mobile phones
    •more …
    

    View Slide

68. @soulislove
    Thank you!
    peter@wikimedia.org
    phedenskog [irc #fx-test]
    soulislove [twitter]
    

    View Slide