LDNWebPerf April 2016 - Andrew Neilson

LDNWebPerf April 2016 - Andrew Neilson

How PerfOps revolutionized performance at Marks & Spencer

Transcript

  1. How PerfOps is revolutionising Performance at M&S Andrew Neilson (

    andrew.neilson@mnscorp.net / @andyjneilson73)
  2. Feb 2014: M&S.com launches on new in-house platform M&S.com launches

    Feb 2014
  3. March 2014: M&S.com begins transition to agile M&S.com launches Feb

    2014 Mar 2014 Agile transition commences
  4. March 2014: M&S.com begins transition to agile M&S.com launches Feb

    2014 Mar 2014 Good news… DevOps key to enabling success for continuous integration Agile transition commences
  5. April 2014: Head up Performance Engineering M&S.com launches Feb 2014

    Mar 2014 Apr 2014 Move to Performance Engineering Agile transition commences
  6. April 2014: Head up Performance Engineering M&S.com launches Feb 2014

    Mar 2014 Apr 2014 Move to Performance Engineering Shape performance for .com Agile transition commences
  7. Summer 2014: Performance Engineering as-is M&S.com launches Feb 2014 Mar

    2014 Apr 2014 May - Sep 2014 Move to Performance Engineering Prepare for Peak 2014 Agile transition commences Ready for Peak 2014
  8. End of September 2014: Performance Engineering ensures M&S.com website should

    be stable for Peak! M&S.com launches Feb 2014 Mar 2014 Apr 2014 May - Sep 2014 Move to Performance Engineering Prepare for Peak 2014 Agile transition commences Go for Peak 2014
  9. End of September 2014: Performance Engineering ensures M&S.com website should

    be stable for Peak! M&S.com launches Feb 2014 Mar 2014 Apr 2014 May - Sep 2014 Move to Performance Engineering Prepare for Peak 2014 Agile transition commences Go for Peak 2014 but…
  10. Transition to agile performance capability pending A full end to

    end view of web performance was required Need more insight into speeds our customer was experiencing?
  11. 1. What does performance mean for a .com business? Research

    Oct 2014
  12. 1. What does performance mean for a .com business? 2.

    How does traditional performance engineering become agile? Research Oct 2014
  13. November 2014 Looking for inspiration…

  14. November 2014

  15. January 2015 PerfOps A Performance Framework for Agile Delivery

  16. Speed is key to meet customer expectations and therefore…

  17. Speed is key to meet customer expectations and therefore… Speed

    is key to generating revenue opportunities
  18. PerfOps Provides a performance test capability throughout agile delivery

  19. PerfOps Provides insight to the customer experience of speed

  20. PerfOps Flags impact to customer experience of speed caused by

    change
  21. PerfOps Drives optimisation and standards across both Business and Delivery

    teams
  22. What are we measuring?

  23. Non Functional Requirements (NFR) What are we measuring?

  24. Non Functional Requirements What are we measuring?

  25. Non Functional Requirements What are we measuring? Performance Transactions

  26. A

  27. A B

  28. Browser Content Delivery Network Request Response Generate Response Render Page

    A B
  29. Browser CDN Request Render Page A B Web Server Web

    Application Order Management Response Generate Response
  30. Browser CDN Request Render Page A B Web Server Web

    Application Order Management Response Generate Response NFR NFR
  31. Browser CDN Request Render Page A B Web Server Web

    Application Order Management Response Generate Response Parent Transaction Child Transaction
  32. What is the definition of B?

  33. Start Render or First Paint? How quickly do we engage

    our customers? What is the definition of B?
  34. What is the definition of B? Speed Index? How efficient

    to visually complete?
  35. What is the definition of B? Page Load? Aligned to

    customer satisfaction?
  36. What is the definition of B? Something bespoke? User timing

    & custom metrics! Steve Souders: https://speedcurve.com/blog/user-timing-and-custom-metrics/
  37. How do we measure B?

  38. Browser CDN Request Response Generate Response Render Page A B

  39. Browser CDN Request Response Generate Response Render Page A B

    ~90% of pages are offloaded
  40. Browser CDN Request Response Generate Response Render Page A B

    ~90% offload Real User Measurements Synthetic Measurements
  41. Browser CDN Request Response Generate Response Render Page A B

    ~90% offload
  42. Browser CDN Request Render Page A B Web Server Web

    Application Order Management Response Generate Response ~10% calls to origin
  43. Browser CDN Request Render Page A B Web Server Web

    Application Order Management Response Generate Response APM ~10% calls to origin
  44. Browser CDN Request Render Page A B Web Server Web

    Application Order Management Response Generate Response ~10% calls to origin
  45. Store Device Integration Request Process Response A B Backend System

    1 Not just a .com opportunity!
  46. Store Device Integration Request Process Response A B Backend System

    1 Not just a .com opportunity! Backend System 1 Backend System 2 Message Process Message A B
  47. Speed is an aspiration not an NFR

  48. Enforce one basic rule: never get slower! Speed is an

    aspiration not an NFR
  49. We must test A to B through delivery

  50. We must test A to B through delivery DevOps tools

    / processes enable us to validate build on build release on release
  51. We must test A to B through delivery DevOps tools

    / processes enable us to validate build on build release on release Enforce one basic DoD: never get slower!
  52. How do we test performance when delivery is agile?

  53. Design Build Test Analysis Product Idea V&P Test

  54. Design Build Test Analysis Product Idea V&P Test Piggy-back on

    DevOps capability
  55. Design Build Test Analysis Product Idea V&P Test Optimisation recommendations

    from PerfOps back-log (analysis & diagnostics)
  56. Design Build Test Analysis Automated performance test capability Product Idea

    V&P Test Optimisation recommendations from PerfOps back-log (analysis & diagnostics)
  57. Design Build Test Analysis Product Idea V&P Test Automated page

    speed test capability Automated performance test capability Automated performance budget test capability Optimisation recommendations from PerfOps back-log (analysis & diagnostics)
  58. Design Build Test Analysis Product Idea V&P Test Future plans…

    Performance by Design including UX guidelines Automated page speed test capability Automated performance test capability Automated performance budget test capability Optimisation recommendations from PerfOps back-log (analysis & diagnostics)
  59. Design Build Test Analysis Product Idea V&P Test Automated page

    speed test capability Automated performance test capability Automated performance budget test capability ? Future plans… Performance by Design including UX guidelines Optimisation recommendations from PerfOps back-log (analysis & diagnostics)
  60. Checkpoint:  What to measure

  61. Checkpoint:  What to measure  How to monitor what

    we measure
  62. Checkpoint:  What to measure  How to monitor what

    we measure  How to test what we measure
  63. Our next challenges…

  64. How do we action what we measure? Our next challenges…

  65. How do we action what we measure? Our next challenges…

    We have insight, who else needs it?
  66. Our next challenges… What will they do with this insight?

    We have insight, who else needs it? How do we action what we measure?
  67. Start basic… don’t drown in data!

  68. Effective communication with simple & powerful of visualisation

  69. Build a performance mind-set

  70. Let’s look at RUM / Synthetic Monitoring and Analysis &

    Diagnostics
  71. Real-time impact caused by…

  72. Real-time impact caused by… third party tags

  73. Real-time impact caused by…

  74. Real-time impact caused by…

  75. Real-time impact caused by… a solar eclipse!

  76. Real-time impact caused by…

  77. Real-time impact caused by… In this case, no impact. marketing

    / email campaigns
  78. Benefits of planned optimisations

  79. Benefits of planned optimisations Step changes can be seen immediately

  80. Insight to device speeds & trends Desktop Tablet Mobile

  81. Insight to device speeds & trends Tablet is slowest Tablet

    is popular
  82. Use of waterfalls to identify optimisations

  83. Basic diagnostics can identify plenty of low hanging fruit Use

    of waterfalls to identify optimisations
  84. Simple but powerful ways to communicate how good we are

    (or not)
  85. Video comparisons Simple but powerful ways to communicate how good

    we are (or not)
  86. Simple but powerful ways to communicate how good we are

    (or not) Filmstrips
  87. Simple but powerful ways to communicate how good we are

    (or not) Competitor benchmarking & analysis
  88. Lots of insight but how do we get everyone to

    buy into it?
  89. Lots of insight but how do we get everyone to

    buy into it? We had a breakthrough!
  90. Site speed trends reporting that degradation over time

  91. PerfOps contacted to see if we could help Majority of

    feedback called out “site is slow”
  92. Problem identified via SpeedCurve…

  93. Request maps by Simon Hearne: http://requestmap.webperf.tools/ Request map of our

    homepage as part of a speed review with NCC Group
  94. If we turned off third party content we be 50%

    faster This does not help the business Third party content must be and can be controlled
  95. Request maps by Simon Hearne: http://requestmap.webperf.tools/ Request map visualisation brings

    to life the impact…
  96. Request map visualisation brings to life the impact… and improvements!

    Request maps by Simon Hearne: http://requestmap.webperf.tools/
  97. Problem identified via powerful visualisation in SpeedCurve…

  98. Customer satisfaction correlation with site speed

  99. Speed must be a priority Impact to speed must be

    monitored Optimisations can also be made by Business not just IT
  100. Maturing Site Speed Analytics There are a lot of studies

    available that highlight why speed is important to retailers in particular. Doing your own study is much more interesting!
  101. • Correlation to single page visits and customer satisfaction •

    How speeds influence SEO and Marketing • Impact v benefit of personalisation and third party tags • Identifying what we cannot control Site Speed Analytics
  102. Site Speed Analytics Studies do not always align to all!

  103. Site Speed Analytics Breakdown via page types and devices as

    well
  104. Site Speed Analytics

  105. Site Speed Analytics Product List Product List The colour of

    the boxes denotes the relative page load whilst the size denotes the weight of the page. M&S has decreased in size and page load, whilst competitors show no improvement A B C D D C A B
  106. Site Speed Analytics

  107. Site Speed Analytics

  108. Site Speed Analytics

  109. Optimisations based on metric driven decisions Business focused metrics ensure

    optimisation priority Site Speed Analytics
  110. Optimisations based on metric driven decisions Business focused metrics ensure

    optimisation priority M&S.com is now 37% faster than it was 8 months ago Site Speed Analytics
  111. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics The

    PerfOps Journey…
  112. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated

    VP The PerfOps Journey…
  113. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated

    VP Automated Budget Tests The PerfOps Journey…
  114. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated

    VP Automated Speed Tests Automated Budget Tests The PerfOps Journey…
  115. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated

    VP Automated Speed Tests Automated Budget Tests More Optimisation Back log The PerfOps Journey…
  116. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated

    VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement The PerfOps Journey…
  117. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated

    VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement The PerfOps Journey…
  118. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated

    VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement Front-end diagnostics The PerfOps Journey…
  119. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated

    VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement Front-end diagnostics More Optimisation Back log The PerfOps Journey…
  120. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated

    VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement Front-end diagnostics More Optimisation Back log Site Speed Dashboards The PerfOps Journey…
  121. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated

    VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement Front-end diagnostics More Optimisation Back log Site Speed Dashboards Third party Governance The PerfOps Journey…
  122. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated

    VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement Front-end diagnostics Third party Governance More Optimisation Back log Site Speed Dashboards Site Speed Consultancy The PerfOps Journey…
  123. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated

    VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement Front-end diagnostics Third party Governance More Optimisation Back log Site Speed Dashboards Site Speed Consultancy Speed Forums The PerfOps Journey…
  124. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated

    VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement Front-end diagnostics Third party Governance More Optimisation Back log Site Speed Dashboards Site Speed Consultancy Performance Transactions Speed Forums The PerfOps Journey…
  125. What next…

  126. What next… Site Speed Analytics

  127. What next… Site Speed Analytics Mature Capability

  128. What next… Site Speed Analytics Mature Capability Performance Budgets

  129. What next… Business Guidelines Site Speed Analytics Mature Capability Performance

    Budgets
  130. What next… Performance by Design Business Guidelines Site Speed Analytics

    Mature Capability Performance Budgets
  131. What next… Performance by Design Business Guidelines Site Speed Analytics

    Mature Capability Performance Budgets Performance must be everywhere
  132. One rule…

  133. One rule… Never get slower!

  134. Thank you www.mandsdigital.com / @mandsdigital