LDNWebPerf April 2016 - Andrew Neilson

How PerfOps revolutionized performance at Marks & Spencer

Transcript

1. How PerfOps is revolutionising Performance at M&S Andrew Neilson ( andrew.neilson@mnscorp.net / @andyjneilson73)
2. Feb 2014: M&S.com launches on new in-house platform M&S.com launches Feb 2014
3. March 2014: M&S.com begins transition to agile M&S.com launches Feb 2014 Mar 2014 Agile transition commences
4. March 2014: M&S.com begins transition to agile M&S.com launches Feb 2014 Mar 2014 Good news… DevOps key to enabling success for continuous integration Agile transition commences
5. April 2014: Head up Performance Engineering M&S.com launches Feb 2014 Mar 2014 Apr 2014 Move to Performance Engineering Agile transition commences
6. April 2014: Head up Performance Engineering M&S.com launches Feb 2014 Mar 2014 Apr 2014 Move to Performance Engineering Shape performance for .com Agile transition commences
7. Summer 2014: Performance Engineering as-is M&S.com launches Feb 2014 Mar 2014 Apr 2014 May - Sep 2014 Move to Performance Engineering Prepare for Peak 2014 Agile transition commences Ready for Peak 2014
8. End of September 2014: Performance Engineering ensures M&S.com website should be stable for Peak! M&S.com launches Feb 2014 Mar 2014 Apr 2014 May - Sep 2014 Move to Performance Engineering Prepare for Peak 2014 Agile transition commences Go for Peak 2014
9. End of September 2014: Performance Engineering ensures M&S.com website should be stable for Peak! M&S.com launches Feb 2014 Mar 2014 Apr 2014 May - Sep 2014 Move to Performance Engineering Prepare for Peak 2014 Agile transition commences Go for Peak 2014 but…
10. Transition to agile performance capability pending A full end to end view of web performance was required Need more insight into speeds our customer was experiencing?
11. 1. What does performance mean for a .com business? Research Oct 2014
12. 1. What does performance mean for a .com business? 2. How does traditional performance engineering become agile? Research Oct 2014
13. November 2014 Looking for inspiration…
14. November 2014
15. January 2015 PerfOps A Performance Framework for Agile Delivery
16. Speed is key to meet customer expectations and therefore…
17. Speed is key to meet customer expectations and therefore… Speed is key to generating revenue opportunities
18. PerfOps Provides a performance test capability throughout agile delivery
19. PerfOps Provides insight to the customer experience of speed
20. PerfOps Flags impact to customer experience of speed caused by change
21. PerfOps Drives optimisation and standards across both Business and Delivery teams
22. What are we measuring?
23. Non Functional Requirements (NFR) What are we measuring?
24. Non Functional Requirements What are we measuring?
25. Non Functional Requirements What are we measuring? Performance Transactions
26. A
27. A B
28. Browser Content Delivery Network Request Response Generate Response Render Page A B
29. Browser CDN Request Render Page A B Web Server Web Application Order Management Response Generate Response
30. Browser CDN Request Render Page A B Web Server Web Application Order Management Response Generate Response NFR NFR
31. Browser CDN Request Render Page A B Web Server Web Application Order Management Response Generate Response Parent Transaction Child Transaction
32. What is the definition of B?
33. Start Render or First Paint? How quickly do we engage our customers? What is the definition of B?
34. What is the definition of B? Speed Index? How efficient to visually complete?
35. What is the definition of B? Page Load? Aligned to customer satisfaction?
36. What is the definition of B? Something bespoke? User timing & custom metrics! Steve Souders: https://speedcurve.com/blog/user-timing-and-custom-metrics/
37. How do we measure B?
38. Browser CDN Request Response Generate Response Render Page A B
39. Browser CDN Request Response Generate Response Render Page A B ~90% of pages are offloaded
40. Browser CDN Request Response Generate Response Render Page A B ~90% offload Real User Measurements Synthetic Measurements
41. Browser CDN Request Response Generate Response Render Page A B ~90% offload
42. Browser CDN Request Render Page A B Web Server Web Application Order Management Response Generate Response ~10% calls to origin
43. Browser CDN Request Render Page A B Web Server Web Application Order Management Response Generate Response APM ~10% calls to origin
44. Browser CDN Request Render Page A B Web Server Web Application Order Management Response Generate Response ~10% calls to origin
45. Store Device Integration Request Process Response A B Backend System 1 Not just a .com opportunity!
46. Store Device Integration Request Process Response A B Backend System 1 Not just a .com opportunity! Backend System 1 Backend System 2 Message Process Message A B
47. Speed is an aspiration not an NFR
48. Enforce one basic rule: never get slower! Speed is an aspiration not an NFR
49. We must test A to B through delivery
50. We must test A to B through delivery DevOps tools / processes enable us to validate build on build release on release
51. We must test A to B through delivery DevOps tools / processes enable us to validate build on build release on release Enforce one basic DoD: never get slower!
52. How do we test performance when delivery is agile?
53. Design Build Test Analysis Product Idea V&P Test
54. Design Build Test Analysis Product Idea V&P Test Piggy-back on DevOps capability
55. Design Build Test Analysis Product Idea V&P Test Optimisation recommendations from PerfOps back-log (analysis & diagnostics)
56. Design Build Test Analysis Automated performance test capability Product Idea V&P Test Optimisation recommendations from PerfOps back-log (analysis & diagnostics)
57. Design Build Test Analysis Product Idea V&P Test Automated page speed test capability Automated performance test capability Automated performance budget test capability Optimisation recommendations from PerfOps back-log (analysis & diagnostics)
58. Design Build Test Analysis Product Idea V&P Test Future plans… Performance by Design including UX guidelines Automated page speed test capability Automated performance test capability Automated performance budget test capability Optimisation recommendations from PerfOps back-log (analysis & diagnostics)
59. Design Build Test Analysis Product Idea V&P Test Automated page speed test capability Automated performance test capability Automated performance budget test capability ? Future plans… Performance by Design including UX guidelines Optimisation recommendations from PerfOps back-log (analysis & diagnostics)
60. Checkpoint:  What to measure
61. Checkpoint:  What to measure  How to monitor what we measure
62. Checkpoint:  What to measure  How to monitor what we measure  How to test what we measure
63. Our next challenges…
64. How do we action what we measure? Our next challenges…
65. How do we action what we measure? Our next challenges… We have insight, who else needs it?
66. Our next challenges… What will they do with this insight? We have insight, who else needs it? How do we action what we measure?
67. Start basic… don’t drown in data!
68. Effective communication with simple & powerful of visualisation
69. Build a performance mind-set
70. Let’s look at RUM / Synthetic Monitoring and Analysis & Diagnostics
71. Real-time impact caused by…
72. Real-time impact caused by… third party tags
73. Real-time impact caused by…
74. Real-time impact caused by…
75. Real-time impact caused by… a solar eclipse!
76. Real-time impact caused by…
77. Real-time impact caused by… In this case, no impact. marketing / email campaigns
78. Benefits of planned optimisations
79. Benefits of planned optimisations Step changes can be seen immediately
80. Insight to device speeds & trends Desktop Tablet Mobile
81. Insight to device speeds & trends Tablet is slowest Tablet is popular
82. Use of waterfalls to identify optimisations
83. Basic diagnostics can identify plenty of low hanging fruit Use of waterfalls to identify optimisations
84. Simple but powerful ways to communicate how good we are (or not)
85. Video comparisons Simple but powerful ways to communicate how good we are (or not)
86. Simple but powerful ways to communicate how good we are (or not) Filmstrips
87. Simple but powerful ways to communicate how good we are (or not) Competitor benchmarking & analysis
88. Lots of insight but how do we get everyone to buy into it?
89. Lots of insight but how do we get everyone to buy into it? We had a breakthrough!
90. Site speed trends reporting that degradation over time
91. PerfOps contacted to see if we could help Majority of feedback called out “site is slow”
92. Problem identified via SpeedCurve…
93. Request maps by Simon Hearne: http://requestmap.webperf.tools/ Request map of our homepage as part of a speed review with NCC Group
94. If we turned off third party content we be 50% faster This does not help the business Third party content must be and can be controlled
95. Request maps by Simon Hearne: http://requestmap.webperf.tools/ Request map visualisation brings to life the impact…
96. Request map visualisation brings to life the impact… and improvements! Request maps by Simon Hearne: http://requestmap.webperf.tools/
97. Problem identified via powerful visualisation in SpeedCurve…
98. Customer satisfaction correlation with site speed
99. Speed must be a priority Impact to speed must be monitored Optimisations can also be made by Business not just IT
100. Maturing Site Speed Analytics There are a lot of studies available that highlight why speed is important to retailers in particular. Doing your own study is much more interesting!
101. • Correlation to single page visits and customer satisfaction • How speeds influence SEO and Marketing • Impact v benefit of personalisation and third party tags • Identifying what we cannot control Site Speed Analytics
102. Site Speed Analytics Studies do not always align to all!
103. Site Speed Analytics Breakdown via page types and devices as well
104. Site Speed Analytics
105. Site Speed Analytics Product List Product List The colour of the boxes denotes the relative page load whilst the size denotes the weight of the page. M&S has decreased in size and page load, whilst competitors show no improvement A B C D D C A B
106. Site Speed Analytics
107. Site Speed Analytics
108. Site Speed Analytics
109. Optimisations based on metric driven decisions Business focused metrics ensure optimisation priority Site Speed Analytics
110. Optimisations based on metric driven decisions Business focused metrics ensure optimisation priority M&S.com is now 37% faster than it was 8 months ago Site Speed Analytics
111. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics The PerfOps Journey…
112. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP The PerfOps Journey…
113. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Budget Tests The PerfOps Journey…
114. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Speed Tests Automated Budget Tests The PerfOps Journey…
115. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Speed Tests Automated Budget Tests More Optimisation Back log The PerfOps Journey…
116. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement The PerfOps Journey…
117. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement The PerfOps Journey…
118. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement Front-end diagnostics The PerfOps Journey…
119. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement Front-end diagnostics More Optimisation Back log The PerfOps Journey…
120. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement Front-end diagnostics More Optimisation Back log Site Speed Dashboards The PerfOps Journey…
121. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement Front-end diagnostics More Optimisation Back log Site Speed Dashboards Third party Governance The PerfOps Journey…
122. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement Front-end diagnostics Third party Governance More Optimisation Back log Site Speed Dashboards Site Speed Consultancy The PerfOps Journey…
123. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement Front-end diagnostics Third party Governance More Optimisation Back log Site Speed Dashboards Site Speed Consultancy Speed Forums The PerfOps Journey…
124. VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement Front-end diagnostics Third party Governance More Optimisation Back log Site Speed Dashboards Site Speed Consultancy Performance Transactions Speed Forums The PerfOps Journey…
125. What next…
126. What next… Site Speed Analytics
127. What next… Site Speed Analytics Mature Capability
128. What next… Site Speed Analytics Mature Capability Performance Budgets
129. What next… Business Guidelines Site Speed Analytics Mature Capability Performance Budgets
130. What next… Performance by Design Business Guidelines Site Speed Analytics Mature Capability Performance Budgets
131. What next… Performance by Design Business Guidelines Site Speed Analytics Mature Capability Performance Budgets Performance must be everywhere
132. One rule…
133. One rule… Never get slower!
134. Thank you www.mandsdigital.com / @mandsdigital