LDNWebPerf April 2016 - Andrew Neilson

by London Web Performance Group

Slide 1

Slide 1 text

How PerfOps is revolutionising Performance at M&S Andrew Neilson ( [email protected] / @andyjneilson73)

Slide 2

Slide 2 text

Feb 2014: M&S.com launches on new in-house platform M&S.com launches Feb 2014

Slide 3

Slide 3 text

March 2014: M&S.com begins transition to agile M&S.com launches Feb 2014 Mar 2014 Agile transition commences

Slide 4

Slide 4 text

March 2014: M&S.com begins transition to agile M&S.com launches Feb 2014 Mar 2014 Good news… DevOps key to enabling success for continuous integration Agile transition commences

Slide 5

Slide 5 text

April 2014: Head up Performance Engineering M&S.com launches Feb 2014 Mar 2014 Apr 2014 Move to Performance Engineering Agile transition commences

Slide 6

Slide 6 text

April 2014: Head up Performance Engineering M&S.com launches Feb 2014 Mar 2014 Apr 2014 Move to Performance Engineering Shape performance for .com Agile transition commences

Slide 7

Slide 7 text

Summer 2014: Performance Engineering as-is M&S.com launches Feb 2014 Mar 2014 Apr 2014 May - Sep 2014 Move to Performance Engineering Prepare for Peak 2014 Agile transition commences Ready for Peak 2014

Slide 8

Slide 8 text

End of September 2014: Performance Engineering ensures M&S.com website should be stable for Peak! M&S.com launches Feb 2014 Mar 2014 Apr 2014 May - Sep 2014 Move to Performance Engineering Prepare for Peak 2014 Agile transition commences Go for Peak 2014

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Transition to agile performance capability pending A full end to end view of web performance was required Need more insight into speeds our customer was experiencing?

Slide 11

Slide 11 text

1. What does performance mean for a .com business? Research Oct 2014

Slide 12

Slide 12 text

1. What does performance mean for a .com business? 2. How does traditional performance engineering become agile? Research Oct 2014

Slide 13

Slide 13 text

November 2014 Looking for inspiration…

Slide 14

Slide 14 text

November 2014

Slide 15

Slide 15 text

January 2015 PerfOps A Performance Framework for Agile Delivery

Slide 16

Slide 16 text

Speed is key to meet customer expectations and therefore…

Slide 17

Slide 17 text

Speed is key to meet customer expectations and therefore… Speed is key to generating revenue opportunities

Slide 18

Slide 18 text

PerfOps Provides a performance test capability throughout agile delivery

Slide 19

Slide 19 text

PerfOps Provides insight to the customer experience of speed

Slide 20

Slide 20 text

PerfOps Flags impact to customer experience of speed caused by change

Slide 21

Slide 21 text

PerfOps Drives optimisation and standards across both Business and Delivery teams

Slide 22

Slide 22 text

What are we measuring?

Slide 23

Slide 23 text

Non Functional Requirements (NFR) What are we measuring?

Slide 24

Slide 24 text

Non Functional Requirements What are we measuring?

Slide 25

Slide 25 text

Non Functional Requirements What are we measuring? Performance Transactions

Slide 26

Slide 26 text

Slide 27

Slide 27 text

A B

Slide 28

Slide 28 text

Browser Content Delivery Network Request Response Generate Response Render Page A B

Slide 29

Slide 29 text

Browser CDN Request Render Page A B Web Server Web Application Order Management Response Generate Response

Slide 30

Slide 30 text

Browser CDN Request Render Page A B Web Server Web Application Order Management Response Generate Response NFR NFR

Slide 31

Slide 31 text

Browser CDN Request Render Page A B Web Server Web Application Order Management Response Generate Response Parent Transaction Child Transaction

Slide 32

Slide 32 text

What is the definition of B?

Slide 33

Slide 33 text

Start Render or First Paint? How quickly do we engage our customers? What is the definition of B?

Slide 34

Slide 34 text

What is the definition of B? Speed Index? How efficient to visually complete?

Slide 35

Slide 35 text

What is the definition of B? Page Load? Aligned to customer satisfaction?

Slide 36

Slide 36 text

What is the definition of B? Something bespoke? User timing & custom metrics! Steve Souders: https://speedcurve.com/blog/user-timing-and-custom-metrics/

Slide 37

Slide 37 text

How do we measure B?

Slide 38

Slide 38 text

Browser CDN Request Response Generate Response Render Page A B

Slide 39

Slide 39 text

Browser CDN Request Response Generate Response Render Page A B ~90% of pages are offloaded

Slide 40

Slide 40 text

Browser CDN Request Response Generate Response Render Page A B ~90% offload Real User Measurements Synthetic Measurements

Slide 41

Slide 41 text

Browser CDN Request Response Generate Response Render Page A B ~90% offload

Slide 42

Slide 42 text

Browser CDN Request Render Page A B Web Server Web Application Order Management Response Generate Response ~10% calls to origin

Slide 43

Slide 43 text

Browser CDN Request Render Page A B Web Server Web Application Order Management Response Generate Response APM ~10% calls to origin

Slide 44

Slide 44 text

Browser CDN Request Render Page A B Web Server Web Application Order Management Response Generate Response ~10% calls to origin

Slide 45

Slide 45 text

Store Device Integration Request Process Response A B Backend System 1 Not just a .com opportunity!

Slide 46

Slide 46 text

Store Device Integration Request Process Response A B Backend System 1 Not just a .com opportunity! Backend System 1 Backend System 2 Message Process Message A B

Slide 47

Slide 47 text

Speed is an aspiration not an NFR

Slide 48

Slide 48 text

Enforce one basic rule: never get slower! Speed is an aspiration not an NFR

Slide 49

Slide 49 text

We must test A to B through delivery

Slide 50

Slide 50 text

We must test A to B through delivery DevOps tools / processes enable us to validate build on build release on release

Slide 51

Slide 51 text

We must test A to B through delivery DevOps tools / processes enable us to validate build on build release on release Enforce one basic DoD: never get slower!

Slide 52

Slide 52 text

How do we test performance when delivery is agile?

Slide 53

Slide 53 text

Design Build Test Analysis Product Idea V&P Test

Slide 54

Slide 54 text

Design Build Test Analysis Product Idea V&P Test Piggy-back on DevOps capability

Slide 55

Slide 55 text

Design Build Test Analysis Product Idea V&P Test Optimisation recommendations from PerfOps back-log (analysis & diagnostics)

Slide 56

Slide 56 text

Design Build Test Analysis Automated performance test capability Product Idea V&P Test Optimisation recommendations from PerfOps back-log (analysis & diagnostics)

Slide 57

Slide 57 text

Design Build Test Analysis Product Idea V&P Test Automated page speed test capability Automated performance test capability Automated performance budget test capability Optimisation recommendations from PerfOps back-log (analysis & diagnostics)

Slide 58

Slide 58 text

Design Build Test Analysis Product Idea V&P Test Future plans… Performance by Design including UX guidelines Automated page speed test capability Automated performance test capability Automated performance budget test capability Optimisation recommendations from PerfOps back-log (analysis & diagnostics)

Slide 59

Slide 59 text

Design Build Test Analysis Product Idea V&P Test Automated page speed test capability Automated performance test capability Automated performance budget test capability ? Future plans… Performance by Design including UX guidelines Optimisation recommendations from PerfOps back-log (analysis & diagnostics)

Slide 60

Slide 60 text

Checkpoint:  What to measure

Slide 61

Slide 61 text

Checkpoint:  What to measure  How to monitor what we measure

Slide 62

Slide 62 text

Checkpoint:  What to measure  How to monitor what we measure  How to test what we measure

Slide 63

Slide 63 text

Our next challenges…

Slide 64

Slide 64 text

How do we action what we measure? Our next challenges…

Slide 65

Slide 65 text

How do we action what we measure? Our next challenges… We have insight, who else needs it?

Slide 66

Slide 66 text

Our next challenges… What will they do with this insight? We have insight, who else needs it? How do we action what we measure?

Slide 67

Slide 67 text

Start basic… don’t drown in data!

Slide 68

Slide 68 text

Effective communication with simple & powerful of visualisation

Slide 69

Slide 69 text

Build a performance mind-set

Slide 70

Slide 70 text

Let’s look at RUM / Synthetic Monitoring and Analysis & Diagnostics

Slide 71

Slide 71 text

Real-time impact caused by…

Slide 72

Slide 72 text

Real-time impact caused by… third party tags

Slide 73

Slide 73 text

Real-time impact caused by…

Slide 74

Slide 74 text

Real-time impact caused by…

Slide 75

Slide 75 text

Real-time impact caused by… a solar eclipse!

Slide 76

Slide 76 text

Real-time impact caused by…

Slide 77

Slide 77 text

Real-time impact caused by… In this case, no impact. marketing / email campaigns

Slide 78

Slide 78 text

Benefits of planned optimisations

Slide 79

Slide 79 text

Benefits of planned optimisations Step changes can be seen immediately

Slide 80

Slide 80 text

Insight to device speeds & trends Desktop Tablet Mobile

Slide 81

Slide 81 text

Insight to device speeds & trends Tablet is slowest Tablet is popular

Slide 82

Slide 82 text

Use of waterfalls to identify optimisations

Slide 83

Slide 83 text

Basic diagnostics can identify plenty of low hanging fruit Use of waterfalls to identify optimisations

Slide 84

Slide 84 text

Simple but powerful ways to communicate how good we are (or not)

Slide 85

Slide 85 text

Video comparisons Simple but powerful ways to communicate how good we are (or not)

Slide 86

Slide 86 text

Simple but powerful ways to communicate how good we are (or not) Filmstrips

Slide 87

Slide 87 text

Simple but powerful ways to communicate how good we are (or not) Competitor benchmarking & analysis

Slide 88

Slide 88 text

Lots of insight but how do we get everyone to buy into it?

Slide 89

Slide 89 text

Lots of insight but how do we get everyone to buy into it? We had a breakthrough!

Slide 90

Slide 90 text

Site speed trends reporting that degradation over time

Slide 91

Slide 91 text

PerfOps contacted to see if we could help Majority of feedback called out “site is slow”

Slide 92

Slide 92 text

Problem identified via SpeedCurve…

Slide 93

Slide 93 text

Request maps by Simon Hearne: http://requestmap.webperf.tools/ Request map of our homepage as part of a speed review with NCC Group

Slide 94

Slide 94 text

If we turned off third party content we be 50% faster This does not help the business Third party content must be and can be controlled

Slide 95

Slide 95 text

Request maps by Simon Hearne: http://requestmap.webperf.tools/ Request map visualisation brings to life the impact…

Slide 96

Slide 96 text

Request map visualisation brings to life the impact… and improvements! Request maps by Simon Hearne: http://requestmap.webperf.tools/

Slide 97

Slide 97 text

Problem identified via powerful visualisation in SpeedCurve…

Slide 98

Slide 98 text

Customer satisfaction correlation with site speed

Slide 99

Slide 99 text

Speed must be a priority Impact to speed must be monitored Optimisations can also be made by Business not just IT

Slide 100

Slide 100 text

Maturing Site Speed Analytics There are a lot of studies available that highlight why speed is important to retailers in particular. Doing your own study is much more interesting!

Slide 101

Slide 101 text

• Correlation to single page visits and customer satisfaction • How speeds influence SEO and Marketing • Impact v benefit of personalisation and third party tags • Identifying what we cannot control Site Speed Analytics

Slide 102

Slide 102 text

Site Speed Analytics Studies do not always align to all!

Slide 103

Slide 103 text

Site Speed Analytics Breakdown via page types and devices as well

Slide 104

Slide 104 text

Site Speed Analytics

Slide 105

Slide 105 text

Site Speed Analytics Product List Product List The colour of the boxes denotes the relative page load whilst the size denotes the weight of the page. M&S has decreased in size and page load, whilst competitors show no improvement A B C D D C A B

Slide 106

Slide 106 text

Site Speed Analytics

Slide 107

Slide 107 text

Site Speed Analytics

Slide 108

Slide 108 text

Site Speed Analytics

Slide 109

Slide 109 text

Optimisations based on metric driven decisions Business focused metrics ensure optimisation priority Site Speed Analytics

Slide 110

Slide 110 text

Optimisations based on metric driven decisions Business focused metrics ensure optimisation priority M&S.com is now 37% faster than it was 8 months ago Site Speed Analytics

Slide 111

Slide 111 text

VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics The PerfOps Journey…

Slide 112

Slide 112 text

VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP The PerfOps Journey…

Slide 113

Slide 113 text

VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Budget Tests The PerfOps Journey…

Slide 114

Slide 114 text

VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Speed Tests Automated Budget Tests The PerfOps Journey…

Slide 115

Slide 115 text

VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Speed Tests Automated Budget Tests More Optimisation Back log The PerfOps Journey…

Slide 116

Slide 116 text

Slide 117

Slide 117 text

Slide 118

Slide 118 text

Slide 119

Slide 119 text

VP Testing Back-end monitoring Optimisation Back logs Back-end diagnostics Automated VP Automated Speed Tests Automated Budget Tests More Optimisation Back log Real User Measurement Synthetic Measurement Front-end diagnostics More Optimisation Back log The PerfOps Journey…

Slide 120

Slide 120 text

Slide 121

Slide 121 text

Slide 122

Slide 122 text

Slide 123

Slide 123 text

Slide 124

Slide 124 text

Slide 125

Slide 125 text

What next…

Slide 126

Slide 126 text

What next… Site Speed Analytics

Slide 127

Slide 127 text

What next… Site Speed Analytics Mature Capability

Slide 128

Slide 128 text

What next… Site Speed Analytics Mature Capability Performance Budgets

Slide 129

Slide 129 text

What next… Business Guidelines Site Speed Analytics Mature Capability Performance Budgets

Slide 130

Slide 130 text

What next… Performance by Design Business Guidelines Site Speed Analytics Mature Capability Performance Budgets

Slide 131

Slide 131 text

What next… Performance by Design Business Guidelines Site Speed Analytics Mature Capability Performance Budgets Performance must be everywhere