If you can't measure it, you can't improve it. But measuring load time is easy, right? "Load time" is an outdated concept, as single page app experiences take over the web. We need better metrics to measure what our users are really feeling.
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics * I know the pain of slow internet
* They did this test when I lived there, in 2009
* The Prime Minister said: “If [the opposition] had their way Australians would be left using carrier pigeons for the future rather than accessing an internationally competitive broadband network”
* So, they tested it. They flew a carrier pigeon with a 700 megabyte usb drive from the central west, to Sydney, about 100kms or 60 miles.
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics ADSL speed in Australia: ~7990kbps * This is kind of shocking for anyone who visits Australia from America
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Good news! Internet gets faster over time * But don’t worry! As technology changes, things get better over time automatically!
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics * And great! Thanks to Moore’s law, CPU speeds have been going up for years!
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics 1% increase in capacity →
Up to 1.1% increase in demand * The “induced demand” effect
for every 1 percent increase in highway capacity, traffic increases 0.29 to 1.1 percent in the long term (about five years out), and up to 0.68 percent in the short term (one or two years)
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Business goals One second delay in Bing results in a 2.8% drop in revenue. Two second delay results in 4.3% drop. * https://wpostats.com/tags/revenue/
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Business goals Walmart: 2% increase in conversions for every 1 second of improvement in load time.
Every 100ms improvement also resulted in up to a 1% increase in revenue. * It’s a UX problem
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Business goals Google's DoubleClick found that publishers whose mobile sites load in 5 seconds earn up to 2x more mobile ad revenue than sites loading in 19 seconds. * It’s a UX problem
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Business goals * Show this picture to your manager and they’ll be immediately convinced
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics The world Southeast Asia, South America, Africa and Siberia. * Actually more traffic from countries with poor connectivity!
* These countries would’ve normally taken two minutes to load!
* Australia not far off
* Actually affects whether people can use it at all
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics User happiness 0.1s 1s 10s Instant “Flow ” Task sw itch * Research has shown that there are three different types of tasks
* <0.1s Instant – the system is reacting immediately
* <1s Flow – I’m in a state of flow and my concentration is not broken between tasks
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics User happiness 0.1s 1s 10s Instant “Flow ” Task sw itch ☹ * Between 1s and 10s is a whole lot of emotion too
* Users are progressively getting frustrated and want to switch to another task in this period of time
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics User happiness 0.1s 1s 10s Instant “Flow ” Task sw itch ☹ Median desktop 5.5s 15.6s Median mobile * Between 1s and 10s is a whole lot of emotion too
* Users are progressively getting frustrated and want to switch to another task in this period of time
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Poor performance is an ignored problem * Not only is it a problem, it’s an ignored problem
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Preaching to the choir * I’m sure I’m preaching to the choir here, if you’ve come to this talk.
* Do you agree with these statistics that I’ve mentioned so far?
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Why measure What to measure ✅ “Good” metrics ⚖ Metric choices ⚒ Ways to measure Protect performance Here’s an overview of what I’m going to talk about today
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics You can’t meaningfully improve what you can’t meaningfully measure I’m going to drop the word “meaningful” in there.
It’s important that you’re measuring the right things
Measuring something is easy, but measuring the right thing is hard
You can’t make meaningful impacts without careful thought about what you’re measuring!
You can’t change the right things if you’re not measuring the right things
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics You can’t meaningfully improve what you can’t meaningfully measure Just want to really emphasise that
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics You can’t meaningfully improve what you can’t meaningfully measure Like, really
This slide deck contains the word meaningful 46 times
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Why measure What to measure ✅ “Good” metrics ⚖ Metric choices ⚒ Ways to measure Protect performance Let’s take a look at what we can measure in order to improve the UX
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics “Performance” We often talk about “performance”. But how does that apply to our users?
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Meaning Behaviour Appearance Media Information * If we’re going to think of it from the users perspective, it’s this
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics HTML Javascript CSS Assets Data { } * Which is, from our perspective, HTML, javascript, CSS, Assets, and data
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics DNS lookup GET request Parse HTML Fetch js Fetch CSS Parse js Parse CSS Layout / paint * How does the computer see what matters to our users
* I’m going to go over the metrics themselves later, slide deck is available for reference
* The way that this is put together is relevant for us so we can know which parts are meaningful for the user
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Fetch deferred js Parse deferred js Fetch more data Event handler * If we go forward in time a bit, we see there’s actually more
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Fetch more data Event handler Push state Load data Rerender * You might reload data on button click
* This is quite a different experience from the initial load, but a load nonetheless!
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics • A HTML document
• A single page app state
• An app state
• Whatever your users think it is So, what’s a “page”? * We need to be keep in mind what our users would think of as a page, and have measurements around that experience
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics This is a slowed down page load. Put your hand up, and lower it when you think the page is loaded
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Load is not a single moment in time — it’s an experience that no one metric can fully capture. There are multiple moments during the load experience that can affect whether a user perceives it as "fast" or “slow"
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Loading is not a boolean state There is no one event, or one easy answer to say when a page has loaded
It might even require knowledge of the future!
Loading is a spectrum
Metrics compress this spectrum into a single number
We need to be careful about how we choose this number
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Why measure What to measure ✅ “Good” metrics ⚖ Metric choices ⚒ Ways to measure Protect performance Let’s take a look at what makes a good metric
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics ⚠ Bad metrics are dangerous * You’ll focus on the wrong things, neglecting real issues
* You’ll change random numbers, but users will remain dissatisfied!
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics ✅ Real devices ✅ Real networks ✅ Sanity check Reflects real experiences * If it is run on real devices, that’s a good sign
* We need to know what real users networks are like
* Do a sanity check. When is your metric being triggered? Is that state actually what you think it is?
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Is a “continuous” function * Another quality of a good metric is one that is a continuous function
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Is a “continuous” function A small improvement in the metric relates to a small improvement in the UX. * Example of something that isn’t this: total bundle size, if we’re code splitting and downloading. Reducing may do nothing
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Is a “continuous” function Side effect: no cheating! * Think of the people looking at this metric as a greedy optimisation algorithm
* If they can cheat, they will
* The shortest path to improve the metric should be the one that will improve the user experience
* You shouldn’t be rewarded for loading a loading spinner very quickly, if that doesn’t result in a correspondingly good UX
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Never send a human To do a machine’s job * As Agent Smith would say, “Never send a human to do a machines job”
* Some metrics you can easily get in a repeatable way through monitoring
* Others (like auditing tools) are tempting to have as one offs
* Spend the time building automatic tooling to report. This is critical
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Why measure What to measure ✅ “Good” metrics ⚖ Metric choices ⚒ Ways to measure Protect performance With the knowledge of what is a good and bad metric, lets look and some, and think about it for our use cases
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics ⚡ Metric sound off! joshnelson.io/pages/metrics * Let’s finally have a look at some metrics
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Good for all apps With the knowledge of what is a good and bad metric, lets look and some, and think about it for our use cases
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Paint timing! First paint First contentful paint First meaningful paint * You can guess my favourite (the one that has meaningful in the name)
* First paint is when the browser first renders anything other than white
* First contentful is when the browser renders any elements (eg. spinner)
* First meaningful is when the browser renders something that is meaningful for the user (eg. data)
* Easy to measure first two, but the last one is the real pot of gold. Harder to measure this in a consistent way.
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Speed index Integrate the % of the page loaded over time * Speed index looks at what % of the page is loaded over time, retroactively
* We then score it based on how much of it was delivered, and how soon
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Load event Triggered after DOMContentLoaded, and after js downloads ✅ Easy to measure
✅ Easy to understand
⚠ No async data requests
⚠ May not be meaningful * This one is super common, available everywhere easy to implement
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics First byte When did the first byte arrive in the browser? ✅ Easy to measure
✅ Measures backend problems
⚠ May not be meaningful * Performance timing API can help you measure this
* It mightn’t mean anything if the first byte isn’t meaningful itself
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Time to interactive The time it takes until buttons work ✅ Measure user interactions
✅ Highly interactive apps
⚠ Need a polyfill
⚠ Less meaningful after page load * Works by detecting CPU idle time, and picks a point where probably buttons on your page will work
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics First input delay On the first click, how long did that take? ✅ Reflects actual user pain
⚠ Depends on user input
(focus on 90th percentile) * First input delay reflects actual user problems
* Will naturally depend on when the user interacts
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Avoid With the knowledge of what is a good and bad metric, lets look and some, and think about it for our use cases
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics DNS lookup GET request Parse HTML Fetch js ⏳DOMContentLoaded ⏳First byte Fetch CSS ⏳Navigation
timing API Parse js Parse CSS Layout / paint ⏳First contentful paint * Remember this timeline from before? Now we can put some events on it to measure the various bits!
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Fetch more data ⏳First input delay Event handler Push state Load data Rerender ⏳Page reloaded * You might reload data on button click
* This is quite a different experience from the initial load, but a load nonetheless!
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Metric starter pack Time to interactive
Bundle size
First input delay ✨ First meaningful paint * If you don’t have time, these three strike a good balance between ease of adoption and in general meaningfulness.
* But if you do, you should investigate first meaningful paint.
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Why measure What to measure ✅ “Good” metrics ⚖ Metric choices ⚒ Ways to measure Protect performance Let’s take a look at what we can measure in order to improve the UX
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics User timing API * User timing API is available so you can get custom insights
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics github.com/jpnelson/react-component-timing * We have a tool that we use to measure component load speeds and first meaningful paint, built into React
* It also marks things using the user timing API, and integrates with tracking tools
* This is still being refined but open to PR’s and comments, please!
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Lighthouse * Gives you actionable things to do, making it a very good debugging tool
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics We need to actually fix the stuff! Great, you’ve chosen a meaningful metric and you have the tools to measure it. Now what?
* You need to fix performance issues
* This is the easy bit! Plenty of info (including later today) about how to do that
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Let’s make this new page! And maybe we have the background put a video in instead?
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Can we add a video? Design: ✅ Product: ✅ Eng: “It’ll be slower” Designers think it’s great
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Performance budgets Performance budgets are a structured way to have this conversation
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics 1. Choose the right metrics
2. Get everyone to agree on a limit
3. Bring it up during planning
4. Figure out how to stay in budget Performance budgets * Getting people to agree might be easier than you think. People will agree to a meaningful metric.
Eg, “Our time to interactive budget will be <=1 second” or “<= 1mb” (assuming the page weight is linked to a better UX)
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics performancebudget.io * Estimate what a target bundle size would be for a given time
* Normal considerations for bundle sizes though: not automatically meaningful
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Apdex * A huge part of performance metrics I haven’t mentioned yet: What about Apdex?
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics 0.1s 1s 10s Instant “Flow ” Task sw itch < < < < < < < < < < < < < 4s < < Load=DOMContentReady * With the wrong metric, you could be reporting incorrectly what real user experiences are like.
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Pick the right metric * Check what your metric is. Sanity check it. Is that really capturing the true user experience?
@nelsonjoshpaul jpnelson https://tinyurl.com/meaningful-performance-metrics Let’s make the internet better * The internet is a vehicle for free information for all
* For me, $’s aren’t inspiring. But the free internet, the open exchange of information, is literally impossible for some people, unless the internet is fast.
jpnelson
okuramasafumi
y0gi
kubode
microcms
yuya4
mski_iksm
leichteckig
77web
akkeylab
punchdrunker
paper2
oliver_diary
scottboms
shpigford
colly
zakiyama
reverentgeek
eileencodes
brettharned
bcantrill
bkeepers
smashingmag
dotmariusz
hannesfritz