Averages Suck

@_jsimonovski Averages Suck not really but there are some better
ways of getting important information from your systems depending on your use-case the talk Jordan Simonovski

@_jsimonovski Do you ever ﬁnd yourself… Not knowing why your
users aren’t happy with your application? Thinking that the metrics you’re using are a good enough indicator of quality of your application?

@_jsimonovski It’s Because You’re Solely Relying on Averages

@_jsimonovski Are Averages Actually That Bad?

@_jsimonovski TL;DR;

@_jsimonovski Should you trust everything that I say?

@_jsimonovski

@_jsimonovski This isn’t really a rant about averages. This is
a rant about meaningful metrics of customer happiness

@_jsimonovski Monitoring as Testing in Prod Cindy Sridharan - https://medium.com/@copyconstruct/testing-in-production-the-safe-way-18ca102d0ef1
• Testing in production doesn’t mean not testing at all. • Monitoring isn’t just about if things are up • Testing in prod means verifying that things are working the way we intended • Continually testing our application in prod is how we determine if our customers are happy

@_jsimonovski Some averages may not be the best indicator of
customer happiness

@_jsimonovski Stats 101 Mean/Average The sum of the values divided
by the number of values.

@_jsimonovski Stats 101 Median The median is the value separating
the higher half from the lower half of a data sample. For a data set, it may be thought of as the "middle" value

@_jsimonovski Stats 101 Mode The mode of a set of
data values is the value that appears most often.

@_jsimonovski Why Are Averages Bad? • Data is easy to
mangle in a small data set with large outliers • Important information can easily be obfuscated in a big enough data set • They can cause false positives/negatives which will probably wake you up at 2am (at some things)

@_jsimonovski Performance Degradation • Alerts based on averages cause false
negatives • In many cases the 90th percentile is treated as though most of our users have been aﬀected by performance degradation • What does the 90th percentile tell us?

@_jsimonovski False Positives/Negatives • Make it harder to identify real
issues • Have people turning oﬀ alerts on things that could potentially be useful • Mean that you’ll be getting up multiple times a night • $$$

@_jsimonovski So, What is Better than Averages? Automatic • Automatic
baselining • A standard deviation represents 33% of the bell curve • 2x standard deviation can be seen as erroneous • e.g. you can raise an alert if response times are in the 2x standard deviation or the slowest 16.5% • One problem. • Most real-world scenarios are not bell- curved. Michael Kopp - https://www.dynatrace.com/news/blog/why-averages-suck-and-percentiles-are-great/

@_jsimonovski What about Percentiles? •✨ Histograms are pretty • Distributions
of data are easier to decipher/debug • Easier to ﬁne-tune • Better for automatic baselining

@_jsimonovski What kind of information can I extract from percentiles?
• My service X gets 12,000 hits a day • My p95 is 95.2ms • Meaning that 5% of my customers get a latency longer than that (~600 customers)

@_jsimonovski How do I get this information with X? •
You can now turn any metric into a distribution • APM Metrics are all available in percentiles • You can calculate percentiles using histogram quantiles • histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) // gets a p95 • Automatically does metric aggregation • Use “agg” object when getting a metric to get the percentile breakdown

@_jsimonovski How Else are we Impacting User Happiness? • Alerting
on Error Rates • Tie your alerting in with your SLOs and detection time of errors will go down If my Error SLO is 99.9%, alerting on a 0.01% breach means I am ﬁnding an error within 10 minutes of deploying a change SRE Workbook - https://landing.google.com/sre/workbook/chapters/alerting-on-slos/

@_jsimonovski Where do Averages Make Sense? • Infrastructure Monitoring •
Alerting on sustained high usage of resources: • CPU • Disk • Applications • Request Rates

@_jsimonovski TL;DW; • Averages are good sometimes • We can
beneﬁt most from looking at load averages like CPU, memory, disk, etc. • Percentiles are good sometimes too. • Percentiles give us an easy way of understanding how our applications are being used

@_jsimonovski MONIT ORING

@_jsimonovski THANKS.

Averages Suck

Averages Suck

Jordan Simonovski

Other Decks in Technology

Featured

Transcript

@_jsimonovski Averages Suck not really but there are some better

@_jsimonovski Do you ever ﬁnd yourself… Not knowing why your

@_jsimonovski It’s Because You’re Solely Relying on Averages

@_jsimonovski Are Averages Actually That Bad?

@_jsimonovski TL;DR;

@_jsimonovski Should you trust everything that I say?

@_jsimonovski

@_jsimonovski This isn’t really a rant about averages. This is

@_jsimonovski Monitoring as Testing in Prod Cindy Sridharan - https://medium.com/@copyconstruct/testing-in-production-the-safe-way-18ca102d0ef1

@_jsimonovski Some averages may not be the best indicator of

@_jsimonovski Stats 101 Mean/Average The sum of the values divided

@_jsimonovski Stats 101 Median The median is the value separating

@_jsimonovski Stats 101 Mode The mode of a set of

@_jsimonovski Why Are Averages Bad? • Data is easy to

@_jsimonovski Performance Degradation • Alerts based on averages cause false

@_jsimonovski False Positives/Negatives • Make it harder to identify real

@_jsimonovski So, What is Better than Averages? Automatic • Automatic

@_jsimonovski What about Percentiles? •✨ Histograms are pretty • Distributions

@_jsimonovski What kind of information can I extract from percentiles?

@_jsimonovski How do I get this information with X? •

@_jsimonovski How Else are we Impacting User Happiness? • Alerting

@_jsimonovski Where do Averages Make Sense? • Infrastructure Monitoring •

@_jsimonovski TL;DW; • Averages are good sometimes • We can

@_jsimonovski MONIT ORING

@_jsimonovski THANKS.