[2019.03 Meetup][TALK #2] João Acabado - Quantiles over Percentiles When and How

When performance matters there should be a clear and easy way of setting service level expectations. The way you measure latency determines how can we define our objectives.
Let's brush up your statistics, Quantiles and Percentiles helps us define performance objectives and indicators and have everyone on the same page whenever our services get too slow.

João Acabado is Resilience Engineering Manager @ Sky Portugal. He's always looking for the easiest hard problem that can be solved. Has been looking to bridge the delivery of software solutions with their operation in the past couple years.

Transcript

1. Quantiles over Percentiles: When and How João Acabado
2. 2 Reliability Manager @ Sky • 7 countries, 80+ devices • 3B+ streams in one year • 10+ exabytes of content delivered in one year • Technology Centre in Lisbon • 200 people working on Back-end, Mobile and Frontend • We’re hiring! • Scaling services up to 10k TPS • 3B+ search queries per month Set up for several million concurrent users Fast ramp-ups
3. 3 Motivation Speed Matters Everyone wants responsive services Monitoring Measure twice cut once Visualizing Tracking the right data A B
4. 4 Percentiles vs Quantiles P 80th = 1,85m 80% Q(1,85m) =
5. 5 How to Measuring Service A A B 0 100 200 300 400 500 600 700 #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 P 80th = 609ms 80% Q(609ms) =
6. 0 100 200 300 400 500 600 700 #2 #8 #1 #3 #6 #5 #4 #7 #9 #10 Response Time Measuring Service A Service Level Objective Q(609ms) = 80% Service Level Indicator P 80th = 609ms
7. 7 Service Level Objective Q(1000ms)=84.31% 1000ms Absolute vs Relative measure Service Level Indicator P50, P99
8. 8 When: Micro-Services (lots of them!)
9. 9 When: EastEnders Effect
10. 10 When: Comparative Analysis SLO: 3s Q(3s)=Y
12. 12 How: Tools and Techniques Tools? Kibana, Grafana with ElasticSearch, Graphite or Prometheus Techniques? Reservoir Sampling, Histograms Further References? "How NOT to Measure Latency" by Gil Tene How? Build it, run it!
13. Set SLO’s for your services and their dependencies and create simple visualizations for them! Questions? joao.acabado@sky.uk
14. 14 Service Radiator