20 April 2019, Tbilisi, Georgia
Latency Measurement and
Results Analysis
Arseniy Marinin, NFT Analyst, Exactpro
Alexander Tregubov, Senior NFT Analyst, Exactpro
Slide 2
Slide 2 text
20 April 2019 Tbilisi, Georgia
Low latency and big data
TESTING is a process of researching software with the intent of finding bugs.
RESEARCH comes with data analysis.
In case of performance testing - HUGE amount of data.
2
Slide 3
Slide 3 text
20 April 2019 Tbilisi, Georgia
Gathering the right data
What do we really want to measure?
How do we want to measure?
3
Slide 4
Slide 4 text
20 April 2019 Tbilisi, Georgia
Gathering the right data
4
Slide 5
Slide 5 text
20 April 2019 Tbilisi, Georgia
How do we want to measure?
B
A
5
Slide 6
Slide 6 text
20 April 2019 Tbilisi, Georgia
Measurement process
User behavior
Load composition
Load configuration: ramp-up vs. constant
6
Slide 7
Slide 7 text
20 April 2019 Tbilisi, Georgia
Gathering raw data
7
Slide 8
Slide 8 text
20 April 2019 Tbilisi, Georgia
Gathering raw data
8
Slide 9
Slide 9 text
20 April 2019 Tbilisi, Georgia
Aggregation
Distribution
Detalization and anomalies
9
Slide 10
Slide 10 text
20 April 2019 Tbilisi, Georgia 10
Slide 11
Slide 11 text
20 April 2019 Tbilisi, Georgia 11
Slide 12
Slide 12 text
20 April 2019 Tbilisi, Georgia
Average color of image
#9d9ca0
12
Slide 13
Slide 13 text
20 April 2019 Tbilisi, Georgia
Aggregation
Distribution
Detalization and anomalies
13
Slide 14
Slide 14 text
20 April 2019 Tbilisi, Georgia
Distribution
14
Slide 15
Slide 15 text
20 April 2019 Tbilisi, Georgia 15
Slide 16
Slide 16 text
20 April 2019 Tbilisi, Georgia 16
Slide 17
Slide 17 text
20 April 2019 Tbilisi, Georgia
Histogram
17
Slide 18
Slide 18 text
20 April 2019 Tbilisi, Georgia
Distribution of data
18
Slide 19
Slide 19 text
20 April 2019 Tbilisi, Georgia 19
Slide 20
Slide 20 text
20 April 2019 Tbilisi, Georgia
What about latency stats distribution?
20
Slide 21
Slide 21 text
20 April 2019 Tbilisi, Georgia
How real distribution looks like?
21
Slide 22
Slide 22 text
20 April 2019 Tbilisi, Georgia
How we should compare 2 releases?
22
Slide 23
Slide 23 text
20 April 2019 Tbilisi, Georgia
How we should compare 2 releases?
23
Slide 24
Slide 24 text
20 April 2019 Tbilisi, Georgia
Aggregation
Distribution
Detalization and anomalies
24
Slide 25
Slide 25 text
20 April 2019 Tbilisi, Georgia
John Snow (281 AC - 302 AC, 302 AC - …)
25
Slide 26
Slide 26 text
20 April 2019 Tbilisi, Georgia
John Snow (1813 - 1858)
One of the founders of modern
epidemiology.
Cholera outbreak in Soho,
London, in 1854.
26
Slide 27
Slide 27 text
20 April 2019 Tbilisi, Georgia
London Cholera Epidemic of 1854
27
Slide 28
Slide 28 text
20 April 2019 Tbilisi, Georgia 28
Slide 29
Slide 29 text
20 April 2019 Tbilisi, Georgia 29
Slide 30
Slide 30 text
20 April 2019 Tbilisi, Georgia 30
Slide 31
Slide 31 text
20 April 2019 Tbilisi, Georgia
Going into details
31
Slide 32
Slide 32 text
20 April 2019 Tbilisi, Georgia
Looking at the extremes and anomalies
32
Slide 33
Slide 33 text
20 April 2019 Tbilisi, Georgia
The Fastest vs. The Slowest
169 requests from 47 domains
4 MB transferred
Pegs CPU for 4 seconds
Rendering doesn’t begin for 20 seconds
33
Slide 34
Slide 34 text
20 April 2019 Tbilisi, Georgia
The Fastest vs. The Slowest
34
Slide 35
Slide 35 text
20 April 2019 Tbilisi, Georgia
Percentile
35
Slide 36
Slide 36 text
20 April 2019 Tbilisi, Georgia
Median
36
Slide 37
Slide 37 text
20 April 2019 Tbilisi, Georgia
Max 99% percentile graph
37
Slide 38
Slide 38 text
20 April 2019 Tbilisi, Georgia
Conclusion
Keep looking at your data
Investigate anomalies
Go into details
38
Inspired by:
Velocity 2011: John Rauser, "Look at Your Data"
Velocity 2012: John Rauser, "Investigating Anomalies"
Slide 39
Slide 39 text
20 April 2019 Tbilisi, Georgia
Thanks for your attention!
39