The Apdex Index Revealed: Triangles Tell The Tale

The purpose of this talk is not to improve upon or redesign the Apdex Index. Rather, it is to explain the underlying construction of the Apdex Index, as it is currently defined. Then, I consider other ways the Apdex metrics might be applied; especially from a performance visualization standpoint.

Transcript

1. The Apdex Index Revealed Triangles Tell The Tale Dr. Neil J. Gunther A.A. Michelson Award 2008 Performance Dynamics July 30, 2009 SM c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 1 / 49
2. Introduction Outline 1 Introduction 2 Review of Apdex Metrics 3 Critique And Defense Of Apdex 4 Now, About That Fraction 5 Seeing It All With Barry3 6 Conclusions c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 2 / 49
3. Introduction What I Won’t Be Doing Remark The purpose of this talk is not to improve upon or redesign the Apdex Index in any way a. My goal is simply want to explain the Apdex Index, as it is currently defined b. Then want to consider other ways the Apdex metrics might be applied; especially from a performance visualization standpoint. aThat’s possibly a different talk. bAs you’ll see, its not to be so simple as it looks. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 3 / 49
4. Introduction Where We’ll End Up User response-time of a single Web application Measured from 5 different locations (dots) Colors rate the Apdex Index value Want dots to cluster near top-left of triangle (animation) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 4 / 49
5. Review of Apdex Metrics Outline 1 Introduction 2 Review of Apdex Metrics 3 Critique And Defense Of Apdex 4 Now, About That Fraction 5 Seeing It All With Barry3 6 Conclusions c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 5 / 49
6. Review of Apdex Metrics Apdex Definitions And Metrics 1 Apdex Thresholds 2 Apdex Zones 3 Apdex Counts 4 Apdex Index 5 Apdex Rating c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 6 / 49
7. Review of Apdex Metrics Apdex Thresholds Definition 1 T is a user-defined time, below which response-time performance is considered acceptable. 2 F = 4T is an Apdex-defined time threshold, above which response-time performance is considered unacceptable. Example In Figure 1, I chose T = 6 seconds. This choice automatically sets F = 4T to 24 seconds. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 7 / 49
8. Review of Apdex Metrics Example Application Data Figure: Histogram of measured response times for a local application. Gold posts: 1 T = 6 seconds (left vertical line) 2 F = 24 seconds (right vertical line) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 8 / 49
9. Review of Apdex Metrics Apdex Zones Definition 1 Satisfied Zone: Any response-time measurement that falls in the range 0 < Ri ≤ T 2 Tolerating Zone: Any response-time measurement that falls in the range T < Ri ≤ F 3 Frustrated Zone: Any response-time measurement that falls in the range Ri > F You can think of as these as buckets. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 9 / 49
10. Review of Apdex Metrics Apdex Counts Definition 1 Count of response times in Satisfied bucket: Cs 2 Count of response times in Tolerating bucket: Ct 3 Count of response times in Frustrated bucket: Cf Example The respective counts for the ATTDISP application with T = 6 are: 1 Satisfied count: Cs = 3 2 Tolerating count: Ct = 678 3 Frustrated count: Cf = 47 c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 10 / 49
11. Review of Apdex Metrics Apdex Index Definition The Apdex Index for threshold T is defined as: AT = Cs + 1 2 Ct Cs + Ct + Cf (1) and its value always lies in the range: 0 < AT < 1. You can think of it as % score. More on this later. Example For ATTDISP with T = 6: A6 = 3 + 339 728 = 342 728 = 0.46978 (2) The color codings are exemplar, not mandatory. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 11 / 49
12. Review of Apdex Metrics Apdex Rating Definition Table: Apdex Index Ratings Index Range Apdex Rating Symbol Color 0.94 to 1.00 Excellent E 0.85 to 0.93 Good G 0.70 to 0.84 Fair F 0.50 to 0.69 Poor P 0.00 to 0.49 Unacceptable U Example Apdex Rating for ATTDISP, AT = 0.46978, is Unacceptable. That’s consistent with why I analyzing its performance problems. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 12 / 49
13. Review of Apdex Metrics The Apdex Process (2) Sorted samples (4) AT (1) Response times (3) Counts (5) Rating Satisfied Tolerating Frustrated c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 13 / 49
14. Critique And Defense Of Apdex Outline 1 Introduction 2 Review of Apdex Metrics 3 Critique And Defense Of Apdex 4 Now, About That Fraction 5 Seeing It All With Barry3 6 Conclusions c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 14 / 49
15. Critique And Defense Of Apdex Common Criticisms A single number is not useful! Where did that 1/2 come from in Definition 6? Apdex is just a poor man’s distribution. Apdex doesn’t accommodate heavy-tail distributions. The same Apdex Index can have different Cs, Ct , and Cf values? I need to review the service levels of hundreds of applications! Why does my Apdex Index stink? c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 15 / 49
16. Critique And Defense Of Apdex Single Numbers Are Useless AT intentially defined as single number for management. Plenty of single-number performance metrics in existence, e.g., SPEC CPU2006. Can resolve details using their web site. I don’t have a problem, as long as I can resolve AT easily into its component Cs, Ct , and Cf values when needed. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 16 / 49
17. Critique And Defense Of Apdex Who Ordered That 1/2!? Where did this come from? AT = Cs + 1 2 Ct Cs + Ct + Cf Good question! Not explained in Apdex Alliance Technical Specification. (PDF) Has bothered me since I first learnt about Apdex ca. 2007. Has bothered other people too. Several presentations at CMG 2008. Let me come back to this shortly. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 17 / 49
18. Critique And Defense Of Apdex Just Poor Man’s Distribution AT is neither a probability or a probability distribution. AT is a figure of merit. More on this in a minute. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 18 / 49
19. Critique And Defense Of Apdex Can’t See Heavy-tailed Distributions That’s right. ... But you really shouldn’t have any. At the application level, heavy-tailed dsns are regarded as undesirable. Let’s not get into network packet self-similarity. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 19 / 49
20. Critique And Defense Of Apdex Hundreds of Applications Good point! Many modern enterprises run 10s of applications per server. There can easily be hundreds or even thousands of servers. Apdex Alliance Technical Specification offers no guidance on this issue. Barry3 solves this one. More later. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 20 / 49
21. Critique And Defense Of Apdex Why Does My AT Stink? Good question! Apdex Alliance Technical Specification offers no guidance on this issue either. Even an executive mgr is likely to ask why AT was green yesterday, but red today. It’s at this point that the single number can become a problem. The Apdex Alliance has over-emphasized simplicity at the expense of information. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 21 / 49
22. Now, About That Fraction Outline 1 Introduction 2 Review of Apdex Metrics 3 Critique And Defense Of Apdex 4 Now, About That Fraction 5 Seeing It All With Barry3 6 Conclusions c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 22 / 49
23. Now, About That Fraction Explanations for That 1/2 Why not 1, ... or 1/3 ... or 1/e ...? Apdex Alliance offers no explanation. Others have also considered this question 1 2. 1A. Akers, ”Is Apdex Sharp Enough?,” Apdex Mtg, CMG 2008 2Y. Ding, ”An Analytical Model for Apdex,” Apdex Mtg, CMG 2008 c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 23 / 49
24. Now, About That Fraction Ding’s Explanation Assumes an exponential response time dsn e.g., M/M/1 Sensitivy analysis to explain the F = 4T Doesn’t really account for the 1/2 Remark Figure 1 is typical and not exponential. A more realistic model would be a chain of M/M/1 servers. The sum of M/M/1 exponential response times is a gamma dsn of response times. But the math analysis becomes much more complicated. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 24 / 49
25. Now, About That Fraction Akers’ Alternative Proposition Refine standard Apdex Index to be more representative of count dsn within the Tolerating Zone by partitioning it into n sub-zones: Example Let n = 4 sub-zones, then Ct becomes: Ct1, Ct2, Ct3, Ct4, and the modified Apdex Index is: A∗ T = Cs + 4 5 Ct1 + 3 5 Ct2 + 2 5 Ct3 + 1 5 Ct4 Cs + Ct + Cf (3) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 25 / 49
26. Now, About That Fraction Numerical Comparison Example Suppose Cs = Cf = 0 and Ct1 = Ct2 = Ct3 = 0, but Ct4 = 100. The proposed Index produces: A∗ T = 100/5 100 = 0.2 (4) By comparison, the standard Apdex Index produces: AT = 100/2 100 = 0.5 (5) The smaller A∗ T value comes from the preponderance of Tolerating counts being closer to the Frustrated bucket than the Satisfied bucket. Standard AT doesn’t see this bias. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 26 / 49
27. Now, About That Fraction Special Case Distribute the 100 Tolerating counts evenly across all n = 4 sub-zones. Example Cs = Cf = 0, but Ct1 = Ct2 = Ct3 = Ct4 = 25. Then numerator is: 4 5 × 25 + 3 5 × 25 + 2 5 × 25 + 1 5 × 25 = 25 5 (4 + 3 + 2 + 1) = 50 (6) Therefore: A∗ T = 50 100 = 0.5 (7) which is the same value as AT in eqn.(5). So A∗ T ≡ AT when the dsn of Tolerating counts is uniform. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 27 / 49
28. Now, About That Fraction Triangles Tell The Tale That was all very interesting but it hasn’t really explained where the 1/2 comes from in the standard Apdex definition. Suppose we simply divide the time-axis by the lower threshold T: 0, T, 2T, 3T, 4T, 5T. The Tolerating Zone covers: T, 2T, 3T, 4T, terminating at F = 4T, consistent with the Apdex definition. We also suppose that the Satisfied Zone has some counts in it. Here’s what that might look like for some arbitrary counts. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 28 / 49
29. Now, About That Fraction Ratio of Areas Consider 5 columns all of width T. dark area light area → 15T + 1 2 (25T + 25T + 25T) 15T + 25T + 25T + 25T + 5T = 105T 2 1 100T (8) Area is measured in event-seconds. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 29 / 49
30. Now, About That Fraction The Tolerating Triangle Ratio of areas: dark area light area = + ∆ All (9) Area of is 15 × T Area of ∆ is half base by height: 25 × 3T/2 Area of all rectangles is 100 × T + ∆ All = 105T 200T = 0.525 (10) Result is the same as using the definition of AT : 15 + 37.5 100 = 52.5 100 = 0.525 (11) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 30 / 49
31. Now, About That Fraction The ATTDISP Triangle Let’s check this conclusion against the ATTDISP data. 3T + 1 2 (3T × 226) 3T + 226T + 226T + 226T + 47T = 342T 728T = 0.46978 (12) Same result as eqn.(2) for A6. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 31 / 49
32. Now, About That Fraction Adaptive Figure of Merit Theorem (Gunther 2009) The Apdex Index AT is a figure of merit that compares the total number of response-time samples (light area) with a profile containing a set of idealized response-time samples (dark area). Ideally all samples should fall into the first (Satisfied) bucket. Failing that, the remainder of the samples in the second (Tolerating) bucket should be monotonically decreasing toward the third (Frustrated) bucket. Corollary The Apdex Index adaptively scores the percentage of application response-time samples that conform to the expected Apdex profile. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 32 / 49
33. Now, About That Fraction Connection with Akers c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 33 / 49
34. Seeing It All With Barry3 Outline 1 Introduction 2 Review of Apdex Metrics 3 Critique And Defense Of Apdex 4 Now, About That Fraction 5 Seeing It All With Barry3 6 Conclusions c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 34 / 49
35. Seeing It All With Barry3 Normalized Counts s = Cs Cs + Ct + Cf (13) t = Ct Cs + Ct + Cf (14) f = Cf Cs + Ct + Cf (15) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 35 / 49
36. Seeing It All With Barry3 Normalized Apdex Index Standard Apdex Index in eqn.(1) AT = Cs + 1 2 Ct Cs + Ct + Cf becomes Definition AT = s + 1 2 t Akers modified Index (n = 4) becomes: A∗ T = s + 4 5 t1 + 3 5 t2 + 2 5 t3 + 1 5 t4 c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 36 / 49
37. Seeing It All With Barry3 Sum Rule Cs Cs + Ct + Cf + Ct Cs + Ct + Cf + Cf Cs + Ct + Cf = Cs + Ct + Cf Cs + Ct + Cf (16) which reduces to: s + t + f = 1 (17) Proposition Three variables or metrics that conform to a sum rule can be represented in a barycentric coordinate system. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 37 / 49
38. Seeing It All With Barry3 Cartesian vs. Barycentric Coordinates c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 38 / 49
39. Seeing It All With Barry3 Tri-angle-ometry 2 2 2
40. Seeing It All With Barry3 Tri-angle-ometry 2 2 2 √ 3 1 1
41. Seeing It All With Barry3 Tri-angle-ometry 2 2 2 √ 3 1 1 × 1/ √ 3
42. Seeing It All With Barry3 Tri-angle-ometry 2 2 2 √ 3 1 1 × 1/ √ 3 2/ √ 3 1 1/ √ 3 c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 39 / 49
43. Seeing It All With Barry3 Centroid in Barry3 b1 + b2 + b3 = 0.3333 + 0.3333 + 0.3333 = 1.0 (height) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 40 / 49
44. Seeing It All With Barry3 California on Day 26 in Barry3 b1 + b2 + b3 = 0.42 + 0.31 + 0.27 = 1.0 (height) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 41 / 49
45. Seeing It All With Barry3 ATTDISP in Barry3 b1 + b2 + b3 = 0.0041 + 0.9300 + 0.0650 = 1.0 (height) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 42 / 49
46. Seeing It All With Barry3 Horizontal Isoclines Lift the apex (s) to a height of z = 1. Leave the other (t, f) vertices at z = 0. AT = (1 × s) + (0 × t) + (0 × f) = s Goal Head ”north” to the blue region. Same as constant-s isoclines. So, can be “east” or “west” of vertical centerline. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 43 / 49
47. Seeing It All With Barry3 Add Tolerating Weight Also lift left (t) vertex to height of z = 1. Leave the right (f) vertex at z = 0. AT = (1 × s) + (1 × t) + (0 × f) = s + t Goal Go ”west” young man! Same as constant-f isoclines. So, could be in “southwest” (t) corner. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 44 / 49
48. Seeing It All With Barry3 Compromise Weights Lower the left vertex to z = 1 2 . Another interpretation of 1/2. AT = (1 × s) + ( 1 2 × t) + (0 × f) = s + 1 2 t Goal Head ”northwest”. Same as constant-AT isoclines. All (s, t, f) are now different. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 45 / 49
49. Seeing It All With Barry3 Bringing It All Together c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 46 / 49
50. Seeing It All With Barry3 Toblerone View of Barry3 c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 47 / 49
51. Conclusions Outline 1 Introduction 2 Review of Apdex Metrics 3 Critique And Defense Of Apdex 4 Now, About That Fraction 5 Seeing It All With Barry3 6 Conclusions c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 48 / 49
52. Conclusions Conclusions The Apdex Index is a figure-of-merit which scores the percentage of application response-times that conform to its adaptive areal profile. The Tolerating triangle explains the factor of 1 2 in the definition of the Apdex Index. The Barry3 triangle provides a visual way to incorporate all the Apdex metrics in one simple diagram. Lifting the Barry3 triangle into a third z-dimension explains both the factor of 1/2 and the Apdex Ratings as diagonal colored bands. Longer article in MeasureIT (Feb 2009 issue?) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 49 / 49