Slide 1

Slide 1 text

The Apdex Index Revealed Triangles Tell The Tale Dr. Neil J. Gunther A.A. Michelson Award 2008 Performance Dynamics July 30, 2009 SM c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 1 / 49

Slide 2

Slide 2 text

Introduction Outline 1 Introduction 2 Review of Apdex Metrics 3 Critique And Defense Of Apdex 4 Now, About That Fraction 5 Seeing It All With Barry3 6 Conclusions c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 2 / 49

Slide 3

Slide 3 text

Introduction What I Won’t Be Doing Remark The purpose of this talk is not to improve upon or redesign the Apdex Index in any way a. My goal is simply want to explain the Apdex Index, as it is currently defined b. Then want to consider other ways the Apdex metrics might be applied; especially from a performance visualization standpoint. aThat’s possibly a different talk. bAs you’ll see, its not to be so simple as it looks. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 3 / 49

Slide 4

Slide 4 text

Introduction Where We’ll End Up User response-time of a single Web application Measured from 5 different locations (dots) Colors rate the Apdex Index value Want dots to cluster near top-left of triangle (animation) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 4 / 49

Slide 5

Slide 5 text

Review of Apdex Metrics Outline 1 Introduction 2 Review of Apdex Metrics 3 Critique And Defense Of Apdex 4 Now, About That Fraction 5 Seeing It All With Barry3 6 Conclusions c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 5 / 49

Slide 6

Slide 6 text

Review of Apdex Metrics Apdex Definitions And Metrics 1 Apdex Thresholds 2 Apdex Zones 3 Apdex Counts 4 Apdex Index 5 Apdex Rating c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 6 / 49

Slide 7

Slide 7 text

Review of Apdex Metrics Apdex Thresholds Definition 1 T is a user-defined time, below which response-time performance is considered acceptable. 2 F = 4T is an Apdex-defined time threshold, above which response-time performance is considered unacceptable. Example In Figure 1, I chose T = 6 seconds. This choice automatically sets F = 4T to 24 seconds. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 7 / 49

Slide 8

Slide 8 text

Review of Apdex Metrics Example Application Data Figure: Histogram of measured response times for a local application. Gold posts: 1 T = 6 seconds (left vertical line) 2 F = 24 seconds (right vertical line) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 8 / 49

Slide 9

Slide 9 text

Review of Apdex Metrics Apdex Zones Definition 1 Satisfied Zone: Any response-time measurement that falls in the range 0 < Ri ≤ T 2 Tolerating Zone: Any response-time measurement that falls in the range T < Ri ≤ F 3 Frustrated Zone: Any response-time measurement that falls in the range Ri > F You can think of as these as buckets. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 9 / 49

Slide 10

Slide 10 text

Review of Apdex Metrics Apdex Counts Definition 1 Count of response times in Satisfied bucket: Cs 2 Count of response times in Tolerating bucket: Ct 3 Count of response times in Frustrated bucket: Cf Example The respective counts for the ATTDISP application with T = 6 are: 1 Satisfied count: Cs = 3 2 Tolerating count: Ct = 678 3 Frustrated count: Cf = 47 c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 10 / 49

Slide 11

Slide 11 text

Review of Apdex Metrics Apdex Index Definition The Apdex Index for threshold T is defined as: AT = Cs + 1 2 Ct Cs + Ct + Cf (1) and its value always lies in the range: 0 < AT < 1. You can think of it as % score. More on this later. Example For ATTDISP with T = 6: A6 = 3 + 339 728 = 342 728 = 0.46978 (2) The color codings are exemplar, not mandatory. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 11 / 49

Slide 12

Slide 12 text

Review of Apdex Metrics Apdex Rating Definition Table: Apdex Index Ratings Index Range Apdex Rating Symbol Color 0.94 to 1.00 Excellent E 0.85 to 0.93 Good G 0.70 to 0.84 Fair F 0.50 to 0.69 Poor P 0.00 to 0.49 Unacceptable U Example Apdex Rating for ATTDISP, AT = 0.46978, is Unacceptable. That’s consistent with why I analyzing its performance problems. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 12 / 49

Slide 13

Slide 13 text

Review of Apdex Metrics The Apdex Process (2) Sorted samples (4) AT (1) Response times (3) Counts (5) Rating Satisfied Tolerating Frustrated c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 13 / 49

Slide 14

Slide 14 text

Critique And Defense Of Apdex Outline 1 Introduction 2 Review of Apdex Metrics 3 Critique And Defense Of Apdex 4 Now, About That Fraction 5 Seeing It All With Barry3 6 Conclusions c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 14 / 49

Slide 15

Slide 15 text

Critique And Defense Of Apdex Common Criticisms A single number is not useful! Where did that 1/2 come from in Definition 6? Apdex is just a poor man’s distribution. Apdex doesn’t accommodate heavy-tail distributions. The same Apdex Index can have different Cs, Ct , and Cf values? I need to review the service levels of hundreds of applications! Why does my Apdex Index stink? c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 15 / 49

Slide 16

Slide 16 text

Critique And Defense Of Apdex Single Numbers Are Useless AT intentially defined as single number for management. Plenty of single-number performance metrics in existence, e.g., SPEC CPU2006. Can resolve details using their web site. I don’t have a problem, as long as I can resolve AT easily into its component Cs, Ct , and Cf values when needed. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 16 / 49

Slide 17

Slide 17 text

Critique And Defense Of Apdex Who Ordered That 1/2!? Where did this come from? AT = Cs + 1 2 Ct Cs + Ct + Cf Good question! Not explained in Apdex Alliance Technical Specification. (PDF) Has bothered me since I first learnt about Apdex ca. 2007. Has bothered other people too. Several presentations at CMG 2008. Let me come back to this shortly. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 17 / 49

Slide 18

Slide 18 text

Critique And Defense Of Apdex Just Poor Man’s Distribution AT is neither a probability or a probability distribution. AT is a figure of merit. More on this in a minute. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 18 / 49

Slide 19

Slide 19 text

Critique And Defense Of Apdex Can’t See Heavy-tailed Distributions That’s right. ... But you really shouldn’t have any. At the application level, heavy-tailed dsns are regarded as undesirable. Let’s not get into network packet self-similarity. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 19 / 49

Slide 20

Slide 20 text

Critique And Defense Of Apdex Hundreds of Applications Good point! Many modern enterprises run 10s of applications per server. There can easily be hundreds or even thousands of servers. Apdex Alliance Technical Specification offers no guidance on this issue. Barry3 solves this one. More later. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 20 / 49

Slide 21

Slide 21 text

Critique And Defense Of Apdex Why Does My AT Stink? Good question! Apdex Alliance Technical Specification offers no guidance on this issue either. Even an executive mgr is likely to ask why AT was green yesterday, but red today. It’s at this point that the single number can become a problem. The Apdex Alliance has over-emphasized simplicity at the expense of information. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 21 / 49

Slide 22

Slide 22 text

Now, About That Fraction Outline 1 Introduction 2 Review of Apdex Metrics 3 Critique And Defense Of Apdex 4 Now, About That Fraction 5 Seeing It All With Barry3 6 Conclusions c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 22 / 49

Slide 23

Slide 23 text

Now, About That Fraction Explanations for That 1/2 Why not 1, ... or 1/3 ... or 1/e ...? Apdex Alliance offers no explanation. Others have also considered this question 1 2. 1A. Akers, ”Is Apdex Sharp Enough?,” Apdex Mtg, CMG 2008 2Y. Ding, ”An Analytical Model for Apdex,” Apdex Mtg, CMG 2008 c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 23 / 49

Slide 24

Slide 24 text

Now, About That Fraction Ding’s Explanation Assumes an exponential response time dsn e.g., M/M/1 Sensitivy analysis to explain the F = 4T Doesn’t really account for the 1/2 Remark Figure 1 is typical and not exponential. A more realistic model would be a chain of M/M/1 servers. The sum of M/M/1 exponential response times is a gamma dsn of response times. But the math analysis becomes much more complicated. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 24 / 49

Slide 25

Slide 25 text

Now, About That Fraction Akers’ Alternative Proposition Refine standard Apdex Index to be more representative of count dsn within the Tolerating Zone by partitioning it into n sub-zones: Example Let n = 4 sub-zones, then Ct becomes: Ct1, Ct2, Ct3, Ct4, and the modified Apdex Index is: A∗ T = Cs + 4 5 Ct1 + 3 5 Ct2 + 2 5 Ct3 + 1 5 Ct4 Cs + Ct + Cf (3) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 25 / 49

Slide 26

Slide 26 text

Now, About That Fraction Numerical Comparison Example Suppose Cs = Cf = 0 and Ct1 = Ct2 = Ct3 = 0, but Ct4 = 100. The proposed Index produces: A∗ T = 100/5 100 = 0.2 (4) By comparison, the standard Apdex Index produces: AT = 100/2 100 = 0.5 (5) The smaller A∗ T value comes from the preponderance of Tolerating counts being closer to the Frustrated bucket than the Satisfied bucket. Standard AT doesn’t see this bias. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 26 / 49

Slide 27

Slide 27 text

Now, About That Fraction Special Case Distribute the 100 Tolerating counts evenly across all n = 4 sub-zones. Example Cs = Cf = 0, but Ct1 = Ct2 = Ct3 = Ct4 = 25. Then numerator is: 4 5 × 25 + 3 5 × 25 + 2 5 × 25 + 1 5 × 25 = 25 5 (4 + 3 + 2 + 1) = 50 (6) Therefore: A∗ T = 50 100 = 0.5 (7) which is the same value as AT in eqn.(5). So A∗ T ≡ AT when the dsn of Tolerating counts is uniform. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 27 / 49

Slide 28

Slide 28 text

Now, About That Fraction Triangles Tell The Tale That was all very interesting but it hasn’t really explained where the 1/2 comes from in the standard Apdex definition. Suppose we simply divide the time-axis by the lower threshold T: 0, T, 2T, 3T, 4T, 5T. The Tolerating Zone covers: T, 2T, 3T, 4T, terminating at F = 4T, consistent with the Apdex definition. We also suppose that the Satisfied Zone has some counts in it. Here’s what that might look like for some arbitrary counts. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 28 / 49

Slide 29

Slide 29 text

Now, About That Fraction Ratio of Areas Consider 5 columns all of width T. dark area light area → 15T + 1 2 (25T + 25T + 25T) 15T + 25T + 25T + 25T + 5T = 105T 2 1 100T (8) Area is measured in event-seconds. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 29 / 49

Slide 30

Slide 30 text

Now, About That Fraction The Tolerating Triangle Ratio of areas: dark area light area = + ∆ All (9) Area of is 15 × T Area of ∆ is half base by height: 25 × 3T/2 Area of all rectangles is 100 × T + ∆ All = 105T 200T = 0.525 (10) Result is the same as using the definition of AT : 15 + 37.5 100 = 52.5 100 = 0.525 (11) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 30 / 49

Slide 31

Slide 31 text

Now, About That Fraction The ATTDISP Triangle Let’s check this conclusion against the ATTDISP data. 3T + 1 2 (3T × 226) 3T + 226T + 226T + 226T + 47T = 342T 728T = 0.46978 (12) Same result as eqn.(2) for A6. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 31 / 49

Slide 32

Slide 32 text

Now, About That Fraction Adaptive Figure of Merit Theorem (Gunther 2009) The Apdex Index AT is a figure of merit that compares the total number of response-time samples (light area) with a profile containing a set of idealized response-time samples (dark area). Ideally all samples should fall into the first (Satisfied) bucket. Failing that, the remainder of the samples in the second (Tolerating) bucket should be monotonically decreasing toward the third (Frustrated) bucket. Corollary The Apdex Index adaptively scores the percentage of application response-time samples that conform to the expected Apdex profile. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 32 / 49

Slide 33

Slide 33 text

Now, About That Fraction Connection with Akers c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 33 / 49

Slide 34

Slide 34 text

Seeing It All With Barry3 Outline 1 Introduction 2 Review of Apdex Metrics 3 Critique And Defense Of Apdex 4 Now, About That Fraction 5 Seeing It All With Barry3 6 Conclusions c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 34 / 49

Slide 35

Slide 35 text

Seeing It All With Barry3 Normalized Counts s = Cs Cs + Ct + Cf (13) t = Ct Cs + Ct + Cf (14) f = Cf Cs + Ct + Cf (15) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 35 / 49

Slide 36

Slide 36 text

Seeing It All With Barry3 Normalized Apdex Index Standard Apdex Index in eqn.(1) AT = Cs + 1 2 Ct Cs + Ct + Cf becomes Definition AT = s + 1 2 t Akers modified Index (n = 4) becomes: A∗ T = s + 4 5 t1 + 3 5 t2 + 2 5 t3 + 1 5 t4 c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 36 / 49

Slide 37

Slide 37 text

Seeing It All With Barry3 Sum Rule Cs Cs + Ct + Cf + Ct Cs + Ct + Cf + Cf Cs + Ct + Cf = Cs + Ct + Cf Cs + Ct + Cf (16) which reduces to: s + t + f = 1 (17) Proposition Three variables or metrics that conform to a sum rule can be represented in a barycentric coordinate system. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 37 / 49

Slide 38

Slide 38 text

Seeing It All With Barry3 Cartesian vs. Barycentric Coordinates c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 38 / 49

Slide 39

Slide 39 text

Seeing It All With Barry3 Tri-angle-ometry 2 2 2

Slide 40

Slide 40 text

Seeing It All With Barry3 Tri-angle-ometry 2 2 2 √ 3 1 1

Slide 41

Slide 41 text

Seeing It All With Barry3 Tri-angle-ometry 2 2 2 √ 3 1 1 × 1/ √ 3

Slide 42

Slide 42 text

Seeing It All With Barry3 Tri-angle-ometry 2 2 2 √ 3 1 1 × 1/ √ 3 2/ √ 3 1 1/ √ 3 c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 39 / 49

Slide 43

Slide 43 text

Seeing It All With Barry3 Centroid in Barry3 b1 + b2 + b3 = 0.3333 + 0.3333 + 0.3333 = 1.0 (height) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 40 / 49

Slide 44

Slide 44 text

Seeing It All With Barry3 California on Day 26 in Barry3 b1 + b2 + b3 = 0.42 + 0.31 + 0.27 = 1.0 (height) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 41 / 49

Slide 45

Slide 45 text

Seeing It All With Barry3 ATTDISP in Barry3 b1 + b2 + b3 = 0.0041 + 0.9300 + 0.0650 = 1.0 (height) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 42 / 49

Slide 46

Slide 46 text

Seeing It All With Barry3 Horizontal Isoclines Lift the apex (s) to a height of z = 1. Leave the other (t, f) vertices at z = 0. AT = (1 × s) + (0 × t) + (0 × f) = s Goal Head ”north” to the blue region. Same as constant-s isoclines. So, can be “east” or “west” of vertical centerline. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 43 / 49

Slide 47

Slide 47 text

Seeing It All With Barry3 Add Tolerating Weight Also lift left (t) vertex to height of z = 1. Leave the right (f) vertex at z = 0. AT = (1 × s) + (1 × t) + (0 × f) = s + t Goal Go ”west” young man! Same as constant-f isoclines. So, could be in “southwest” (t) corner. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 44 / 49

Slide 48

Slide 48 text

Seeing It All With Barry3 Compromise Weights Lower the left vertex to z = 1 2 . Another interpretation of 1/2. AT = (1 × s) + ( 1 2 × t) + (0 × f) = s + 1 2 t Goal Head ”northwest”. Same as constant-AT isoclines. All (s, t, f) are now different. c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 45 / 49

Slide 49

Slide 49 text

Seeing It All With Barry3 Bringing It All Together c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 46 / 49

Slide 50

Slide 50 text

Seeing It All With Barry3 Toblerone View of Barry3 c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 47 / 49

Slide 51

Slide 51 text

Conclusions Outline 1 Introduction 2 Review of Apdex Metrics 3 Critique And Defense Of Apdex 4 Now, About That Fraction 5 Seeing It All With Barry3 6 Conclusions c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 48 / 49

Slide 52

Slide 52 text

Conclusions Conclusions The Apdex Index is a figure-of-merit which scores the percentage of application response-times that conform to its adaptive areal profile. The Tolerating triangle explains the factor of 1 2 in the definition of the Apdex Index. The Barry3 triangle provides a visual way to incorporate all the Apdex metrics in one simple diagram. Lifting the Barry3 triangle into a third z-dimension explains both the factor of 1/2 and the Apdex Ratings as diagonal colored bands. Longer article in MeasureIT (Feb 2009 issue?) c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 49 / 49