Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Apdex Index Revealed: Triangles Tell The Tale

The Apdex Index Revealed: Triangles Tell The Tale

The purpose of this talk is not to improve upon or redesign the Apdex Index. Rather, it is to explain the underlying construction of the Apdex Index, as it is currently defined. Then, I consider other ways the Apdex metrics might be applied; especially from a performance visualization standpoint.

Dr. Neil Gunther

December 07, 2009
Tweet

More Decks by Dr. Neil Gunther

Other Decks in Research

Transcript

  1. The Apdex Index Revealed
    Triangles Tell The Tale
    Dr. Neil J. Gunther
    A.A. Michelson Award 2008
    Performance Dynamics
    July 30, 2009
    SM
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 1 / 49

    View full-size slide

  2. Introduction
    Outline
    1 Introduction
    2 Review of Apdex Metrics
    3 Critique And Defense Of Apdex
    4 Now, About That Fraction
    5 Seeing It All With Barry3
    6 Conclusions
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 2 / 49

    View full-size slide

  3. Introduction
    What I Won’t Be Doing
    Remark
    The purpose of this talk is not to improve upon or redesign the Apdex
    Index in any way a. My goal is simply want to explain the Apdex Index,
    as it is currently defined b.
    Then want to consider other ways the Apdex metrics might be applied;
    especially from a performance visualization standpoint.
    aThat’s possibly a different talk.
    bAs you’ll see, its not to be so simple as it looks.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 3 / 49

    View full-size slide

  4. Introduction
    Where We’ll End Up
    User response-time of a single Web application
    Measured from 5 different locations (dots)
    Colors rate the Apdex Index value
    Want dots to cluster near top-left of triangle (animation)
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 4 / 49

    View full-size slide

  5. Review of Apdex Metrics
    Outline
    1 Introduction
    2 Review of Apdex Metrics
    3 Critique And Defense Of Apdex
    4 Now, About That Fraction
    5 Seeing It All With Barry3
    6 Conclusions
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 5 / 49

    View full-size slide

  6. Review of Apdex Metrics
    Apdex Definitions And Metrics
    1 Apdex Thresholds
    2 Apdex Zones
    3 Apdex Counts
    4 Apdex Index
    5 Apdex Rating
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 6 / 49

    View full-size slide

  7. Review of Apdex Metrics
    Apdex Thresholds
    Definition
    1 T is a user-defined time, below which response-time performance
    is considered acceptable.
    2 F = 4T is an Apdex-defined time threshold, above which
    response-time performance is considered unacceptable.
    Example
    In Figure 1, I chose T = 6 seconds.
    This choice automatically sets F = 4T to 24 seconds.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 7 / 49

    View full-size slide

  8. Review of Apdex Metrics
    Example Application Data
    Figure: Histogram of measured response times for a local application.
    Gold posts:
    1 T = 6 seconds (left vertical line)
    2 F = 24 seconds (right vertical line)
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 8 / 49

    View full-size slide

  9. Review of Apdex Metrics
    Apdex Zones
    Definition
    1 Satisfied Zone: Any response-time measurement that falls in the
    range 0 < Ri ≤ T
    2 Tolerating Zone: Any response-time measurement that falls in
    the range T < Ri ≤ F
    3 Frustrated Zone: Any response-time measurement that falls in
    the range Ri > F
    You can think of as these as buckets.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 9 / 49

    View full-size slide

  10. Review of Apdex Metrics
    Apdex Counts
    Definition
    1 Count of response times in Satisfied bucket: Cs
    2 Count of response times in Tolerating bucket: Ct
    3 Count of response times in Frustrated bucket: Cf
    Example
    The respective counts for the ATTDISP application with T = 6 are:
    1 Satisfied count: Cs = 3
    2 Tolerating count: Ct = 678
    3 Frustrated count: Cf = 47
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 10 / 49

    View full-size slide

  11. Review of Apdex Metrics
    Apdex Index
    Definition
    The Apdex Index for threshold T is defined as:
    AT =
    Cs + 1
    2
    Ct
    Cs + Ct + Cf
    (1)
    and its value always lies in the range: 0 < AT < 1.
    You can think of it as % score. More on this later.
    Example
    For ATTDISP with T = 6:
    A6 =
    3 + 339
    728
    =
    342
    728
    = 0.46978 (2)
    The color codings are exemplar, not mandatory.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 11 / 49

    View full-size slide

  12. Review of Apdex Metrics
    Apdex Rating
    Definition
    Table: Apdex Index Ratings
    Index Range Apdex Rating Symbol Color
    0.94 to 1.00 Excellent E
    0.85 to 0.93 Good G
    0.70 to 0.84 Fair F
    0.50 to 0.69 Poor P
    0.00 to 0.49 Unacceptable U
    Example
    Apdex Rating for ATTDISP, AT = 0.46978, is Unacceptable.
    That’s consistent with why I analyzing its performance problems.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 12 / 49

    View full-size slide

  13. Review of Apdex Metrics
    The Apdex Process
    (2) Sorted
    samples
    (4) AT
    (1) Response times
    (3) Counts
    (5) Rating
    Satisfied
    Tolerating
    Frustrated
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 13 / 49

    View full-size slide

  14. Critique And Defense Of Apdex
    Outline
    1 Introduction
    2 Review of Apdex Metrics
    3 Critique And Defense Of Apdex
    4 Now, About That Fraction
    5 Seeing It All With Barry3
    6 Conclusions
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 14 / 49

    View full-size slide

  15. Critique And Defense Of Apdex
    Common Criticisms
    A single number is not useful!
    Where did that 1/2 come from in Definition 6?
    Apdex is just a poor man’s distribution.
    Apdex doesn’t accommodate heavy-tail distributions.
    The same Apdex Index can have different Cs, Ct , and Cf
    values?
    I need to review the service levels of hundreds of applications!
    Why does my Apdex Index stink?
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 15 / 49

    View full-size slide

  16. Critique And Defense Of Apdex
    Single Numbers Are Useless
    AT
    intentially defined as single number for management.
    Plenty of single-number performance metrics in existence, e.g.,
    SPEC CPU2006.
    Can resolve details using their web site.
    I don’t have a problem, as long as I can resolve AT
    easily into its
    component Cs, Ct , and Cf
    values when needed.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 16 / 49

    View full-size slide

  17. Critique And Defense Of Apdex
    Who Ordered That 1/2!?
    Where did this come from?
    AT =
    Cs + 1
    2
    Ct
    Cs + Ct + Cf
    Good question!
    Not explained in Apdex Alliance Technical Specification. (PDF)
    Has bothered me since I first learnt about Apdex ca. 2007.
    Has bothered other people too. Several presentations at CMG
    2008.
    Let me come back to this shortly.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 17 / 49

    View full-size slide

  18. Critique And Defense Of Apdex
    Just Poor Man’s Distribution
    AT
    is neither a probability or a probability distribution.
    AT
    is a figure of merit.
    More on this in a minute.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 18 / 49

    View full-size slide

  19. Critique And Defense Of Apdex
    Can’t See Heavy-tailed Distributions
    That’s right. ... But you really shouldn’t have any.
    At the application level, heavy-tailed dsns are regarded as
    undesirable.
    Let’s not get into network packet self-similarity.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 19 / 49

    View full-size slide

  20. Critique And Defense Of Apdex
    Hundreds of Applications
    Good point!
    Many modern enterprises run 10s of applications per server.
    There can easily be hundreds or even thousands of servers.
    Apdex Alliance Technical Specification offers no guidance on this
    issue.
    Barry3 solves this one. More later.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 20 / 49

    View full-size slide

  21. Critique And Defense Of Apdex
    Why Does My AT
    Stink?
    Good question!
    Apdex Alliance Technical Specification offers no guidance on this
    issue either.
    Even an executive mgr is likely to ask why AT
    was green
    yesterday, but red today.
    It’s at this point that the single number can become a problem.
    The Apdex Alliance has over-emphasized simplicity at the
    expense of information.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 21 / 49

    View full-size slide

  22. Now, About That Fraction
    Outline
    1 Introduction
    2 Review of Apdex Metrics
    3 Critique And Defense Of Apdex
    4 Now, About That Fraction
    5 Seeing It All With Barry3
    6 Conclusions
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 22 / 49

    View full-size slide

  23. Now, About That Fraction
    Explanations for That 1/2
    Why not 1, ... or 1/3 ... or 1/e ...?
    Apdex Alliance offers no explanation.
    Others have also considered this question 1 2.
    1A. Akers, ”Is Apdex Sharp Enough?,” Apdex Mtg, CMG 2008
    2Y. Ding, ”An Analytical Model for Apdex,” Apdex Mtg, CMG 2008
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 23 / 49

    View full-size slide

  24. Now, About That Fraction
    Ding’s Explanation
    Assumes an exponential response time dsn e.g., M/M/1
    Sensitivy analysis to explain the F = 4T
    Doesn’t really account for the 1/2
    Remark
    Figure 1 is typical and not exponential. A more realistic model would
    be a chain of M/M/1 servers. The sum of M/M/1 exponential
    response times is a gamma dsn of response times. But the math
    analysis becomes much more complicated.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 24 / 49

    View full-size slide

  25. Now, About That Fraction
    Akers’ Alternative
    Proposition
    Refine standard Apdex Index to be more representative of count dsn
    within the Tolerating Zone by partitioning it into n sub-zones:
    Example
    Let n = 4 sub-zones, then Ct becomes: Ct1, Ct2, Ct3, Ct4, and the
    modified Apdex Index is:
    A∗
    T
    =
    Cs +
    4
    5
    Ct1 +
    3
    5
    Ct2 +
    2
    5
    Ct3 +
    1
    5
    Ct4
    Cs + Ct + Cf
    (3)
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 25 / 49

    View full-size slide

  26. Now, About That Fraction
    Numerical Comparison
    Example
    Suppose Cs = Cf = 0 and Ct1 = Ct2 = Ct3 = 0, but Ct4 = 100. The
    proposed Index produces:
    A∗
    T
    =
    100/5
    100
    = 0.2 (4)
    By comparison, the standard Apdex Index produces:
    AT =
    100/2
    100
    = 0.5 (5)
    The smaller A∗
    T
    value comes from the preponderance of Tolerating
    counts being closer to the Frustrated bucket than the Satisfied bucket.
    Standard AT
    doesn’t see this bias.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 26 / 49

    View full-size slide

  27. Now, About That Fraction
    Special Case
    Distribute the 100 Tolerating counts evenly across all n = 4 sub-zones.
    Example
    Cs = Cf = 0, but Ct1 = Ct2 = Ct3 = Ct4 = 25. Then numerator is:
    4
    5
    × 25 +
    3
    5
    × 25 +
    2
    5
    × 25 +
    1
    5
    × 25 =
    25
    5
    (4 + 3 + 2 + 1) = 50 (6)
    Therefore:
    A∗
    T
    =
    50
    100
    = 0.5 (7)
    which is the same value as AT
    in eqn.(5).
    So A∗
    T
    ≡ AT
    when the dsn of Tolerating counts is uniform.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 27 / 49

    View full-size slide

  28. Now, About That Fraction
    Triangles Tell The Tale
    That was all very interesting but it hasn’t really explained where the
    1/2 comes from in the standard Apdex definition.
    Suppose we simply divide the time-axis by the lower threshold T:
    0, T, 2T, 3T, 4T, 5T.
    The Tolerating Zone covers: T, 2T, 3T, 4T, terminating at F = 4T,
    consistent with the Apdex definition.
    We also suppose that the Satisfied Zone has some counts in it.
    Here’s what that might look like for some arbitrary counts.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 28 / 49

    View full-size slide

  29. Now, About That Fraction
    Ratio of Areas
    Consider 5 columns all of width T.
    dark area
    light area

    15T + 1
    2
    (25T + 25T + 25T)
    15T + 25T + 25T + 25T + 5T
    =
    105T
    2
    1
    100T
    (8)
    Area is measured in event-seconds.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 29 / 49

    View full-size slide

  30. Now, About That Fraction
    The Tolerating Triangle
    Ratio of areas:
    dark area
    light area
    =
    + ∆
    All
    (9)
    Area of is 15 × T
    Area of ∆ is half base by height: 25 × 3T/2
    Area of all rectangles is 100 × T
    + ∆
    All
    =
    105T
    200T
    = 0.525 (10)
    Result is the same as using the definition of AT
    :
    15 + 37.5
    100
    =
    52.5
    100
    = 0.525 (11)
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 30 / 49

    View full-size slide

  31. Now, About That Fraction
    The ATTDISP Triangle
    Let’s check this conclusion against the ATTDISP data.
    3T + 1
    2
    (3T × 226)
    3T + 226T + 226T + 226T + 47T
    =
    342T
    728T
    = 0.46978 (12)
    Same result as eqn.(2) for A6.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 31 / 49

    View full-size slide

  32. Now, About That Fraction
    Adaptive Figure of Merit
    Theorem (Gunther 2009)
    The Apdex Index AT
    is a figure of merit that compares the total number
    of response-time samples (light area) with a profile containing a set of
    idealized response-time samples (dark area). Ideally all samples
    should fall into the first (Satisfied) bucket. Failing that, the remainder of
    the samples in the second (Tolerating) bucket should be monotonically
    decreasing toward the third (Frustrated) bucket.
    Corollary
    The Apdex Index adaptively scores the percentage of application
    response-time samples that conform to the expected Apdex profile.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 32 / 49

    View full-size slide

  33. Now, About That Fraction
    Connection with Akers
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 33 / 49

    View full-size slide

  34. Seeing It All With Barry3
    Outline
    1 Introduction
    2 Review of Apdex Metrics
    3 Critique And Defense Of Apdex
    4 Now, About That Fraction
    5 Seeing It All With Barry3
    6 Conclusions
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 34 / 49

    View full-size slide

  35. Seeing It All With Barry3
    Normalized Counts
    s =
    Cs
    Cs + Ct + Cf
    (13)
    t =
    Ct
    Cs + Ct + Cf
    (14)
    f =
    Cf
    Cs + Ct + Cf
    (15)
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 35 / 49

    View full-size slide

  36. Seeing It All With Barry3
    Normalized Apdex Index
    Standard Apdex Index in eqn.(1)
    AT =
    Cs + 1
    2
    Ct
    Cs + Ct + Cf
    becomes
    Definition
    AT = s +
    1
    2
    t
    Akers modified Index (n = 4) becomes:
    A∗
    T
    = s +
    4
    5
    t1 +
    3
    5
    t2 +
    2
    5
    t3 +
    1
    5
    t4
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 36 / 49

    View full-size slide

  37. Seeing It All With Barry3
    Sum Rule
    Cs
    Cs + Ct + Cf
    +
    Ct
    Cs + Ct + Cf
    +
    Cf
    Cs + Ct + Cf
    =
    Cs + Ct + Cf
    Cs + Ct + Cf
    (16)
    which reduces to:
    s + t + f = 1 (17)
    Proposition
    Three variables or metrics that conform to a sum rule can be
    represented in a barycentric coordinate system.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 37 / 49

    View full-size slide

  38. Seeing It All With Barry3
    Cartesian vs. Barycentric Coordinates
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 38 / 49

    View full-size slide

  39. Seeing It All With Barry3
    Tri-angle-ometry
    2
    2
    2

    View full-size slide

  40. Seeing It All With Barry3
    Tri-angle-ometry
    2
    2
    2

    3
    1 1

    View full-size slide

  41. Seeing It All With Barry3
    Tri-angle-ometry
    2
    2
    2

    3
    1 1
    × 1/

    3

    View full-size slide

  42. Seeing It All With Barry3
    Tri-angle-ometry
    2
    2
    2

    3
    1 1
    × 1/

    3
    2/

    3
    1
    1/

    3
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 39 / 49

    View full-size slide

  43. Seeing It All With Barry3
    Centroid in Barry3
    b1 + b2 + b3 = 0.3333 + 0.3333 + 0.3333 = 1.0 (height)
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 40 / 49

    View full-size slide

  44. Seeing It All With Barry3
    California on Day 26 in Barry3
    b1 + b2 + b3 = 0.42 + 0.31 + 0.27 = 1.0 (height)
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 41 / 49

    View full-size slide

  45. Seeing It All With Barry3
    ATTDISP in Barry3
    b1 + b2 + b3 = 0.0041 + 0.9300 + 0.0650 = 1.0 (height)
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 42 / 49

    View full-size slide

  46. Seeing It All With Barry3
    Horizontal Isoclines
    Lift the apex (s) to a height of
    z = 1.
    Leave the other (t, f) vertices at
    z = 0.
    AT = (1 × s) + (0 × t) + (0 × f)
    = s
    Goal
    Head ”north” to the blue region.
    Same as constant-s isoclines. So,
    can be “east” or “west” of vertical
    centerline.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 43 / 49

    View full-size slide

  47. Seeing It All With Barry3
    Add Tolerating Weight
    Also lift left (t) vertex to height of
    z = 1.
    Leave the right (f) vertex at z = 0.
    AT = (1 × s) + (1 × t) + (0 × f)
    = s + t
    Goal
    Go ”west” young man!
    Same as constant-f isoclines. So,
    could be in “southwest” (t) corner.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 44 / 49

    View full-size slide

  48. Seeing It All With Barry3
    Compromise Weights
    Lower the left vertex to z = 1
    2
    .
    Another interpretation of 1/2.
    AT = (1 × s) + (
    1
    2
    × t) + (0 × f)
    = s +
    1
    2
    t
    Goal
    Head ”northwest”.
    Same as constant-AT
    isoclines.
    All (s, t, f) are now different.
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 45 / 49

    View full-size slide

  49. Seeing It All With Barry3
    Bringing It All Together
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 46 / 49

    View full-size slide

  50. Seeing It All With Barry3
    Toblerone View of Barry3
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 47 / 49

    View full-size slide

  51. Conclusions
    Outline
    1 Introduction
    2 Review of Apdex Metrics
    3 Critique And Defense Of Apdex
    4 Now, About That Fraction
    5 Seeing It All With Barry3
    6 Conclusions
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 48 / 49

    View full-size slide

  52. Conclusions
    Conclusions
    The Apdex Index is a figure-of-merit which scores the percentage
    of application response-times that conform to its adaptive areal
    profile.
    The Tolerating triangle explains the factor of 1
    2
    in the definition of
    the Apdex Index.
    The Barry3 triangle provides a visual way to incorporate all the
    Apdex metrics in one simple diagram.
    Lifting the Barry3 triangle into a third z-dimension explains both
    the factor of 1/2 and the Apdex Ratings as diagonal colored
    bands.
    Longer article in MeasureIT (Feb 2009 issue?)
    c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 49 / 49

    View full-size slide