The Apdex Index Revealed: Triangles Tell The Tale

The Apdex Index Revealed
Triangles Tell The Tale
Dr. Neil J. Gunther
A.A. Michelson Award 2008
Performance Dynamics
July 30, 2009
SM
c 2019 Performance Dynamics The Apdex Index Revealed July 30, 2009 1 / 49

View Slide

Introduction
Outline
1 Introduction
2 Review of Apdex Metrics
3 Critique And Defense Of Apdex
4 Now, About That Fraction
5 Seeing It All With Barry3
6 Conclusions

View Slide

Introduction
What I Won’t Be Doing
Remark
The purpose of this talk is not to improve upon or redesign the Apdex
Index in any way a. My goal is simply want to explain the Apdex Index,
as it is currently deﬁned b.
Then want to consider other ways the Apdex metrics might be applied;
especially from a performance visualization standpoint.
aThat’s possibly a different talk.
bAs you’ll see, its not to be so simple as it looks.

View Slide

Introduction
Where We’ll End Up
User response-time of a single Web application
Measured from 5 different locations (dots)
Colors rate the Apdex Index value
Want dots to cluster near top-left of triangle (animation)

View Slide

Review of Apdex Metrics
Outline
1 Introduction
6 Conclusions

View Slide

Apdex Deﬁnitions And Metrics
1 Apdex Thresholds
2 Apdex Zones
3 Apdex Counts
4 Apdex Index
5 Apdex Rating

View Slide

Apdex Thresholds
Definition
1 T is a user-defined time, below which response-time performance
is considered acceptable.
2 F = 4T is an Apdex-defined time threshold, above which
response-time performance is considered unacceptable.
Example
In Figure 1, I chose T = 6 seconds.
This choice automatically sets F = 4T to 24 seconds.

View Slide

Example Application Data
Figure: Histogram of measured response times for a local application.
Gold posts:
1 T = 6 seconds (left vertical line)
2 F = 24 seconds (right vertical line)

View Slide

Apdex Zones
Deﬁnition
1 Satisﬁed Zone: Any response-time measurement that falls in the
range 0 < Ri ≤ T
2 Tolerating Zone: Any response-time measurement that falls in
the range T < Ri ≤ F
3 Frustrated Zone: Any response-time measurement that falls in
the range Ri > F
You can think of as these as buckets.

View Slide

Apdex Counts
Definition
1 Count of response times in Satisfied bucket: Cs
2 Count of response times in Tolerating bucket: Ct
3 Count of response times in Frustrated bucket: Cf
Example
The respective counts for the ATTDISP application with T = 6 are:
1 Satisfied count: Cs = 3
2 Tolerating count: Ct = 678
3 Frustrated count: Cf = 47

View Slide

Apdex Index
Deﬁnition
The Apdex Index for threshold T is deﬁned as:
AT =
Cs + 1
2
Ct
Cs + Ct + Cf
(1)
and its value always lies in the range: 0 < AT < 1.
You can think of it as % score. More on this later.
Example
For ATTDISP with T = 6:
A6 =
3 + 339
728
=
342
728
= 0.46978 (2)
The color codings are exemplar, not mandatory.

View Slide

Apdex Rating
Deﬁnition
Table: Apdex Index Ratings
Index Range Apdex Rating Symbol Color
0.94 to 1.00 Excellent E
0.85 to 0.93 Good G
0.70 to 0.84 Fair F
0.50 to 0.69 Poor P
0.00 to 0.49 Unacceptable U
Example
Apdex Rating for ATTDISP, AT = 0.46978, is Unacceptable.
That’s consistent with why I analyzing its performance problems.

View Slide

The Apdex Process
(2) Sorted
samples
(4) AT
(1) Response times
(3) Counts
(5) Rating
Satisfied
Tolerating
Frustrated

View Slide

Critique And Defense Of Apdex
Outline
1 Introduction
6 Conclusions

View Slide

Common Criticisms
A single number is not useful!
Where did that 1/2 come from in Deﬁnition 6?
Apdex is just a poor man’s distribution.
Apdex doesn’t accommodate heavy-tail distributions.
The same Apdex Index can have different Cs, Ct , and Cf
values?
I need to review the service levels of hundreds of applications!
Why does my Apdex Index stink?

View Slide

Single Numbers Are Useless
AT
intentially deﬁned as single number for management.
Plenty of single-number performance metrics in existence, e.g.,
SPEC CPU2006.
Can resolve details using their web site.
I don’t have a problem, as long as I can resolve AT
easily into its
component Cs, Ct , and Cf
values when needed.

View Slide

Who Ordered That 1/2!?
Where did this come from?
AT =
Cs + 1
2
Ct
Cs + Ct + Cf
Good question!
Not explained in Apdex Alliance Technical Speciﬁcation. (PDF)
Has bothered me since I ﬁrst learnt about Apdex ca. 2007.
Has bothered other people too. Several presentations at CMG
2008.
Let me come back to this shortly.

View Slide

Just Poor Man’s Distribution
AT
is neither a probability or a probability distribution.
AT
is a ﬁgure of merit.
More on this in a minute.

View Slide

Can’t See Heavy-tailed Distributions
That’s right. ... But you really shouldn’t have any.
At the application level, heavy-tailed dsns are regarded as
undesirable.
Let’s not get into network packet self-similarity.

View Slide

Hundreds of Applications
Good point!
Many modern enterprises run 10s of applications per server.
There can easily be hundreds or even thousands of servers.
Apdex Alliance Technical Speciﬁcation offers no guidance on this
issue.
Barry3 solves this one. More later.

View Slide

Why Does My AT
Stink?
Good question!
Apdex Alliance Technical Speciﬁcation offers no guidance on this
issue either.
Even an executive mgr is likely to ask why AT
was green
yesterday, but red today.
It’s at this point that the single number can become a problem.
The Apdex Alliance has over-emphasized simplicity at the
expense of information.

View Slide

Now, About That Fraction
Outline
1 Introduction
6 Conclusions

View Slide

Explanations for That 1/2
Why not 1, ... or 1/3 ... or 1/e ...?
Apdex Alliance offers no explanation.
Others have also considered this question 1 2.
1A. Akers, ”Is Apdex Sharp Enough?,” Apdex Mtg, CMG 2008
2Y. Ding, ”An Analytical Model for Apdex,” Apdex Mtg, CMG 2008

View Slide

Ding’s Explanation
Assumes an exponential response time dsn e.g., M/M/1
Sensitivy analysis to explain the F = 4T
Doesn’t really account for the 1/2
Remark
Figure 1 is typical and not exponential. A more realistic model would
be a chain of M/M/1 servers. The sum of M/M/1 exponential
response times is a gamma dsn of response times. But the math
analysis becomes much more complicated.

View Slide

Akers’ Alternative
Proposition
Reﬁne standard Apdex Index to be more representative of count dsn
within the Tolerating Zone by partitioning it into n sub-zones:
Example
Let n = 4 sub-zones, then Ct becomes: Ct1, Ct2, Ct3, Ct4, and the
modiﬁed Apdex Index is:
A∗
T
=
Cs +
4
5
Ct1 +
3
5
Ct2 +
2
5
Ct3 +
1
5
Ct4
Cs + Ct + Cf
(3)

View Slide

Numerical Comparison
Example
Suppose Cs = Cf = 0 and Ct1 = Ct2 = Ct3 = 0, but Ct4 = 100. The
proposed Index produces:
A∗
T
=
100/5
100
= 0.2 (4)
By comparison, the standard Apdex Index produces:
AT =
100/2
100
= 0.5 (5)
The smaller A∗
T
value comes from the preponderance of Tolerating
counts being closer to the Frustrated bucket than the Satisﬁed bucket.
Standard AT
doesn’t see this bias.

View Slide

Special Case
Distribute the 100 Tolerating counts evenly across all n = 4 sub-zones.
Example
Cs = Cf = 0, but Ct1 = Ct2 = Ct3 = Ct4 = 25. Then numerator is:
4
5
× 25 +
3
5
× 25 +
2
5
× 25 +
1
5
× 25 =
25
5
(4 + 3 + 2 + 1) = 50 (6)
Therefore:
A∗
T
=
50
100
= 0.5 (7)
which is the same value as AT
in eqn.(5).
So A∗
T
≡ AT
when the dsn of Tolerating counts is uniform.

View Slide

Triangles Tell The Tale
That was all very interesting but it hasn’t really explained where the
1/2 comes from in the standard Apdex definition.
Suppose we simply divide the time-axis by the lower threshold T:
0, T, 2T, 3T, 4T, 5T.
The Tolerating Zone covers: T, 2T, 3T, 4T, terminating at F = 4T,
consistent with the Apdex definition.
We also suppose that the Satisfied Zone has some counts in it.
Here’s what that might look like for some arbitrary counts.

View Slide

Ratio of Areas
Consider 5 columns all of width T.
dark area
light area
→
15T + 1
2
(25T + 25T + 25T)
15T + 25T + 25T + 25T + 5T
=
105T
2
1
100T
(8)
Area is measured in event-seconds.

View Slide

The Tolerating Triangle
Ratio of areas:
dark area
light area
=
+ ∆
All
(9)
Area of is 15 × T
Area of ∆ is half base by height: 25 × 3T/2
Area of all rectangles is 100 × T
+ ∆
All
=
105T
200T
= 0.525 (10)
Result is the same as using the deﬁnition of AT
:
15 + 37.5
100
=
52.5
100
= 0.525 (11)

View Slide

The ATTDISP Triangle
Let’s check this conclusion against the ATTDISP data.
3T + 1
2
(3T × 226)
3T + 226T + 226T + 226T + 47T
=
342T
728T
= 0.46978 (12)
Same result as eqn.(2) for A6.

View Slide

Adaptive Figure of Merit
Theorem (Gunther 2009)
The Apdex Index AT
is a figure of merit that compares the total number
of response-time samples (light area) with a profile containing a set of
idealized response-time samples (dark area). Ideally all samples
should fall into the first (Satisfied) bucket. Failing that, the remainder of
the samples in the second (Tolerating) bucket should be monotonically
decreasing toward the third (Frustrated) bucket.
Corollary
The Apdex Index adaptively scores the percentage of application
response-time samples that conform to the expected Apdex profile.

View Slide

Connection with Akers

View Slide

Seeing It All With Barry3
Outline
1 Introduction
6 Conclusions

View Slide

Normalized Counts
s =
Cs
Cs + Ct + Cf
(13)
t =
Ct
Cs + Ct + Cf
(14)
f =
Cf
Cs + Ct + Cf
(15)

View Slide

Normalized Apdex Index
Standard Apdex Index in eqn.(1)
AT =
Cs + 1
2
Ct
Cs + Ct + Cf
becomes
Deﬁnition
AT = s +
1
2
t
Akers modiﬁed Index (n = 4) becomes:
A∗
T
= s +
4
5
t1 +
3
5
t2 +
2
5
t3 +
1
5
t4

View Slide

Sum Rule
Cs
Cs + Ct + Cf
+
Ct
Cs + Ct + Cf
+
Cf
Cs + Ct + Cf
=
Cs + Ct + Cf
Cs + Ct + Cf
(16)
which reduces to:
s + t + f = 1 (17)
Proposition
Three variables or metrics that conform to a sum rule can be
represented in a barycentric coordinate system.

View Slide

Cartesian vs. Barycentric Coordinates

View Slide

Tri-angle-ometry
2
2
2

View Slide

Tri-angle-ometry
2
2
2
√
3
1 1

View Slide

Tri-angle-ometry
2
2
2
√
3
1 1
× 1/
√
3

View Slide

Tri-angle-ometry
2
2
2
√
3
1 1
× 1/
√
3
2/
√
3
1
1/
√
3

View Slide

Centroid in Barry3
b1 + b2 + b3 = 0.3333 + 0.3333 + 0.3333 = 1.0 (height)

View Slide

California on Day 26 in Barry3
b1 + b2 + b3 = 0.42 + 0.31 + 0.27 = 1.0 (height)

View Slide

ATTDISP in Barry3
b1 + b2 + b3 = 0.0041 + 0.9300 + 0.0650 = 1.0 (height)

View Slide

Horizontal Isoclines
Lift the apex (s) to a height of
z = 1.
Leave the other (t, f) vertices at
z = 0.
AT = (1 × s) + (0 × t) + (0 × f)
= s
Goal
Head ”north” to the blue region.
Same as constant-s isoclines. So,
can be “east” or “west” of vertical
centerline.

View Slide

Add Tolerating Weight
Also lift left (t) vertex to height of
z = 1.
Leave the right (f) vertex at z = 0.
AT = (1 × s) + (1 × t) + (0 × f)
= s + t
Goal
Go ”west” young man!
Same as constant-f isoclines. So,
could be in “southwest” (t) corner.

View Slide

Compromise Weights
Lower the left vertex to z = 1
2
.
Another interpretation of 1/2.
AT = (1 × s) + (
1
2
× t) + (0 × f)
= s +
1
2
t
Goal
Head ”northwest”.
Same as constant-AT
isoclines.
All (s, t, f) are now different.

View Slide

Bringing It All Together

View Slide

Toblerone View of Barry3

View Slide

Conclusions
Outline
1 Introduction
6 Conclusions

View Slide

Conclusions
Conclusions
The Apdex Index is a figure-of-merit which scores the percentage
of application response-times that conform to its adaptive areal
profile.
The Tolerating triangle explains the factor of 1
2
in the definition of
the Apdex Index.
The Barry3 triangle provides a visual way to incorporate all the
Apdex metrics in one simple diagram.
Lifting the Barry3 triangle into a third z-dimension explains both
the factor of 1/2 and the Apdex Ratings as diagonal colored
bands.
Longer article in MeasureIT (Feb 2009 issue?)

View Slide

The Apdex Index Revealed: Triangles Tell The Tale

The Apdex Index Revealed: Triangles Tell The Tale

Dr. Neil Gunther

More Decks by Dr. Neil Gunther

Other Decks in Research

Featured

Transcript