The Data Analytics of Application Scaling and Why There Are No Giants

The Data Analytics of Application Scaling
and Why There Are No Giants
Neil J. Gunther @DrQz
Performance Dynamics
SF Bay ACM Meetup
Walmart Labs, Sunnyvale, California
Wed, Nov 14, 2018
SM
c 2018 Performance Dynamics The Data Analytics of Application Scaling November 15, 2018 1 / 74

View Slide

The Topic—Scalability Performance Analysis
What is performance analysis?
What it isn’t: Not monitoring computer systems
Not debugging code
Not “performance testing” (load testing)
Not on anyone’s business card
What it is: All of the above!
Multidisciplinary: maths, stats, coding, skepticism,
architecture, critical thinking, market trends, etc.
The art of knowing what data to throw away
How to quantify application scalability
Data analytics applied to computer efﬁciency
Data is only “half” the story (at best)
Other “half” requires performance models

View Slide

The Topic—Scalability Performance Analysis
What is performance analysis?
What it isn’t: Not monitoring computer systems
Not debugging code
Not “performance testing” (load testing)
Not on anyone’s business card
What it is: All of the above!
Multidisciplinary: maths, stats, coding, skepticism,
architecture, critical thinking, market trends, etc.
The art of knowing what data to throw away
How to quantify application scalability
Data analytics applied to computer efﬁciency
Data is only “half” the story (at best)
Other “half” requires performance models
Working motto: Trust nothing and verify

View Slide

I’ve written some things

View Slide

Guerrilla performance classes
Performance training is a very serious business (so, lunches need to be long)
Guerrilla training classes local, online, and in-house (textbooks included)
Guerrilla Data Analytics class — linear regression to machine learning in R

View Slide

Tall Tales About Giants
Outline
1 Tall Tales About Giants
2 Computational Scaling
3 Universal Scalability Law (USL)
4 Determining USL Coefﬁcients
5 Application Scalability
6 Using Production Data
7 Wrap Up

View Slide

Mythical giants (and beanstalks)
J&B giant was reputedly about 30 tall in some accounts (no data)
Let’s not even get into beanstalks in the clouds!

View Slide

Galileo was onto this
1 Discorsi e Dimostrazioni Matematiche Intorno a Due Nuove Scienze1, Galileo
Galilei (1638)
2 On Being the Right Size, J. B. S. Haldane (1928)
1
The two new sciences: (1) materials science and (2) kinematics.

View Slide

Allometric scaling
Robert P. Wadlow
Tallest human giant
b. Alton, Illinois in 1918
Reached 8.925 feet
(2.72 meters)
Guinness world record
Died in 1940 at 22 yo
Father was 5.958 feet
(1.82 meters) Leg braces
Dear old Dad

View Slide

Why there are no 30 ft giants
0.0 0.5 1.0 1.5 2.0
0.0 0.5 1.0 1.5 2.0
Weight
Scaling
Stable region Unstable region
load line
support line critical point
1 Weight (mass) grows ∼ L3 with volume but support ∼ L2 cross-sectional area
2 Above critical point things break!!!
3 Going to use a similar approach to computer software scalability

View Slide

Computational Scaling
Outline
7 Wrap Up

View Slide

Goggle up — real science ahead

View Slide

Scaling property 1: Equal bang for the buck
α = 0
β = 0
Processes
Capacity
Everybody wants this

View Slide

Scaling property 2: Diminishing returns
α > 0
β = 0
Processes
Capacity
Everybody usually gets this

View Slide

Scaling property 3: Bottleneck limit
α >> 0
β = 0
1/α
Processes
Capacity
Everybody hates this

View Slide

Scaling property 4: Retrograde throughput
α >> 0
β > 0
1/N
1/α
Processes
Capacity
Everybody thinks this never happens

View Slide

Universal Scalability Law (USL)
Outline
7 Wrap Up

View Slide

How to quantify computational scalability
N: processes provide system stimulus or load
2
NJG. “A Simple Capacity Model of Massively Parallel Transaction Systems,” CMG Conference (1993)

View Slide

CN
: response function or relative capacity
2

View Slide

CN
Question: What kind of function?
2

View Slide

CN
Answer: Nonlinear rational function 2
2

View Slide

CN
CN
(α, β, γ) =
γN
1 + α (N − 1) + β N(N − 1)
2

View Slide

CN
CN
(α, β, γ) =
γN
1 + α (N − 1) + β N(N − 1)
Three Cs:
2

View Slide

CN
CN
(α, β, γ) =
γN
1 + α (N − 1) + β N(N − 1)
Three Cs:
1
Concurrency (0 < γ < ∞)
2

View Slide

CN
CN
(α, β, γ) =
γN
1 + α (N − 1) + β N(N − 1)
Three Cs:
1
2
Contention (0 < α < 1)
2

View Slide

CN
CN
(α, β, γ) =
γN
1 + α (N − 1) + β N(N − 1)
Three Cs:
1
2
Contention (0 < α < 1)
3
Coherency (0 < β < 1)
2

View Slide

Measurement meets model
X(N)
X(1)
Thruput data

View Slide

X(N)
X(1)
Thruput data
−→ CN
Scalability metric

View Slide

X(N)
X(1)
Thruput data
−→ CN
Scalability metric
←−
γN
1 + α (N − 1) + β N(N − 1)
USL model

View Slide

X(N)
X(1)
Thruput data
−→ CN
Scalability metric
←−
γN
1 + α (N − 1) + β N(N − 1)
USL model
0 20 40 60 80 100
5 10 15 20
Processes (N)
Relative capacity, C(N)
Linear scaling Amdahl−like scaling
Retrograde scaling

View Slide

Determining USL Coefﬁcients
Outline
7 Wrap Up

View Slide

Finding α, β, and γ
Throughput measurements XN
at various process loads N sourced from:
1 Load testing platform
2 Production monitoring
Want to determine the α, β, γ that best model the XN
data
XN
(α, β, γ) =
γ
1 + α (N − 1) + β N(N − 1)
XN
is a nonlinear rational function (tricky)
Brute force (ugh!)
Clever ways: Solve for α, β, γ coefﬁcients in one swell foop
1 nls() nonlinear regression in R
2 Solver optimizer Excel Add-in

View Slide

Everybody’s data scientist but ...
NASA Dawn spacecraft is orbiting the dwarf planet Ceres (gone dark)

View Slide

A little least-squares history
Ceres was ﬁrst observed over 200 years ago
Gauss accurately estimated the (then unknown) orbit of Ceres c.1801
Already developed least squares statistical regression at 18 yo
He used little data when everyone else assumed big data was necessary
Data errors represented by Gaussian distribution

View Slide

Tips at different restaurants
0 1 2 3 4 5 6
0 5 10 15
Restaurant
Tip ($)

View Slide

Tip deviations from mean
0 1 2 3 4 5 6
0 5 10 15
Restaurant
Tip ($)
−5
7
1
−2
4
−5

View Slide

Are tips related to the bill?
0 20 40 60 80 100
0 5 10 15
Bill ($)
Tip ($)

View Slide

Least squares are real
0 20 40 60 80 100
0 5 10 15
Bill ($)
Tip ($)

View Slide

Relative areas: R2 = 0.7494
0 20 40 60 80 100
0 5 10 15
Bill ($)
Tip ($)
Tip
error
Model
error

View Slide

Conﬁdence bands
See GCaP book Chap. 5 & App. B
α = 0.04979728
β = 1.143438e−05
R2 = 0.9883438
0
100
200
300
0 20 40 60
Processors
Benchmark throughput (Krays/s)
USL Analysis of SGI Origin 2000

View Slide

Application Scalability
Outline
7 Wrap Up

View Slide

Varnish Scalability
Data provided by
Darius Popa, DigitAir and Stefan Parvu, Nokia

View Slide

Varnish architecture
HTTP accelerator
Reverse web proxy caching system
Sits in front of classic web server
Caching handled by virtual memory
Claim: Highly scalable (read: linear)

View Slide

Varnish architecture
HTTP accelerator
Reverse web proxy caching system
Sits in front of classic web server
Caching handled by virtual memory
Claim: Highly scalable (read: linear) ... but is it?

View Slide

JMeter measurements
Example (Read in raw data and plot it in R)
df.test <- read.table(fname, header=TRUE, sep="\t")
plot(df.test$N, df.test$X_N, type="b")
print(df.test)
0 100 200 300 400
0 100 200 300
USL 1
Load generators (N)
Relative capacity C(N)
N X_N Capacity Efficiency
1 1 1.4 1.000000 1.0000000
2 2 2.7 1.928571 0.9642857
3 5 6.4 4.571429 0.9142857
4 10 12.8 9.142857 0.9142857
5 25 32.0 22.857143 0.9142857
6 50 64.0 45.714286 0.9142857
7 75 98.0 70.000000 0.9333333
8 100 131.0 93.571429 0.9357143
9 150 197.0 140.714286 0.9380952
10 250 320.0 228.571429 0.9142857
11 300 392.0 280.000000 0.9333333
12 400 518.0 370.000000 0.9250000

View Slide

Near linear scaling
0 100 200 300 400
0 100 200 300
USL 2
Load generators (N)

View Slide

Varnish meets the USL
0 100 200 300 400
0 100 200 300
USL 3
Load generators (N)
α = 1e−04
β = 0
γ = 0.955364
pmax = NaN
Cmax = NaN
Croof = 10000

View Slide

USL beyond the data
0 1000 2000 3000 4000 5000
0 500 1000 1500 2000 2500 3000
USL 4
Load generators (N)
α = 1e−04
β = 0
γ = 0.955364
pmax = NaN
Cmax = NaN
Croof = 10000

View Slide

Linux Network Driver
XPD: eXpress Data Path
“XDP: A new fast and programmable network layer”
Jesper Brouer, Red Hat

View Slide

RedHat(IBM?) benchmark data

View Slide

USL sees beyond the data
Projected from 6 to 20 cores

View Slide

Memcached
“Hidden Scalability Gotchas in Memcached and Friends”
NJG, Shanti Subramanyam, and Stefan Parvu Sun Microsystems
Presented at Velocity 2010

View Slide

Memcached scalability
Scaleup Scaleout

View Slide

Memcache scaleout strategy
Distributed cache of key-value pairs
Data pre-loaded from RDBMS backend
Deploy memcache on cheaper older CPUs (but not multicore)
Single worker thread ok — until next hardware roll (multicore)

View Slide

Memcache scalability data
0 2 4 6 8 10 12
0 50 100 150 200 250 300
Worker threads (N)
Throughput KOPS X(N)

View Slide

Explains these conﬁguration warnings
Conﬁguring the memcached server
Threading is used to scale memcached across CPU’s. The model
is by "worker threads", meaning that each thread handles concurrent
connections. ... By default 4 threads are allocated. ... Setting it
to very large values (80+) will make it run considerably slower.
Linux man pages - memcached (1)
-t
Number of threads to use to process incoming requests. ... It is
typically not useful to set this higher than the number of CPU cores
on the memcached server. Setting a high number (64 or more) of worker
threads is not recommended. The default is 4.

View Slide

Memcached load-test data

View Slide

Memcached regression analysis

View Slide

Memcache scalability model
0 2 4 6 8 10 12
0 50 100 150 200 250 300
Worker threads (N)
α = 0.0468
β = 0.021016
γ = 84.89
Nmax = 6.73
Xmax = 274.87
Xroof = 1814.82

View Slide

Concurrency parameter
0 2 4 6 8 10 12
0 50 100 150 200 250 300
Worker threads (N)
α = 0.0468
β = 0.021016
γ = 84.89
Nmax = 6.73
Xmax = 274.87
Xroof = 1814.82
α = 0
β = 0
Processes
Capacity
1 γ = 84.89
2 Slope of linear bound as Kops/thread
3 Estimate of throughput X(1) = 84.89 Kops at N = 1 thread

View Slide

Contention parameter
0 2 4 6 8 10 12
0 50 100 150 200 250 300
Worker threads (N)
α = 0.0468
β = 0.021016
γ = 84.89
Nmax = 6.73
Xmax = 274.87
Xroof = 1814.82
α >> 0
β = 0
1/α
Processes
Capacity
α = 0.0468
Waiting or queueing for resources about 4.6% of the time
Max possible throughput is X(1)/α = 1814.78 Kops (Xroof
)

View Slide

Coherency parameter
0 2 4 6 8 10 12
0 50 100 150 200 250 300
Worker threads (N)
α = 0.0468
β = 0.021016
γ = 84.89
Nmax = 6.73
Xmax = 274.87
Xroof = 1814.82
α >> 0
β > 0
1/N
1/α
Processes
Capacity
β = 0.0210 corresponds to retrograde throughput
Distributed copies of data (e.g., caches) have to be exchanged/updated
about 2.1% of the time to be consistent
Peak occurs at Nmax = (1 − α)/β = 6.73 threads

View Slide

Improving scalability performance
0 10 20 30 40 50
0 5 10 15 20 25
Threads (N)
Speedup S(N)
mcd 1.2.8
mcd 1.3.2
mcd 1.3.2 + patch

View Slide

Sirius and Zookeeper
“Sirius: Distributing and Coordinating Application”
Michael Bevilacqua-Linn, Maulan Byron, Peter Cline,
Jon Moore, and Steve Muir, Comcast
Presented at USENIX Annual Technical Conf. 2014

View Slide

Distributed voting throughput data
All downhill ... which looked crazy! (to me)

View Slide

USL scalability model

View Slide

α = 0
β = 0
Processes
Capacity
1 γ = 1024.98
2 Single node is meaningless (need N ≥ 3 for majority)
3 Interpret γ as N = 1 virtual throughput
4 USL estimates X(1) = 1024.98 WPS (black square)

View Slide

α >> 0
β = 0
1/α
Processes
Capacity
α = 0.05
Queueing for resources about 5% of the time
Max possible throughput is X(1)/α = 20499.54 WPS (Xroof
)
But Xroof
not feasible in these systems

View Slide

Coherency parameter
α >> 0
β > 0
1/N
1/α
Processes
Capacity
β = 0.1651 says retrograde throughput dominates!
Distributed data being exchanged (compared?) about 16.5% of the time
(virtual) Peak at Nmax = (1 − α)/β = 2.4 cluster nodes
Shocking but that’s exactly how it’s supposed to work!

View Slide

Using Production Data
Outline
7 Wrap Up

View Slide

AWS Cloud Application
“Exposing the Cost of Performance Hidden in the Cloud”
NJG and Mohit Chawla, Germany
Presented at CMG cloudXchange 2018

View Slide

Production data
Previously measured X and R directly on test rig
Table 1: Converting data to performance metrics
Data Meaning Metrics Meaning
T Elapsed time X = C/T Throughtput
Tp
Processing time R = (Tp/T)(T/C) Response time
C Completed work N = X × R Concurrent threads
Ucpu
CPU utilization S = Ucpu/X Service time
Example (Coalesced metrics)
Linux epoch Timestamp interval between rows is 300 seconds
Timestamp, X, N, S, R, U_cpu
1486771200000, 502.171674, 170.266663, 0.000912, 0.336740, 0.458120
1486771500000, 494.403035, 175.375000, 0.001043, 0.355975, 0.515420
1486771800000, 509.541751, 188.866669, 0.000885, 0.360924, 0.450980
1486772100000, 507.089094, 188.437500, 0.000910, 0.367479, 0.461700
1486772400000, 532.803039, 191.466660, 0.000880, 0.362905, 0.468860
...

View Slide

Tomcat data from AWS
0 100 200 300 400 500
0 200 400 600 800
Tomcat threads
Throughput (RPS)

View Slide

USL nonlinear analysis
0 100 200 300 400 500
0 200 400 600 800
Tomcat threads
Throughput (RPS)
α = 0
β = 3e−06
γ = 3
Nmax = 539.2
Xmax = 809.55
Nopt = 274.8

View Slide

0 100 200 300 400 500
0 200 400 600 800
Tomcat threads
Throughput (RPS)
α = 0
β = 3e−06
γ = 3
Nmax = 539.2
Xmax = 809.55
Nopt = 274.8
α = 0
β = 0
Processes
Capacity
1 γ = 3.0
2 Smallest number of threads during 24 hr sample is N > 100
3 Nonetheless USL estimates throughput X(1) = 3 RPS

View Slide

0 100 200 300 400 500
0 200 400 600 800
Tomcat threads
Throughput (RPS)
α = 0
β = 3e−06
γ = 3
Nmax = 539.2
Xmax = 809.55
Nopt = 274.8
α >> 0
β = 0
1/α
Processes
Capacity
α = 0
No signiﬁcant waiting or queueing
Max possible throughput Xroof
not deﬁned

View Slide

Coherency parameter
0 100 200 300 400 500
0 200 400 600 800
Tomcat threads
Throughput (RPS)
α = 0
β = 3e−06
γ = 3
Nmax = 539.2
Xmax = 809.55
Nopt = 274.8
α >> 0
β > 0
1/N
1/α
Processes
Capacity
β = 3 × 10−6 implies very weak retrograde throughput
Extremely little data exchange
But entirely responsible for sublinearity
And peak throughput Xmax = 809.55 RPS
Peak occurs at Nmax = 539.2 threads

View Slide

Revised USL analysis
Parallel threads implies linear scaling
Linear slope γ ∼ 3: γ = 2.65
Should be no contention, i.e., α = 0
Discontinuity at N ∼ 275 threads
Throughput plateaus, i.e., β = 0
Saturation occurs at processor
utilization UCPU
≥ 75%
Linux OS can’t do that!
Pseudo-saturation due to AWS Auto
Scaling policy (hypervisor?)
Many EC2 instances spun up and down
during 24 hrs

View Slide

Corrected USL linear model
0 100 200 300 400 500
0 200 400 600 800
Tomcat threads
Throughput (RPS)
α = 0
β = 0
γ = 2.65
Nmax = NaN
Xmax = 727.03
Nopt = 274.8
Parallel threads Pseudo−saturation

View Slide

MySQL Big Data
2,000 production database logs
500,000 data points
NJG with Baron Schwartz and Preetam Jinka

View Slide

How do you comprehend big data?

View Slide

MySQL big data — the (horror) movie

View Slide

USL analysis
Analyzed some 2,000 production database logs
I resorted to animation as a visualization tool
About 500,000 data points in aggregate
0 1 2 3 4 5 6 7
0 5000 10000 15000 20000
USL3: [ 917 ] mysql 10.0.24−MariaDB
Concurrent processes
Queries per second
α = 0.0469166
β = 0.0067516
γ = 5444.43
X1 = 5196.11
Npeak = 11.88
Xpeak = 25910.07
Nopt = 21.31
Xmax = 116044.9
0 10 20 30 40
0 5000 10000 15000 20000
USL3: [ 1403 ] mysql 5.5.52−0ubuntu0.12.04.1−log ...
Concurrent processes
Queries per second
α = 0.0456189
β = 0.00118014
γ = 2271.33
X1 = 3195.96
Npeak = 28.44
Xpeak = 24074.24
Nopt = 21.92
Xmax = 49789.25
Comparison of USL parameters
USL found unexpected progressive changes in scalability

View Slide

Wrap Up
Outline
7 Wrap Up

View Slide

Wrap Up
R packages for the USL
1 SATK on R-Forge
Author: Paul Puglia (Guerrilla graduate)
Applies multiple USL coefﬁcient models for best ﬁt
install.packages("SATK", repos="http://R-Forge.R-project.org")
library(SATK)
data(USLcalc)
uslcalc.zones <- zones(USLcalc)
plot(uslcalc.zones)
2 usl on CRAN
Author: Stefan M¨
oding
Uses both nls() from base and nlxb() from nlsr package
install.packages("usl")
library(usl)
data(specsdm91)
usl.model <- usl(throughput ˜ load, specsdm91)
summary(usl.model)
peak.scalability(usl.model)
plot(specsdm91, pch=16)
plot(usl.model, col="red", add=TRUE)

View Slide

Wrap Up
Response time scalability
Presented throughput scalability
Response time scalability?
Brooks’ law (too many cooks)
Queueing theory foundations
Queueing simulations
XN
=
γN
1 + α (N − 1) + β N(N − 1)
RN =
N
XN
− Z
0 2 4 6 8 10
People
2
4
6
8
10
12
Months
Fixed delay due to meetings
0 2 4 6 8 10
People
2
4
6
8
10
Months
Growing delay due to 1 on 1 mtgs
0 2 4 6 8 10
People
5
10
15
Months
0 2 4 6 8 10
People
0.5
1.0
1.5
2.0
Output
Parallel
Amdahl
Brooks
USL

View Slide

Wrap Up
Questions?
www.perfdynamics.com
Castro Valley, California
Twitter twitter.com/DrQz
Facebook facebook.com/PerformanceDynamics
Blog perfdynamics.blogspot.com
Training classes perfdynamics.com/Classes
[email protected]
+1-510-537-5758

View Slide

The Data Analytics of Application Scaling and Why There Are No Giants

The Data Analytics of Application Scaling and Why There Are No Giants

Dr. Neil Gunther

More Decks by Dr. Neil Gunther

Other Decks in Research

Featured

Transcript