Predicting Performance Changes of Distributed Applications

PREDICTING PERFORMANCE
CHANGES OF DISTRIBUTED
APPLICATIONS
Wojciech Rząsa
@wrzasa
Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

View Slide

Passion for informatics
PhD, but primarily an engineer
Rzeszow University of Technology
Research: distributed systems
Teaching: Ruby, Rails, ...
Rzeszow Ruby User Group
ABOUT ME
http://rrug.pl

View Slide

THE GRID

View Slide

(c) e-ScienceCity , .
http://www.e-sciencecity.org/ Creative Commons Attribution-ShareAlike 3.0 Unported License

View Slide

GRID MONITORING — OCM-G
Debugging
Interactive applications
Shared infrastructure
Distributed
No central management
Standard interface for tools
Tight security requirements

View Slide

SECURITY VS. PERFORMANCE

View Slide

EMPOWER DEVELOPERS TO
ASSESS PERFORMANCE!

View Slide

AGENDA
How to predict performance changes
Basic example
Two case studies

View Slide

Rząsa W.: Timed colored Petri net based estimation of efficiency of the grid applications. Supervisor: E. Nawarecki, AGH-UST, Kraków, 2011.
Baliś B., Bubak M., Rząsa W., Szepieniec T., Wismüller R.: Security in the OCM-G Grid Application Monitoring System. PPAM 2003, LNCS 3019,
pp. 779-787, 2004, Eds. R. Wyrzykowski et al.
Baliś B., Bubak M., Rząsa W., Szepieniec T., Wismüller R.: Two Aspects of Security Solution for Distributed Systems in the Grid on the Example
of the OCM-G. In proc. of CGW'03, pp.197-206, Kraków 2004 ISBN 83-915141-3-7.
Baliś B., Bubak M., Rząsa W., Szepieniec T.: Efficiency of the GSI Secured Network Transmission. ICCS 2004, LNCS 3036, p. 107-115, 2004, Eds.
M. Bubak et al.
Rząsa W., Bubak M., Baliś B., Szepieniec T.: Simulation Method for Estimation of Security Overhead of Grid Applications. In proc. of CGW'05,
pp. 300-307, Kraków 2006 ISBN 83-915141-5-3, EAN 9788391514153.
Rząsa W., Bubak M., Baliś B., Szepieniec T.: Overhead Verification for Cryptographically Secured Transmission in the Grid. Computing and
Informatics, Vol. 26, 2007, 89-101.
Rząsa W., Bubak M.: Application of Petri Nets to Evaluation of Grid Applications Efficiency. In proc. of CGW'08, pp. 261-269, Kraków 2009,
ISBN 978-83-61433-00-2.
Rząsa W.: Combining Timed Colored Petri Nets and Real TCP Implementation to Reliably Simulate Distributed Applications. CN 2009, CCIS 39,
pp. 79-86, 2009, Eds. A. Kwiecień, P. Gaj, and P. Stera.
Dec G, Jędrzejec B, Rząsa W.: Kolorowana sieć Petriego jako model systemu podejmowania decyzji kredytowej. STUDIA INFORMATICA 2010,
Volume 31, Number 2A (89).
Rząsa W., Bubak M.: Simulation Method Supporting Development of Parallel Applications for Grids. In proc. of CGW'10, pp. 194-201, Kraków
2011, ISBN 978-83-61433-03-3.
Dec G., Rząsa W.: Modelowanie wielowarstwowej rozproszonej aplikacji www z zastosowaniem TCPN. Praca zbiorowa pod red. L. Trybusa i
S. Samoleja: Projektowanie, analiza i implementacja systemów czasu rzeczywistego, ISBN 878-83-206-1822-8, Wyd. Komunikacji i Łączności,
Warszawa 2011, pp. 137-148.
Rząsa W., Rzońca D., Stec A., Trybus B.: Analysis of Challenge-Response Authentication in a Networked Control System, in: Kwiecien A., Gaj
P., and Stera P. (Eds.): Computer Networks 2012, Communications in Computer and Information Science 291, Springer-Verlag Berlin
Heidelberg 2012, pp. 271-279.
Rząsa W., Bubak M., Nawarecki E.: High-Level Model for Performance Evaluation of Distributed Applications, in: Balicki J., Krawczyk H.,
Nawarecki E. (Eds.): Grid and Volunteer Computing, Gdansk University of Technology Faculty of Elektronics, Telecomunication and
Informatics Press, Gdańsk 2012, pp. 7-23.
Rząsa W.: Synchronization Algorithm for Timed Colored Petri Nets and Ns-2 Simulators, in: Kwiecień A., Gaj P., and Stera P. (Eds): CN2013,
CCIS 370, pp. 1-10, Springer-Verlag Berlin Heidelberg, 2013, ISSN 1865-0929, ISBN 978-3-642-38864-4.
Kowalski, M.; Rzasa, W., "Object-oriented approach to Timed Colored Petri Net simulation," Computer Science and Information Systems
(FedCSIS), 2013 Federated Conference on, pp.1401,1404, 8-11 Sept. 2013, ISBN 978-1-4673-4471-5 (Web), 978-83-60810-53-8 (USB), IEEE Catalog
Number: CFP1385N-ART (Web),CFP1385N-USB (USB) Jamro M., Rzońca D., Rząsa W.: Testing communication tasks in distributed control
systems with SysML and Timed Colored Petri Nets model. Computers in Industry, Vol. 71, August 2015, pp. 77-87.
Rząsa W.: "Simulation-Based Analysis of a Platform as a Service Infrastructure Performance from a User Perspective", P. Gaj et al. (Eds.): CN
2015, CCIS 522, pp. 182–192, 2015 ISBN: 978-3-319-19418-9. Rząsa W., Rzońca D.: Event-Driven Approach to Modeling and Performance
Estimation of a Distributed Control System, in: Gaj P., Kwiecień A., and Stera P. (Eds.): Computer Networks 2016, Communications in
Computer and Information Science 608, Springer International Publishing 2016, pp. 168-179.

View Slide

SOLUTION BASICS
Simulation
Model described in Ruby-based DSL
Simulator based on a formalism
Stats available via Ruby iterators

View Slide

BASIC EXAMPLE
Web server
Web clients
Resources

View Slide

WEB SERVER
program :apache do
on_event :data_received do |data|
stats_start server: :apache, name: process.name
cpu do |cpu|
(100 * data.size.in_bytes / cpu.performance).miliseconds
end
send_data to: data.src, size: data.size * 10,
type: :response, content: data.content
stats_stop server: :apache, name: process.name
end
end

View Slide

WEB CLIENT
program :wget do |opts|
sent = 0
on_event :send do
# . . .
end
# . . .
end
register_event :send
end

View Slide

WEB CLIENT — SEND
sent = 0
on_event :send do
cpu { |cpu| (150 / cpu.performance).miliseconds }
send_data to: opts[:target], size: 1024.bytes,
type: :request, content: sent
sent += 1
if sent < opts[:count]
register_event :send, delay: 5.miliseconds
end
end
# . . .
end
end

View Slide

WEB CLIENT — RECEIVE
sent = 0
on_event :send do
# . . .
end
log "Got data #{data} in process #{process.name}"
stats event: :request_served, client: process.name
end
end

View Slide

RESOURCES
node :desktop do
cpu 100
end
node :gandalf do
cpu 1400
end
net :net01, bw: 1024.bps
net :net02, bw: 510.bps
route from: :desktop, to: :gandalf,
via: [ :net01, :net02 ], twoway: true

View Slide

PROCESSES ON NODES
new_process :client1, program: :wget,
args: { target: :server, count: 10 }
new_process :client2, program: :wget,
args: { target: :server, count: 10 }
new_process :server, program: :apache
put :server, on: :gandalf
put :client1, on: :desktop
put :client2, on: :desktop

View Slide

SAVE YOUR MODEL
e.g. in model.rb

View Slide

RUN SIMULATION
class Experiment < RBSim::Experiment
end
params = { }
sim = Experiment.new
sim.run './model.rb', params
sim.save_stats 'simulation.stats'

View Slide

PROCESS SIMULATION STATS
class Experiment < RBSim::Experiment
def print_req_times_for(s)
app_stats.durations(server: s) do |tags, start, stop|
puts "Req. time #{(stop - start).in_miliseconds} ms."
end
end
end
all_stats = Experiment.read_stats 'simulation.stats'
first_experiment = all_stats.first
first_experiment.print_req_times_for(:apache)

View Slide

BASIC EXAMPLE SUMMARY
Model in DSL
No boilerplate
Web server and client (~30 LoC)
Resources (~12 LoC)
Mapping processes to resources (~6 LoC)
Running simulation (~8 LoC)
Loading saved stats (~3 LoC)

View Slide

CASE STUDY #1
RAPGENIUS VS. HEROKU
FEBRUARY 2013
https://genius.com/James-somers-herokus-ugly-
secret-annotated

View Slide

HEROKU HTTP "ROUTING"
"Intelligent"
Random

View Slide

FOR HEROKU
Perfect scalability
Don't have to detect idle/busy dynos

View Slide

FOR A CLIENT
Is "random routing" worse?
How much?

View Slide

https://genius.com/James-somers-herokus-ugly-
secret-annotated

View Slide

-- E. Dijkstra
In the good old days physicists
repeated each other's experiments,
just to be sure. Today they stick to
FORTRAN, so that they can share each
other's programs, bugs included.

View Slide

ITEMS TO MODEL
Random HTTP router
"Intelligent" HTTP

View Slide

RANDOM HTTP ROUTER
program :random_router do |servers|
if data.type == :request
server = servers.sample
send_data to: server, size: data.size, type: :request,
content: { from: data.src, content: data.content }
elsif data.type == :response
send_data to: data.content[:from], size: data.size,
type: :response,
content: data.content[:content]
else
raise "Unknown data type #{data.type} received " +
"by #{process.name}"
end
end
end

View Slide

INTELLIGENT HTTP ROUTER
program :router do |servers|
request_queue = []
# . . .
end
on_event :process_request do
# . . .
end
end

View Slide

if data.type == :request
request_queue << data
register_event :process_request
elsif data.type == :response
servers << data.src
send_data to: data.content[:from], size: data.size,
type: :response, content: data.content[:content]
else
raise "Unknown data type #{data.type} received " +
"by #{process.name}"
end
end

View Slide

on_event :process_request do
unless servers.empty? or request_queue.empty?
data = request_queue.shift
server = servers.shift
send_data to: server, size: data.size, type: :request,
content: { from: data.src, content: data.content }
unless request_queue.empty?
end
end
end

View Slide

RESOURCES
(EXAMPLE)
servers.each do |s|
node s do
cpu 1
end
new_process s, program: :webserver,
args: { request_times: params[:request_times] }
put s, on: s
end

View Slide

RESULTS

View Slide


View Slide

WANT APDEX?

View Slide


View Slide

WHAT IF?
we had more independent "intelligent" routers?
good scalability + better performance for users?

View Slide


View Slide

CASE STUDY #1 SUMMARY
Reusability (model items)
Flexibility (arbitrary algorithms in routers)
Different levels of details for results
histograms
apdex
What ifs

View Slide

CASE STUDY #2
TO SCALE HEROKU APPLICATION ...
...OR NOT TO SCALE?

View Slide

HEROKU DYNOS
Name RAM CPU
share
Compute Price per
dyno-month
standard-
1x
512MB 1x 1x-4x $25
standard-
2x
1024MB 2x 4x-8x $50

View Slide

API BACKEND
Rails
Unicorn
CPU intensive
6 standard-1x dynos
scale to 3 standard-2x dynos?

View Slide

UNICORN
Master process
Balancing load of worker processes
Like Heroku's old "intelligent router"!

View Slide

APPLICATION SCALING
2x faster dynos
2x fewer dynos
More Unicorn workers per dyno
Same number of Unicorn workers per application
Same price
More RAM for peaks
Better load balancing?

View Slide

APPLICATION PARAMETERS
Load (req/min)
Response times (distribution)

View Slide

ITEMS TO MODEL

View Slide

REUSED ITEMS
HTTP client
HTTP server
Random HTTP router
Unicorn master process (Intelligent Heroku router)

View Slide

MODELING DYNOS
OR
node :standard1x do
cpu 1
end
node :standard2x do
cpu 2
end
node :standard2x do
cpu 1
cpu 1
end

View Slide

HEROKU DYNOS
Name RAM CPU
share
Compute Price per
dyno-month
standard-
1x
512MB 1x 1x-4x $25
standard-
2x
1024MB 2x 4x-8x $50

View Slide

DYNO SCALING
HORIZONTAL OR VERTICAL?
DOES IT MATTER!?

View Slide

SIMULATE
AND
node :standard2x do
cpu 2
end
node :standard2x do
cpu 1
cpu 1
end

View Slide

HORIZONTAL DYNO SCALING

View Slide

VERTICAL DYNO SCALING

View Slide

IT DOES MATTER HOW DYNOS
ARE SCALED!
HOW TO FIND OUT?
documentation does not help...
cat /proc/cpuinfo does not help...

View Slide

EXPERIMENT
(ON HEROKU DYNOS)

View Slide

SINGLE CPU INTENSIVE TASK
Comparable time on both dyno types
def cpu_intensive_task(n)
start = Time.now
(1..n).reduce(:*)
Time.now - start
end

View Slide

16 CPU INTENSIVE TASKS AT
ONCE
on standard-1x (2 CPUs)
on standard-2x (4 CPUs)
real 1m8.690s
user 2m13.360s
sys 0m3.871s
real 0m29.182s
user 2m17.570s
sys 0m4.053s

View Slide

CONCLUSION
Dynos are scaled horizontally (more CPUs)
We shouldn't change dyno config

View Slide

PAY THE BILL!
$0.09

View Slide

CASE STUDY #2 SUMMARY
Modeling with reusable components
Simulation-tested alternative configurations
Simple, cheap experiments to verify crucial factors

View Slide

SUMMARY
Easier, cheaper, faster
DSL, no boilerplate code
What ifs
No magic — just software science
Simulation as a Service ;-)
Rubber duck

View Slide

Predicting Performance Changes of Distributed Applications

Predicting Performance Changes of Distributed Applications

wrzasa

More Decks by wrzasa

Other Decks in Programming

Featured

Transcript