Predicting Performance Changes of Distributed Applications

1.

PREDICTING PERFORMANCE CHANGES OF DISTRIBUTED APPLICATIONS Wojciech Rząsa @wrzasa Wojciech
Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

2.

Passion for informatics PhD, but primarily an engineer Rzeszow University
of Technology Research: distributed systems Teaching: Ruby, Rails, ... Rzeszow Ruby User Group ABOUT ME http://rrug.pl Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

3.

THE GRID Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed
Applications

4.

(c) e-ScienceCity , . http://www.e-sciencecity.org/ Creative Commons Attribution-ShareAlike 3.0 Unported
License Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

5.

GRID MONITORING — OCM-G Debugging Interactive applications Shared infrastructure Distributed
No central management Standard interface for tools Tight security requirements Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

6.

SECURITY VS. PERFORMANCE Wojciech Rząsa @wrzasa Predicting Performance Changes of
Distributed Applications

7.

EMPOWER DEVELOPERS TO ASSESS PERFORMANCE! Wojciech Rząsa @wrzasa Predicting Performance
Changes of Distributed Applications

8.

AGENDA How to predict performance changes Basic example Two case
studies Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

9.

Rząsa W.: Timed colored Petri net based estimation of efficiency
of the grid applications. Supervisor: E. Nawarecki, AGH-UST, Kraków, 2011. Baliś B., Bubak M., Rząsa W., Szepieniec T., Wismüller R.: Security in the OCM-G Grid Application Monitoring System. PPAM 2003, LNCS 3019, pp. 779-787, 2004, Eds. R. Wyrzykowski et al. Baliś B., Bubak M., Rząsa W., Szepieniec T., Wismüller R.: Two Aspects of Security Solution for Distributed Systems in the Grid on the Example of the OCM-G. In proc. of CGW'03, pp.197-206, Kraków 2004 ISBN 83-915141-3-7. Baliś B., Bubak M., Rząsa W., Szepieniec T.: Efficiency of the GSI Secured Network Transmission. ICCS 2004, LNCS 3036, p. 107-115, 2004, Eds. M. Bubak et al. Rząsa W., Bubak M., Baliś B., Szepieniec T.: Simulation Method for Estimation of Security Overhead of Grid Applications. In proc. of CGW'05, pp. 300-307, Kraków 2006 ISBN 83-915141-5-3, EAN 9788391514153. Rząsa W., Bubak M., Baliś B., Szepieniec T.: Overhead Verification for Cryptographically Secured Transmission in the Grid. Computing and Informatics, Vol. 26, 2007, 89-101. Rząsa W., Bubak M.: Application of Petri Nets to Evaluation of Grid Applications Efficiency. In proc. of CGW'08, pp. 261-269, Kraków 2009, ISBN 978-83-61433-00-2. Rząsa W.: Combining Timed Colored Petri Nets and Real TCP Implementation to Reliably Simulate Distributed Applications. CN 2009, CCIS 39, pp. 79-86, 2009, Eds. A. Kwiecień, P. Gaj, and P. Stera. Dec G, Jędrzejec B, Rząsa W.: Kolorowana sieć Petriego jako model systemu podejmowania decyzji kredytowej. STUDIA INFORMATICA 2010, Volume 31, Number 2A (89). Rząsa W., Bubak M.: Simulation Method Supporting Development of Parallel Applications for Grids. In proc. of CGW'10, pp. 194-201, Kraków 2011, ISBN 978-83-61433-03-3. Dec G., Rząsa W.: Modelowanie wielowarstwowej rozproszonej aplikacji www z zastosowaniem TCPN. Praca zbiorowa pod red. L. Trybusa i S. Samoleja: Projektowanie, analiza i implementacja systemów czasu rzeczywistego, ISBN 878-83-206-1822-8, Wyd. Komunikacji i Łączności, Warszawa 2011, pp. 137-148. Rząsa W., Rzońca D., Stec A., Trybus B.: Analysis of Challenge-Response Authentication in a Networked Control System, in: Kwiecien A., Gaj P., and Stera P. (Eds.): Computer Networks 2012, Communications in Computer and Information Science 291, Springer-Verlag Berlin Heidelberg 2012, pp. 271-279. Rząsa W., Bubak M., Nawarecki E.: High-Level Model for Performance Evaluation of Distributed Applications, in: Balicki J., Krawczyk H., Nawarecki E. (Eds.): Grid and Volunteer Computing, Gdansk University of Technology Faculty of Elektronics, Telecomunication and Informatics Press, Gdańsk 2012, pp. 7-23. Rząsa W.: Synchronization Algorithm for Timed Colored Petri Nets and Ns-2 Simulators, in: Kwiecień A., Gaj P., and Stera P. (Eds): CN2013, CCIS 370, pp. 1-10, Springer-Verlag Berlin Heidelberg, 2013, ISSN 1865-0929, ISBN 978-3-642-38864-4. Kowalski, M.; Rzasa, W., "Object-oriented approach to Timed Colored Petri Net simulation," Computer Science and Information Systems (FedCSIS), 2013 Federated Conference on, pp.1401,1404, 8-11 Sept. 2013, ISBN 978-1-4673-4471-5 (Web), 978-83-60810-53-8 (USB), IEEE Catalog Number: CFP1385N-ART (Web),CFP1385N-USB (USB) Jamro M., Rzońca D., Rząsa W.: Testing communication tasks in distributed control systems with SysML and Timed Colored Petri Nets model. Computers in Industry, Vol. 71, August 2015, pp. 77-87. Rząsa W.: "Simulation-Based Analysis of a Platform as a Service Infrastructure Performance from a User Perspective", P. Gaj et al. (Eds.): CN 2015, CCIS 522, pp. 182–192, 2015 ISBN: 978-3-319-19418-9. Rząsa W., Rzońca D.: Event-Driven Approach to Modeling and Performance Estimation of a Distributed Control System, in: Gaj P., Kwiecień A., and Stera P. (Eds.): Computer Networks 2016, Communications in Computer and Information Science 608, Springer International Publishing 2016, pp. 168-179. Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

10.

SOLUTION BASICS Simulation Model described in Ruby-based DSL Simulator based
on a formalism Stats available via Ruby iterators Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

11.

BASIC EXAMPLE Web server Web clients Resources Wojciech Rząsa @wrzasa
Predicting Performance Changes of Distributed Applications

12.

WEB SERVER program :apache do on_event :data_received do |data| stats_start
server: :apache, name: process.name cpu do |cpu| (100 * data.size.in_bytes / cpu.performance).miliseconds end send_data to: data.src, size: data.size * 10, type: :response, content: data.content stats_stop server: :apache, name: process.name end end Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

13.

WEB CLIENT program :wget do |opts| sent = 0 on_event
:send do # . . . end on_event :data_received do |data| # . . . end register_event :send end Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

14.

WEB CLIENT — SEND program :wget do |opts| sent =
0 on_event :send do cpu { |cpu| (150 / cpu.performance).miliseconds } send_data to: opts[:target], size: 1024.bytes, type: :request, content: sent sent += 1 if sent < opts[:count] register_event :send, delay: 5.miliseconds end end on_event :data_received do |data| # . . . end register_event :send end Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

15.

WEB CLIENT — RECEIVE program :wget do |opts| sent =
0 on_event :send do # . . . end on_event :data_received do |data| log "Got data #{data} in process #{process.name}" stats event: :request_served, client: process.name end register_event :send end Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

16.

RESOURCES node :desktop do cpu 100 end node :gandalf do
cpu 1400 end net :net01, bw: 1024.bps net :net02, bw: 510.bps route from: :desktop, to: :gandalf, via: [ :net01, :net02 ], twoway: true Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

17.

PROCESSES ON NODES new_process :client1, program: :wget, args: { target:
:server, count: 10 } new_process :client2, program: :wget, args: { target: :server, count: 10 } new_process :server, program: :apache put :server, on: :gandalf put :client1, on: :desktop put :client2, on: :desktop Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

18.

SAVE YOUR MODEL e.g. in model.rb Wojciech Rząsa @wrzasa Predicting
Performance Changes of Distributed Applications

19.

RUN SIMULATION class Experiment < RBSim::Experiment end params = {
} sim = Experiment.new sim.run './model.rb', params sim.save_stats 'simulation.stats' Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

20.

PROCESS SIMULATION STATS class Experiment < RBSim::Experiment def print_req_times_for(s) app_stats.durations(server:
s) do |tags, start, stop| puts "Req. time #{(stop - start).in_miliseconds} ms." end end end all_stats = Experiment.read_stats 'simulation.stats' first_experiment = all_stats.first first_experiment.print_req_times_for(:apache) Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

21.

BASIC EXAMPLE SUMMARY Model in DSL No boilerplate Web server
and client (~30 LoC) Resources (~12 LoC) Mapping processes to resources (~6 LoC) Running simulation (~8 LoC) Loading saved stats (~3 LoC) Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

22.

CASE STUDY #1 RAPGENIUS VS. HEROKU FEBRUARY 2013 https://genius.com/James-somers-herokus-ugly- secret-annotated
Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

23.

HEROKU HTTP "ROUTING" "Intelligent" Random Wojciech Rząsa @wrzasa Predicting Performance
Changes of Distributed Applications

24.

FOR HEROKU Perfect scalability Don't have to detect idle/busy dynos

25.

FOR A CLIENT Is "random routing" worse? How much? Wojciech
Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

26.

https://genius.com/James-somers-herokus-ugly- secret-annotated Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed
Applications

27.

-- E. Dijkstra In the good old days physicists repeated
each other's experiments, just to be sure. Today they stick to FORTRAN, so that they can share each other's programs, bugs included. Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

28.

ITEMS TO MODEL Random HTTP router "Intelligent" HTTP Wojciech Rząsa
@wrzasa Predicting Performance Changes of Distributed Applications

29.

RANDOM HTTP ROUTER program :random_router do |servers| on_event :data_received do
|data| if data.type == :request server = servers.sample send_data to: server, size: data.size, type: :request, content: { from: data.src, content: data.content } elsif data.type == :response send_data to: data.content[:from], size: data.size, type: :response, content: data.content[:content] else raise "Unknown data type #{data.type} received " + "by #{process.name}" end end end Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

30.

INTELLIGENT HTTP ROUTER program :router do |servers| request_queue = []
on_event :data_received do |data| # . . . end on_event :process_request do # . . . end end Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

31.

INTELLIGENT HTTP ROUTER on_event :data_received do |data| if data.type ==
:request request_queue << data register_event :process_request elsif data.type == :response servers << data.src send_data to: data.content[:from], size: data.size, type: :response, content: data.content[:content] register_event :process_request else raise "Unknown data type #{data.type} received " + "by #{process.name}" end end Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

32.

INTELLIGENT HTTP ROUTER on_event :process_request do unless servers.empty? or request_queue.empty?
data = request_queue.shift server = servers.shift send_data to: server, size: data.size, type: :request, content: { from: data.src, content: data.content } unless request_queue.empty? register_event :process_request end end end Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

33.

RESOURCES (EXAMPLE) servers.each do |s| node s do cpu 1
end new_process s, program: :webserver, args: { request_times: params[:request_times] } put s, on: s end Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

34.

RESULTS Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

35.


36.


37.


38.

WANT APDEX? Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed
Applications

39.


40.

WHAT IF? we had more independent "intelligent" routers? good scalability
+ better performance for users? Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

41.


42.


43.


44.


45.


46.


47.

CASE STUDY #1 SUMMARY Reusability (model items) Flexibility (arbitrary algorithms
in routers) Different levels of details for results histograms apdex What ifs Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

48.

CASE STUDY #2 TO SCALE HEROKU APPLICATION ... ...OR NOT
TO SCALE? Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

49.

HEROKU DYNOS Name RAM CPU share Compute Price per dyno-month
standard- 1x 512MB 1x 1x-4x $25 standard- 2x 1024MB 2x 4x-8x $50 Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

50.

API BACKEND Rails Unicorn CPU intensive 6 standard-1x dynos scale
to 3 standard-2x dynos? Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

51.

UNICORN Master process Balancing load of worker processes Like Heroku's
old "intelligent router"! Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

52.

APPLICATION SCALING 2x faster dynos 2x fewer dynos More Unicorn
workers per dyno Same number of Unicorn workers per application Same price More RAM for peaks Better load balancing? Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

53.

APPLICATION PARAMETERS Load (req/min) Response times (distribution) Wojciech Rząsa @wrzasa
Predicting Performance Changes of Distributed Applications

54.

ITEMS TO MODEL Wojciech Rząsa @wrzasa Predicting Performance Changes of

55.

REUSED ITEMS HTTP client HTTP server Random HTTP router Unicorn
master process (Intelligent Heroku router) Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

56.

MODELING DYNOS OR node :standard1x do cpu 1 end node
:standard2x do cpu 2 end node :standard2x do cpu 1 cpu 1 end Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

57.

HEROKU DYNOS Name RAM CPU share Compute Price per dyno-month
standard- 1x 512MB 1x 1x-4x $25 standard- 2x 1024MB 2x 4x-8x $50 Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

58.

DYNO SCALING HORIZONTAL OR VERTICAL? DOES IT MATTER!? Wojciech Rząsa
@wrzasa Predicting Performance Changes of Distributed Applications

59.

SIMULATE AND node :standard2x do cpu 2 end node :standard2x
do cpu 1 cpu 1 end Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

60.

HORIZONTAL DYNO SCALING Wojciech Rząsa @wrzasa Predicting Performance Changes of

61.

VERTICAL DYNO SCALING Wojciech Rząsa @wrzasa Predicting Performance Changes of

62.

IT DOES MATTER HOW DYNOS ARE SCALED! HOW TO FIND
OUT? documentation does not help... cat /proc/cpuinfo does not help... Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

63.

EXPERIMENT (ON HEROKU DYNOS) Wojciech Rząsa @wrzasa Predicting Performance Changes
of Distributed Applications

64.

SINGLE CPU INTENSIVE TASK Comparable time on both dyno types
def cpu_intensive_task(n) start = Time.now (1..n).reduce(:*) Time.now - start end Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

65.

16 CPU INTENSIVE TASKS AT ONCE on standard-1x (2 CPUs)
on standard-2x (4 CPUs) real 1m8.690s user 2m13.360s sys 0m3.871s real 0m29.182s user 2m17.570s sys 0m4.053s Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

66.

CONCLUSION Dynos are scaled horizontally (more CPUs) We shouldn't change
dyno config Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

67.

PAY THE BILL! $0.09 Wojciech Rząsa @wrzasa Predicting Performance Changes
of Distributed Applications

68.

CASE STUDY #2 SUMMARY Modeling with reusable components Simulation-tested alternative
configurations Simple, cheap experiments to verify crucial factors Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

69.

SUMMARY Easier, cheaper, faster DSL, no boilerplate code What ifs
No magic — just software science Simulation as a Service ;-) Rubber duck Wojciech Rząsa @wrzasa Predicting Performance Changes of Distributed Applications

Predicting Performance Changes of Distributed Applications

Predicting Performance Changes of Distributed Applications

Want more?

Transcript