Measuring the Performance of an XML-Based Communication Primitive Gregory M. Kapfhammer Department of Computer Science Allegheny College http : //cs.allegheny.edu/˜gkapfham/ Measuring the Performance of an XML-Based Communication Primitive, RICSS, September 22, 2006 – p. 1/15 

A Unique Invocation The London Times asks “What’s Wrong with the World?” Dear Sirs, I am. Sincerely yours, G. K. Chesterton Measuring the Performance of an XML-Based Communication Primitive, RICSS, September 22, 2006 – p. 2/15 

Storage and Communication Primitives How does object encoding impact performance? Contribution: A benchmarking framework to compare the performance of sockets and XML-RPC Measuring the Performance of an XML-Based Communication Primitive, RICSS, September 22, 2006 – p. 3/15 

Remote Communication and OpenDHT Clients can put and get with Sun RPC or XML-RPC Does the communication primitive impact performance? How do we measure performance and/or correctness? Measuring the Performance of an XML-Based Communication Primitive, RICSS, September 22, 2006 – p. 4/15 

Program Execution with a JVM Program Stack Fast? Interpreter? Machine Virtual JIT? Adaptive? Native Code Cache Heap methodA testOne Input Output Byte Code P JVM implementation and configuration impacts performance Measuring the Performance of an XML-Based Communication Primitive, RICSS, September 22, 2006 – p. 5/15 

Micro Benchmarks Experiment Sent by client Received by client SS Single primitive Single primitive SV Single primitive Vector VS Vector Single primitive VV Vector Vector Use benchmarks similar to those proposed by Allman et al. Implement the benchmarks in the Java language ExperimentCampaign framework uses Perl and Mathematica Measuring the Performance of an XML-Based Communication Primitive, RICSS, September 22, 2006 – p. 6/15 

Micro Benchmarks II Experiment Sent by client Received by client FIND (SS) Single primitive Single primitive FACT (SV) Single primitive Vector GCD (VS) Vector Single primitive REV (VV) Vector Vector Benchmarks use sockets and Apache XML-RPC Benchmarks perform a simple computation on the server Configure the client and server to execute on same node Measuring the Performance of an XML-Based Communication Primitive, RICSS, September 22, 2006 – p. 7/15 

Experiment Design Select Java 1.5.0, GNU/Linux with kernel 2.6.12, 3 GHz P4, 1 GB main memory, 1 MB L1 Cache, CPU hyperthreading Use operating system and language-based timers to calculate R(B, P), R∆ (B, P, P ), and R% ∆ (B, P, P ) Replace the socket communication primitive with XML-RPC Execute ten trials and calculate arithmetic means, standard deviations, and confidence intervals Formulate the null hypothesis as H0 : µR(B,P) = µR(B,P ) Use the Welch’s approximate t-test with α = .01 Measuring the Performance of an XML-Based Communication Primitive, RICSS, September 22, 2006 – p. 8/15 

Micro Benchmark I S SS X SS S SV X SV S VS X VS S VV X VV Benchmarks 0.02 0.04 0.06 0.08 0.1 0.12 0.14 Time seconds S SS X SS S SV X SV S VS X VS S VV X VV Micro Experiments Language Based Timer 0.0004 0.0859 0.0013 0.0897 0.0012 0.0908 0.0237 0.0897 XML-RPC shows greater response time with more dispersion Measuring the Performance of an XML-Based Communication Primitive, RICSS, September 22, 2006 – p. 9/15 

Micro Benchmark II S FIND X FIND S FACT X FACT S GCD X GCD S REV X REV Benchmarks 0.25 0.5 0.75 1 1.25 1.5 1.75 2 Time seconds S FIND X FIND S FACT X FACT S GCD X GCD S REV X REV Macro Experiments Language Based Timer 0.0017 0.0863 0.0023 0.093 0.0022 0.0857 0.0033 1.7605 X-REV exhibits high response time due to string parsing Measuring the Performance of an XML-Based Communication Primitive, RICSS, September 22, 2006 – p. 10/15 

Using Very Large Vectors size(V ) size(V ) (bytes) R(VV, S) (sec) R(VV, X) (sec) 5000 80,520 0.298 0.347 10000 161,000 0.598 0.523 50000 927,720 18.784 1.697 At smaller vector sizes sockets demonstrate slightly better response times XML-RPC shows better response time when size(V ) = 50000 : why? Measuring the Performance of an XML-Based Communication Primitive, RICSS, September 22, 2006 – p. 11/15 

Explanatory Power of GC size(V ) YGC (count) YGC (sec) FGC (count) FGC (sec) 5000 16 .008 0 0 10000 63 .023 4 .050 50000 1645 .697 663 10.375 size(V ) YGC (count) YGC (sec) FGC (count) FGC (sec) 5000 14 .016 0 0 10000 27 .022 1 .020 50000 123 .695 5 .143 Varying the heap size of socket configuration yields similar results Measuring the Performance of an XML-Based Communication Primitive, RICSS, September 22, 2006 – p. 12/15 

GC Allocation Rate S-VV allocates 710, 374, 184 bytes and X-VV only allocates 54, 101, 312 bytes At benchmark termination, S-VV has 4, 773, 224 bytes and X-VV has 7, 234, 520 bytes of live objects Sockets use char[] and XML-RPC uses java.nio.CharBuffer Can we use past GC behavior to predict future program performance? Measuring the Performance of an XML-Based Communication Primitive, RICSS, September 22, 2006 – p. 13/15 

Conclusions A suite of micro benchmarks to measure the performance of communication primitives A comparison of sockets and XML-RPC that we can extend to other primitives Experiments reveal a trade-off in the performance of the two primitives Extend the study to new primitives and JVMs Focus on remote communication, long running benchmarks, and the measurement of throughput What are your suggestions? Measuring the Performance of an XML-Based Communication Primitive, RICSS, September 22, 2006 – p. 14/15 

An Invitation to Participate I value your comments, suggestions, and participation! http : //cs.allegheny.edu/˜gkapfham/research/ Measuring the Performance of an XML-Based Communication Primitive, RICSS, September 22, 2006 – p. 15/15