The (not so) dark art of performance tuning (from Newts to Newton)

Copyright 2014 Kodewerk Ltd. All rights reserved The (not so)
Dark Art of Performance Tuning From Newts to Newton Kirk Peperdine @kcpeppe Aleksey Shiplev @shipilev

Kodewerk Java Performance Services tm Copyright 2014 Kodewerk Ltd. All
rights reserved About Me • Consultant (www.kodewerk.com) • performance tuning and training seminar • Co-author www.javaperformancetuning.com • Member of Java Champion program • Other stuff... (google is you care to)

rights reserved Disclaimer The resemblance of any opinion, recommendation or comment made during this presentation to performance tuning advice is merely coincidental.

rights reserved Our Typical Customer Application isn’t performing to project sponsors expectations Development team has been tuning for weeks some improvements but... Different team experts come to the table with different opinions finger pointing

rights reserved Where is the problem?

rights reserved Commonly Heard I see StringBuffer is being used all over, lets change it to StringBuilder I think our DBMS is the problem, we need to migrate to [buzzword] Not sure where the problem is but we’ve been changing code to make things better

rights reserved Better Action Understand your performance requirements typically response time or throughput Your goal is to improve this metric

rights reserved What if the code you’ve changed is not used at all? it accounts for just a few microseconds of time Maybe a good idea is changes are small, isolated and painless to make

rights reserved What if CPU utilization is 6.25% method precomutes something reused later? “I can see that method bar() is accounts for 5% of time, let’s remoe it”

rights reserved What if you just depleted disk bandwidth IT had reshaped the network connection database just needs a cleanup? “I think our database is the problem! lets migrate to [buzzword]”

rights reserved Perf tuning 101 The space can be humongous you can’t traverse it all assume something is or isn’t part of the problem Prefer hypothesis free investigation be methodical step wise process to arrive at a conclusion

rights reserved Measure Don’t Guess®

rights reserved Typical Dev Teams Developments teams are often very skilled but don’t understand performance testing Performance data often taken out of context mis-understood completely missed

rights reserved As developers we tend to be too focused on the code!!!!

rights reserved public class Software { public static void main( String[] args) { System.out.println(“Software is abstract”); } }

rights reserved Hardware is Real!

Copyright 2014 Kodewerk Ltd. All rights reserved Physical Limits CPU
capacity throughput : clock speed granularity: cache line Memory capacity : volume Bus throughput : clock speed width : 32 bits

Copyright 2014 Kodewerk Ltd. All rights reserved Limits of Hardware
disk ~1 Bbit/sec (SATA ~3Gbits/sec) granularity: disk sector (512 bytes) Network frame buffer/packet frequency (clock)

Copyright 2014 Kodewerk Ltd. All rights reserved Other Limits other
hardware devices video/sound heat battery time

Copyright 2014 Kodewerk Ltd. All rights reserved public class Software
{ public static void main( String[] args) { System.out.println(“Software is abstract”); } }

rights reserved But we already have measurements!

rights reserved Yeah but…. ! do you have a context to understand the measurements?

rights reserved Are they the right measurements

rights reserved Are you blind to the real problem?

Copyright 2014 Kodewerk Ltd. All rights reserved Application Developers Live
Here business logic, non-shareable soft resources

Copyright 2014 Kodewerk Ltd. All rights reserved Question? Which is
faster? a) Bubble sort b) Quick sort In Big O notation... - Bubble sort is N^2 - Quick sort of Nlog(N)

Copyright 2014 Kodewerk Ltd. All rights reserved Application Performance is
rooted in dynamics dynamics algorithmic strength

Copyright 2014 Kodewerk Ltd. All rights reserved OS/Hardware JVM Application
Application Runs Here CPU, memory, disk I/O network I/O, Locks manage memory, execution dynamics algorithmic strength

Process Diagnostic Model CPU, memory, disk I/O network I/O, Locks manage memory, execution Actors usage patterns algorithmic strength

Actors Mixing in the dynamics CPU, memory, disk I/O network I/O. Locks manage memory, execution usage patterns Actors drives application Application drives JVM JVM drives OS/Hardware Hardware consumed function of actors function of application cannot exceed capacity algorithmic strength

rights reserved Hardware counters show how the hardware is being consumed subscription rates to non-shareable components tells us what our application is doing Hardware consumtion is a fuction of load Hardware Patterns

Actors Dominating Consumer CPU, memory, disk I/O network I/O. Locks manage memory, execution usage patterns Activity that dominates how the CPU is utilized Determine by analyizing breakout of CPU counters garbage collection logs algorithmic strength

Copyright 2014 Kodewerk Ltd. All rights reserved Dominating Consumer Choices
OS/Hardware Application Actors CPU, memory, disk I/O network I/O, Locks manage memory, execution algorithmic strength usage patterns Application JVM Liveliness System JVM

Copyright 2014 Kodewerk Ltd. All rights reserved Dominating Consumer Conditions
sys cpu > 10% of user cpu user CPU ~= 100% memory efficient? GC Logs Application JVM Liveliness System system profiling: netstat, mpstat, iostat, sar, strace, gc logs, etc… Thread starvation Thread dump app/CPU profiling GC tuning, pool sizes, collectors, ... Memory profiling, size frequency, life span,... yes yes yes no no no

Copyright 2014 Kodewerk Ltd. All rights reserved Expression of Consumption
Application JVM Liveliness System passively dominant aggressivly dominant

Copyright 2014 Kodewerk Ltd. All rights reserved Measuing Consumption Application
JVM Liveliness System Kernel time Idle User time

rights reserved r b swpd free buff cache si so bi bo in cs us sy id wa ! 3 9 100 24496 11096 13267036 0 0 0 5 0 1 2 1 96 1! 3 2 100 23420 11088 13268328 0 0 0 0 77330 175352 17 26 39 17! 3 9 100 20836 11088 13270628 0 0 0 68 105118 227382 14 40 21 25! 8 4 100 23356 11080 13268272 0 0 0 0 80062 164387 12 30 29 30! 7 7 100 23180 11084 13267068 0 0 0 72 98353 234851 15 43 28 15! 11 2 100 25820 11088 13263676 0 0 0 120 100749 214921 11 42 17 30! 13 1 100 22316 11088 13267176 0 0 0 0 103878 246723 16 56 19 9 ! 4 3 100 21824 11088 13269140 0 0 0 0 48625 97288 15 16 9 60! 11 2 100 20932 11080 13269808 0 0 0 0 110760 236774 14 41 24 20! 1 12 100 23624 11084 13267488 0 0 0 204 69117 148611 15 27 25 33! 7 5 100 24996 11096 13267476 0 0 0 164 24495 48552 13 10 30 48! 1 12 100 20792 11096 13271872 0 0 0 0 25659 54331 8 9 26 56! 6 8 100 21984 11080 13269920 0 0 0 20 46309 101404 16 18 51 15! 4 9 100 22764 11080 13268956 0 0 16 0 88553 229557 17 35 38 11 Dominating Consumer???

rights reserved Performance benchmarking

rights reserved

rights reserved Relevant: reproduces the phenomena Isolated: leavesout unwanted effects Measurable: provides the needed metrics Reliable: produces consisten results Experimental Setup You can’t go anywhere without a proper test environment

rights reserved Define: Unit of Test the thing you are testing Test Harness everything else!!!! Experimental Setup

rights reserved Hardware production like configuration hidden bottlenecks phantom bottlenecks Software test harness Load injector Relevant

rights reserved Software test harness load injector Data-Lite performance testing anti-pattern production like in volume and veracity beware of the effects of caching it is all over the place and it works!!!!! Relevant

rights reserved Quiet All activity counts against your hardware budget affects performance counters (global) introduces noise which will corrupt the diagnosis phantom bottlenecks Isolated

rights reserved Usage Patterns defines workload use case + tx rate + velocity Performance requirements Validation test the test make sure bottleneck isn’t in the test harness Measurable and Reliable

rights reserved Script usage patterns into a load test Install/configure app to the same as prod Setup monitoring OS performance counters and GC logging Lill everything else running on your system Spike test to ensure correctness validate to ensure test tests what needs to be tested Setup

rights reserved Benchmark benchmark = new Benchmark() benchmark.configure(); performance = benchmark.baseline(application); user.setHappy(performance.meets(requirements)); while (( ! user.isHappy()) && (user.hasMoney())) { Profiler profiler = performance.identifyDominatingConsumer(); profilingResults = benchmark.profile(profiler); application.fixUsing( profilingResults); while ( application.failsQA()) application.debug(); performance = benchmark.baseline(application); user.setHappy(performance.meets(requirements)); } Process

rights reserved Time for a demo

rights reserved Throughput (Bandwidth) rate at which operations are retired ops/sec, MB/sec, frags/sec easy to measure and interpret Time (latency) how much time on operation took generally hard to measure reliably Metrics

rights reserved What is in a response time execution dead time response time = ∑ execution + ∑ dead time

rights reserved Many thing affect dead time and consequently execution time these can change from request to request Metrics

rights reserved Bandwidth vs Latency

rights reserved Reliability

rights reserved Any errors in the test environment will affect results throwing exceptions running out of memory other failures that short-circuit Coordinated omission will back pressure from the server interfere with the test harness’s ability to apply load? Ensure Test is Reliable

rights reserved Generational Counts Quiz how to identify a memory leak

rights reserved Generational Counts Answer

rights reserved Speeding up the Application

rights reserved Change something in some specific way! response to some localized condition How to Speed up the application

rights reserved Change something in some specific way! How to Speed up the application What prevents the application from performing? ! ! Where does it reside? ! ! How can we stop it from messing with performance?

rights reserved Identify transaction not meeting functional requirement pin point area of concern Identify dominating consumer determine the nature of the problem System -> Application -> Microarchitecture Top-Down

rights reserved Entry point is CPU utilization use tools to expose performance counter System Dominant/Liveliness

rights reserved Consider network, scheduling, swapping or other kernel activities System Dominant

rights reserved In some cases hardware/OS configuration is needed May need to change the application Other contributors

rights reserved Unbalanced threading lots of voluntary context switches (thrashing) Lots of involuntary context switches (saturation) Scheduling

rights reserved Swapping is a performance killer for Java avoid swapping at all costs kill other processes to save memory Swapping

rights reserved Sometimes the kernel is your enemy Unusual API chooices from the JVM and/or application (un)known bugs Kernel

rights reserved Usual things when interacting with devices IRQ balancing sometimes this is expensive IRQ%, SOFT%

rights reserved Watch for Disk activity, throughput and IOPS IOWait%

rights reserved Is all that disk activity necessary? caching, buffering are your friends faster disk can solve throughput/IOPS problems IOWait%

rights reserved More caching helps? free up memory for caches trade performance for consistency IOWait%

rights reserved CPU are free and no one is using them easy to diagnose but commonly missed Idle%

rights reserved Running low-threaded application on multicore improve paralyzation of algorithms throw away CPUs Idle%

rights reserved There are not enough threads ready to run Locking? Waiting for something else? Socket? Idle%

rights reserved Surprising case application is highly threaded misconfigured Garbage collector Idle%

rights reserved Application/JVM

rights reserved Application/JVM is finally busy This is where most people start and profilers start to be actually useful user%

rights reserved Memory The gem and the curse of von-Neumann architectures cache misses dominate in most applications user%

rights reserved TLB very important for memory-bound workloads “Invisible” artifact of virtual memory system user%

rights reserved CPU caches: capacity Important to hide memory latency (and bandwidth) issues Virtually all applications today are memory/ cache-bounded user%

rights reserved CPU Caches: coherence Inter-CPU communication is managed via cache coherence Understanding this is the road to master the communication user%

rights reserved Memory Bandwidth once caches run out, you face the memory dominates the cache miss performance faster memory, multiple channels help Application level (Bandwidth)

rights reserved Primitive of Queuing theory L = λτ Implications if you know the arrival rate and service time you know the length of the queue for a given L, throughput is inversely proportional to service time Little’s Law

rights reserved Plain unshared memory Plain shared memory provide communication Volatile All above, plus visibility Atomics All above, plus atomicity Atomic sections All above, plus group atomicity Spin-locks All above, plus mutual exclusion Wait-locks All above, plus blocking Coherency Primitives power speed

rights reserved It is possible at times to make an optimistic check Fallback to pessimistic version on failure The optimistic check has less power, but more performant Coherency: Optimistic Checks AtomicBoolean isSet = ...; if ( !isSet.get() && isSet.compareAndSet(false, true) { // oneshot action }

rights reserved Identify It is possible at times to make an optimistic check Fallback to pessimistic version on failure The optimistic check has less power, but more performant Coherency: Optimistic Checks ReentrantLock lock = ...; int count = LIMIT; while (!lock.tryLock()) { if (count++ > 0) { lock.lock(); break; } }

rights reserved It is possible at times to split the shared state Much less contention on modifying the local state The total state is the superposition of local states Example: thread-safe counter synchronized { i++; } AtomicInteger.inc(); ThreadLocal.set(ThreadLocal.get() + 1); AtomicInteger[random.nextInt(count)].inc(); Coherency: Striping

rights reserved If you can remove the communication, do that Immutability to enforce Thread local states Example: ThreadLocalRandom @ JDK7 Random: use CAS to maintain the state ThreadLocalRandom: essentially, ThreadLocal<Random> Can use plain memory ops to maintain the state Coherence: No-coherence

rights reserved Identify Communication quanta = cache line 32 – 128 bytes long Helps with bulk memory transfers, cache architecture Coherence protocols working on cache line ! False Sharing CPUs updating the adjacent fields? Cache line ping-pong! Coherence: False Sharing ...][AABB][...

rights reserved JVM is the new abstraction level Interacts with the application, mangles into application JVM performance affects application performance JVM Level

rights reserved Most usual contender in JVM layer Lots of things to try fixing (not covered here, see elsewhere) JVM Level GC

rights reserved Very cool to have your code compiled Sometimes it's even cooler to get the code compiled better JVM Level JIT

rights reserved Important for startup metrics; not really relevant for others Removing obstacles is the road to awe JVM Level (Classloading)

rights reserved What about the Customer? Need to gather clear requirements Develop a sound benchmarking/testing environment get better measurements Identify dominating consumer refocus team on problems that matter not always easy

rights reserved Questions

The (not so) dark art of performance tuning (from Newts to Newton)

The (not so) dark art of performance tuning (from Newts to Newton)

More Decks by Kirk Pepperdine

Other Decks in Programming

Featured

Transcript