Tweet
Tweet

Slide 1

Slide 1 text

! ! ! Threads, Needles, Stacks, Heaps DevNexus, 2014 Ovidiu Dimulescu ! @odimulescu www.odimulescu.com

Slide 2

Slide 2 text

The opinions and views expressed in this talk are my own, and do not necessarily reflect the opinions or views of my employer.

Slide 3

Slide 3 text

• Landscape • Flags • Threads • OOME • Questions? Agenda

Slide 4

Slide 4 text

• Oracle HotSpot • OpenJDK HotSpot • Oracle JRockit * • IBM JVM • Azul Systems Zing JVM * • Waratek CloudVM * Landscape - JVMs

Slide 5

Slide 5 text

• Why is the application slow? • Why does the application have hiccups? • Why is CPU pegged? • Why are we running out of memory? • … • Why _______ ? Landscape - Pain points

Slide 6

Slide 6 text

Landscape - Challenges

Slide 7

Slide 7 text

App Servers Message Queues Caches Schedulers Workers Data Stores Clients

Slide 8

Slide 8 text

#### Assumption is the mother of all F***UPS! - Travis Dane

Slide 9

Slide 9 text

#### Measure Anything, Measure Everything ! “ If Engineering at Etsy has a religion, it’s the Church of Graphs. If it moves, we track it …” http://codeascraft.com/2011/02/15/measure-anything-measure-everything/

Slide 10

Slide 10 text

* Images Credit: Wikimedia Commons

Slide 11

Slide 11 text

• Complex apps architecture and environments • Insufficient historical data • Weak inter-systems correlation • Platforms disparity • No one size fits all • Folklore, outdated information • High barrier to entry for effective diagnostics Landscape - Challenges

Slide 12

Slide 12 text

The JVM

Slide 13

Slide 13 text

JPDA - Platform Debugger Architecture * http://docs.oracle.com/javase/6/docs/technotes/guides/jpda/architecture.html

Slide 14

Slide 14 text

JPDA - Platform Debugger Architecture * http://docs.oracle.com/javase/6/docs/technotes/guides/jpda/architecture.html

Slide 15

Slide 15 text

• Profiling, debugging, monitoring, thread analysis, coverage analysis, etc. JVM TI - JVM Tool Interface

Slide 16

Slide 16 text

• Profiling, debugging, monitoring, thread analysis, coverage analysis, etc. ! • Native Agents - Full » -agentlib:= » -agentpath:= » hprof, jdwp agents included JVM TI - JVM Tool Interface

Slide 17

Slide 17 text

• Profiling, debugging, monitoring, thread analysis, coverage analysis, etc. ! • Native Agents - Full » -agentlib:= » -agentpath:= » hprof, jdwp agents included ! • Java Agents - Instrumentation only (java.lang.instrument.*) » -javaagent:agent-name.jar » multiple supported, management-agent.jar provided JVM TI - JVM Tool Interface

Slide 18

Slide 18 text

JVM TI - JVM Tool Interface TI Agent I JVM TI Agent II event action event action TI Client Attach API load agent

Slide 19

Slide 19 text

JMX - Java Management Extensions MBeanServer Standard MBeans Dynamic MBeans MXBeans Connector Connector Adaptor Adaptor Probes Agent Clients Remote Management JMX Console Custom Console RMI HTTP Browser HTTP SNMP SNMP J V M

Slide 20

Slide 20 text

JMX - Java Management Extensions MBeanServer Standard MBeans Dynamic MBeans MXBeans Connector Connector Adaptor Adaptor Probes Agent Clients Remote Management JMX Console Custom Console RMI HTTP Browser HTTP SNMP SNMP J V M

Slide 21

Slide 21 text

JMX - Java Management Extensions http://docs.oracle.com/javase/7/docs/api/java/lang/management/package-summary.html

Slide 22

Slide 22 text

JMX - Java Management Extensions Explicit ! -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=1234 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=/path/to/pw/file -Dcom.sun.management.jmxremote.access.file=/path/to/access/file -Djava.rmi.server.hostname= ! Attach API ! Unless -XX:+DisableAttachMechanism

Slide 23

Slide 23 text

JMX - Java Management Extensions • De facto standard • Wide support • RMI not firewall friendly, dynamic ports • RMI connector heavy, Observer effect • MBeans need to behave

Slide 24

Slide 24 text

• jcmd (JDK 7+) • jinfo • jhat • jmap • jps • jstat / jstatd • jstack HotSpot - Command Line Tools http://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/tooldescr.html

Slide 25

Slide 25 text

• jhat • JConsole • VisualVM • MissionControl (JDK >= 7u40) HotSpot - Graphical Tools

Slide 26

Slide 26 text

Vendor Tools - Sample set

Slide 27

Slide 27 text

Heap Analysers ! » MAT » Heapster » HeapAudit Open Source Tools GC Visualization ! » VisualVM » GCViewer » GCHisto » gcviz JMX Clients ! » MC5J » VisualVM » jmxterm » CRaSH Code Libraries * ! » Coda Hale Metrics » Javalution » Trove » FastUtil

Slide 28

Slide 28 text

* Image Credit: Wikimedia Commons

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

-X are non-standard, not guaranteed to be supported on all VM implementations, and are subject to change without notice in subsequent releases of the JDK. ! $ java -X … -Xloggc:<file> log GC status to a file with time stamps -Xms set initial Java heap size -Xmx set maximum Java heap size -Xss set java thread stack size -Xprof output cpu profiling data … Flags

Slide 31

Slide 31 text

-XX are not stable and are subject to change without notice ! Boolean -XX:+ or -XX:- ! Numeric -XX:=[unit] Unit: ‘k’, ‘K’, ‘m’, ‘M’, ‘g’, ‘G’ - 32768 or 32k ! String -XX:= Flags

Slide 32

Slide 32 text

$ java -XX:+PrintFlagsInitial -version $ java -XX:+PrintFlagsFinal -version $ java -XX:+PrintCommandLineFlags -version ! • product - An officially supported, internal JVM option • rw - Dynamically writable • C1 - Client JIT Compiler • C2 - Server JIT Compiler • pd - Platform Dependent • lp64 - 64bit VM only • manageable - Externally defined and dynamically writable • diagnostic - VM debugging • experimental - Not officially supported ! -XX:+UnlockExperimentalVMOptions -XX:+UnlockDiagnosticVMOptions Flags

Slide 33

Slide 33 text

• ms • mx • PermSize • MaxPermSize • NewSize • MaxNewSize • NewRatio • UseCompressedOps • HeapDumpOnOutOfMemoryError • HeapDumpPath Flags - Memory

Slide 34

Slide 34 text

Flags - GC Logging • verbose:gc • loggc • PrintGCDetails • PrintGCTimeStamps • PrintGCDateStamps • UseGCLogFileRotation • NumberOfGCLogFiles=10 • GCLogFileSize=50M • PrintTenuringDistribution • PrintGCApplicationStoppedTime

Slide 35

Slide 35 text

Flags - GC • verbose:class • TraceClassLoading • TraceClassUnloading • TraceClassResolution • UseParallelGC • UseParallelOldGC • UseParNewGC • UseConcMarkSweepGC • GCTimeLimit • GCHeapFreeLimit

Slide 36

Slide 36 text

Legacy ! -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=host:port ! Modern ! -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=host:port ! Help ! $ java -agentlib:jdwp=help Flags - Debugging

Slide 37

Slide 37 text

Flags - Demo

Slide 38

Slide 38 text

* Image Credit: Wikimedia Commons

Slide 39

Slide 39 text

Threads Type Regular Daemon

Slide 40

Slide 40 text

Threads Type Regular Daemon Name Automatic Custom *

Slide 41

Slide 41 text

Threads Type Regular Daemon Name Automatic Custom * Parent Group Not Null*

Slide 42

Slide 42 text

Threads Type Regular Daemon Name Automatic Custom * Parent Group Not Null* Priority MIN - NORM - MAX

Slide 43

Slide 43 text

Threads Type Regular Daemon Name Automatic Custom * Parent Group Not Null* Priority MIN - NORM - MAX State Thread.State

Slide 44

Slide 44 text

Thread States NEW RUNNABLE TERMINATED BLOCKED WAITING TIMED_WAITING

Slide 45

Slide 45 text

Thread Dump $ jstack $ jcmd Thread.print (JDK 7+) ! Output to terminal ! $ kill -3 or kill -SIGQUIT ! Outputs to JVM’s stdout ! JMX ! VisualVM, JConsole, etc.

Slide 46

Slide 46 text

Thread Dump » Dump identifier ! 2014-02-15 15:05:42 Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.45-b08 mixed mode)

Slide 47

Slide 47 text

Thread Dump » Dump identifier ! » Application Threads !

Slide 48

Slide 48 text

Thread Dump "main" prio=5 tid=0x0000000101028800 nid=0x507 runnable [0x00007fff5fbfd000] java.lang.Thread.State: RUNNABLE at org.eclipse.swt.internal.cocoa.OS.objc_msgSend_bool(Native Method) at org.eclipse.swt.internal.cocoa.NSRunLoop.runMode(NSRunLoop.java:42) at org.eclipse.swt.widgets.Display.sleep(Display.java:4565) at org.eclipse.ui.application.WorkbenchAdvisor.eventLoopIdle(WorkbenchAdvisor.java:364) at org.eclipse.ui.internal.ide.application.IDEWorkbenchAdvisor.eventLoopIdle(IDEWorkbenchAdvisor.java:917) at org.eclipse.ui.internal.Workbench$3.eventLoopIdle(Workbench.java:459) at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1026) at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332) at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:916) at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:86) at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:585) at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332) at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:540) at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149) at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:124) at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:353) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:180) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:629) at org.eclipse.equinox.launcher.Main.basicRun(Main.java:584) at org.eclipse.equinox.launcher.Main.run(Main.java:1438)

Slide 49

Slide 49 text

Thread Dump » Dump identifier ! » Application Threads ! » VM Threads

Slide 50

Slide 50 text

Thread Dump "C2 CompilerThread1" daemon prio=5 tid=0x0033 nid=0x6703 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE ! "C2 CompilerThread0" daemon prio=5 tid=0x0011 nid=0x6503 waiting on condition [0x0000000000] java.lang.Thread.State: RUNNABLE ! "Signal Dispatcher" daemon prio=5 tid=0x0022 nid=0x6303 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE ! "Finalizer" daemon prio=5 tid=0x0044 nid=0x5f03 in Object.wait() [0x000000010b88b000] java.lang.Thread.State: WAITING (on object monitor) ! "Reference Handler" daemon prio=5 tid=0x0055 nid=0x5d03 in Object.wait() [0x000000010b788000] java.lang.Thread.State: WAITING (on object monitor) ! "VM Thread" prio=5 tid=0x0066 nid=0x5b03 runnable ! "GC task thread#0 (ParallelGC)" prio=5 tid=0x000000010102d000 nid=0x5303 runnable ! "GC task thread#1 (ParallelGC)" prio=5 tid=0x0000000101038800 nid=0x5503 runnable ! "GC task thread#2 (ParallelGC)" prio=5 tid=0x0000000101039000 nid=0x5703 runnable ! "GC task thread#3 (ParallelGC)" prio=5 tid=0x0000000101039800 nid=0x5903 runnable ! "VM Periodic Task Thread" prio=5 tid=0x0000000101058000 nid=0x6b03 waiting on condition In general the VM thread is in one of three states: waiting to execute a VM operation, synchronizing all threads in preparation for a VM operation, or executing a VM operation. If you suspect that a hang is a HotSpot VM bug rather than an application or class library deadlock, then pay special attention to the VM thread.

Slide 51

Slide 51 text

Thread Dump » Dump identifier ! » Application Threads ! » VM Threads ! » JNI References Count ! JNI global references: 759

Slide 52

Slide 52 text

Thread Dump » Dump identifier ! » Application Threads ! » VM Threads ! » JNI References Count ! » Heap Utilization Summary *

Slide 53

Slide 53 text

Thread Dump PSYoungGen total 75776K, used 69708K [0x07f55, 0x07fb48, 0x08) eden space 75264K, 92% used [0x07f550,0x07f98fb028,0x07f9e8) from space 512K, 18% used [0x07fa18,0x07fa1980,0x07fa20) to space 11264K, 0% used [0x07fa98,0x07fa98,0x07fb48) ! ParOldGen total 59392K, used 46432K [0x07e0, 0x07e3a0, 0x07f550) object space 59392K, 78% used [0x07e0,0x07e2d58270,0x07e3a0) ! PSPermGen total 71680K, used 47149K [0x07d0, 0x07d460, 0x07e0) object space 71680K, 65% used [0x07d0,0x07d2e0b6c8,0x07d460)

Slide 54

Slide 54 text

Thread Dump “Thread I" prio=5 tid=0x0100 nid=0x1ef runnable [0x01acff] java.lang.Thread.State: RUNNABLE ! “Thread 3" prio=5 tid=0x0300 nid=0x3dd in Object.wait() [0x0366] java.lang.Thread.State: TIMED_WAITING (on object monitor) ! “Thread 4" prio=5 tid=0x0400 nid=0x4ab in Object.wait() [0x0477] java.lang.Thread.State: WAITING (on object monitor) ! “Thread 5" prio=5 tid=0x0500 nid=0x5bd waiting for monitor entry [0x0523] java.lang.Thread.State: BLOCKED (on object monitor) ! “Thread 2" prio=5 tid=0x0200 nid=0x2ef waiting on condition [0x02b8] java.lang.Thread.State: TIMED_WAITING (sleeping) ! “Thread 6" prio=5 tid=0x0600 nid=0x6ef waiting on condition [0x06b8] java.lang.Thread.State: TIMED_WAITING (sleeping)

Slide 55

Slide 55 text

Threads - Demo

Slide 56

Slide 56 text

OOEM

Slide 57

Slide 57 text

Exception in thread "main" java.lang.OutOfMemoryError: OOEM - Flavors

Slide 58

Slide 58 text

• Java heap space OOEM - Flavors

Slide 59

Slide 59 text

• Java heap space • PermGen space OOEM - Flavors

Slide 60

Slide 60 text

• Java heap space • PermGen space • GC overhead limit exceeded OOEM - Flavors

Slide 61

Slide 61 text

• Java heap space • PermGen space • GC overhead limit exceeded • unable to create new native thread OOEM - Flavors

Slide 62

Slide 62 text

• Java heap space • PermGen space • GC overhead limit exceeded • unable to create new native thread • Requested array size exceeds VM limit • request bytes for . Out of swap space? • (Native method) • Direct buffer memory OOEM - Flavors

Slide 63

Slide 63 text

Guilty until proven innocent * Image Credit: http://www.sodahead.com/living/which-do-you-think-is-more-important-in-the-running-of-a-society-garbage-collector-or-surgeon/question-3564735/

Slide 64

Slide 64 text

• Sizing » Heap Areas (Eden, Old, PermGen, etc.) » Tenuring Threshold » Thread Stack ! • Collectors choice » Type (Parallel, ParallelOld, CMS, G1) » Options (# of threads, etc.) OOEM - JVM Configuration

Slide 65

Slide 65 text

• Containers » Classloaders leaks ! • Frameworks » Reflection » Dynamic proxies OOEM - Application Runtime

Slide 66

Slide 66 text

• Objects structure overhead ! • Collections waste » Fill ratio consideration » Never shrink » Up to 2 x expansion algorithms ! • Use of Finalizers, 2 x GC cycles OOEM - Application

Slide 67

Slide 67 text

Heap Demo

Slide 68

Slide 68 text

• JVM Fundamentals • Tools of the trade • Instrument the app • Telemetry • Culture Summary

Slide 69

Slide 69 text

Questions ? Ovidiu Dimulescu ! @odimulescu www.odimulescu.com

Slide 70

Slide 70 text

http://www.kdgregory.com/index.php?page=java.outOfMemory http://www.infoq.com/articles/Java_Garbage_Collection_Distilled http://java-is-the-new-c.blogspot.com/2013/07/tuning-and-benchmarking-java-7s-garbage.html http://blog.ragozin.info/2012/10/safepoints-in-hotspot-jvm.html https://code.google.com/p/java-allocation-instrumenter/ https://weblogs.java.net/blog/kcpeppe/archive/2013/12/11/case-study-jvm-hotspot-flags http://resources.ej-technologies.com/jprofiler/help/doc/indexRedirect.html?http&&&resources.ej-technologies.com/jprofiler/help/doc/helptopics/config/ background.html http://openjdk.java.net/groups/hotspot/docs/RuntimeOverview.html http://docs.oracle.com/javase/7/docs/platform/jvmti/jvmti.html http://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/toc.html http://docs.oracle.com/javase/7/docs/technotes/guides/rmi/faq.html http://olegz.wordpress.com/2009/03/23/jmx-connectivity-through-the-firewall/ http://www.uml-diagrams.org/examples/java-6-thread-state-machine-diagram-example.html http://www.javacodegeeks.com/2012/03/jvm-how-to-analyze-thread-dump.html http://www.javabeat.net/introduction-to-java-agents/ https://blog.heroku.com/archives/2013/12/3/end_monolithic_app http://openjdk.java.net/jeps/137 http://docs.oracle.com/javase/7/docs/technotes/guides/jfr/toc.html http://www.youtube.com/playlist?list=PLKCk3OyNwIzsEVDq6zErLW7HSkY7aqdeT Resources