Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Voyeurs in the JVM land

Voyeurs in the JVM land

Jaroslaw Palka

March 18, 2018
Tweet

More Decks by Jaroslaw Palka

Other Decks in Programming

Transcript

  1. Voyeurs in the
    JVM land

    View Slide

  2. about me
    Jarek Pałka
    Allegro.tech, doing stu , back to coding, hell yeah!!!
    JDD, 4Developers and one more conference (still under
    development) where I serve as a dictator for life
    JVM, bytecode, parsers, graphs and other cool things (like ponies)
    owner at Symentis trainings,
    former chief architect, development manager, head of
    development, backend developer and performance guy

    View Slide

  3. You are all invited!

    View Slide

  4. agenda
    JDK with batteries included
    JVM logging and tracking
    Linux tools for curious
    other tools for weirdos

    View Slide

  5. JDK with batteries included
    jps
    jmap
    jstack
    jstat
    jcmd

    View Slide

  6. how it works
    JVM stores metrics in memory mapped les
    /tmp/hsperfdata_[username]/[pid]

    View Slide

  7. test
    lsof +d /tmp/hsperfdata_jarek

    View Slide

  8. jps
    lists all running JVM processes

    View Slide

  9. jstack
    dumps stacks of all JVM threads (in a selected process)
    jstack -l [pid] # to include locks info

    View Slide

  10. jmap
    prints heap information, histogram or dump heap content to a le

    View Slide

  11. jmap -heap [pid] # to print heap usage
    jmap -histo [pid] # to print histogram
    jmap -dump: le=jvm.dump # to dump heap

    View Slide

  12. jstat
    samples running JVM for selected metrics
    jstat -gc [pid] 1000

    View Slide

  13. jcmd

    View Slide

  14. one tool to rule them all,
    one stop shop for all commands available in JVM

    View Slide

  15. let’s play with it
    jcmd [pid] help

    View Slide

  16. JVM logging and tracking

    View Slide

  17. JVM has tons of diagnostic
    options

    View Slide

  18. garbage collection

    View Slide

  19. jstat -gc [pid] [interval]
    or

    View Slide

  20. -Xloggc:gc.log
    -XX:+PrintGCDetails
    -XX:+PrintGCDateStamps
    -XX:+PrintTenuringDistribution
    -XX:+PrintGCApplicationStoppedTime
    -XX:+PrintClassHistogramAfterFullGC
    -XX:+PrintClassHistogramBeforeFullGC
    -XX:+UseGCLogFileRotation
    -XX:NumberOfGCLogFiles=5
    -XX:GCLogFileSize=10M

    View Slide

  21. safepoint

    View Slide

  22. what?
    — Nitsan Wakart
    Imagine if you will a JVM full of mutator threads,
    all busy, sweating, mutating the heap. Some of
    them have shared mutable state. They’re
    mutating each others state, concurrently, like
    animals. Some stand in corners mutating their
    own state (go blind they will). Suddenly a neon sign
    ashes the word PINEAPPLES. One by one the
    mutators stop their rampant heap romping and
    wait, sweat dripping. When the last mutator stops,
    a bunch of elves come in, empty the ashtrays, ll
    up all the drinks, mop up the puddles, and quickly
    as they can they vanish back to the north pole. The
    sign is turned o and the threads go back to it

    View Slide

  23. — Nitsan Wakart
    At a safepoint the mutator thread is at a known
    and well de ned point in it’s interaction with the
    heap. This means that all the references on the
    stack are mapped (at known locations) and the
    JVM can account for all of them. As long as the
    thread remains at a safepoint we can safely
    manipulate the heap + stack such that the thread’s
    view of the world remains consistent when it
    leaves the safepoint.

    View Slide

  24. -XX:+PrintSafepointStatistics
    -XX:PrintSafepointStatisticsCount=1
    Debugging JVM safepoint pauses

    View Slide

  25. just in time compilation

    View Slide

  26. -XX:+UnlockDiagnosticVMOptions
    -XX:+PrintCompilation
    -XX:+PrintInlining
    -XX:+UnlockDiagnosticVMOptions
    -XX:+TraceClassLoading
    -XX:+LogCompilation
    -XX:LogFile=mylog le.log
    -XX:+PrintAssembly

    View Slide

  27. TLAB

    View Slide

  28. what?
    — Ross K
    A Thread Local Allocation Bu er (TLAB) is a region
    of Eden that is used for allocation by a single
    thread. It enables a thread to do object allocation
    using thread local top and limit pointers, which is
    faster than doing an atomic operation on a top
    pointer that is shared across threads. A thread
    acquires a TLAB at it’s rst object allocation after a
    GC scavenge. The size of the TLAB is computed via
    a somewhat complex process discribed below. The
    TLAB is released when it is full (or nearly so), or the
    next GC scavenge occurs. TLABs are allocated only
    in Eden, never from From-Space or the OldGen.

    View Slide

  29. should I care?
    you want as much of allocations to happen in TLABs, period

    View Slide

  30. -XX:+PrintTLAB
    The Real Thing

    View Slide

  31. native memory tracking

    View Slide

  32. Stackover ow
    Java process taking more memory than its max heap size

    View Slide

  33. java -XX:NativeMemoryTracking=[o |summary|detail]
    jcmd [pid] VM.native_memory summary

    View Slide

  34. a weapon of mass destruction

    View Slide

  35. or pair made in heaven

    View Slide

  36. FlightRecorder
    — Oracle Help Center
    Java Flight Recorder (JFR) is a tool for collecting
    diagnostic and pro ling data about a running Java
    application. It is integrated into the Java Virtual
    Machine (JVM) and causes almost no performance
    overhead, so it can be used even in heavily loaded
    production environments.

    View Slide

  37. java -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -
    XX:StartFlightRecording=duration=60s, lename=myrecording.jfr

    View Slide

  38. warning
    as of now, you can’t use it to analyze production systems

    View Slide

  39. until JDK 10 comes out,
    this is o cial statement now

    View Slide

  40. java -XX:+UnlockCommercialFeatures -XX:+FlightRecorder
    jcmd [pid] JFR.start name=recording
    jcmd [pid] JFR.start name=recording lename=recording.jfr

    View Slide

  41. Java Mission Control

    View Slide

  42. Linux tools for curious
    sysstat
    sysdig
    perf

    View Slide

  43. sysstat
    pidstat -t -d -p [pid] 1 # IO usage per thread
    pidstat -t -w -p [pid] 1 # task switching per thread
    pidstat -r -p [pid] 1 # page faults per process

    View Slide

  44. warning
    forget about strace, ptrace syscall is not what you want :)

    View Slide

  45. tracing syscalls

    View Slide

  46. sysdig
    sysdig prod.pid=[pid] -w [pid].scap # record events
    csysdig -r [pid].scap # analyze

    View Slide

  47. perf
    perf record -p [pid] -o [pid].perf # record events
    perf report -i [pid].perf # analyze

    View Slide

  48. tools for weirdos
    honest pro ler
    amegraphs

    View Slide

  49. honest pro ler
    it uses uno cial JVM API call AsyncGetCallTrace
    as opposed to other pro lers which use JVMTI (JVM tool interface)

    View Slide

  50. here goes long boring discussion about complexity of OpenJDK
    global safepoint mechanism

    View Slide

  51. — Honest pro ler wiki
    It accurately pro les applications, avoiding an
    inherent bias towards places that have safepoints.
    It pro les applications with signi cantly lower
    overhead than traditional pro ling techniques,
    making it suitable for use in production.

    View Slide

  52. The Pros and Cons of AGCT

    View Slide

  53. java -agentpath:../honest-pro ler/liblagent.so=logPath=honest.logs Main

    View Slide

  54. tools I didn’t mention
    GCviewer
    JITWatch
    PrintAssembly
    Solaris Studio
    Censum
    Memory Analyzer Tool

    View Slide

  55. Q&A

    View Slide

  56. links
    JVM Anatomy Park
    Nitsan’s blog
    Chris Newland blog, JITwatch author
    Marcus Hirt blog, all stu JMC
    System calls in the Linux kernel
    sysdig
    perf: Linux pro ling with performance counters

    View Slide

  57. Java Microbenchmark
    Harness

    View Slide

  58. — Wes Dyer
    Make it correct, make it clear, make it concise,
    make it fast. In that order.

    View Slide

  59. — JMH wiki
    JMH is a Java harness for building, running, and
    analysing nano/micro/milli/macro benchmarks
    written in Java and other languages targetting the
    JVM.

    View Slide

  60. mvn archetype:generate \
    -DinteractiveMode=false \
    -DarchetypeGroupId=org.openjdk.jmh \
    -DarchetypeArtifactId=jmh-java-benchmark-archetype \
    -DgroupId=org.sample \
    -DartifactId=test \
    -Dversion=1.0
    http://openjdk.java.net/projects/code-tools/jmh/

    View Slide

  61. benchmarks
    these are public non-static methods annotated with @Benchmark
    import org.openjdk.jmh.annotations.Benchmark;
    public class CodeBenchmark {
    @Benchmark
    public void testMethod(){
    }
    }

    View Slide

  62. managing state & life cycle
    more complex examples will need to work with some data (state),
    this is what for state objects are for

    View Slide

  63. @State(Scope.Benchmark)
    public class CodeBenchmarkState{
    public final ArrayList list = new ArrayList<>();
    }

    View Slide

  64. public class CodeBenchmark{
    @Benchmark
    public void testMethod(CodeBenchmarkState state){
    state.add(0);
    }
    }

    View Slide

  65. note on scopes

    View Slide

  66. Scope.Benchmark
    With benchmark scope, all instances of the same type will be shared
    across all worker threads

    View Slide

  67. Scope.Group
    With group scope, all instances of the same type will be shared
    across all threads within the same group. Each thread group will be
    supplied with its own state object

    View Slide

  68. Scope.Thread
    With thread scope, all instances of the same type are distinct, even if
    multiple state objects are injected in the same benchmark

    View Slide

  69. lifecycle
    every state object can have @Setup and @TearDown xture methods

    View Slide

  70. time for rst benchmark
    let’s compare iteration speed over primitive array, ArrayList and
    LinkedList

    View Slide

  71. running benchmarks
    mvn package
    java -jar target/benchmark.jar

    View Slide

  72. forks, warm ups and iterations
    by default JMH forks JVM for each run of benchmark,
    within each fork you have two phases
    warm up
    iteration
    number of repetitions of each phase can be controlled over
    command line

    View Slide

  73. command line
    -f - number of forks
    -wi - number of warm ups
    -i - number of iterations

    View Slide

  74. java -jar target/benchmark.jar -f 1 -i 5 -wi 5

    View Slide

  75. parameterized tests
    JMH supports parameterized tests through @Param annotation
    Test parameters should be public non- nal elds on state objects
    they are injected right before call to setup xture methods

    View Slide

  76. @State(Scope.Benchmark)
    public class CodeBenchmark {
    @Param{"0.1","0.2","0.5","0.75","1.0"}
    public float loadFactor;
    private Map map;
    @Setup
    public void setUp(){
    map = new HashMap<>(16,loadFactor);
    }
    }

    View Slide

  77. controlling parameters
    you overwrite values of the parameters with command line options
    java -jar target/benchmarks.jar -p loadFactor=0.8,0.9

    View Slide

  78. dead code

    View Slide

  79. … and black holes

    View Slide

  80. one of the dangers JMH tries to mitigate is dead code optimization
    from JIT,
    to avoid it, consume return values from functions with black holes
    @Benchmark
    public void testMethod(Blackhole blackhole){
    blackhole.consume(codeBenchmark());
    }

    View Slide

  81. asymmetric tests

    View Slide

  82. sometimes you want to benchmark your concurrent code,
    like performance of read and write paths
    this is where @Group and @GroupThreads come in

    View Slide

  83. @State(Scope.Benchmark)
    public class CodeBenchmark {
    @Benchmark
    @Group("benchmarkGroup")
    @GroupThreads(1)
    public void testWrites() {
    }
    @Benchmark
    @Group("bechmarkGroup")
    @GroupThreads(1)
    public void testReads(Blackhole blackhole) {
    }
    }

    View Slide

  84. time for third benchmark
    compare performance of various thread-safe counter
    implementations
    public class Counter {
    private long counter;
    public void inc() {
    ++counter;
    }
    public long counter() {
    return counter;
    }
    }

    View Slide

  85. pro lers
    they can provide some insights into your code
    java -jar benchmark.jar -lprof
    java -jar benchmark.jar -prof hs_gc

    View Slide

  86. reporters
    and last but not least, writing test results to les
    java -jar benchmark.jar -lr
    java -jar benchmark.jar -rf csv -rff results.csv

    View Slide

  87. tips and tricks
    on laptops governors can trick you,
    it’s easy to control them on linux with cpufreq-set

    View Slide