Voyeurs in the JVM land

Transcript

1. Voyeurs in the
   JVM land
   

   View Slide

2. about me
   Jarek Pałka
   Allegro.tech, doing stu , back to coding, hell yeah!!!
   JDD, 4Developers and one more conference (still under
   development) where I serve as a dictator for life
   JVM, bytecode, parsers, graphs and other cool things (like ponies)
   owner at Symentis trainings,
   former chief architect, development manager, head of
   development, backend developer and performance guy
   

   View Slide

3. You are all invited!
   

   View Slide

4. agenda
   JDK with batteries included
   JVM logging and tracking
   Linux tools for curious
   other tools for weirdos
   

   View Slide

5. JDK with batteries included
   jps
   jmap
   jstack
   jstat
   jcmd
   

   View Slide

6. how it works
   JVM stores metrics in memory mapped les
   /tmp/hsperfdata_[username]/[pid]
   

   View Slide

7. test
   lsof +d /tmp/hsperfdata_jarek
   

   View Slide

8. jps
   lists all running JVM processes
   

   View Slide

9. jstack
   dumps stacks of all JVM threads (in a selected process)
   jstack -l [pid] # to include locks info
   

   View Slide

10. jmap
    prints heap information, histogram or dump heap content to a le
    

    View Slide

11. jmap -heap [pid] # to print heap usage
    jmap -histo [pid] # to print histogram
    jmap -dump: le=jvm.dump # to dump heap
    

    View Slide

12. jstat
    samples running JVM for selected metrics
    jstat -gc [pid] 1000
    

    View Slide

13. jcmd
    

    View Slide

14. one tool to rule them all,
    one stop shop for all commands available in JVM
    

    View Slide

15. let’s play with it
    jcmd [pid] help
    

    View Slide

16. JVM logging and tracking
    

    View Slide

17. JVM has tons of diagnostic
    options
    

    View Slide

18. garbage collection
    

    View Slide

19. jstat -gc [pid] [interval]
    or
    

    View Slide

20. -Xloggc:gc.log
    -XX:+PrintGCDetails
    -XX:+PrintGCDateStamps
    -XX:+PrintTenuringDistribution
    -XX:+PrintGCApplicationStoppedTime
    -XX:+PrintClassHistogramAfterFullGC
    -XX:+PrintClassHistogramBeforeFullGC
    -XX:+UseGCLogFileRotation
    -XX:NumberOfGCLogFiles=5
    -XX:GCLogFileSize=10M
    

    View Slide

21. safepoint
    

    View Slide

22. what?
    — Nitsan Wakart
    Imagine if you will a JVM full of mutator threads,
    all busy, sweating, mutating the heap. Some of
    them have shared mutable state. They’re
    mutating each others state, concurrently, like
    animals. Some stand in corners mutating their
    own state (go blind they will). Suddenly a neon sign
    ashes the word PINEAPPLES. One by one the
    mutators stop their rampant heap romping and
    wait, sweat dripping. When the last mutator stops,
    a bunch of elves come in, empty the ashtrays, ll
    up all the drinks, mop up the puddles, and quickly
    as they can they vanish back to the north pole. The
    sign is turned o and the threads go back to it
    

    View Slide

23. — Nitsan Wakart
    At a safepoint the mutator thread is at a known
    and well de ned point in it’s interaction with the
    heap. This means that all the references on the
    stack are mapped (at known locations) and the
    JVM can account for all of them. As long as the
    thread remains at a safepoint we can safely
    manipulate the heap + stack such that the thread’s
    view of the world remains consistent when it
    leaves the safepoint.
    

    View Slide

24. -XX:+PrintSafepointStatistics
    -XX:PrintSafepointStatisticsCount=1
    Debugging JVM safepoint pauses
    

    View Slide

25. just in time compilation
    

    View Slide

26. -XX:+UnlockDiagnosticVMOptions
    -XX:+PrintCompilation
    -XX:+PrintInlining
    -XX:+UnlockDiagnosticVMOptions
    -XX:+TraceClassLoading
    -XX:+LogCompilation
    -XX:LogFile=mylog le.log
    -XX:+PrintAssembly
    

    View Slide

27. TLAB
    

    View Slide

28. what?
    — Ross K
    A Thread Local Allocation Bu er (TLAB) is a region
    of Eden that is used for allocation by a single
    thread. It enables a thread to do object allocation
    using thread local top and limit pointers, which is
    faster than doing an atomic operation on a top
    pointer that is shared across threads. A thread
    acquires a TLAB at it’s rst object allocation after a
    GC scavenge. The size of the TLAB is computed via
    a somewhat complex process discribed below. The
    TLAB is released when it is full (or nearly so), or the
    next GC scavenge occurs. TLABs are allocated only
    in Eden, never from From-Space or the OldGen.
    

    View Slide

29. should I care?
    you want as much of allocations to happen in TLABs, period
    

    View Slide

30. -XX:+PrintTLAB
    The Real Thing
    

    View Slide

31. native memory tracking
    

    View Slide

32. Stackover ow
    Java process taking more memory than its max heap size
    

    View Slide

33. java -XX:NativeMemoryTracking=[o |summary|detail]
    jcmd [pid] VM.native_memory summary
    

    View Slide

34. a weapon of mass destruction
    

    View Slide

35. or pair made in heaven
    

    View Slide

36. FlightRecorder
    — Oracle Help Center
    Java Flight Recorder (JFR) is a tool for collecting
    diagnostic and pro ling data about a running Java
    application. It is integrated into the Java Virtual
    Machine (JVM) and causes almost no performance
    overhead, so it can be used even in heavily loaded
    production environments.
    

    View Slide

37. java -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -
    XX:StartFlightRecording=duration=60s, lename=myrecording.jfr
    

    View Slide

38. warning
    as of now, you can’t use it to analyze production systems
    

    View Slide

39. until JDK 10 comes out,
    this is o cial statement now
    

    View Slide

40. java -XX:+UnlockCommercialFeatures -XX:+FlightRecorder
    jcmd [pid] JFR.start name=recording
    jcmd [pid] JFR.start name=recording lename=recording.jfr
    

    View Slide

41. Java Mission Control
    

    View Slide

42. Linux tools for curious
    sysstat
    sysdig
    perf
    

    View Slide

43. sysstat
    pidstat -t -d -p [pid] 1 # IO usage per thread
    pidstat -t -w -p [pid] 1 # task switching per thread
    pidstat -r -p [pid] 1 # page faults per process
    

    View Slide

44. warning
    forget about strace, ptrace syscall is not what you want :)
    

    View Slide

45. tracing syscalls
    

    View Slide

46. sysdig
    sysdig prod.pid=[pid] -w [pid].scap # record events
    csysdig -r [pid].scap # analyze
    

    View Slide

47. perf
    perf record -p [pid] -o [pid].perf # record events
    perf report -i [pid].perf # analyze
    

    View Slide

48. tools for weirdos
    honest pro ler
    amegraphs
    

    View Slide

49. honest pro ler
    it uses uno cial JVM API call AsyncGetCallTrace
    as opposed to other pro lers which use JVMTI (JVM tool interface)
    

    View Slide

50. here goes long boring discussion about complexity of OpenJDK
    global safepoint mechanism
    

    View Slide

51. — Honest pro ler wiki
    It accurately pro les applications, avoiding an
    inherent bias towards places that have safepoints.
    It pro les applications with signi cantly lower
    overhead than traditional pro ling techniques,
    making it suitable for use in production.
    

    View Slide

52. The Pros and Cons of AGCT
    

    View Slide

53. java -agentpath:../honest-pro ler/liblagent.so=logPath=honest.logs Main
    

    View Slide

54. tools I didn’t mention
    GCviewer
    JITWatch
    PrintAssembly
    Solaris Studio
    Censum
    Memory Analyzer Tool
    

    View Slide

55. Q&A
    

    View Slide

56. links
    JVM Anatomy Park
    Nitsan’s blog
    Chris Newland blog, JITwatch author
    Marcus Hirt blog, all stu JMC
    System calls in the Linux kernel
    sysdig
    perf: Linux pro ling with performance counters
    

    View Slide

57. Java Microbenchmark
    Harness
    

    View Slide

58. — Wes Dyer
    Make it correct, make it clear, make it concise,
    make it fast. In that order.
    

    View Slide

59. — JMH wiki
    JMH is a Java harness for building, running, and
    analysing nano/micro/milli/macro benchmarks
    written in Java and other languages targetting the
    JVM.
    

    View Slide

60. mvn archetype:generate \
    -DinteractiveMode=false \
    -DarchetypeGroupId=org.openjdk.jmh \
    -DarchetypeArtifactId=jmh-java-benchmark-archetype \
    -DgroupId=org.sample \
    -DartifactId=test \
    -Dversion=1.0
    http://openjdk.java.net/projects/code-tools/jmh/
    

    View Slide

61. benchmarks
    these are public non-static methods annotated with @Benchmark
    import org.openjdk.jmh.annotations.Benchmark;
    public class CodeBenchmark {
    @Benchmark
    public void testMethod(){
    }
    }
    

    View Slide

62. managing state & life cycle
    more complex examples will need to work with some data (state),
    this is what for state objects are for
    

    View Slide

63. @State(Scope.Benchmark)
    public class CodeBenchmarkState{
    public final ArrayList list = new ArrayList<>();
    }
    

    View Slide

64. public class CodeBenchmark{
    @Benchmark
    public void testMethod(CodeBenchmarkState state){
    state.add(0);
    }
    }
    

    View Slide

65. note on scopes
    

    View Slide

66. Scope.Benchmark
    With benchmark scope, all instances of the same type will be shared
    across all worker threads
    

    View Slide

67. Scope.Group
    With group scope, all instances of the same type will be shared
    across all threads within the same group. Each thread group will be
    supplied with its own state object
    

    View Slide

68. Scope.Thread
    With thread scope, all instances of the same type are distinct, even if
    multiple state objects are injected in the same benchmark
    

    View Slide

69. lifecycle
    every state object can have @Setup and @TearDown xture methods
    

    View Slide

70. time for rst benchmark
    let’s compare iteration speed over primitive array, ArrayList and
    LinkedList
    

    View Slide

71. running benchmarks
    mvn package
    java -jar target/benchmark.jar
    

    View Slide

72. forks, warm ups and iterations
    by default JMH forks JVM for each run of benchmark,
    within each fork you have two phases
    warm up
    iteration
    number of repetitions of each phase can be controlled over
    command line
    

    View Slide

73. command line
    -f - number of forks
    -wi - number of warm ups
    -i - number of iterations
    

    View Slide

74. java -jar target/benchmark.jar -f 1 -i 5 -wi 5
    

    View Slide

75. parameterized tests
    JMH supports parameterized tests through @Param annotation
    Test parameters should be public non- nal elds on state objects
    they are injected right before call to setup xture methods
    

    View Slide

76. @State(Scope.Benchmark)
    public class CodeBenchmark {
    @Param{"0.1","0.2","0.5","0.75","1.0"}
    public float loadFactor;
    private Map map;
    @Setup
    public void setUp(){
    map = new HashMap<>(16,loadFactor);
    }
    }
    

    View Slide

77. controlling parameters
    you overwrite values of the parameters with command line options
    java -jar target/benchmarks.jar -p loadFactor=0.8,0.9
    

    View Slide

78. dead code
    

    View Slide

79. … and black holes
    

    View Slide

80. one of the dangers JMH tries to mitigate is dead code optimization
    from JIT,
    to avoid it, consume return values from functions with black holes
    @Benchmark
    public void testMethod(Blackhole blackhole){
    blackhole.consume(codeBenchmark());
    }
    

    View Slide

81. asymmetric tests
    

    View Slide

82. sometimes you want to benchmark your concurrent code,
    like performance of read and write paths
    this is where @Group and @GroupThreads come in
    

    View Slide

83. @State(Scope.Benchmark)
    public class CodeBenchmark {
    @Benchmark
    @Group("benchmarkGroup")
    @GroupThreads(1)
    public void testWrites() {
    }
    @Benchmark
    @Group("bechmarkGroup")
    @GroupThreads(1)
    public void testReads(Blackhole blackhole) {
    }
    }
    

    View Slide

84. time for third benchmark
    compare performance of various thread-safe counter
    implementations
    public class Counter {
    private long counter;
    public void inc() {
    ++counter;
    }
    public long counter() {
    return counter;
    }
    }
    

    View Slide

85. pro lers
    they can provide some insights into your code
    java -jar benchmark.jar -lprof
    java -jar benchmark.jar -prof hs_gc
    

    View Slide

86. reporters
    and last but not least, writing test results to les
    java -jar benchmark.jar -lr
    java -jar benchmark.jar -rf csv -rff results.csv
    

    View Slide

87. tips and tricks
    on laptops governors can trick you,
    it’s easy to control them on linux with cpufreq-set
    

    View Slide