$30 off During Our Annual Pro Sale. View Details »

Unleash the Power of Open Source Java Profilers

Unleash the Power of Open Source Java Profilers

Profilers help to analyze performance bottlenecks of your application - if you know which to use and how to work with them.

There are many open-source profilers, like async-profiler or JMC. This talk will give you insights into these tools, focusing on:

- Understanding the basic concepts of profiling like flame graphs, ...
- Usage of async-profiler and JMC
- Advantages and disadvantages of the different tools

Slides for my JavaZone 2023 talk, similar to the slides for other conferences. You can find a recording at vimeo and more information on my profiling talks page.

Johannes Bechberger

September 15, 2023
Tweet

More Decks by Johannes Bechberger

Other Decks in Programming

Transcript

  1. Unleash the Power of Open
    Source Java Profilers
    Johannes Bechberger
    The best
    JDKTM
    Krzysztof Ślusarski

    View Slide

  2. View Slide

  3. Client
    Admin
    Server
    Question?
    Set Question
    JSON

    View Slide

  4. Premature Optimization
    We should forget about small efficiencies,
    say about 97% of the time: premature
    optimization is the root of all evil.
    Yet we should not pass up our
    opportunities in that critical 3%. A good
    programmer will not be lulled into
    complacency by such reasoning, he will be
    wise to look carefully at the critical code;
    but only after that code has been
    identified.
    — Donald Knuth

    Source: Jacob Appelbaum, Wikipedia

    View Slide

  5. So the event came...
    ... and the server was overwhelmed

    View Slide

  6. Premature Optimization
    We should forget about small efficiencies,
    say about 97% of the time: premature
    optimization is the root of all evil.
    Yet we should not pass up our
    opportunities in that critical 3%.
    A good programmer will not be lulled into
    complacency by such reasoning, he will be
    wise to look carefully at the critical code;
    but only after that code has been
    identified.
    — Donald Knuth

    Source: Jacob Appelbaum, Wikipedia

    View Slide

  7. Profilers to the rescue

    View Slide

  8. View Slide

  9. Source: ARD Degeto/WistingSesong/Viaplay

    View Slide

  10. main
    serverLoop
    handleQuestionRequest
    currentQuestion isQuestionEnabled
    fun main() {
    ...
    serverLoop()
    }
    fun serverLoop() {
    while (true) {
    req = ...
    if (req.isQuestionRequest)
    handleQuestionRequest(req)
    ...
    }
    }
    fun handleQuestionRequest(req) {
    if (isQuestionEnabled()) {
    emit(currentQuestion().json)
    } else {
    emit({})
    }
    }
    parseJSON parseJSON

    View Slide

  11. View Slide

  12. main
    serverLoop
    handleQuestionRequest
    fun main() {
    ...
    serverLoop()
    }
    fun serverLoop() {
    while (true) {
    req = ...
    if (req.isQuestionRequest)
    handleQuestionRequest(req)
    ...
    }
    }
    fun handleQuestionRequest(req) {
    if (isQuestionEnabled()) {
    emit(currentQuestion().json)
    } else {
    emit({})
    }
    }

    View Slide

  13. Debugging
    Testing
    Profiling
    Toolbox

    View Slide

  14. View Slide

  15. View Slide

  16. Who of you used a
    profiler?

    View Slide

  17. yes
    no
    JetBrains 2022 survey

    View Slide

  18. View Slide

  19. What is profiling?
    A report on the amounts of time spent in
    each routine of a program, used to find
    and tune away the hot spots in it.
    — The Jargon File

    View Slide

  20. Different Profilers

    View Slide

  21. Instrumenting Profilers Inserting instructions into the code automatically
    fun serverLoop() {
    logEntry(“serverLoop”)
    while (true) {
    req = ...
    if (req.isQuestionRequest)
    handleQuestionRequest(req)
    ...
    }
    logExit(“serverLoop”)
    }
    Profiled Code

    Production Code
    long time = time();
    println(“serverLoop: “ + (time() – start))
    on class load

    View Slide

  22. Sampling Profilers
    fun serverLoop() {
    while (true) {
    req = ...
    if (req.isQuestionRequest)
    handleQuestionRequest(req)
    ...
    }
    }
    Profiled Code
    =
    Production Code

    View Slide

  23. Profilers aren’t
    rocket science

    View Slide

  24. So let’s write our own
    https://mostlynerdless.de

    View Slide

  25. Main
    agentmain(args)
    premain(args)
    Profiler
    sample
    sleep
    Store
    start store

    View Slide

  26. Flamegraphs would be great
    main
    serverLoop
    handleQuestionRequest
    currentQuestion isQuestionEnabled
    parseJSON parseJSON

    View Slide

  27. Turning traces into flamegraphs
    parseJSON
    currentQuestion
    handleQR
    serverLoop
    main
    private static class Node {
    final String method;
    final Map children;
    long samples = 0;
    void addTrace(List trace)
    }

    View Slide

  28. Turning traces into flamegraphs
    parseJSON
    currentQuestion
    handleQR
    serverLoop
    main main
    serverLoop
    handleQR
    currentQuestion
    parseJSON
    1
    1
    1
    1
    1

    View Slide

  29. Flamegraphs would be great
    main
    serverLoop
    handleQuestionRequest
    currentQuestion isQuestionEnabled
    parseJSON parseJSON

    View Slide

  30. Turning traces into flamegraphs
    parseJSON
    currentQuestion
    handleQR
    serverLoop
    main main
    serverLoop
    handleQR
    currentQuestion
    parseJSON
    2
    2
    2
    2
    2

    View Slide

  31. Flamegraphs would be great
    main
    serverLoop
    handleQuestionRequest
    currentQuestion isQuestionEnabled
    parseJSON parseJSON

    View Slide

  32. Turning traces into flamegraphs
    currentQuestion
    handleQR
    serverLoop
    main main
    serverLoop
    handleQR
    currentQuestion
    parseJSON
    3
    3
    3
    3
    2

    View Slide

  33. Flamegraphs would be great
    main
    serverLoop
    handleQuestionRequest
    currentQuestion isQuestionEnabled
    parseJSON parseJSON

    View Slide

  34. Turning traces into flamegraphs
    isQuestionEnabled
    handleQR
    serverLoop
    main main
    serverLoop
    handleQR
    currentQuestion
    parseJSON
    4
    4
    4
    3
    2
    parseJSON
    isQuestionEnabled
    parseJSON
    1
    1

    View Slide

  35. Turning traces into flamegraphs
    main
    serverLoop
    handleQR
    currentQuestion
    parseJSON
    4
    4
    4
    3
    2
    isQuestionEnabled
    parseJSON
    1
    1
    main
    serverLoop
    handleQuestionRequest
    currentQuestion isQEnabled
    parseJSON
    parseJSON

    View Slide

  36. With the power of d3-flame-graph

    View Slide

  37. Reality

    View Slide

  38. Java/JDK
    Mission Control
    Application Performance
    Monitors, ...
    Sampling Profiler
    External
    VisualVM
    Netbeans
    async-profiler
    Sync Async
    Forte Analyzer
    Built-In
    Java/JDK Flight Recorder
    2016
    1991
    ASGCT 2002
    2012
    2018 Open Source
    2010

    View Slide

  39. Don’t trust them

    View Slide

  40. View Slide

  41. AsyncGetCallTrace

    View Slide

  42. Tests in the OpenJDK?

    View Slide

  43. 1

    View Slide

  44. That’s my work

    View Slide

  45. Take profiles with a
    grain of salt

    View Slide

  46. Profiling Loop
    Model Hypothesis Evaluation

    View Slide

  47. Obtaining a profile

    View Slide

  48. Async-Profiler
    java \
    -agentpath:libasyncProfiler.so=start,\
    event=cpu,\
    file=flame.html,flamegraph \
    arguments
    java –jar ap-loader.jar profiler PID ...
    Download from GitHub
    file=file.jfr,jfr,jfrsync,alloc \

    View Slide

  49. Some of its features
    • Many events can trigger sampling
    • locks
    • perf-events like cache-misses
    • methods (via instrumentation)
    • Embeddable
    • via ap-loader
    • Hackable
    Async-profiler - manual by use cases
    by Krzysztof Ślusarski

    View Slide

  50. JDK Flight Recorder (JFR)
    java \
    -XX:+UnlockDiagnosticVMOptions \
    -XX:+DebugNonSafepoints \
    -XX:+FlightRecorder \
    -XX:StartFlightRecording=filename=file.jfr \
    arguments
    jcmd PID JFR.start
    jcmd PID JFR.dump filename=file.jfr
    jcmd PID JFR.stop
    Already included in your JDK 8+

    View Slide

  51. https://sapmachine.io/jfrevents

    View Slide

  52. Fibonacci Server
    maas.com/fib/{n}

    View Slide

  53. Custom JFR Events
    class SessionEvent extends jdk.jfr.Event {
    int sessionId;
    int n;
    ....
    }
    var event = new SessionEvent(sessionId, n);
    event.begin();
    ctx.result("fibonacci: " + fib(n));
    event.commit();

    View Slide

  54. View Slide

  55. View Slide

  56. Inspecting a profile

    View Slide

  57. JDK Mission Control

    View Slide

  58. View Slide

  59. View Slide

  60. View Slide

  61. View Slide

  62. JFR Plugin for IntelliJ Still a prototype

    View Slide

  63. View Slide

  64. View Slide

  65. View Slide

  66. View Slide

  67. View Slide

  68. Impact on performance
    in some benchmark, 48s, 8 cores, 4% standard deviation
    Overhead
    JFR (reduced setting) 0 - 5%, typically < 2%
    JFR (profiling setting) 1 - 8%, typically < 5%
    async-profiler 3 - 6%, typically < 2%
    async-profiler with jfrsync 3 - 10%

    View Slide

  69. Continuous Profiling

    View Slide

  70. asprof -e wall \
    --loop 1m -f profile-%t.jfr

    View Slide

  71. Pyroscope

    View Slide

  72. View Slide

  73. View Slide

  74. Parca

    View Slide

  75. View Slide

  76. ... and many more

    View Slide

  77. Debugging
    Testing
    Profiling
    Toolbox

    View Slide

  78. @parttimen3rd on Twitter
    parttimenerd on GitHub
    mostlynerdless.de
    @SweetSapMachine
    sapmachine.io

    View Slide