Unleash the Power of Open Source Java Profilers

Profilers help to analyze performance bottlenecks of your application - if you know which to use and how to work with them.

There are many open-source profilers, like async-profiler or JMC. This talk will give you insights into these tools, focusing on:

- Understanding the basic concepts of profiling like flame graphs, ...
- Usage of async-profiler and JMC
- Advantages and disadvantages of the different tools

Slides for my JavaZone 2023 talk, similar to the slides for other conferences. You can find a recording at vimeo and more information on my profiling talks page.

Transcript

1. Unleash the Power of Open
   Source Java Profilers
   Johannes Bechberger
   The best
   JDKTM
   Krzysztof Ślusarski
   

   View Slide

2. View Slide

3. Client
   Admin
   Server
   Question?
   Set Question
   JSON
   

   View Slide

4. Premature Optimization
   We should forget about small efficiencies,
   say about 97% of the time: premature
   optimization is the root of all evil.
   Yet we should not pass up our
   opportunities in that critical 3%. A good
   programmer will not be lulled into
   complacency by such reasoning, he will be
   wise to look carefully at the critical code;
   but only after that code has been
   identified.
   — Donald Knuth
   “
   Source: Jacob Appelbaum, Wikipedia
   

   View Slide

5. So the event came...
   ... and the server was overwhelmed
   

   View Slide

6. Premature Optimization
   We should forget about small efficiencies,
   say about 97% of the time: premature
   optimization is the root of all evil.
   Yet we should not pass up our
   opportunities in that critical 3%.
   A good programmer will not be lulled into
   complacency by such reasoning, he will be
   wise to look carefully at the critical code;
   but only after that code has been
   identified.
   — Donald Knuth
   “
   Source: Jacob Appelbaum, Wikipedia
   

   View Slide

7. Profilers to the rescue
   

   View Slide

8. View Slide

9. Source: ARD Degeto/WistingSesong/Viaplay
   

   View Slide

10. main
    serverLoop
    handleQuestionRequest
    currentQuestion isQuestionEnabled
    fun main() {
    ...
    serverLoop()
    }
    fun serverLoop() {
    while (true) {
    req = ...
    if (req.isQuestionRequest)
    handleQuestionRequest(req)
    ...
    }
    }
    fun handleQuestionRequest(req) {
    if (isQuestionEnabled()) {
    emit(currentQuestion().json)
    } else {
    emit({})
    }
    }
    parseJSON parseJSON
    

    View Slide

11. View Slide

12. main
    serverLoop
    handleQuestionRequest
    fun main() {
    ...
    serverLoop()
    }
    fun serverLoop() {
    while (true) {
    req = ...
    if (req.isQuestionRequest)
    handleQuestionRequest(req)
    ...
    }
    }
    fun handleQuestionRequest(req) {
    if (isQuestionEnabled()) {
    emit(currentQuestion().json)
    } else {
    emit({})
    }
    }
    

    View Slide

13. Debugging
    Testing
    Profiling
    Toolbox
    

    View Slide

14. View Slide

15. View Slide

16. Who of you used a
    profiler?
    

    View Slide

17. yes
    no
    JetBrains 2022 survey
    

    View Slide

18. View Slide

19. What is profiling?
    A report on the amounts of time spent in
    each routine of a program, used to find
    and tune away the hot spots in it.
    — The Jargon File
    “
    

    View Slide

20. Different Profilers
    

    View Slide

21. Instrumenting Profilers Inserting instructions into the code automatically
    fun serverLoop() {
    logEntry(“serverLoop”)
    while (true) {
    req = ...
    if (req.isQuestionRequest)
    handleQuestionRequest(req)
    ...
    }
    logExit(“serverLoop”)
    }
    Profiled Code
    ≠
    Production Code
    long time = time();
    println(“serverLoop: “ + (time() – start))
    on class load
    

    View Slide

22. Sampling Profilers
    fun serverLoop() {
    while (true) {
    req = ...
    if (req.isQuestionRequest)
    handleQuestionRequest(req)
    ...
    }
    }
    Profiled Code
    =
    Production Code
    

    View Slide

23. Profilers aren’t
    rocket science
    

    View Slide

24. So let’s write our own
    https://mostlynerdless.de
    

    View Slide

25. Main
    agentmain(args)
    premain(args)
    Profiler
    sample
    sleep
    Store
    start store
    

    View Slide

26. Flamegraphs would be great
    main
    serverLoop
    handleQuestionRequest
    currentQuestion isQuestionEnabled
    parseJSON parseJSON
    

    View Slide

27. Turning traces into flamegraphs
    parseJSON
    currentQuestion
    handleQR
    serverLoop
    main
    private static class Node {
    final String method;
    final Map children;
    long samples = 0;
    void addTrace(List trace)
    }
    

    View Slide

28. Turning traces into flamegraphs
    parseJSON
    currentQuestion
    handleQR
    serverLoop
    main main
    serverLoop
    handleQR
    currentQuestion
    parseJSON
    1
    1
    1
    1
    1
    

    View Slide

29. Flamegraphs would be great
    main
    serverLoop
    handleQuestionRequest
    currentQuestion isQuestionEnabled
    parseJSON parseJSON
    

    View Slide

30. Turning traces into flamegraphs
    parseJSON
    currentQuestion
    handleQR
    serverLoop
    main main
    serverLoop
    handleQR
    currentQuestion
    parseJSON
    2
    2
    2
    2
    2
    

    View Slide

31. Flamegraphs would be great
    main
    serverLoop
    handleQuestionRequest
    currentQuestion isQuestionEnabled
    parseJSON parseJSON
    

    View Slide

32. Turning traces into flamegraphs
    currentQuestion
    handleQR
    serverLoop
    main main
    serverLoop
    handleQR
    currentQuestion
    parseJSON
    3
    3
    3
    3
    2
    

    View Slide

33. Flamegraphs would be great
    main
    serverLoop
    handleQuestionRequest
    currentQuestion isQuestionEnabled
    parseJSON parseJSON
    

    View Slide

34. Turning traces into flamegraphs
    isQuestionEnabled
    handleQR
    serverLoop
    main main
    serverLoop
    handleQR
    currentQuestion
    parseJSON
    4
    4
    4
    3
    2
    parseJSON
    isQuestionEnabled
    parseJSON
    1
    1
    

    View Slide

35. Turning traces into flamegraphs
    main
    serverLoop
    handleQR
    currentQuestion
    parseJSON
    4
    4
    4
    3
    2
    isQuestionEnabled
    parseJSON
    1
    1
    main
    serverLoop
    handleQuestionRequest
    currentQuestion isQEnabled
    parseJSON
    parseJSON
    

    View Slide

36. With the power of d3-flame-graph
    

    View Slide

37. Reality
    

    View Slide

38. Java/JDK
    Mission Control
    Application Performance
    Monitors, ...
    Sampling Profiler
    External
    VisualVM
    Netbeans
    async-profiler
    Sync Async
    Forte Analyzer
    Built-In
    Java/JDK Flight Recorder
    2016
    1991
    ASGCT 2002
    2012
    2018 Open Source
    2010
    

    View Slide

39. Don’t trust them
    

    View Slide

40. View Slide

41. AsyncGetCallTrace
    

    View Slide

42. Tests in the OpenJDK?
    

    View Slide

43. View Slide

44. That’s my work
    

    View Slide

45. Take profiles with a
    grain of salt
    

    View Slide

46. Profiling Loop
    Model Hypothesis Evaluation
    

    View Slide

47. Obtaining a profile
    

    View Slide

48. Async-Profiler
    java \
    -agentpath:libasyncProfiler.so=start,\
    event=cpu,\
    file=flame.html,flamegraph \
    arguments
    java –jar ap-loader.jar profiler PID ...
    Download from GitHub
    file=file.jfr,jfr,jfrsync,alloc \
    

    View Slide

49. Some of its features
    • Many events can trigger sampling
    • locks
    • perf-events like cache-misses
    • methods (via instrumentation)
    • Embeddable
    • via ap-loader
    • Hackable
    Async-profiler - manual by use cases
    by Krzysztof Ślusarski
    

    View Slide

50. JDK Flight Recorder (JFR)
    java \
    -XX:+UnlockDiagnosticVMOptions \
    -XX:+DebugNonSafepoints \
    -XX:+FlightRecorder \
    -XX:StartFlightRecording=filename=file.jfr \
    arguments
    jcmd PID JFR.start
    jcmd PID JFR.dump filename=file.jfr
    jcmd PID JFR.stop
    Already included in your JDK 8+
    

    View Slide

51. https://sapmachine.io/jfrevents
    

    View Slide

52. Fibonacci Server
    maas.com/fib/{n}
    

    View Slide

53. Custom JFR Events
    class SessionEvent extends jdk.jfr.Event {
    int sessionId;
    int n;
    ....
    }
    var event = new SessionEvent(sessionId, n);
    event.begin();
    ctx.result("fibonacci: " + fib(n));
    event.commit();
    

    View Slide

54. View Slide

55. View Slide

56. Inspecting a profile
    

    View Slide

57. JDK Mission Control
    

    View Slide

58. View Slide

59. View Slide

60. View Slide

61. View Slide

62. JFR Plugin for IntelliJ Still a prototype
    

    View Slide

63. View Slide

64. View Slide

65. View Slide

66. View Slide

67. View Slide

68. Impact on performance
    in some benchmark, 48s, 8 cores, 4% standard deviation
    Overhead
    JFR (reduced setting) 0 - 5%, typically < 2%
    JFR (profiling setting) 1 - 8%, typically < 5%
    async-profiler 3 - 6%, typically < 2%
    async-profiler with jfrsync 3 - 10%
    

    View Slide

69. Continuous Profiling
    

    View Slide

70. asprof -e wall \
    --loop 1m -f profile-%t.jfr
    

    View Slide

71. Pyroscope
    

    View Slide

72. View Slide

73. View Slide

74. Parca
    

    View Slide

75. View Slide

76. ... and many more
    

    View Slide

77. Debugging
    Testing
    Profiling
    Toolbox
    

    View Slide

78. @parttimen3rd on Twitter
    parttimenerd on GitHub
    mostlynerdless.de
    @SweetSapMachine
    sapmachine.io
    

    View Slide