$30 off During Our Annual Pro Sale. View Details »

Continuous Performance Regression Testing with JfrUnit (JChampionsConf)

Continuous Performance Regression Testing with JfrUnit (JChampionsConf)

Functional unit and integration tests are a common practice to detect and prevent regressions within a software component or application’s behavior.

Things look different, though, when it comes to performance-related aspects: how to identify an application is slower than it used to be? How to spot higher memory consumption than before? How to find out about sub-optimal SQL queries that sneaked in? Any performance tests based on metrics like wall-clock time or through-put are not portable. They are subject to a specific execution environment such as a developer laptop, CI, or production-like environment.

Welcome JfrUnit: based on the JDK Flight Recorder (JFR), it allows you to implement assertions based on all kinds of JFR events emitted by the JVM or your application. JfrUnit makes it very easy to identify potential performance issues by asserting metrics that may impact your application’s performance, like an increased object allocation rate, retrieval of redundant data from the database, loading of unneeded classes, and much more. Come and join us for this code-centric session to learn about:

- Using JDK Flight Recorder and JfrUnit for implementing performance regression tests
- Emitting JFR events from 3rd party libraries using JMC Agent
- Analyzing performance regressions in JDK Mission Control

Recording at https://www.youtube.com/watch?v=OS7yZJQ2Q-0

Gunnar Morling

January 24, 2022
Tweet

More Decks by Gunnar Morling

Other Decks in Programming

Transcript

  1. Continuous Performance Regression
    Testing With JfrUnit
    Gunnar Morling
    Software Engineer, Red Hat
    @gunnarmorling

    View Slide

  2. #JfrUnit @gunnarmorling
    Today’s Objective: Learn about...
    Implementing performance regression tests with
    JDK Flight Recorder and JfrUnit
    Analyzing performance regressions in JDK Mission
    Control
    Emitting JFR events from 3rd party libraries
    1
    2
    3

    View Slide

  3. #JfrUnit @gunnarmorling
    Gunnar Morling
    ● Open source software engineer at Red Hat
    ○ Debezium
    ○ Quarkus, Hibernate
    ● Spec Lead for Bean Validation 2.0
    ● Java Champion
    ● @gunnarmorling

    View Slide

  4. #JfrUnit @gunnarmorling
    Challenges of Performance Tests
    ● Tests based on throughput or latency are dependent on
    specifics of the environment
    ○ Require production-like set-up
    ○ Impacted by unrelated concurrent load (e.g. on CI server)

    View Slide

  5. #JfrUnit @gunnarmorling
    ● Assert proxy metrics like memory
    allocation or IO
    ● Failures may indicate a
    performance regression
    ● Plain unit tests
    ● Analyse in JDK Mission Control
    JfrUnit — Performance Regression Tests
    “Unit Testing, for Performance”

    View Slide

  6. #JfrUnit @gunnarmorling
    Hello, JfrUnit!

    View Slide

  7. #JfrUnit @gunnarmorling
    Hello, JfrUnit!

    View Slide

  8. #JfrUnit @gunnarmorling
    Hello, JfrUnit!

    View Slide

  9. #JfrUnit @gunnarmorling
    Hello, JfrUnit!

    View Slide

  10. #JfrUnit @gunnarmorling
    Hello, JfrUnit!

    View Slide

  11. #JfrUnit @gunnarmorling
    Tools
    © James Petts https://flic.kr/p/LgibJR (CC BY-SA 2.0)

    View Slide

  12. #JfrUnit @gunnarmorling
    ● A JVM “troubleshooting, monitoring,
    and profiling framework”
    ● 150+ built-in event types; custom ones
    ● Open-source since Java 11
    ● Low overhead, meant for “always-on”
    OpenJDK Flight Recorder

    View Slide

  13. #JfrUnit @gunnarmorling
    https://bestsolution-at.github.io/jfr-doc/
    JFR Event Types

    View Slide

  14. #JfrUnit @gunnarmorling
    JEP 349
    ● Processing JFR events in “real-time”
    ● Use Cases
    ○ Insight into live data
    ○ Export to monitoring services
    ○ Predictive analytics
    ○ JfrUnit 😎
    JFR Event Streaming

    View Slide

  15. #JfrUnit @gunnarmorling
    API
    JFR Event Streaming

    View Slide

  16. #JfrUnit @gunnarmorling
    ● Control, visualize, and
    analyze JFR recordings
    ● And more:
    ○ JMX console
    ○ heap dump analyzer
    ○ etc.
    OpenJDK Mission Control

    View Slide

  17. #JfrUnit @gunnarmorling
    Demo
    © Wall Boat https://flic.kr/p/Y6zkmX (Public Domain)

    View Slide

  18. #JfrUnit @gunnarmorling
    JMC Agent
    © Fitz1187 https://flic.kr/p/6pGGe2 (CC BY-ND 2.0)

    View Slide

  19. #JfrUnit @gunnarmorling
    ● Emit JFR events from
    existing 3rd party code
    ● Attaches to the JVM
    and instruments classes
    ● Configured via XML
    or JMC Agent Plug-in
    JMC Agent

    View Slide

  20. #JfrUnit @gunnarmorling
    Bonus: JFR Analytics
    © Steve Jurvetson https://flic.kr/p/Vv7m (CC BY 2.0)

    View Slide

  21. #JfrUnit @gunnarmorling
    ● An exploration for querying JFR using SQL
    ○ Pull-based (e.g. for automated analyses)
    ○ Streaming queries (e.g. for predictive
    analytics)
    ● Work-in-progress at
    https://github.com/moditect/jfr-analytics
    JFR Analytics

    View Slide

  22. #JfrUnit @gunnarmorling
    Wrap-Up
    © Maria Eklind https://flic.kr/p/251HmVw (CC BY-SA 2.0)

    View Slide

  23. #JfrUnit @gunnarmorling
    ● Robust tests with very fast feedback cycle
    ○ Independent from environment
    ○ Proactive identification of issues
    ● Metrics need solid understanding; e.g. increased allocation may
    or may not be a problem
    ● Cannot identify all issues, e.g. locks showing up only under load
    ● Won’t help with issues elsewhere, e.g. a bad query execution plan
    JfrUnit — Discussion and Limitations

    View Slide

  24. #JfrUnit @gunnarmorling
    ● Asserting proxy metrics allows for reliable performance
    regression tests
    ● JfrUnit: a tool in the performance testing box
    ○ Complement - no replacement - for other tests
    ● JMC Agent: produce events from code not
    under your control
    ● Move beyond Java 1.8
    Take Aways

    View Slide

  25. #JfrUnit @gunnarmorling
    https://horreum.hyperfoil.io/
    Outlook — Keeping Track of Historic Values

    View Slide

  26. #JfrUnit @gunnarmorling
    ● JfrUnit
    https://github.com/moditect/jfrunit
    ● Introduction to JfrUnit
    https://www.morling.dev/blog/towards-continuous-performance-regression
    -testing/
    ● JVM performance monitoring with JMC Agent
    https://developers.redhat.com/articles/2021/11/16/jvm-performance-monito
    ring-jmc-agent
    ● Profiling Java inside containers with ContainerJFR
    https://developers.redhat.com/devnation/tech-talks/containerjfr
    Resources

    View Slide

  27. #JfrUnit @gunnarmorling
    Q&A
    [email protected]
    @gunnarmorling
    📧

    View Slide

  28. View Slide