Continuous Performance Regression Testing With JfrUnit

--- Presented at DevNation Tech Talks ---

Functional unit and integration tests are a common practice to detect and prevent regressions within a software component or application's behavior.

Things look different, though, when it comes to performance-related aspects: how to identify an application is slower than it used to be? How to spot higher memory consumption than before? How to find out about sub-optimal SQL queries that sneaked in? Any performance tests based on metrics like wall-clock time or through-put are not portable. They are subject to a specific execution environment such as a developer laptop, CI, or production-like environment.

Welcome JfrUnit: based on the JDK Flight Recorder (JFR), it allows you to implement assertions based on all kinds of JFR events emitted by the JVM or your application. JfrUnit makes it very easy to identify potential performance issues by asserting metrics that may impact your application's performance, like an increased object allocation rate, retrieval of redundant data from the database, loading of unneeded classes, and much more.

Come and join us for this code-centric session to learn about:

* Using JDK Flight Recorder and JfrUnit for implementing performance regression tests
* Emitting JFR events from 3rd party libraries using JMC Agent
* Analyzing performance regressions in JDK Mission Control

URL: https://developers.redhat.com/devnation/tech-talks/jfrunit

Transcript

1. Continuous Performance Regression
   Testing With JfrUnit
   Gunnar Morling
   Software Engineer, Red Hat
   @gunnarmorling
   

   View Slide

2. #JfrUnit @gunnarmorling
   Today’s Objective: Learn about...
   1. Implementing performance regression tests with JDK
   Flight Recorder and JfrUnit
   2. Analyzing performance regressions in JDK Mission Control
   3. Emitting JFR events from 3rd party libraries
   

   View Slide

3. #JfrUnit @gunnarmorling
   3
   Gunnar Morling
   ● Open source software engineer at Red Hat
   ○ Debezium
   ○ Quarkus
   ● Spec Lead for Bean Validation 2.0
   ● Java Champion
   ● @gunnarmorling
   

   View Slide

4. #JfrUnit @gunnarmorling
   Challenges of Performance Tests
   ● Tests based on throughput or latency are dependent on
   specifics of the environment
   ○ Require production-like set-up
   ○ Impacted by unrelated concurrent load (e.g. on CI server)
   

   View Slide

5. #JfrUnit @gunnarmorling
   ● Assert metrics like memory allocation, or IO
   ○ Based on JDK Flight
   Recorder events
   ○ Failures may indicate a
   performance regression
   ● Plain unit tests
   ● Analyse in JDK Mission Control
   JfrUnit — Assert Performance Impacting Metrics
   

   View Slide

6. #JfrUnit @gunnarmorling
   Tools
   © James Petts https://flic.kr/p/LgibJR (CC BY-SA 2.0)
   

   View Slide

7. #JfrUnit @gunnarmorling
   ● “Troubleshooting, monitoring, and profiling
   framework embedded into the JVM”
   ● 150+ built-in event types; custom ones
   ● Open-source since Java 11
   (backported to 8)
   ● Low overhead, meant for “always-on”
   ● Accessible via recording files
   and streaming
   OpenJDK Flight Recorder
   

   View Slide

8. #JfrUnit @gunnarmorling
   https://bestsolution-at.github.io/jfr-doc/
   JFR Event Types
   

   View Slide

9. #JfrUnit @gunnarmorling
   ● Control, visualize, and
   analyze JFR recordings
   ● And more:
   ○ JMX console
   ○ heap dump analyzer
   ○ etc.
   OpenJDK Mission Control
   

   View Slide

10. #JfrUnit @gunnarmorling
    Demo
    © Wall Boat https://flic.kr/p/Y6zkmX (Pubic Domain)
    

    View Slide

11. #JfrUnit @gunnarmorling
    Hello, JfrUnit!
    

    View Slide

12. #JfrUnit @gunnarmorling
    ● Robust tests with very fast feedback cycle
    ○ Independent from environment
    ○ Proactive identification of issues
    ● Metrics need solid understanding; e.g. increased allocation may
    or may not be a problem
    ● Cannot identify all issues, e.g. locks showing up only under load
    ● Won’t help with issues elsewhere, e.g. a bad query execution plan
    JfrUnit — Discussion and Limitations
    

    View Slide

13. #JfrUnit @gunnarmorling
    Wrap-Up
    © Maria Eklind https://flic.kr/p/251HmVw (CC BY-SA 2.0)
    

    View Slide

14. #JfrUnit @gunnarmorling
    ● JfrUnit: a tool in the performance testing box
    ○ Complement - no replacement - for other tests
    ● JMC Agent: produce events from code not under your control
    ● Move beyond Java 1.8 🚀
    Take Aways
    

    View Slide

15. #JfrUnit @gunnarmorling
    https://horreum.hyperfoil.io/
    Outlook — Keeping Track of Historic Values
    

    View Slide

16. #JfrUnit @gunnarmorling
    ● JfrUnit
    https://github.com/moditect/jfrunit
    ● Introduction to JfrUnit
    https://www.morling.dev/blog/towards-continuous-performance-regression-
    testing/
    ● Introduction to JMC Agent
    https://developers.redhat.com/blog/2020/10/29/collect-jdk-flight-recorder
    -events-at-runtime-with-jmc-agent/
    ● Profiling Java inside containers with ContainerJFR
    https://developers.redhat.com/devnation/tech-talks/containerjfr
    Resources
    

    View Slide

17. #JfrUnit @gunnarmorling
    Q&A
    gunnar@hibernate.org
    @gunnarmorling
    📧
    

    View Slide

18. View Slide