Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Continuous Performance Regression Testing with JfrUnit (P99Conf)

Continuous Performance Regression Testing with JfrUnit (P99Conf)

Functional unit and integration tests are a common practice to detect and prevent regressions within a software component or application's behavior.

Things look different, though, when it comes to performance-related aspects: how to identify an application is slower than it used to be? How to spot higher memory consumption than before? How to find out about sub-optimal SQL queries that sneaked in? Any performance tests based on metrics like wall-clock time or through-put are not portable. They are subject to a specific execution environment such as a developer laptop, CI, or production-like environment.

Welcome JfrUnit: based on the JDK Flight Recorder (JFR), it allows you to implement assertions based on all kinds of JFR events emitted by the JVM or your application. JfrUnit makes it very easy to identify potential performance issues by asserting metrics that may impact your application's performance, like an increased object allocation rate, retrieval of redundant data from the database, loading of unneeded classes, and much more.

Come and join us for this code-centric session to learn about:

* Using JDK Flight Recorder and JfrUnit for implementing performance regression tests
* Emitting JFR events from 3rd party libraries using JMC Agent
* Analyzing performance regressions in JDK Mission Control

Gunnar Morling

October 07, 2021
Tweet

More Decks by Gunnar Morling

Other Decks in Programming

Transcript

  1. Brought to you by
    Continuous Performance
    Regression Testing with JfrUnit
    Gunnar Morling
    Principal Software Engineer at

    View full-size slide

  2. Today’s Objective: Learn About...
    ■ Implementing performance regression tests with JDK Flight Recorder
    and JfrUnit
    ■ Analyzing performance regressions in JDK Mission Control
    ■ Emitting JFR events from 3rd party libraries

    View full-size slide

  3. Gunnar Morling
    Principal Software Engineer at Red Hat
    ■ Debezium
    ■ Quarkus
    ■ JfrUnit, ModiTect, kcctl
    ■ Spec Lead for Bean Validation 2.0
    ■ Java Champion
    ■ @gunnarmorling

    View full-size slide

  4. Challenges of Performance Tests
    ■ Tests based on throughput or latency are dependent on specifics of the
    environment
    ● Require production-like set-up
    ● Impacted by unrelated concurrent load (e.g. on CI server)

    View full-size slide

  5. JfrUnit — Assert Performance Impacting Metrics
    ■ Assert proxy metrics like memory allocation or IO
    ● Based on JDK Flight
    Recorder events
    ● Failures may indicate a
    performance regression
    ■ Plain unit tests
    ■ Analyse in JDK Mission Control

    View full-size slide

  6. Hello, JfrUnit!

    View full-size slide

  7. OpenJDK Flight Recorder
    ■ “Troubleshooting, monitoring, and profiling framework
    embedded into the JVM”
    ■ 150+ built-in event types; custom ones
    ■ Open-source since Java 11 (backported to 8)
    ■ Low overhead, meant for “always-on”
    ■ Accessible via recording files and streaming

    View full-size slide

  8. JFR Event Types
    https://bestsolution-at.github.io/jfr-doc/

    View full-size slide

  9. OpenJDK Mission Control
    ■ Control, visualize, and
    analyze JFR recordings
    ■ And more:
    ● JMX console
    ● heap dump analyzer
    ● etc.

    View full-size slide

  10. JfrUnit — Discussion and Limitations
    ■ Robust tests with very fast feedback cycle
    ● Independent from environment
    ● Proactive identification of issues
    ■ Metrics need solid understanding; e.g. increased allocation may or may not
    be a problem
    ■ Cannot identify all issues, e.g. locks showing up only under load
    ■ Won’t help with issues elsewhere, e.g. a bad query execution plan

    View full-size slide

  11. Take Aways
    ■ JfrUnit: a tool in the performance testing box
    ● Complement - no replacement - for other tests
    ■ JMC Agent: produce events from code not under your control
    ■ Move beyond Java 1.8 🚀

    View full-size slide

  12. Outlook — Keeping Track of Historic Values
    https://horreum.hyperfoil.io/

    View full-size slide

  13. Resources
    ■ JfrUnit
    https://github.com/moditect/jfrunit
    ■ Introduction to JfrUnit
    https://www.morling.dev/blog/towards-continuous-performance-regression-t
    esting/
    ■ Introduction to JMC Agent
    https://developers.redhat.com/blog/2020/10/29/collect-jdk-flight-recorder-ev
    ents-at-runtime-with-jmc-agent/
    ■ Profiling Java inside containers with ContainerJFR
    https://developers.redhat.com/devnation/tech-talks/containerjfr

    View full-size slide

  14. Brought to you by
    Gunnar Morling
    [email protected]
    @gunnarmorling

    View full-size slide