Continuous Performance Regression Testing with JfrUnit (JChampionsConf)

Functional unit and integration tests are a common practice to detect and prevent regressions within a software component or application’s behavior.

Things look different, though, when it comes to performance-related aspects: how to identify an application is slower than it used to be? How to spot higher memory consumption than before? How to find out about sub-optimal SQL queries that sneaked in? Any performance tests based on metrics like wall-clock time or through-put are not portable. They are subject to a specific execution environment such as a developer laptop, CI, or production-like environment.

Welcome JfrUnit: based on the JDK Flight Recorder (JFR), it allows you to implement assertions based on all kinds of JFR events emitted by the JVM or your application. JfrUnit makes it very easy to identify potential performance issues by asserting metrics that may impact your application’s performance, like an increased object allocation rate, retrieval of redundant data from the database, loading of unneeded classes, and much more. Come and join us for this code-centric session to learn about:

- Using JDK Flight Recorder and JfrUnit for implementing performance regression tests
- Emitting JFR events from 3rd party libraries using JMC Agent
- Analyzing performance regressions in JDK Mission Control

Recording at https://www.youtube.com/watch?v=OS7yZJQ2Q-0

Transcript

1. Continuous Performance Regression Testing With JfrUnit Gunnar Morling Software Engineer, Red Hat @gunnarmorling
2. #JfrUnit @gunnarmorling Today’s Objective: Learn about... Implementing performance regression tests with JDK Flight Recorder and JfrUnit Analyzing performance regressions in JDK Mission Control Emitting JFR events from 3rd party libraries 1 2 3
3. #JfrUnit @gunnarmorling Gunnar Morling • Open source software engineer at Red Hat ◦ Debezium ◦ Quarkus, Hibernate • Spec Lead for Bean Validation 2.0 • Java Champion • @gunnarmorling
4. #JfrUnit @gunnarmorling Challenges of Performance Tests • Tests based on throughput or latency are dependent on specifics of the environment ◦ Require production-like set-up ◦ Impacted by unrelated concurrent load (e.g. on CI server)
5. #JfrUnit @gunnarmorling • Assert proxy metrics like memory allocation or IO • Failures may indicate a performance regression • Plain unit tests • Analyse in JDK Mission Control JfrUnit — Performance Regression Tests “Unit Testing, for Performance”
6. #JfrUnit @gunnarmorling Hello, JfrUnit!
7. #JfrUnit @gunnarmorling Hello, JfrUnit!
8. #JfrUnit @gunnarmorling Hello, JfrUnit!
9. #JfrUnit @gunnarmorling Hello, JfrUnit!
10. #JfrUnit @gunnarmorling Hello, JfrUnit!
11. #JfrUnit @gunnarmorling Tools © James Petts https://flic.kr/p/LgibJR (CC BY-SA 2.0)
12. #JfrUnit @gunnarmorling • A JVM “troubleshooting, monitoring, and profiling framework” • 150+ built-in event types; custom ones • Open-source since Java 11 • Low overhead, meant for “always-on” OpenJDK Flight Recorder
13. #JfrUnit @gunnarmorling https://bestsolution-at.github.io/jfr-doc/ JFR Event Types
14. #JfrUnit @gunnarmorling JEP 349 • Processing JFR events in “real-time” • Use Cases ◦ Insight into live data ◦ Export to monitoring services ◦ Predictive analytics ◦ JfrUnit 😎 JFR Event Streaming
15. #JfrUnit @gunnarmorling API JFR Event Streaming
16. #JfrUnit @gunnarmorling • Control, visualize, and analyze JFR recordings • And more: ◦ JMX console ◦ heap dump analyzer ◦ etc. OpenJDK Mission Control
17. #JfrUnit @gunnarmorling Demo © Wall Boat https://flic.kr/p/Y6zkmX (Public Domain)
18. #JfrUnit @gunnarmorling JMC Agent © Fitz1187 https://flic.kr/p/6pGGe2 (CC BY-ND 2.0)
19. #JfrUnit @gunnarmorling • Emit JFR events from existing 3rd party code • Attaches to the JVM and instruments classes • Configured via XML or JMC Agent Plug-in JMC Agent
20. #JfrUnit @gunnarmorling Bonus: JFR Analytics © Steve Jurvetson https://flic.kr/p/Vv7m (CC BY 2.0)
21. #JfrUnit @gunnarmorling • An exploration for querying JFR using SQL ◦ Pull-based (e.g. for automated analyses) ◦ Streaming queries (e.g. for predictive analytics) • Work-in-progress at https://github.com/moditect/jfr-analytics JFR Analytics
22. #JfrUnit @gunnarmorling Wrap-Up © Maria Eklind https://flic.kr/p/251HmVw (CC BY-SA 2.0)
23. #JfrUnit @gunnarmorling • Robust tests with very fast feedback cycle ◦ Independent from environment ◦ Proactive identification of issues • Metrics need solid understanding; e.g. increased allocation may or may not be a problem • Cannot identify all issues, e.g. locks showing up only under load • Won’t help with issues elsewhere, e.g. a bad query execution plan JfrUnit — Discussion and Limitations
24. #JfrUnit @gunnarmorling • Asserting proxy metrics allows for reliable performance regression tests • JfrUnit: a tool in the performance testing box ◦ Complement - no replacement - for other tests • JMC Agent: produce events from code not under your control • Move beyond Java 1.8 Take Aways
25. #JfrUnit @gunnarmorling https://horreum.hyperfoil.io/ Outlook — Keeping Track of Historic Values
26. #JfrUnit @gunnarmorling • JfrUnit https://github.com/moditect/jfrunit • Introduction to JfrUnit https://www.morling.dev/blog/towards-continuous-performance-regression -testing/ • JVM performance monitoring with JMC Agent https://developers.redhat.com/articles/2021/11/16/jvm-performance-monito ring-jmc-agent • Profiling Java inside containers with ContainerJFR https://developers.redhat.com/devnation/tech-talks/containerjfr Resources
27. #JfrUnit @gunnarmorling Q&A gunnar@hibernate.org @gunnarmorling 📧