Continuous Profiling mit Grafana Pyroscope

Transcript

2. • 🏡 Oldenburg • *1979 • Freelancer • michaelkotten.com • @michaelkotten
4. • Identify • performance bottlenecks • memory leaks • resource utilization patterns • Traditionally (non-continuous) • console.log debugging • Sample-based • Instrumentation-based • Profiling tools • JDK Mission Control • VisualVM • Other commercial tools
5. • Better performance • Less operational costs • Increased customer satisfaction • Faster incident resolution • More revenue!!!
6. 1. Collect profiling data 2. Save profiles 3. Visualize and query 4. Optimize code 5. Repeat
7. • Continuous Profiling Platform • Open Source • Built-in UI • Grafana integration • Pyroscope data source • Flamegraph panel • Multi-tenancy support
8. • Distributor • receives profiling data • divides data into batches • sends batches to ingester • Ingester • writes data to Object store • Compactor • increases query performance • reduces long-term storage usage
9. • Query-frontend • receives query • send to queue of query-scheduler • Querier • picks queries from query-scheduler • evaluates query expressions • reads from ingester and store-gateway • Store-gateway • lookup data from long-term storage
10. Setup
11. • Monolithic mode
12. • Monolithic mode • Microservices mode
13. • Monolithic mode • Microservices mode • Installation using • Docker version: '3.9' services: pyroscope: image: 'grafana/pyroscope:latest' ports: - 4040:4040
14. • Monolithic mode • Microservices mode • Installation using • Docker • K8s with helm (optionally in microservices mode) helm repo add grafana https://grafana.github.io/helm-charts helm repo update # install pyroscope in microservices mode helm upgrade --install -n pyroscope pyroscope grafana/pyroscope \ --values values-micro-services.yaml
16. • Supports Prometheus, OTEL, Grafana ecosystem (Loki, ..., Pyroscope) • Built-in UI • Scalable • Runs in different modes • Static mode • Static mode Kubernetes operator • Flow mode * • Uses async-profiler for Java * currently only flow mode for Pyroscope
17. • Supports Prometheus, OTEL, Grafana ecosystem (Loki, ..., Pyroscope) • Built-in UI • Scalable • Runs in different modes • Static mode • Static mode Kubernetes operator • Flow mode * • Uses async-profiler for Java • Deprecated, will be replaced by Grafana Alloy * currently only flow mode for Pyroscope
18. • Support for multiple languages
19. • Support for multiple languages • As Maven or Gradle artifact <dependencies> ... <dependency> <groupId>io.pyroscope</groupId> <artifactId>agent</artifactId> <version>0.13.1</version> </dependency> ... </dependencies>
20. • Support for multiple languages • As Maven or Gradle artifact PyroscopeAgent.start( new Config.Builder() .setApplicationName("demo-app-java") .setProfilingEvent(EventType.ITIMER) .setFormat(Format.JFR) .setServerAddress("http://localhost:4040") .build() );
21. • Support for multiple languages • As Maven or Gradle artifact • As javaagent export PYROSCOPE_APPLICATION_NAME=demo-app-java export PYROSCOPE_SERVER_ADDRESS=http://localhost:4040 export PYROSCOPE_FORMAT=jfr java -javaagent:pyroscope.jar -jar app.jar
22. • Filter data in the UI • Dynamic labels Pyroscope.LabelsWrapper.run(new LabelsSet("controller“, “slow_controller“), () -> { slowCode(); });
23. • Filter data in the UI • Dynamic labels • Static labels Pyroscope.setStaticLabels(Map.of("region", System.getenv("REGION"))); // or with Config.Builder if you start pyroscope with PyroscopeAgent.start PyroscopeAgent.start(new Config.Builder() .setLabels(Map.of("region", System.getenv("REGION"))) // ... .build() );
24. Flamegraphs
25. • Visualization of profiling data • Shows stacktrace with time spent • 100% on top node • width = time spent • narrower node = less time • call tree from top to bottom
26. • Consumed CPU time * supported in Java
27. • Consumed CPU time • Memory allocation • Types: • Alloc objects • Alloc space • amount + frequency * supported in Java
28. • Consumed CPU time • Memory allocation • Types: • Alloc objects • Alloc space • amount + frequency • Locks • Frequency • Duration * supported in Java
29. Code dive
30. • Traditional profiling • development and testing phase • detailed insights • under control of developer • Continuous profiling • production environment or extended performance tests • continuous view of system performance • less detailed due to minimized overhead
32. https://flamegraph.com
34. @michaelkotten https://michaelkotten.com consulting@michaelkotten.com