Devoxx FR 2016 Flamegraphs

Devoxx FR
Flamegraph, Java vous déclare sa flamme
21/04/2016
Nicolas Peters & Vincent Beretti

Transcript

1. #DevoxxFR
   Flamegraphs
   Java vous déclare sa flamme
   Nicolas Peters @nicolaspeters
   Vincent Beretti @vberetti
   1
   

   View Slide

2. #DevoxxFR
   Qui ?
   Nicolas Peters
   Lead Dev
   Canal Plus VOD
   @nicolaspeters
   Vincent Beretti
   Architecte pôle conseil
   @vberetti
   2
   

   View Slide

3. #DevoxxFR
   CPU 100%
   3
   

   View Slide

4. #DevoxxFR 4
   

   View Slide

5. #DevoxxFR
   Inspirations
   http://www.brendangregg.com/
   @brendangregg
   https://blog.virtual-void.net/
   @virtualvoid
   5
   

   View Slide

6. #DevoxxFR
   perf
   perf record
   perf script
   data
   stacks
   perf
   6
   

   View Slide

7. #DevoxxFR
   out.stacks
   7
   

   View Slide

8. #DevoxxFR
   out.stacks
   java 4502 2664.295070: 10101010 cpu-clock:
   7fe0a9d2b55b [unknown] (/tmp/perf-4067.map)
   7fe0a9d2b520 [unknown] (/tmp/perf-4067.map)
   7fe0a9ffc1545 [unknown] (/tmp/perf-4067.map)
   7fe0aa234a30 [unknown] (/tmp/perf-4067.map)
   7fe0aa26b68c [unknown] (/tmp/perf-4067.map)
   7fe0aa1cac1c [unknown] (/tmp/perf-4067.map)
   7fe0aa2cf09cc [unknown] (/tmp/perf-4067.map)
   7fe0aa29f70cc [unknown] (/tmp/perf-4067.map)
   7fe0aa1bf44cc [unknown] (/tmp/perf-4067.map)
   7fe0aa25a824 [unknown] (/tmp/perf-4067.map)
   7fe0aa15e374 [unknown] (/tmp/perf-4067.map)
   7fe0aa2538ac [unknown] (/tmp/perf-4067.map)
   7fe0aa275404 [unknown] (/tmp/perf-4067.map)
   7fe0aa27227c [unknown] (/tmp/perf-4067.map)
   7fe0aa16d3e4 [unknown] (/tmp/perf-4067.map)
   7fe0aa27d2cc [unknown] (/tmp/perf-4067.map)
   ….
   7fe0a9ef3da4 [unknown] (/tmp/perf-4067.map)
   7fe0a9ef1f9cc [unknown] (/tmp/perf-4067.map)
   7fe0a90004e7 call_stub (/tmp/perf-4067.map)
   63d6ea JavaCalls::call_helper (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so)
   63aba4 JavaCalls::call_virtual (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so)
   63b1b9 JavaCalls::call_virtual (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so)
   67e6d1 thread_entry (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so)
   9b40aa JavaThread::thread_main_inner (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so)
   869702 java_start (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so)
   697dc5 start_thread (/usr/lib64/libpthread-2.17.so)
   8
   ns
   tid
   

   View Slide

9. #DevoxxFR
   7fe0a9ffc1545 Lsun/reflect/GeneratedMethodAccessor65;::invoke
   fe0aa26b68c Lorg/springframework/web/method/support/InvocableHandlerMethod;::doInvoke
   perf + perf-agent
   perf record
   perf script
   data
   stacks
   perf-agent map
   perf
   java -XX:+PreserveFramePointer -jar myapp.jar
   Java > 1.8.0_66
   perf-map-agent
   9
   

   View Slide

10. #DevoxxFR
    java 4502 2664.295070: 10101010 cpu-clock:
    7fe0a9d2b520 Lsample/flamegraph/web/FiboController;::fibonacci (/tmp/perf-4067.map)
    7fe0a9ffc1545 Lsun/reflect/GeneratedMethodAccessor65;::invoke (/tmp/perf-4067.map)
    7fe0aa234a30 Ljava/lang/reflect/Method;::invoke (/tmp/perf-4067.map)
    7fe0aa26b68c Lorg/springframework/web/method/support/InvocableHandlerMethod;::doInvoke (/tmp/perf-4067.map)
    7fe0aa1cac1c Lorg/springframework/web/method/support/InvocableHandlerMethod;::invokeForRequest (/tmp/perf-4067.map)
    7fe0aa2cf09cc Lorg/springframework/web/servlet/mvc/method/annotation/ServletInvocableHandlerMethod;::invokeAndHandle (/tmp/perf-4067.map)
    7fe0aa29f70cc Lorg/springframework/web/servlet/mvc/method/annotation/RequestMappingHandlerAdapter;::invokeHandlerMethod (/tmp/perf-4067.map)
    7fe0aa1bf44cc Lorg/springframework/web/servlet/mvc/method/annotation/RequestMappingHandlerAdapter;::handleInternal (/tmp/perf-4067.map)
    7fe0aa25a824 Lorg/springframework/web/servlet/mvc/method/AbstractHandlerMethodAdapter;::handle (/tmp/perf-4067.map)
    7fe0aa15e374 Lorg/springframework/web/servlet/DispatcherServlet;::doDispatch (/tmp/perf-4067.map)
    7fe0aa2538ac Lorg/springframework/web/servlet/DispatcherServlet;::doService (/tmp/perf-4067.map)
    7fe0aa275404 Lorg/springframework/web/servlet/FrameworkServlet;::processRequest (/tmp/perf-4067.map)
    7fe0aa27227c Lorg/springframework/web/servlet/FrameworkServlet;::doGet (/tmp/perf-4067.map)
    7fe0aa16d3e4 Ljavax/servlet/http/HttpServlet;::service (/tmp/perf-4067.map)
    7fe0aa27d2cc Lorg/springframework/web/servlet/FrameworkServlet;::service (/tmp/perf-4067.map)
    ….
    7fe0a9ef3da4 Lorg/apache/tomcat/util/net/NioEndpoint$SocketProcessor;::doRun (/tmp/perf-4067.map)
    7fe0a9ef1f9cc Lorg/apache/tomcat/util/net/NioEndpoint$SocketProcessor;::run (/tmp/perf-4067.map)
    7fe0a90004e7 call_stub (/tmp/perf-4067.map)
    63d6ea JavaCalls::call_helper (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so)
    63aba4 JavaCalls::call_virtual (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so)
    63b1b9 JavaCalls::call_virtual (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so)
    67e6d1 thread_entry (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so)
    9b40aa JavaThread::thread_main_inner (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so)
    869702 java_start (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so)
    697dc5 start_thread (/usr/lib64/libpthread-2.17.so)
    symboles de l’application
    out.stacks avec perf.map
    10
    

    View Slide

11. #DevoxxFR
    perf-java-flames
    perf record
    perf script
    data
    stacks
    perf-agent map
    stack-collapsed
    flamegraph.pl
    perf
    java -XX:+PreserveFramePointer -jar myapp.jar
    Java > 1.8.0_66
    perf-map-agent
    flamegraph
    11
    

    View Slide

12. #DevoxxFR
    Installation & Lancement
    Installation
    yum install -y perf
    yum install -y java-1.8.0-openjdk-devel # > 1.8.0_66
    git clone https://github.com/brendangregg/FlameGraph /opt/FlameGraph
    git clone https://github.com/jrudolph/perf-map-agent /opt/perf-map-agent
    cmake . && make && bin/create-links-in /usr/bin
    Lancement
    java -XX:+PreserveFramePointer
    perf-java-flames
    12
    

    View Slide

13. #DevoxxFR
    Live Demo
    13
    

    View Slide

14. #DevoxxFR
    Fibonacci
    14
    

    View Slide

15. #DevoxxFR
    GC
    15
    

    View Slide

16. #DevoxxFR 16
    

    View Slide

17. #DevoxxFR
    Off - CPU
    17
    http://agentzh.org/misc/slides/off-cpu-flame-graphs.pdf
    Yichun Zhang (agentzh)
    

    View Slide

18. #DevoxxFR
    Off-CPU
    18
    http://fr.slideshare.net/brendangregg/linux-bpf-superpowers
    

    View Slide

19. #DevoxxFR
    Évènements
    19
    

    View Slide

20. #DevoxxFR
    ● Facile
    ● Rapide
    ● Peu d’impact
    ● Java + kernel Linux
    ● Vision processus interne de la JVM
    ● N’importe quel process
    ● Extensible : event + api
    ● Privilège root pour perf
    ● Documentation de perf
    Conclusion
    +
    20
    

    View Slide

21. #DevoxxFR
    Conclusion
    CPU == €
    21
    BPE/BCC
    

    View Slide

22. #DevoxxFR
    BCC
    http://fr.slideshare.net/brendangregg/linux-bpf-superpowers
    

    View Slide

23. #DevoxxFR
    Références
    ●
    Java Mixed-Mode Flame Graph
    http://www.brendangregg.com/blog/2015-11-06/java-mixed-
    mode-flame-graphs.html
    ●
    Saving 13 Million Computational Minutes per Day
    http://techblog.netflix.com/2016/04/saving-13-million-
    computational-minutes.html?m=1
    ●
    Le code de la presentation
    https://github.com/npeters/devoxx-flamegraph
    23
    

    View Slide

24. #DevoxxFR
    Questions ?
    24
    

    View Slide