Upgrade to Pro — share decks privately, control downloads, hide ads and more …

FlameGraph: What you need to know to get started

FlameGraph: What you need to know to get started

These are the slides from my FlameGraph talk from Linuxfest Northwest 2014. I will be putting the youtube video up in the next month or so (it takes a while for me to edit).

shawn-sterling

April 28, 2014
Tweet

More Decks by shawn-sterling

Other Decks in Technology

Transcript

  1. Who am I email: [email protected] • web: www.systemtemplar.org • github:

    github.com/shawn-sterling • https://speakerdeck.com/shawnsterling member of --> <-- trust me, I have a huge beard aerodynamic --> <-- Sysadmin for 18 years, small start ups -> large enterprises <-- I am the guy (google: I wanna be the guy + sysadmin) <-- contributes to several open source projects including FlameGraph <-- Author of graphios (nagios plugin for graphite)
  2. Example Stack Trace def a(): b() def b(): c() def

    c(): error() a() 1 • 2 • 3 • 4 • 5 • 6 • 7 • 8 • 9 • 10
  3. Traceback (most recent call last): File "tb.py", line 10, in

    <module> a() File "tb.py", line 2, in a b() File "tb.py", line 5, in b c() File "tb.py", line 8, in c error() NameError: global name 'error' is not defined Example Stack Trace def a(): b() def b(): c() def c(): error() a() 1 • 2 • 3 • 4 • 5 • 6 • 7 • 8 • 9 • 10
  4. Traceback (most recent call last): File "tb.py", line 10, in

    <module> a() File "tb.py", line 2, in a b() File "tb.py", line 5, in b c() File "tb.py", line 8, in c error() NameError: global name 'error' is not defined Example Stack Trace def a(): b() def b(): c() def c(): error() a() 1 • 2 • 3 • 4 • 5 • 6 • 7 • 8 • 9 • 10
  5. A single stack trace can take a long time to

    go through and figure out what is going on.
  6. . rhel: yum install $package-debuginfo ie: yum install mysql-debuginfo debian/ubuntu:

    apt-get install $package-dbg ie: apt-get install varnish-dbg fedora/rhel7: debuginfo-install $package ie: debuginfo-install kernel
  7. Using DTrace # dtrace -x ustackframes=100 -n 'profile-97 /execname ==

    "mysqld" && arg1/ { @[ustack()] = count(); } tick-60s { exit(0); }' -o out.stacks # ./stackcollapse.pl out.stacks > out.folded # ./flamegraph.pl out.folded > out.svg
  8. Using Perf # perf record -a -g -F 99 sleep

    60 # perf script | ./stackcollapse-perf.pl > out.perf-folded # ./flamegraph.pl out.perf-folded > perf-kernel.svg
  9. Using System Tap # stap -s 32 -D MAXBACKTRACE=100 -D

    MAXSTRINGLEN=4096 -D MAXMAPENTRIES=10240 \ -D MAXACTION=10000 -D STP_OVERLOAD_THRESHOLD=5000000000 --all-modules \ -ve 'global s; probe timer.profile { s[backtrace()] <<< 1; } probe end { foreach (i in s+) { print_stack(i); printf("\t%d\n", @count(s[i])); } } probe timer.s(60) { exit(); }' \ > out.stap-stacks # ./stackcollapse-stap.pl out.stap-stacks > out.stap-folded # cat out.stap-folded | ./flamegraph.pl > stap-kernel.svg
  10. Using ktap # ktap -e 's = ptable(); profile-1ms {

    s[backtrace(12, -1)] <<< 1 } trace_end { for (k, v in pairs(s)) { print(k, count(v), "\n") } } tick-30s { exit(0) }' -o out.kstacks # sed 's/ //g' out.kstacks | stackcollapse.pl > out.kstacks.folded # ./flamegraph.pl out.kstacks.folded > out.kstacks.svg
  11. Using Xperf (Windows) C:>xperf_to_collapsedstacks.py name.etl starttime stoptime C:>flamegraph.pl output.txt >

    graph.svg * See http://randomascii.wordpress.com/2013/03/26/summarizing- xperf-cpu-usage-with-flame-graphs/
  12. dtrace: ./stackcollapse.pl broken.dtrace > broken.folded perf: perf script -i broken.perf

    | ./stackcollapse-perf.pl > broken.folded stap: ./stackcollapse-stap.pl broken.stap > broken.folded ktap: sed 's/ //g' broken.ktap | stackcollapse.pl > broken.folded hprof: hprof2flamegraph broken.hprof > broken.folded get folding
  13. cat working.folded | ./flamegraph.pl --cp > working.svg cat broken.folded |

    ./flamegraph.pl --cp --colors mem > broken.svg rm palette.map broken.svg
  14. # ./check-debug-info -p 16882 File /usr/bin/bash has no debug info

    embedded. File /usr/lib64/ld-2.16.so has no debug info embedded. File /usr/lib64/libc-2.16.so has no debug info embedded. File /usr/lib64/libdl-2.16.so has no debug info embedded. File /usr/lib64/libnss_files-2.16.so has no debug info embedded. # rpm -qf /usr/lib64/ld-2.16.so glibc-2.16-34.fc18.x86_64 # yum install glibc-debuginfo
  15. ## Kernel calls # ./sample-bt -p 1784 -k -t 60

    > kern.st ## User calls # ./sample-bt -p 1784 -u -t 60 > user.st ## Kernal and User calls # ./sample-bt -p 1784 -k -u -t 60 > both.st
  16. ## Kernel calls # ./sample-bt -p 1784 -k -t 60

    > kern.st $ ./stackcollapse-stap.pl kern.st > kern.folded $ ./flamegraph.pl kern.folded > kern.svg
  17. ## User calls # ./sample-bt -p 1784 -u -t 60

    > user.st $ ./stackcollapse-stap.pl user.st > user.folded $ ./flamegraph.pl user.folded > user.svg
  18. ## Kernal and User calls # ./sample-bt -p 1784 -k

    -u -t 60 > both.st $ ./stackcollapse-stap.pl both.st > both.folded $ ./flamegraph.pl both.folded > both.svg
  19. ## Kernal and User calls # ./sample-bt -p 1784 -k

    -u -t 60 > both.st $ ./stackcollapse-stap.pl both.st > both.folded $ cat both.folded | grep -v execute_command | ./flamegraph.pl both.folded > nex.svg
  20. ## Debug flag to dump stap file: # ./sample-bt -p

    1784 -k -d > test.stap # stap -x 1794 test.stap # cat test.stap probe begin { warn(sprintf("Tracing %d (/usr/bin/bash) in kernel-space only...\n", target())) } global bts; probe timer.profile { if (pid() == target() && !user_mode()) { bts[backtrace()] <<< 1 } } probe end { nstacks = 0 foreach (bt in bts limit 1) { nstacks++ } if (nstacks == 0) { warn("No backtraces found. Quitting now...\n") } else { foreach (bt in bts- limit 1024) { print_stack(bt) printf("\t%d\n", @count(bts[bt])) } } } probe timer.s(5) { warn("Time's up. Quitting now...(it may take a while)\n") exit() }
  21. # ./sample-bt-off-cpu -k -u -p 12861 -t 10 > bash.st

    $ ./stackcollapse-stap.pl bash.st > bash.folded $ ./flamegraph.pl bash.folded > bash.svg
  22. sample-bt-vfs # ./sample-bt-vfs -p 15206 -t 10 > vfs.st or

    # ./sample-bt-vfs -p 15206 -t 10 --latency > vfs.st
  23. # ./pl-sample-bt -t 60 -p 754 > nginx.st $ ./stackcollapse-stap.pl

    nginx.st > nginx.folded $ ./flamegraph.pl nginx.folded > nginx.svg
  24. email: [email protected] • web: www.systemtemplar.org • github: github.com/shawn-sterling • https://speakerdeck.com/shawnsterling

    And thanks to Brendan Gregg for writing FlameGraph Also thanks to Yichun Zhang (agentzh) for his toolkits
  25. Image Credits Most of these images were found in google's

    image search with “labeled for non commercial reuse” turned on. No money will be made from these slides if it remains under my control. If I have incorrectly used your image please let me know and I will remove it. Fire Breather http://commons.wikimedia.org/wiki/File:Fire_breathing_20060715_7005_collien.jpg Rick and Morty Screenshot http://video.adultswim.com/rick-and-morty/ Stack Trace http://www.flickr.com/photos/miltown77/327120002/ Failboat http://www.flickr.com/photos/jeffmcneill/4252968654/ Kcachegrind http://itarato.blogspot.ca/2013/01/drupal-and-symfony-with-xdebug-and.html Code path map http://talks.php.net/show/confoo10/10 Snowflake http://www.flickr.com/photos/chaoticmind75/10152925944/ Snow Scene http://www.flickr.com/photos/vesiaphotography/12544068844/ Hand drawn flamegraph http://agentzh.org/ Demo http://www.flickr.com/photos/democonference/3948252064/ Show me http://csd-berlin.de/blog/2013/05/29/show-me-glamour-is-back-19-30-uhr-im-friedrichstadt-palast/ Picard meme http://foolz.us Jackie Chan meme http://alltheragefaces.com/ CPU pins http://ocgold.com/blog/?p=7240 Willy Wonka meme http://memegenerator.net/ Cereal Guy meme http://alltheragefaces.com/face/cereal-guy-cereal-guy-spitting cpu picture http://uk.hardware.info/ Symbols http://wojtas19.deviantart.com/art/Transmutation-Circle-115246997 Fire Dancer http://www.flickr.com/photos/sunphotoaz/4487436356/ My little ponies http://ex0artefact.deviantart.com/art/Mlp-and-What-the-fuck-386595361 Whats the problem pony http://mylittlefacewhen.com/f/5170/ lighter fluid guy http://www.flickr.com/photos/silvermarquis/477978519/ telephone pole http://www.flickr.com/photos/strollers/90725657/ nuts http://www.flickr.com/photos/cifor/9944748885/ Tell me more http://ww.jduensing.com/ off hours waiting https://www.flickr.com/photos/tiptoe/5608253489/ Safe http://safemanitoba.com/sites/default/files/styles/node_display_image/public/safeproduction650x290.png paper phoenix http://www.flickr.com/photos/jon_tucker/2707620458/ do it live http://www.mixcrate.com/img/ugc/covers/1/7/176688_l.jpg?v=713201235 debug car https://plus.google.com/103443672885327262273/about drive like you stole it http://cdn.a1decals.com/wp-content/uploads/drive-it-like-you-stole-it-w-n-1024x1024.jpeg please wait https://www.flickr.com/photos/askpang/6773635892/ linux filesystems http://cs.jhu.edu/~razvanm/fs-expedition/tux3.html what else http://www.enjoy-your-car.com/31-what-else.html sample pic http://images.cdn.fotopedia.com/flickr-3465412768-hd.jpg try it http://www.flickr.com/photos/mag3737/5191418684/ question marks http://www.flickr.com/photos/oberazzi/318947873/ thank you http://www.flickr.com/photos/nateone/3768979925/