Upgrade to Pro — share decks privately, control downloads, hide ads and more …

FlameGraph: What you need to know to get started

FlameGraph: What you need to know to get started

These are the slides from my FlameGraph talk from Linuxfest Northwest 2014. I will be putting the youtube video up in the next month or so (it takes a while for me to edit).

27b1eaa5f4f04eda6b5947ac64056fcd?s=128

shawn-sterling

April 28, 2014
Tweet

More Decks by shawn-sterling

Other Decks in Technology

Transcript

  1. FlameGraph What you need to know to get started

  2. Who am I email: shawn@systemtemplar.org • web: www.systemtemplar.org • github:

    github.com/shawn-sterling • https://speakerdeck.com/shawnsterling member of --> <-- trust me, I have a huge beard aerodynamic --> <-- Sysadmin for 18 years, small start ups -> large enterprises <-- I am the guy (google: I wanna be the guy + sysadmin) <-- contributes to several open source projects including FlameGraph <-- Author of graphios (nagios plugin for graphite)
  3. What is my purpose? * you should totally watch this

    show (Rick and Morty).
  4. Audience pole

  5. None
  6. Example Stack Trace def a(): b() def b(): c() def

    c(): error() a() 1 • 2 • 3 • 4 • 5 • 6 • 7 • 8 • 9 • 10
  7. Traceback (most recent call last): File "tb.py", line 10, in

    <module> a() File "tb.py", line 2, in a b() File "tb.py", line 5, in b c() File "tb.py", line 8, in c error() NameError: global name 'error' is not defined Example Stack Trace def a(): b() def b(): c() def c(): error() a() 1 • 2 • 3 • 4 • 5 • 6 • 7 • 8 • 9 • 10
  8. Traceback (most recent call last): File "tb.py", line 10, in

    <module> a() File "tb.py", line 2, in a b() File "tb.py", line 5, in b c() File "tb.py", line 8, in c error() NameError: global name 'error' is not defined Example Stack Trace def a(): b() def b(): c() def c(): error() a() 1 • 2 • 3 • 4 • 5 • 6 • 7 • 8 • 9 • 10
  9. Actual Stack Trace *Java is a nightmare

  10. A single stack trace can take a long time to

    go through and figure out what is going on.
  11. FlameGraph lets you analyze MANY stack stack traces at once

  12. None
  13. None
  14. None
  15. The top edge is where cpu time is consumed

  16. • steal agentz's picture Agentzh made this awesome picture.

  17. None
  18. None
  19. None
  20. None
  21. • Click to add text

  22. None
  23. Full stack problems <-- Slow

  24. Full stack problems <-- Slow

  25. None
  26. None
  27. . • get the real flamegraph here!

  28. None
  29. None
  30. None
  31. Comparison of working system and broken system working broken

  32. Same side by side, consistent palette working broken

  33. Comparison mode with different palette option

  34. None
  35. None
  36. None
  37. • getting started Let's get started

  38. . rhel: yum install $package-debuginfo ie: yum install mysql-debuginfo debian/ubuntu:

    apt-get install $package-dbg ie: apt-get install varnish-dbg fedora/rhel7: debuginfo-install $package ie: debuginfo-install kernel
  39. None
  40. Gather Data

  41. Using DTrace # dtrace -x ustackframes=100 -n 'profile-97 /execname ==

    "mysqld" && arg1/ { @[ustack()] = count(); } tick-60s { exit(0); }' -o out.stacks # ./stackcollapse.pl out.stacks > out.folded # ./flamegraph.pl out.folded > out.svg
  42. Using Perf # perf record -a -g -F 99 sleep

    60 # perf script | ./stackcollapse-perf.pl > out.perf-folded # ./flamegraph.pl out.perf-folded > perf-kernel.svg
  43. Using System Tap # stap -s 32 -D MAXBACKTRACE=100 -D

    MAXSTRINGLEN=4096 -D MAXMAPENTRIES=10240 \ -D MAXACTION=10000 -D STP_OVERLOAD_THRESHOLD=5000000000 --all-modules \ -ve 'global s; probe timer.profile { s[backtrace()] <<< 1; } probe end { foreach (i in s+) { print_stack(i); printf("\t%d\n", @count(s[i])); } } probe timer.s(60) { exit(); }' \ > out.stap-stacks # ./stackcollapse-stap.pl out.stap-stacks > out.stap-folded # cat out.stap-folded | ./flamegraph.pl > stap-kernel.svg
  44. Using ktap # ktap -e 's = ptable(); profile-1ms {

    s[backtrace(12, -1)] <<< 1 } trace_end { for (k, v in pairs(s)) { print(k, count(v), "\n") } } tick-30s { exit(0) }' -o out.kstacks # sed 's/ //g' out.kstacks | stackcollapse.pl > out.kstacks.folded # ./flamegraph.pl out.kstacks.folded > out.kstacks.svg
  45. Using hprof (java) # java -agentlib:hprof=cpu=samples,depth=100,interval=1ms,lineno=y,thr ead=y,file=output.hprof[...] # hprof2flamegraph output.hprof

    > output-folded.txt # flamegraph.pl output-folded.txt > output.svg Also google 'spf4j'
  46. Using Xperf (Windows) C:>xperf_to_collapsedstacks.py name.etl starttime stoptime C:>flamegraph.pl output.txt >

    graph.svg * See http://randomascii.wordpress.com/2013/03/26/summarizing- xperf-cpu-usage-with-flame-graphs/
  47. None
  48. None
  49. get folding

  50. dtrace: ./stackcollapse.pl broken.dtrace > broken.folded perf: perf script -i broken.perf

    | ./stackcollapse-perf.pl > broken.folded stap: ./stackcollapse-stap.pl broken.stap > broken.folded ktap: sed 's/ //g' broken.ktap | stackcollapse.pl > broken.folded hprof: hprof2flamegraph broken.hprof > broken.folded get folding
  51. cat broken.folded | ./flamegraph.pl > broken.svg

  52. cat broken.folded | ./flamegraph.pl --hash > broken.svg

  53. cat broken.folded | ./flamegraph.pl --cp > broken.svg cat working.folded |

    ./flamegraph.pl --cp > working.svg
  54. cat working.folded | ./flamegraph.pl --cp > working.svg cat broken.folded |

    ./flamegraph.pl --cp --colors mem > broken.svg rm palette.map broken.svg
  55. what else ya got

  56. off cpu flamegraphs

  57. off cpu flamegraphs

  58. Script VM (Perl, Python, Ruby, etc) Userspace Kernel Debug Flow

  59. git clone https://github.com/agentzh/nginx-systemtap-toolkit.git

  60. check-debug-info

  61. # ./check-debug-info -p 16882 File /usr/bin/bash has no debug info

    embedded. File /usr/lib64/ld-2.16.so has no debug info embedded. File /usr/lib64/libc-2.16.so has no debug info embedded. File /usr/lib64/libdl-2.16.so has no debug info embedded. File /usr/lib64/libnss_files-2.16.so has no debug info embedded. # rpm -qf /usr/lib64/ld-2.16.so glibc-2.16-34.fc18.x86_64 # yum install glibc-debuginfo
  62. sample-bt

  63. ## Kernel calls # ./sample-bt -p 1784 -k -t 60

    > kern.st ## User calls # ./sample-bt -p 1784 -u -t 60 > user.st ## Kernal and User calls # ./sample-bt -p 1784 -k -u -t 60 > both.st
  64. ## Kernel calls # ./sample-bt -p 1784 -k -t 60

    > kern.st $ ./stackcollapse-stap.pl kern.st > kern.folded $ ./flamegraph.pl kern.folded > kern.svg
  65. ## User calls # ./sample-bt -p 1784 -u -t 60

    > user.st $ ./stackcollapse-stap.pl user.st > user.folded $ ./flamegraph.pl user.folded > user.svg
  66. ## Kernal and User calls # ./sample-bt -p 1784 -k

    -u -t 60 > both.st $ ./stackcollapse-stap.pl both.st > both.folded $ ./flamegraph.pl both.folded > both.svg
  67. ## Kernal and User calls # ./sample-bt -p 1784 -k

    -u -t 60 > both.st $ ./stackcollapse-stap.pl both.st > both.folded $ cat both.folded | grep -v execute_command | ./flamegraph.pl both.folded > nex.svg
  68. ## Debug flag to dump stap file: # ./sample-bt -p

    1784 -k -d > test.stap # stap -x 1794 test.stap # cat test.stap probe begin { warn(sprintf("Tracing %d (/usr/bin/bash) in kernel-space only...\n", target())) } global bts; probe timer.profile { if (pid() == target() && !user_mode()) { bts[backtrace()] <<< 1 } } probe end { nstacks = 0 foreach (bt in bts limit 1) { nstacks++ } if (nstacks == 0) { warn("No backtraces found. Quitting now...\n") } else { foreach (bt in bts- limit 1024) { print_stack(bt) printf("\t%d\n", @count(bts[bt])) } } } probe timer.s(5) { warn("Time's up. Quitting now...(it may take a while)\n") exit() }
  69. sample-bt-off-cpu

  70. sample-bt-off-cpu # ./sample-bt-off-cpu -k -u -p 12861 -t 10 >

    bash.st
  71. # ./sample-bt-off-cpu -k -u -p 12861 -t 10 > bash.st

    $ ./stackcollapse-stap.pl bash.st > bash.folded $ ./flamegraph.pl bash.folded > bash.svg
  72. sample-bt-vfs

  73. sample-bt-vfs # ./sample-bt-vfs -p 15206 -t 10 > vfs.st or

    # ./sample-bt-vfs -p 15206 -t 10 --latency > vfs.st
  74. None
  75. git clone https://github.com/agentzh/perl-systemtap-toolkit.git

  76. # ./pl-sample-bt -t 60 -p 754 > nginx.st $ ./stackcollapse-stap.pl

    nginx.st > nginx.folded $ ./flamegraph.pl nginx.folded > nginx.svg
  77. None
  78. None
  79. Google flamegraph to download

  80. email: shawn@systemtemplar.org • web: www.systemtemplar.org • github: github.com/shawn-sterling • https://speakerdeck.com/shawnsterling

    And thanks to Brendan Gregg for writing FlameGraph Also thanks to Yichun Zhang (agentzh) for his toolkits
  81. Questions? email: shawn@systemtemplar.org • web: www.systemtemplar.org • github: github.com/shawn-sterling •

    https://speakerdeck.com/shawnsterling
  82. Image Credits Most of these images were found in google's

    image search with “labeled for non commercial reuse” turned on. No money will be made from these slides if it remains under my control. If I have incorrectly used your image please let me know and I will remove it. Fire Breather http://commons.wikimedia.org/wiki/File:Fire_breathing_20060715_7005_collien.jpg Rick and Morty Screenshot http://video.adultswim.com/rick-and-morty/ Stack Trace http://www.flickr.com/photos/miltown77/327120002/ Failboat http://www.flickr.com/photos/jeffmcneill/4252968654/ Kcachegrind http://itarato.blogspot.ca/2013/01/drupal-and-symfony-with-xdebug-and.html Code path map http://talks.php.net/show/confoo10/10 Snowflake http://www.flickr.com/photos/chaoticmind75/10152925944/ Snow Scene http://www.flickr.com/photos/vesiaphotography/12544068844/ Hand drawn flamegraph http://agentzh.org/ Demo http://www.flickr.com/photos/democonference/3948252064/ Show me http://csd-berlin.de/blog/2013/05/29/show-me-glamour-is-back-19-30-uhr-im-friedrichstadt-palast/ Picard meme http://foolz.us Jackie Chan meme http://alltheragefaces.com/ CPU pins http://ocgold.com/blog/?p=7240 Willy Wonka meme http://memegenerator.net/ Cereal Guy meme http://alltheragefaces.com/face/cereal-guy-cereal-guy-spitting cpu picture http://uk.hardware.info/ Symbols http://wojtas19.deviantart.com/art/Transmutation-Circle-115246997 Fire Dancer http://www.flickr.com/photos/sunphotoaz/4487436356/ My little ponies http://ex0artefact.deviantart.com/art/Mlp-and-What-the-fuck-386595361 Whats the problem pony http://mylittlefacewhen.com/f/5170/ lighter fluid guy http://www.flickr.com/photos/silvermarquis/477978519/ telephone pole http://www.flickr.com/photos/strollers/90725657/ nuts http://www.flickr.com/photos/cifor/9944748885/ Tell me more http://ww.jduensing.com/ off hours waiting https://www.flickr.com/photos/tiptoe/5608253489/ Safe http://safemanitoba.com/sites/default/files/styles/node_display_image/public/safeproduction650x290.png paper phoenix http://www.flickr.com/photos/jon_tucker/2707620458/ do it live http://www.mixcrate.com/img/ugc/covers/1/7/176688_l.jpg?v=713201235 debug car https://plus.google.com/103443672885327262273/about drive like you stole it http://cdn.a1decals.com/wp-content/uploads/drive-it-like-you-stole-it-w-n-1024x1024.jpeg please wait https://www.flickr.com/photos/askpang/6773635892/ linux filesystems http://cs.jhu.edu/~razvanm/fs-expedition/tux3.html what else http://www.enjoy-your-car.com/31-what-else.html sample pic http://images.cdn.fotopedia.com/flickr-3465412768-hd.jpg try it http://www.flickr.com/photos/mag3737/5191418684/ question marks http://www.flickr.com/photos/oberazzi/318947873/ thank you http://www.flickr.com/photos/nateone/3768979925/