Upgrade to Pro — share decks privately, control downloads, hide ads and more …

toward_systematization_of_linux_trace_tool

Takuma Kume
December 23, 2017
140

 toward_systematization_of_linux_trace_tool

Takuma Kume

December 23, 2017
Tweet

More Decks by Takuma Kume

Transcript

  1. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint ໨࣍ wഎܠɺ՝୊ɺલఏɺ໨త wτϨʔεख๏ͷ੔ཧ wτϨʔεख๏ͷ࢓૊Έ wࠓޙͷ՝୊
  2. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint എܠɾ՝୊ɾલఏ
  3. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint എܠɾ՝୊ wݪҼಛఆ͕Ͱ͖ͳ͍ߴෛՙ͕ൃੜ͢Δɻ wݪҼಛఆ͕Ͱ͖ͳ͍αʔόμ΢ϯɺϓϩηεμ΢ϯ͕ൃੜ͢Δɻ wݪҼ͕ಛఆͰ͖ͳ͍ϨΠςϯγʔͷஶ͍͠௿Լ͕ൃੜ͢Δɻ ฐ͕ࣾఏڙ͢ΔϗεςΟϯάαʔϏεͰ͓٬༷ʹି͠ग़͍ͯ͠Δ -JOVYͷαʔόͷӡ༻ɾ։ൃʹ͓͍ͯҎԼͷΑ͏ͳ՝୊͕͋Δɻ 
  4. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint લఏɿϗεςΟϯάαʔϏεͷఏڙํࣜ wϗεςΟϯάαʔϏεͰ͸γϯάϧߏ੒ͷ୆ͷαʔόΛෳ਺ਓ Ͱڞ༗ͯ͠͝ར༻͍ͨͩ͘ํࣜΛͱ͍ͬͯΔɻ w୆ͷαʔόΛΑΓଟ͘ͷਓʹ͝ར༻͍ͨͩ͘͜ͱͰɺརӹ͸େ ͖͘ͳΔɻ ୆ͷαʔό౰ͨΓͷӨڹ౓߹͍ɾίετޮ཰ͷ؍఺Ͱɺো֐ͷݪ Ҽಛఆ΍ଟछଟ༷ͷϓϩάϥϜΛޮ཰Ϋ࣮ߦͯ͠๯಄Ͱड़΂ͨ՝୊ Λղܾ͢Δ͜ͱ͸ඇৗʹॏཁͰ͋Δɻ 
  5. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint લఏɿϗεςΟϯάαʔϏεαʔόͷಛੑ w͓٬༷͸ݖݶͷൣғͰࣗ༝ʹϓϩάϥϜΛ࣮ߦͰ͖Δɻ wϓϩάϥϜͷҰ෦͸͞·͟·ͳෛՙΛੜΉɻ w$16 ϝϞϦ σΟεΫ*0 τϥϑΟοΫ ϓϩηεϒϩοΩϯά wͳͲʜ wݱ࣌఺ͰಛఆͰ͖ͳ͍ෛՙ΋͋Δɻ 
  6. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint લఏɿϗεςΟϯάαʔϏεαʔόͷػೳ sshd crond UserA UserB … UserA UserB … ϗεςΟϯάαʔϏεͷ8FCαʔό಺ͷओͳఏڙػೳ httpd UserA UserB … ͓٬༷͕ࣗ༝ʹૢ࡞Ͱ͖ΔՕॴ͕ෳ਺͋ΔͨΊ੾Γ෼͚͕ඞཁ 
  7. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint લఏɿϗεςΟϯάαʔϏεͷো֐ରԠ wUPQ΍QTίϚϯυʹΑͬͯɺͲͷϓϩηε͕ݪҼͱͳ͍ͬͯΔ ͔Λಛఆ͢Δɻ wTUSBDF΍MTPGίϚϯυʹΑͬͯɺγεςϜίʔϧ΍Φʔϓϯ͠ ͍ͯΔϑΝΠϧΛ֬ೝͯ͠ෛՙݪҼΛಛఆ͢Δɻ w͜ΕͰಛఆͰ͖Δ৔߹΋͋Δ͕ɺಛఆͰ͖ͳ͍৔߹͸औΓ׶͑ͣ ݪҼͰ͋ΔϓϩηεΛ࠶ىಈ࣏ͯ͠Δ͜ͱ΋ʜ ྫ͑͹ϗεςΟϯάαʔόͷ-"΍$16͕ߴ͍ঢ়ଶ 
  8. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint લఏɿϗεςΟϯάαʔϏεͷো֐ରࡦ αʔόͷো֐༧๷΍ݪҼಛఆͷͨΊͷ৘ใͱͯ͠ɺ$16ɾϝϞϦɾ ϩά΍αʔϏεϨεϙϯελΠϜͳͲͷ৘ใΛऩू͍ͯ͠Δɻ ˠ͔͠͠ɺ͜Ε͚ͩͰ͸ো֐ͷݪҼ͕ಛఆͰ͖ͳ͍͜ͱ͕͠͹͠ ͹ൃੜ͢Δɻ 
  9. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint લఏɿٻΊͨπʔϧ ͜͜·ͰͰɺαʔόͷো֐ରԠͰγεςϜίʔϧͷτϨʔεͳͲ͕ ༗ޮ׆༻Ͱ͖Δ͜ͱɺͦΕ͚ͩͰ͸଍Γͳ͍͜ͱ͕͕෼͔ͬͨɻ wҎԼͷπʔϧΛٻΊΔɻ wγεςϜίʔϧͷΈͳΒͣࣗ༝౓͕ߴ͘LFSOFMϨϕϧͰͷτ ϨʔεΛͰ͖Δɻ wτϨʔεʹࡍͯ͠Φʔόʔϔου͕খ͍͞ɻ 
  10. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint લఏɿ௿Φʔόʔϔουͷඞཁੑ wಈ࡞தͷϓϩμΫγϣϯ؀ڥͰ࣮ࢪ͍ͨͨ͠ΊɺύϑΥʔϚϯε ʹӨڹΛ༩͑ͨ͘ͳ͍ɻ w͍ͭো֐͕ൃੜ͢Δ͔෼͔Βͳ͍ɻՄೳͰ͋Ε͹ɺৗʹτϨʔε Λͯ͠৘ใΛऩू͍ͨ͠ɻ wܧଓͯ͠ΦʔόʔϔουΛ૿΍ͨ͘͠ͳ͍ɻ 
  11. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint લఏɿΦʔόʔϔουͷߴ͍τϨʔεख๏ TUSBDF  TUSBDF QSPDFTT UBSHFU QSPDFTT strace -p {target PID} ptrace(PTRACE_ATTACH, PID…) waitpid SFHJTUFS systemcall SIGTRAP ϓϩηεఀࢭ ղੳ ࠶։ … ϓϩηεऴྃ exit 1. 2. 3. 4. 5. 6. 
  12. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint લఏɿΦʔόʔϔουͷߴ͍τϨʔεख๏ TUSBDF  TUSBDF QSPDFTT UBSHFU QSPDFTT strace -p {target PID} ptrace(PTRACE_ATTACH, PID…) waitpid SFHJTUFS systemcall SIGTRAP ϓϩηεఀࢭ ղੳ ࠶։ … ϓϩηεऴྃ exit 1. 2. 3. 4. 5. 6. γεςϜίʔϧͷղੳதʹ ϓϩάϥϜ͕ॠؒతʹఀࢭ͢Δ 
  13. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint લఏɿΦʔόʔϔουͷ௿͍τϨʔεख๏ ref: http://www.brendangregg.com/linuxperf.html 
  14. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint ໨త 
  15. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint ໨త wݪҼෆ໌ͷো֐ʹ͍ͭͯɺ6TFSMFWFM ,FSOFMͷτϨʔε৘ใΛ ׆༻ͯ͠ղܾ͢ΔՄೳੑʹ͍ͭͯݕূ͢Δɻ wͦͷҝʹɺܧଓతʹτϨʔε৘ใΛऩूͯ͠ো֐࣌ʹ৘ใΛ׆༻ ͢Δ͜ͱͷ༗ޮੑΛݕূ͢Δɻ wαʔϏεͷΦϖϨʔγϣϯΤϯδχΞશһ͕-JOVYͷτϨʔεʹ ਫ਼௨͍ͯ͠Δ͜ͱ͸كͰ͋ΔɻͦͷͨΊɺଟ͘ͷΦϖϨʔγϣϯ ΤϯδχΞͰ΋-JOVYͷτϨʔε͕Ͱ͖ΔΑ͏ʹମܥԽ͢Δɻ 
  16. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint ࠓճͷൣғ 
  17. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint ࠓճͷൣғ wطଘͷ௿ΦʔόʔϔουͳτϨʔεπʔϧ͸ଟ͘ଘࡏ͍ͯ͠Δɻ ͦΕͧΕͷπʔϧ͕Ͳ͏͍͏ػೳ΍໾ׂ͕͋Γɺզʑ͕ԿΛબ୒ ͢Ε͹ྑ͍ͷ͔ΛҎԼͷ؍఺͔Βߟ͑ͨɻ wଟ͘ͷτϨʔεख๏ͷ໾ׂΛ੔ཧͯ͠શମ૾Λ௫Ή͜ͱɻ wͦΕͧΕͷπʔϧͷ࢓૊Έʹ͍ͭͯಛ௃Λཧղ͢Δ͜ͱɻ 
  18. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint τϨʔεख๏ͷ੔ཧ 
  19. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint 
  20. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint ࠞཚ͠·͢ʂʂ 
  21. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint ͭͷϨΠϠʹ෼ྨ wσʔλιʔεϨΠϠ wϋʔυ΢ΣΞɾιϑτ΢ΣΞͷΠϕϯτΛτϨʔε͢Δػߏ wϑϨʔϜϫʔΫϨΠϠ wσʔλιʔεϨΠϠʹΞΫηε͢ΔͨΊͷ൚༻తͳϑϨʔϜϫʔΫ wΞϓϦέʔγϣϯϨΠϠ 6TFSMFWFM  wϑϨʔϜϫʔΫϨΠϠʹର໋ͯ͠ྩΛग़ͨ͠Γɺऔಘͨ݁͠ՌΛू ܭɾදࣔͳͲΛ͢Δ 
  22. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint Πϕϯτͷ෼ྨ wτϨʔεͱ͸ɺ֤ॴ͔Βൃߦ͞ΕΔΠϕϯτΛτϨʔε͢Δͱ͍ ͏͜ͱɻ Software Event Hardware Event GPU Event Memory Event CPU Event Dynamic Event Static Event 
  23. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint τϨʔεπʔϧͷϨΠϠʹΑΔ෼ྨ bcc (eBPF scripts) systemtap perf-tool (ftrace scripts) perf eBPF kernel module (Inserted by tool) Ftrace (debugfs) perf_event uprobe Tracepoint kprobe PMU Static Event Dynamic Event CPU Event Software Event Hardware Event Πϕϯτ σʔλιʔεϨΠϠ ϑϨʔϜϫʔΫϨΠϠ ΞϓϦέʔγϣϯϨΠϠ user kernel 
  24. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint τϨʔεख๏ͷ࢓૊Έ 
  25. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint σʔλιʔεϨΠϠ w1.6 1FSGPSNBODF.POJUPSJOH6OJUT  w$16ͳͲͷϋʔυ΢ΣΞͷύϑΥʔϚϯεʹؔ͢ΔΠϕϯτ ʹΞΫηε͢ΔͨΊͷΠϯλʔϑΣΠε wLQSPCFVQSPCF wಈ࡞தͷιϑτ΢ΣΞʹΧελϜΠϕϯτΛ࡞੒͢Δɻ w,FSOFM͕ର৅ͳΒLQSPCF 6TBFMFWFM͕ର৅ͳΒVQSPCFΛ ར༻ͯ͠ɺιϑτ΢ΣΞͷΠϕϯτΛτϨʔε͢Δɻ w5SBDFQPJOU wLFSOFMʹ૊Έࠐ·Ε͍ͯΔ੩తͳτϨʔεϙΠϯτ 
  26. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint ϑϨʔϜϫʔΫϨΠϠ w'USBDF wEFCVHGTͱ͍͏ಛघͳϑΝΠϧγεςϜΛΠϯλʔϑΣΠε ʹ͍࣋ͬͯΔɻTZTLFSOFMEFCVHUSBDJOH഑ԼʹτϨʔ εͷ༗ޮԽ΍ϑΟϧλϦϯάͷͨΊͷϑΝΠϧɺτϨʔε݁Ռ Λग़ྗ͢ΔͨΊͷϑΝΠϧ͕ଘࡏ͍ͯ͠Δɻ wUSBDFQPJOU LQSPCF VQSPCFΛ࢖ͬͯτϨʔεΛߦ͏͜ͱ͕ Ͱ͖Δɻ w୯ҰϢʔβ SPPU ͷΈͰಈ࡞͢Δɻ wϓϩάϥϚϒϧͰ͸ͳ͍ɻ 
  27. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint ϑϨʔϜϫʔΫϨΠϠ wQFSG@FWFOUT wτϨʔεର৅ ϋʔυ΢ΣΞΠϕϯτ ιϑτ΢ΣΞΠϕϯτ  τϨʔεϙΠτ Λࢦఆͯ͠QFSG@FWFOU@PQFO  Λ࣮ߦ͠ ͯΠϕϯτΛొ࿥͢Δɻ w1.6 USBDFQPJOUΛ࢖ͬͯτϨʔεΛߦ͏͜ͱ͕Ͱ͖Δɻ wLQSPCFʹΑΔಈతτϨʔε͸-JOVYҎ͔߱ΒՄೳͰ CQG  Λར༻͢Δɻ 
  28. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint ϑϨʔϜϫʔΫϨΠϠ w F#1' w 1.6 USBDFQPJOU LQSPCF VQSPCFΛ࢖ͬͯτϨʔεΛߦ͏͜ͱ͕Ͱ͖Δɻ w F#1'͸6TFSMFWFMͰ࡞੒ͨ͠#ZUFDPEFΛCQG  Λ࢖ͬͯLFSOFMۭؒʹϩʔ υ͠ɺ࣮ߦ͢Δ͜ͱ͕Ͱ͖ΔɻF#1'ʹΑΔτϨʔεͰ͸ɺલड़ͨ͠σʔλιʔ εʹΠϕϯτΛొ࿥͢Δ͜ͱͰ࣮ݱͰ͖Δɻ w F#1'ͷಛ௃ͱͯ͠ɺ఺্͛Δɻ w LFSOFMۭؒʹಡΈࠐ·ΕΔ#ZUFDPEF͸F#1'ͷ7FSJpFSʹΑͬͯίʔυͷ ҆શੑ͕νΣοΫ͞Εෆਖ਼ͳϝϞϦɺελοΫ΁ͷΞΫηε΍ແݶϧʔϓ ͳͲΛݕ஌͢Δ͜ͱͰ҆શੑΛ֬อ͍ͯ͠Δɻ w LFSOFMۭؒͱ6TFSMFWFMͷؒͰσʔλͷ΍ΓऔΓΛ͢ΔͨΊʹF#1'ͷ.BQ Λ࡞੒͢Δ͜ͱ͕Ͱ͖ΔɻͦΕ͸)BTI "SSBZͳͲͷܗࣜΛ΋͍ͬͯΔɻ w Φʔόʔϔου͕ඇৗʹগͳ͍ɻ 
  29. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint bcc (eBPF scripts) systemtap perf-tool (ftrace scripts) perf eBPF kernel module (Inserted by tool) Ftrace (debugfs) perf_event uprobe Tracepoint kprobe PMU Static Event Dynamic Event CPU Event Software Event Hardware Event Πϕϯτ σʔλιʔεϨΠϠ ϑϨʔϜϫʔΫϨΠϠ ΞϓϦέʔγϣϯϨΠϠ user kernel eBPF kprobe Dynamic Event Software Event Πϕϯτ σʔλιʔεϨΠϠ ϑϨʔϜϫʔΫϨΠϠ ΞϓϦέʔγϣϯϨΠϠ user kernel 
  30. VTFS LFSOFM ϓϩάϥϜ #1'CZUFDPEF F#1' WFSJpFS ੜ੒ MPBE #1'CZUFDPEF NBQ

    ੜ੒ 6TFS,FSOFMؒͰσʔλͷ΍ΓऔΓΛ͢Δ )BTI "SSBZ FUDʜ LQSPCFT LQSPCFT F#1' 
  31. LQSPCFT F#1' VTFS LFSOFM ϓϩάϥϜ #1'CZUFDPEF F#1' WFSJpFS ੜ੒ MPBE

    #1'CZUFDPEF NBQ USBDJOH ॻ͖ࠐΈ ࢀর LQSPCFT 
  32. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint F#1'ͷΦʔόʔϔου͕௿͍ཧ༝ user kernel user kernel ैདྷख๏ eBPF ϓϩάϥϜ τϨʔαʔ ϑϨʔϜϫʔΫ σʔλιʔε ϑΟϧλ ूܭ දࣔ ϓϩάϥϜ τϨʔαˍूܭ ϑϨʔϜϫʔΫ σʔλιʔε දࣔ ࡞੒ ࡞੒ Map ऩू ॻࠐΈ 
  33. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint F#1'ͷΦʔόʔϔου͕௿͍ཧ༝ user kernel user kernel ैདྷख๏ eBPF ϓϩάϥϜ τϨʔαʔ ϑϨʔϜϫʔΫ σʔλιʔε ϑΟϧλ ूܭ දࣔ ϓϩάϥϜ τϨʔαˍूܭ ϑϨʔϜϫʔΫ σʔλιʔε දࣔ ࡞੒ ࡞੒ Map ऩू ॻࠐΈ F#1'͸ϓϩάϥϚϒϧͰ NBQʹΑΔσʔλͷڞ༗͕ՄೳͰ͋Δ τϨʔε݁ՌͷϑΟϧλ΍ूܭΛLFSOFM ଆͰ׬݁Ͱ͖Δɻ Φʔόʔϔου͕௿͍ʂ 
  34. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint ͍͞͝ʹ w-JOVYͷτϨʔεπʔϧ͸ɺσʔλιʔεϨΠϠɾϑϨʔϜϫʔ ΫϨΠϠɾΞϓϦέʔγϣϯϨΠϠͷΑ͏ͳ֊૚ߏ଄Ͱ෼͚Δͱ Πϝʔδ͠΍͍͢ɻ wπʔϧͷબ୒ͷ؍఺ͱͯ͠͸ɺͲͷϑϨʔϜϫʔΫΛ࢖ͬͯτϨʔ εΛ͢Δ͔ʹΑͬͯΦʔόʔϔου΍τϨʔεର৅͕มΘΔɻ wͲͷτϨʔεख๏Ͱ΋༻్ʹΑͬͯ͸ඞཁे෼Ͱ͋Δ͕ɺ৽͍͠ -JOVYΛ࢖͍ͬͯΔͳΒF#1'Λ࠾༻͍ͨ͠ɻ௿Φʔόʔϔουʹ ͩ͜ΘΔඞཁ͕͋Ε͹ಛʹɻ 
  35. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint ࠓޙͷ՝୊ wࠓճ͸τϨʔεख๏ͷશମ૾΍ͦΕͧΕͷಛੑʹ͍ͭͯΛௐࠪ͠ ͨɻ͔͠͠ɺ࣮ࡍʹΦʔόʔϔουͷଌఆ΍ɺ͸͡Ίʹड़΂ͨ՝ ୊ʹରͯ͠ͷ༗ޮੑͷݕূ͕Ͱ͖͍ͯͳ͍ͨΊࠓޙ΍͍ͬͯ͘ɻ w͜ΕΒͷπʔϧΛମܥԽ͢Δʹ͋ͨͬͯ͸ɺ༗ޮੑΛࣔͨ͠ޙʹ ந৅Խ͞ΕͨϢʔεέʔεͱϓϥΫςΟεΛ໢ཏ͢Δ͜ͱͱɺඞ ཁʹԠͯ͡τϨʔεϑϨʔϜϫʔΫΛ࢖ͬͨιϑτ΢ΣΞͷ։ൃ ͕ඞཁʹͳΔɻ 
  36. Systemtap eBPF kprobe perf Dynamic Trace uprobe Static Trace BCC

    ftrace /sys/kernel/debug/tracing/ Kernel Module BPF Bytecode strace Events Tracepoint