Upgrade to Pro — share decks privately, control downloads, hide ads and more …

BPF/bccによる トレーシング入門 (仮)

734096b490c456ce1e8670d279ac30cf?s=47 mmisono
November 05, 2018

BPF/bccによる トレーシング入門 (仮)

2018/11/5 OSSセキュリティ技術の会 第四回勉強会
https://secureoss-sig.connpass.com/event/103763/

734096b490c456ce1e8670d279ac30cf?s=128

mmisono

November 05, 2018
Tweet

Transcript

  1. #1'CDDʹΑΔ τϨʔγϯάೖ໳ Ծ 2018/11/5 OSS   

  2. ͜ͷൃදͷ಺༰ 2 BPFʹΑΔτϨʔγϯάͷ಺෦ಈ࡞ͷઆ໌͕ओʹͳΓ·͢ɽ ۩ମతͳπʔϧͷ࢖͍ํͷઆ໌΍ɼτϨʔγϯάͷηΦϦʔɾఆੴͳͲͷ ࿩͸͋·Γ͋Γ·ͤΜɽ ͜ͷࢿྉͷBPF = eBPFͰ͢

  3. 01 02 03 ΞδΣϯμ 3 Linux Tracing ͷ֓ཁ BPFʹΑΔτϨʔγϯά bccʹΑΔτϨʔγϯά

  4. 1 Linux Tracing System 4

  5. ͸͡Ίʹ 5 BPFͰͷτϨʔγϯά = º ׬શʹ৽͍͠τϨʔγϯάϑϨʔϜϫʔΫ ̋ طଘͷτϨʔγϯάϑϨʔϜϫʔΫΛิ͏΋ͷ

  6. -JOVY5SBDJOH4ZTUFN$PNQPOFOU 6 Performance Counter (PMU) Tracepoint (Static Tracing) Kprobe (Dynamic

    Tracing) perf_event ftrace Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng    *O,FSOFM 'SBNFXPSL 6TFSMBOE5PPM %BUBTPVSDF
  7. -JOVY5SBDJOH4ZTUFN$PNQPOFOU 7 Tracepoint (Static Tracing) Kprobe (Dynamic Tracing) perf_event ftrace

    Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng    *O,FSOFM 'SBNFXPSL 6TFSMBOE5PPM %BUBTPVSDF Performance Counter (PMU) zzzzzz zzzzzz 
  8. σʔλιʔε 8 ɾ$16ݻ༗ͷػೳ ɾ.43ܦ༝Ͱ৘ใΛऔಘ ɾ*1$ Ωϟογϡώοτ཰ ʜ Performance Counter (PMU)

    ɾ4UBUJD5SBDJOH ɾΧʔωϧ಺ʹຒΊࠐ·Ε͍ͯΔ ɾ$BMMCBDLؔ਺Λొ࿥Ͱ͖Δ Tracepoint ɾ%ZOBNJD5SBDJOH ɾCSFBLQPJOUʹΑΔ ಈతϑοΫ ɾ$BMMCBDLؔ਺Λొ࿥Ͱ͖Δ Kprobe 1 2 3
  9. 1FSGPSNBODF$PVOUFS 1.6 9 ɾ$16ݻ༗ͷػೳ ɾαΠΫϧ਺ *1$ Ωϟογϡώοτ཰ ෼ذ༧ଌώοτ཰ ʜ ɾ*OUFMͷ৔߹

    ɾ.43 .PEFM4QFDJGJD3FHJTUFS ͔Βऔಘ ɾΞʔΩςΫνϟʹΑͬͯdݸఔ౓ ɾͲͷ৘ใΛಘ͍͔ͨ.43Ͱઃఆ͢Δ ɾಛఆͷ஋ʹୡͨ͠৔߹ׂΓࠐΈΛൃੜ͢Δػೳ͋Γ ɾ.43ݸ਺Ҏ্ͷ৘ใΛऔಘ͍ͨ͠৔߹͏·࣌͘෼ׂ͢Δඞཁ͕͋Δ
  10. 5SBDFQPJOU 10 ɾΧʔωϧιʔεதʹ௚઀ఆٛ ) ) ( (( ɾUSBDF@ ͱ͍͏໊લͷఆ͕ٛ͋Ε͹ ͍͍ͩͨ5SBDFQPJOUͷఆٛ

    ɾΧʔωϧόʔδϣϯ͕ҟͳͬͯ΋Πϯ λϑΣʔεతͳޓ׵ੑ͕͋Δʢ͸ͣʣ https://github.com/torvalds/linux/blob/v4.18/fs/exec.c#L1697
  11. ,QSPCF 11 Insn Break point pre handler post handler Insn

    ( ) ɾϒϨʔΫϙΠϯτΛར༻ͨ͠ ಈతϑοΫ ɾΧʔωϧ಺ͷେ෦෼͕ϑοΫՄೳ ɾΧʔωϧόʔδϣϯʹґଘ #
  12. ɾ-JOVYඪ४૷උͷϓϩϑΝΠϥ ɾΧʔωϧ಺ϑϨʔϜϫʔΫ ( ) ( ) ) ɾϢʔβπεϖʔεπʔϧ ( ɾQFSGͰͰ͖Δ͜ͱ

    ɾΠϕϯτͷൃੜճ਺ͷΧ΢ϯτ ( ɾ)BSEXBSF&WFOU 1FSGPSNBODF$PVOUFS ɾ5SBDFQPJOU &WFOU 5SBDFQPJOU ,QSPCF ɾ4PGUXBSF&WFOU QFSGಠࣗͷΠϕϯτ ɾαϯϓϦϯά ( ɾ1.6ͷׂΓࠐΈΛར༻ͨ͠αϯϓϦϯά ҰൠʹαΠΫϧ਺Λར༻ 1FSG DGQFSGGUSBDFͷ࢓૊Έ IUUQNNJIBUFOBCMPHDPNFOUSZ 12 kprobe Performance Counter perf_event tracepoint perf_event_open(2) Hardware Tracepoint perf mmaped ring buffer Software
  13. 2 Tracing with BPF

  14. 5SBDJOHXJUI#1' Tracepoint Kporbe Perf software event Perf hardware event Event

    Call BPF Program Helper Function pid uid  … eBPF Map perf buffer
  15. 15 bpf(2) system call Create BPF map Kernel Userland BPF

    map User Program
  16. 16 bpf(2) system call Verifier C source BPF Program JIT

    (Optional) Load BPF Program Kernel Userland BPF map LLVM/Clang User Program Event Attach BPF bytecode Tracepoint Kporbe Performane counter
  17. 17 bpf(2) system call C source BPF Program Load BPF

    Program Kernel Userland BPF map LLVM/Clang User Program Event Call Return value Access BPF bytecode Tracepoint Kporbe Performane counter Call Return value Helper Function
  18. 18 bpf(2) system call C source BPF Program Load BPF

    Program Kernel Userland BPF map LLVM/Clang User Program Event BPF bytecode Tracepoint Kporbe Performane counter Read BPF map
  19. #1'ϓϩάϥϜͷྫ 19 ɾF#1' NBQ͔Β͜Ε·Ͱͷܭ਺݁ՌΛऔಘ ɾ݁ՌʹΛ଍ͯ͠NBQʹॻ͖໭͢ Πϕϯτൃੜճ਺ͷܭ਺ ɾϖΞͱͳΔؔ਺Λݟ͚ͭΔ FH BMMPDGSFF ɾQSPMPHVFͷؔ਺Ͱ࣌ࠁΛऔಘɼNBQʹ֨ೲ

    ɾFQJMPHVFͷؔ਺ͰNBQʹ֨ೲͨ࣌͠ࠁͱͷࠩΛܭࢉ ϨΠςϯγͷଌఆ
  20. #1'ϓϩάϥϜྫ 20 https://github.com/torvalds/linux/blob/v4.18/samples/bpf/tracex3_kern.c ɾCMLJP MBUFODZͷଌఆ * , ( ( (

    ( ( * , * ,( * ( ( ( *) ( * ( 
  21. Χʔωϧαϙʔτঢ়گ 21 ػೳ -JOVY7FSTJPO #1'1SPHSBN5ZQF ,QSPCF    

       6QSPCF        5SBDFQPJOU         1FSGTPGUXBSF IBSEXBSFFWFOU         https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md
  22. #1'ͷ࣮ࡍͷར༻ํ๏ 22 Linux Sample https://github.com/torvalds/linux/tree/master/samples/bpf bpf(2) http://man7.org/linux/man-pages/man2/bpf.2.html pef_event_open(2) http://man7.org/linux/man-pages/man2/perf_event_open.2.html

  23. 3 Tracing with bcc

  24. #1'ϓϩάϥϜ࡞੒Ͱେมͳ఺ • υΩϡϝϯτෆ଍ɼγεςϜίʔϧͷཧղ͕େม • CQG  QFSG@FWFOU@PQFO  ͱ͍͏ڧఢ •

    $ݴޠͷจ๏ͱͯ͠ؾΛ͚ͭΔ͜ͱ͕ଟʑ͋Δ • FH จࣈྻఆ਺͸ελοΫʹ഑ஔ͢Δ • #1'NBQͷऔΓѻ͍ • #1'ϓϩάϥϜʹ#1'NBQͷGJMFEFTDSJQUPSΛຒΊࠐΉඞཁ͕͋Δ • Ұํ$MBOHͰ࡞੒ͨ͠#1'ϓϩάϥϜ͸&-'όΠφϦ • &-'όΠφϦΛద੾ʹϩʔυ͢Δϩʔμʔ͕ඞཁ • -JOVYͷαϯϓϧʹଘࡏ͢Δ͕ɼҰൠͷΞϓϦέʔγϣϯ͔Β͸࢖͍ʹ͍͘
  25. CDD #1'$PNQJMFS$PMMFDUJPO 25 ɾIUUQTHJUIVCDPNJPWJTPSCDD ɾ#1'ϓϩάϥϜ࡞੒Λαϙʔτ͢ΔͨΊͷϥΠϒϥϦ ஫τϨʔγϯάʹݶఆ͢Δ΋ͷͰ͸ͳ͍ ɾ#1'༻NPEJGJFE$ίϯύΠϥ ϩʔμ ɾଞݴޠόΠϯσΟϯά -VB

    1ZUIPO (P  ˞τϨʔγϯάίʔυࣗମ͸$Ͱهड़ ɾCDDΛ༻͍ͨτϨʔγϯάπʔϧ܈
  26. CDDͰͷϓϩάϥϜྫ 26   finish_task_switch() kprobe Python map  

     ! https://github.com/iovisor/bcc/blob/master/examples/tracing/task_switch.py
  27. .PEJGJFE$ 27 https://github.com/iovisor/bcc/blob/master/examples/tracing/vfsreadlat.c &bcc (5BPF.4, modified C &BPF map206&-3 &#'

    7+19 &… &Clang$%%)5 :%#* AST  & eBPF map/8  &"!% https://github.com/iovisor/bcc/blob/master/docs/reference_g uide.md
  28. CDDDPNQJMFS 28 ) : : (

  29. πʔϧͱͯ͠ͷCDD 29 CDDͷϦϙδτϦʹɼCDDΛར༻ͨ͠τϨʔγϯάπʔϧؚ͕·Ε͍ͯΔ IUUQTHJUIVCDPNJPWJTPSCDDUSFFNBTUFSUPPMT ओཁEJTUSPʹQBDLBHF͕ଘࡏ ɾ6CVOUV ɾ'FEPSB ɾ"SDI ɾ(FOUPP ɾPQFO464&

    ɾ3)&- IUUQTHJUIVCDPNJPWJTPSCDDCMPCNBTUFS*/45"--NE
  30. πʔϧͱͯ͠ͷCDD 30

  31. πʔϧͱͯ͠ͷCDD 31

  32. CDDͷܽ఺ ࢓༷ 32 ࣮ߦ࣌͝ͱʹίϯύΠϧ͕ൃੜ ͨͩ͠ɼBPFʹ͸جຊతʹҾ਺ͷ֓೦͕ͳ͍ͨΊಈతίϯύΠϧ͕ඞཁͳ৔໘͸ଟʑ͋Δ ґଘؔ܎͕૿Ճ͢Δ ݱঢ়Python3ରԠ͕͍·͍ͪ

  33. Complementary 33

  34. ΍ͬͺΓ$Ҏ֎ͰτϨʔγϯά͍ͨ͠ 34 bpftrace (Dtrace-like ⇨ LLVM ⇨ eBPF) https://github.com/iovisor/bpftrace ply

    (Dtrace-like ⇨ eBPF) https://github.com/iovisor/ply py2bpf (Python byte code ⇨ eBPF) https://github.com/facebookresearch/py2bpf
  35. (P͔Βͷར༻ 35 gobpf https://github.com/iovisor/gobpf github.com/iovisor/gobpf/bcc bcc binding (libbcc͕ඞཁ) github.com/iovisor/gobpf/elf elf

    loader (elfόΠφϦ͸ࣗ෼ͰίϯύΠϧ͢Δ)
  36. ຊ೔આ໌͍ͯ͠ͳ͍͜ͱ 36 uprobe (⇔ kprobe) USDT (⇔ tracepoint) ftrace

  37. ·ͱΊ ैདྷͷLinux͔Βଘࡏ͢ΔτϨʔγϯάػߏΛBPFͰϓϩάϥϚϥϒϧʹ ར༻͢Δ͜ͱ͕Ͱ͖·͢ bccΛ࢖͏ͱBPFͰͷτϨʔεϓϩάϥϜ࡞੒͕͙ͬͱָʹͳΓ·͢ bccʹΑͬͯ(BPFͷ͜ͱΛԿ΋஌Βͳͯ͘΋)؆୯ʹBPFʹΑΔτϨʔγϯά ͕࣮ߦͰ͖·͢ Let’s try! 37

  38. 38