BPF/bccによる トレーシング入門 (仮)

2018/11/5 OSSセキュリティ技術の会 第四回勉強会
https://secureoss-sig.connpass.com/event/103763/

Transcript

1. #1'CDDʹΑΔ τϨʔγϯάೖ໳ Ծ 2018/11/5 OSS
      
2. ͜ͷൃදͷ಺༰ 2 BPFʹΑΔτϨʔγϯάͷ಺෦ಈ࡞ͷઆ໌͕ओʹͳΓ·͢ɽ ۩ମతͳπʔϧͷ࢖͍ํͷઆ໌΍ɼτϨʔγϯάͷηΦϦʔɾఆੴͳͲͷ ࿩͸͋·Γ͋Γ·ͤΜɽ ͜ͷࢿྉͷBPF = eBPFͰ͢
3. 01 02 03 ΞδΣϯμ 3 Linux Tracing ͷ֓ཁ BPFʹΑΔτϨʔγϯά bccʹΑΔτϨʔγϯά
4. 1 Linux Tracing System 4
5. ͸͡Ίʹ 5 BPFͰͷτϨʔγϯά = º ׬શʹ৽͍͠τϨʔγϯάϑϨʔϜϫʔΫ ̋ طଘͷτϨʔγϯάϑϨʔϜϫʔΫΛิ͏΋ͷ
6. -JOVY
   5SBDJOH
   4ZTUFN
   $PNQPOFOU 6 Performance Counter (PMU) Tracepoint (Static Tracing) Kprobe (Dynamic Tracing) perf_event ftrace Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng
   
   
   
   
   
   
   
   
   *O
   ,FSOFM
   'SBNFXPSL 6TFSMBOE
   5PPM %BUB
   TPVSDF
7. -JOVY
   5SBDJOH
   4ZTUFN
   $PNQPOFOU 7 Tracepoint (Static Tracing) Kprobe (Dynamic Tracing) perf_event ftrace Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng
   
   
   
   
   
   
   
   
   *O
   ,FSOFM
   'SBNFXPSL 6TFSMBOE
   5PPM %BUB
   TPVSDF Performance Counter (PMU) zzzzzz zzzzzz 
8. σʔλιʔε 8 ɾ$16ݻ༗ͷػೳ ɾ.43ܦ༝Ͱ৘ใΛऔಘ ɾ*1$
   Ωϟογϡώοτ཰
   ʜ Performance Counter (PMU) ɾ4UBUJD
   5SBDJOH ɾΧʔωϧ಺ʹຒΊࠐ·Ε͍ͯΔ ɾ$BMMCBDLؔ਺Λొ࿥Ͱ͖Δ Tracepoint ɾ%ZOBNJD
   5SBDJOH ɾCSFBLQPJOUʹΑΔ ಈతϑοΫ ɾ$BMMCBDLؔ਺Λొ࿥Ͱ͖Δ Kprobe 1 2 3
9. 1FSGPSNBODF
   $PVOUFS
   1.6 9 ɾ$16ݻ༗ͷػೳ ɾαΠΫϧ਺
   *1$
   Ωϟογϡώοτ཰
   ෼ذ༧ଌώοτ཰ ʜ ɾ*OUFMͷ৔߹ ɾ.43
   .PEFM
   4QFDJGJD
   3FHJTUFS
   ͔Βऔಘ ɾΞʔΩςΫνϟʹΑͬͯdݸఔ౓ ɾͲͷ৘ใΛಘ͍͔ͨ.43Ͱઃఆ͢Δ ɾಛఆͷ஋ʹୡͨ͠৔߹ׂΓࠐΈΛൃੜ͢Δػೳ͋Γ ɾ.43ݸ਺Ҏ্ͷ৘ใΛऔಘ͍ͨ͠৔߹͏·࣌͘෼ׂ͢Δඞཁ͕͋Δ
10. 5SBDFQPJOU 10 ɾΧʔωϧιʔεதʹ௚઀ఆٛ ) ) ( (( ɾUSBDF@
    ͱ͍͏໊લͷఆ͕ٛ͋Ε͹ ͍͍ͩͨ5SBDFQPJOUͷఆٛ ɾΧʔωϧόʔδϣϯ͕ҟͳͬͯ΋Πϯ λϑΣʔεతͳޓ׵ੑ͕͋Δʢ͸ͣʣ https://github.com/torvalds/linux/blob/v4.18/fs/exec.c#L1697
11. ,QSPCF 11 Insn Break point pre handler post handler Insn ( ) ɾϒϨʔΫϙΠϯτΛར༻ͨ͠ ಈతϑοΫ ɾΧʔωϧ಺ͷେ෦෼͕ϑοΫՄೳ ɾΧʔωϧόʔδϣϯʹґଘ #
12. ɾ-JOVYඪ४૷උͷϓϩϑΝΠϥ ɾΧʔωϧ಺ϑϨʔϜϫʔΫ
    ( ) ( ) ) ɾϢʔβπεϖʔεπʔϧ
    ( ɾQFSGͰͰ͖Δ͜ͱ ɾΠϕϯτͷൃੜճ਺ͷΧ΢ϯτ ( ɾ)BSEXBSF
    &WFOU
    1FSGPSNBODF
    $PVOUFS ɾ5SBDFQPJOU &WFOU
    5SBDFQPJOU
    ,QSPCF ɾ4PGUXBSF
    &WFOU
    QFSGಠࣗͷΠϕϯτ ɾαϯϓϦϯά ( ɾ1.6ͷׂΓࠐΈΛར༻ͨ͠αϯϓϦϯά ҰൠʹαΠΫϧ਺Λར༻ 1FSG DG
    QFSGGUSBDFͷ࢓૊Έ IUUQNNJIBUFOBCMPHDPNFOUSZ 12 kprobe Performance Counter perf_event tracepoint perf_event_open(2) Hardware Tracepoint perf mmaped ring buffer Software
13. 2 Tracing with BPF
14. 5SBDJOH
    XJUI
    #1' Tracepoint Kporbe Perf software event Perf hardware event Event Call BPF Program Helper Function pid
     uid
     
     … eBPF Map perf buffer
15. 15 bpf(2) system call Create BPF map Kernel Userland BPF map User Program
16. 16 bpf(2) system call Verifier C source BPF Program JIT (Optional) Load BPF Program Kernel Userland BPF map LLVM/Clang User Program Event Attach BPF bytecode
    Tracepoint
    Kporbe
    Performane counter
17. 17 bpf(2) system call C source BPF Program Load BPF Program Kernel Userland BPF map LLVM/Clang User Program Event Call Return value Access BPF bytecode
    Tracepoint
    Kporbe
    Performane counter Call Return value Helper Function
18. 18 bpf(2) system call C source BPF Program Load BPF Program Kernel Userland BPF map LLVM/Clang User Program Event BPF bytecode
    Tracepoint
    Kporbe
    Performane counter Read BPF map
19. #1'ϓϩάϥϜͷྫ 19 ɾF#1' NBQ͔Β͜Ε·Ͱͷܭ਺݁ՌΛऔಘ ɾ݁ՌʹΛ଍ͯ͠NBQʹॻ͖໭͢ Πϕϯτൃੜճ਺ͷܭ਺ ɾϖΞͱͳΔؔ਺Λݟ͚ͭΔ FH
    BMMPDGSFF ɾQSPMPHVFͷؔ਺Ͱ࣌ࠁΛऔಘɼNBQʹ֨ೲ ɾFQJMPHVFͷؔ਺ͰNBQʹ֨ೲͨ࣌͠ࠁͱͷࠩΛܭࢉ ϨΠςϯγͷଌఆ
20. #1'ϓϩάϥϜྫ 20 https://github.com/torvalds/linux/blob/v4.18/samples/bpf/tracex3_kern.c ɾCMLJP MBUFODZͷଌఆ * , ( ( ( ( ( * , * ,( * ( ( ( *) ( * (
    
    
    
21. Χʔωϧαϙʔτঢ়گ 21 ػೳ -JOVY
    7FSTJPO #1'
    1SPHSBN
    5ZQF ,QSPCF        6QSPCF        5SBDFQPJOU      
      1FSG
    TPGUXBSF
    
    IBSEXBSF
    FWFOU         https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md
22. #1'ͷ࣮ࡍͷར༻ํ๏ 22 Linux Sample https://github.com/torvalds/linux/tree/master/samples/bpf bpf(2) http://man7.org/linux/man-pages/man2/bpf.2.html pef_event_open(2) http://man7.org/linux/man-pages/man2/perf_event_open.2.html
23. 3 Tracing with bcc
24. #1'ϓϩάϥϜ࡞੒Ͱେมͳ఺ • υΩϡϝϯτෆ଍ɼγεςϜίʔϧͷཧղ͕େม • CQG  QFSG@FWFOU@PQFO  ͱ͍͏ڧఢ • $ݴޠͷจ๏ͱͯ͠ؾΛ͚ͭΔ͜ͱ͕ଟʑ͋Δ • FH
    จࣈྻఆ਺͸ελοΫʹ഑ஔ͢Δ • #1'
    NBQͷऔΓѻ͍ • #1'ϓϩάϥϜʹ#1'
    NBQͷGJMF
    EFTDSJQUPSΛຒΊࠐΉඞཁ͕͋Δ • Ұํ$MBOHͰ࡞੒ͨ͠#1'ϓϩάϥϜ͸&-'όΠφϦ • &-'όΠφϦΛద੾ʹϩʔυ͢Δϩʔμʔ͕ඞཁ • -JOVYͷαϯϓϧʹଘࡏ͢Δ͕ɼҰൠͷΞϓϦέʔγϣϯ͔Β͸࢖͍ʹ͍͘
25. CDD
    #1'
    $PNQJMFS
    $PMMFDUJPO 25 ɾIUUQTHJUIVCDPNJPWJTPSCDD ɾ#1'ϓϩάϥϜ࡞੒Λαϙʔτ͢ΔͨΊͷϥΠϒϥϦ ஫
    τϨʔγϯάʹݶఆ͢Δ΋ͷͰ͸ͳ͍ ɾ#1'༻NPEJGJFE
    $ίϯύΠϥ
    ϩʔμ ɾଞݴޠόΠϯσΟϯά -VB
    1ZUIPO
    (P
    
    ˞
    τϨʔγϯάίʔυࣗମ͸$Ͱهड़ ɾCDDΛ༻͍ͨτϨʔγϯάπʔϧ܈
26. CDDͰͷϓϩάϥϜྫ 26   finish_task_switch() kprobe Python
    map    ! https://github.com/iovisor/bcc/blob/master/examples/tracing/task_switch.py
27. .PEJGJFE
    $ 27 https://github.com/iovisor/bcc/blob/master/examples/tracing/vfsreadlat.c &bcc (5BPF.4, modified C &BPF map206&-3 &#' 7+19 &… &Clang$%%)5 :%#* AST
    
     & eBPF map/8
     &"!% https://github.com/iovisor/bcc/blob/master/docs/reference_g uide.md
28. CDD
    DPNQJMFS 28 ) : : (
29. πʔϧͱͯ͠ͷCDD 29 CDDͷϦϙδτϦʹɼCDDΛར༻ͨ͠τϨʔγϯάπʔϧؚ͕·Ε͍ͯΔ IUUQTHJUIVCDPNJPWJTPSCDDUSFFNBTUFSUPPMT ओཁEJTUSPʹQBDLBHF͕ଘࡏ ɾ6CVOUV ɾ'FEPSB ɾ"SDI ɾ(FOUPP ɾPQFO464& ɾ3)&- IUUQTHJUIVCDPNJPWJTPSCDDCMPCNBTUFS*/45"--NE
30. πʔϧͱͯ͠ͷCDD 30
31. πʔϧͱͯ͠ͷCDD 31
32. CDDͷܽ఺ ࢓༷ 32 ࣮ߦ࣌͝ͱʹίϯύΠϧ͕ൃੜ ͨͩ͠ɼBPFʹ͸جຊతʹҾ਺ͷ֓೦͕ͳ͍ͨΊಈతίϯύΠϧ͕ඞཁͳ৔໘͸ଟʑ͋Δ ґଘؔ܎͕૿Ճ͢Δ ݱঢ়Python3ରԠ͕͍·͍ͪ
33. Complementary 33
34. ΍ͬͺΓ$Ҏ֎ͰτϨʔγϯά͍ͨ͠ 34 bpftrace (Dtrace-like ⇨ LLVM ⇨ eBPF) https://github.com/iovisor/bpftrace ply (Dtrace-like ⇨ eBPF) https://github.com/iovisor/ply py2bpf (Python byte code ⇨ eBPF) https://github.com/facebookresearch/py2bpf
35. (P͔Βͷར༻ 35 gobpf https://github.com/iovisor/gobpf github.com/iovisor/gobpf/bcc bcc binding (libbcc͕ඞཁ) github.com/iovisor/gobpf/elf elf loader (elfόΠφϦ͸ࣗ෼ͰίϯύΠϧ͢Δ)
36. ຊ೔આ໌͍ͯ͠ͳ͍͜ͱ 36 uprobe (⇔ kprobe) USDT (⇔ tracepoint) ftrace
37. ·ͱΊ ैདྷͷLinux͔Βଘࡏ͢ΔτϨʔγϯάػߏΛBPFͰϓϩάϥϚϥϒϧʹ ར༻͢Δ͜ͱ͕Ͱ͖·͢ bccΛ࢖͏ͱBPFͰͷτϨʔεϓϩάϥϜ࡞੒͕͙ͬͱָʹͳΓ·͢ bccʹΑͬͯ(BPFͷ͜ͱΛԿ΋஌Βͳͯ͘΋)؆୯ʹBPFʹΑΔτϨʔγϯά ͕࣮ߦͰ͖·͢ Let’s try! 37
38. 38