BPF/bccによる トレーシング入門 (仮)

Transcript

1. 1.

   #1'CDDʹΑΔ τϨʔγϯάೖ໳ Ծ 2018/11/5 OSS
      

2. 2.

   ͜ͷൃදͷ಺༰ 2 BPFʹΑΔτϨʔγϯάͷ಺෦ಈ࡞ͷઆ໌͕ओʹͳΓ·͢ɽ ۩ମతͳπʔϧͷ࢖͍ํͷઆ໌΍ɼτϨʔγϯάͷηΦϦʔɾఆੴͳͲͷ ࿩͸͋·Γ͋Γ·ͤΜɽ ͜ͷࢿྉͷBPF = eBPFͰ͢

3. 3.

   01 02 03 ΞδΣϯμ 3 Linux Tracing ͷ֓ཁ BPFʹΑΔτϨʔγϯά bccʹΑΔτϨʔγϯά

4. 4.

   1 Linux Tracing System 4

5. 5.

   ͸͡Ίʹ 5 BPFͰͷτϨʔγϯά = º ׬શʹ৽͍͠τϨʔγϯάϑϨʔϜϫʔΫ ̋ طଘͷτϨʔγϯάϑϨʔϜϫʔΫΛิ͏΋ͷ

6. 6.

   -JOVY
   5SBDJOH
   4ZTUFN
   $PNQPOFOU 6 Performance Counter (PMU) Tracepoint (Static Tracing) Kprobe (Dynamic

   Tracing) perf_event ftrace Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng
   
   
   
   
   
   
   
   
   *O
   ,FSOFM
   'SBNFXPSL 6TFSMBOE
   5PPM %BUB
   TPVSDF
7. 7.

   -JOVY
   5SBDJOH
   4ZTUFN
   $PNQPOFOU 7 Tracepoint (Static Tracing) Kprobe (Dynamic Tracing) perf_event ftrace

   Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng
   
   
   
   
   
   
   
   
   *O
   ,FSOFM
   'SBNFXPSL 6TFSMBOE
   5PPM %BUB
   TPVSDF Performance Counter (PMU) zzzzzz zzzzzz 
8. 8.

   σʔλιʔε 8 ɾ$16ݻ༗ͷػೳ ɾ.43ܦ༝Ͱ৘ใΛऔಘ ɾ*1$
   Ωϟογϡώοτ཰
   ʜ Performance Counter (PMU)

   ɾ4UBUJD
   5SBDJOH ɾΧʔωϧ಺ʹຒΊࠐ·Ε͍ͯΔ ɾ$BMMCBDLؔ਺Λొ࿥Ͱ͖Δ Tracepoint ɾ%ZOBNJD
   5SBDJOH ɾCSFBLQPJOUʹΑΔ ಈతϑοΫ ɾ$BMMCBDLؔ਺Λొ࿥Ͱ͖Δ Kprobe 1 2 3
9. 9.

   1FSGPSNBODF
   $PVOUFS
   1.6 9 ɾ$16ݻ༗ͷػೳ ɾαΠΫϧ਺
   *1$
   Ωϟογϡώοτ཰
   ෼ذ༧ଌώοτ཰ ʜ ɾ*OUFMͷ৔߹

   ɾ.43
   .PEFM
   4QFDJGJD
   3FHJTUFS
   ͔Βऔಘ ɾΞʔΩςΫνϟʹΑͬͯdݸఔ౓ ɾͲͷ৘ใΛಘ͍͔ͨ.43Ͱઃఆ͢Δ ɾಛఆͷ஋ʹୡͨ͠৔߹ׂΓࠐΈΛൃੜ͢Δػೳ͋Γ ɾ.43ݸ਺Ҏ্ͷ৘ใΛऔಘ͍ͨ͠৔߹͏·࣌͘෼ׂ͢Δඞཁ͕͋Δ
10. 10.

    5SBDFQPJOU 10 ɾΧʔωϧιʔεதʹ௚઀ఆٛ ) ) ( (( ɾUSBDF@
    ͱ͍͏໊લͷఆ͕ٛ͋Ε͹ ͍͍ͩͨ5SBDFQPJOUͷఆٛ

    ɾΧʔωϧόʔδϣϯ͕ҟͳͬͯ΋Πϯ λϑΣʔεతͳޓ׵ੑ͕͋Δʢ͸ͣʣ https://github.com/torvalds/linux/blob/v4.18/fs/exec.c#L1697
11. 11.

    ,QSPCF 11 Insn Break point pre handler post handler Insn

    ( ) ɾϒϨʔΫϙΠϯτΛར༻ͨ͠ ಈతϑοΫ ɾΧʔωϧ಺ͷେ෦෼͕ϑοΫՄೳ ɾΧʔωϧόʔδϣϯʹґଘ #
12. 12.

    ɾ-JOVYඪ४૷උͷϓϩϑΝΠϥ ɾΧʔωϧ಺ϑϨʔϜϫʔΫ
    ( ) ( ) ) ɾϢʔβπεϖʔεπʔϧ
    ( ɾQFSGͰͰ͖Δ͜ͱ

    ɾΠϕϯτͷൃੜճ਺ͷΧ΢ϯτ ( ɾ)BSEXBSF
    &WFOU
    1FSGPSNBODF
    $PVOUFS ɾ5SBDFQPJOU &WFOU
    5SBDFQPJOU
    ,QSPCF ɾ4PGUXBSF
    &WFOU
    QFSGಠࣗͷΠϕϯτ ɾαϯϓϦϯά ( ɾ1.6ͷׂΓࠐΈΛར༻ͨ͠αϯϓϦϯά ҰൠʹαΠΫϧ਺Λར༻ 1FSG DG
    QFSGGUSBDFͷ࢓૊Έ IUUQNNJIBUFOBCMPHDPNFOUSZ 12 kprobe Performance Counter perf_event tracepoint perf_event_open(2) Hardware Tracepoint perf mmaped ring buffer Software
13. 13.

    2 Tracing with BPF

14. 14.

    5SBDJOH
    XJUI
    #1' Tracepoint Kporbe Perf software event Perf hardware event Event

    Call BPF Program Helper Function pid
     uid
     
     … eBPF Map perf buffer
15. 15.

    15 bpf(2) system call Create BPF map Kernel Userland BPF

    map User Program
16. 16.

    16 bpf(2) system call Verifier C source BPF Program JIT

    (Optional) Load BPF Program Kernel Userland BPF map LLVM/Clang User Program Event Attach BPF bytecode
    Tracepoint
    Kporbe
    Performane counter
17. 17.

    17 bpf(2) system call C source BPF Program Load BPF

    Program Kernel Userland BPF map LLVM/Clang User Program Event Call Return value Access BPF bytecode
    Tracepoint
    Kporbe
    Performane counter Call Return value Helper Function
18. 18.

    18 bpf(2) system call C source BPF Program Load BPF

    Program Kernel Userland BPF map LLVM/Clang User Program Event BPF bytecode
    Tracepoint
    Kporbe
    Performane counter Read BPF map
19. 19.

    #1'ϓϩάϥϜͷྫ 19 ɾF#1' NBQ͔Β͜Ε·Ͱͷܭ਺݁ՌΛऔಘ ɾ݁ՌʹΛ଍ͯ͠NBQʹॻ͖໭͢ Πϕϯτൃੜճ਺ͷܭ਺ ɾϖΞͱͳΔؔ਺Λݟ͚ͭΔ FH
    BMMPDGSFF ɾQSPMPHVFͷؔ਺Ͱ࣌ࠁΛऔಘɼNBQʹ֨ೲ

    ɾFQJMPHVFͷؔ਺ͰNBQʹ֨ೲͨ࣌͠ࠁͱͷࠩΛܭࢉ ϨΠςϯγͷଌఆ
20. 20.

    #1'ϓϩάϥϜྫ 20 https://github.com/torvalds/linux/blob/v4.18/samples/bpf/tracex3_kern.c ɾCMLJP MBUFODZͷଌఆ * , ( ( (

    ( ( * , * ,( * ( ( ( *) ( * (
    
    
    
21. 21.

    Χʔωϧαϙʔτঢ়گ 21 ػೳ -JOVY
    7FSTJPO #1'
    1SPHSBN
    5ZQF ,QSPCF    

       6QSPCF        5SBDFQPJOU      
      1FSG
    TPGUXBSF
    
    IBSEXBSF
    FWFOU         https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md
22. 22.

    #1'ͷ࣮ࡍͷར༻ํ๏ 22 Linux Sample https://github.com/torvalds/linux/tree/master/samples/bpf bpf(2) http://man7.org/linux/man-pages/man2/bpf.2.html pef_event_open(2) http://man7.org/linux/man-pages/man2/perf_event_open.2.html

23. 23.

    3 Tracing with bcc

24. 24.

    #1'ϓϩάϥϜ࡞੒Ͱେมͳ఺ • υΩϡϝϯτෆ଍ɼγεςϜίʔϧͷཧղ͕େม • CQG  QFSG@FWFOU@PQFO  ͱ͍͏ڧఢ •

    $ݴޠͷจ๏ͱͯ͠ؾΛ͚ͭΔ͜ͱ͕ଟʑ͋Δ • FH
    จࣈྻఆ਺͸ελοΫʹ഑ஔ͢Δ • #1'
    NBQͷऔΓѻ͍ • #1'ϓϩάϥϜʹ#1'
    NBQͷGJMF
    EFTDSJQUPSΛຒΊࠐΉඞཁ͕͋Δ • Ұํ$MBOHͰ࡞੒ͨ͠#1'ϓϩάϥϜ͸&-'όΠφϦ • &-'όΠφϦΛద੾ʹϩʔυ͢Δϩʔμʔ͕ඞཁ • -JOVYͷαϯϓϧʹଘࡏ͢Δ͕ɼҰൠͷΞϓϦέʔγϣϯ͔Β͸࢖͍ʹ͍͘
25. 25.

    CDD
    #1'
    $PNQJMFS
    $PMMFDUJPO 25 ɾIUUQTHJUIVCDPNJPWJTPSCDD ɾ#1'ϓϩάϥϜ࡞੒Λαϙʔτ͢ΔͨΊͷϥΠϒϥϦ ஫
    τϨʔγϯάʹݶఆ͢Δ΋ͷͰ͸ͳ͍ ɾ#1'༻NPEJGJFE
    $ίϯύΠϥ
    ϩʔμ ɾଞݴޠόΠϯσΟϯά -VB

    
    1ZUIPO
    (P
    
    ˞
    τϨʔγϯάίʔυࣗମ͸$Ͱهड़ ɾCDDΛ༻͍ͨτϨʔγϯάπʔϧ܈
26. 26.

    CDDͰͷϓϩάϥϜྫ 26   finish_task_switch() kprobe Python
    map  

     ! https://github.com/iovisor/bcc/blob/master/examples/tracing/task_switch.py
27. 27.

    .PEJGJFE
    $ 27 https://github.com/iovisor/bcc/blob/master/examples/tracing/vfsreadlat.c &bcc (5BPF.4, modified C &BPF map206&-3 &#'

    7+19 &… &Clang$%%)5 :%#* AST
    
     & eBPF map/8
     &"!% https://github.com/iovisor/bcc/blob/master/docs/reference_g uide.md
28. 28.

    CDD
    DPNQJMFS 28 ) : : (

29. 29.

    πʔϧͱͯ͠ͷCDD 29 CDDͷϦϙδτϦʹɼCDDΛར༻ͨ͠τϨʔγϯάπʔϧؚ͕·Ε͍ͯΔ IUUQTHJUIVCDPNJPWJTPSCDDUSFFNBTUFSUPPMT ओཁEJTUSPʹQBDLBHF͕ଘࡏ ɾ6CVOUV ɾ'FEPSB ɾ"SDI ɾ(FOUPP ɾPQFO464&

    ɾ3)&- IUUQTHJUIVCDPNJPWJTPSCDDCMPCNBTUFS*/45"--NE
30. 30.

    πʔϧͱͯ͠ͷCDD 30

31. 31.

    πʔϧͱͯ͠ͷCDD 31

32. 32.

    CDDͷܽ఺ ࢓༷ 32 ࣮ߦ࣌͝ͱʹίϯύΠϧ͕ൃੜ ͨͩ͠ɼBPFʹ͸جຊతʹҾ਺ͷ֓೦͕ͳ͍ͨΊಈతίϯύΠϧ͕ඞཁͳ৔໘͸ଟʑ͋Δ ґଘؔ܎͕૿Ճ͢Δ ݱঢ়Python3ରԠ͕͍·͍ͪ

33. 33.

    Complementary 33

34. 34.

    ΍ͬͺΓ$Ҏ֎ͰτϨʔγϯά͍ͨ͠ 34 bpftrace (Dtrace-like ⇨ LLVM ⇨ eBPF) https://github.com/iovisor/bpftrace ply

    (Dtrace-like ⇨ eBPF) https://github.com/iovisor/ply py2bpf (Python byte code ⇨ eBPF) https://github.com/facebookresearch/py2bpf
35. 35.

    (P͔Βͷར༻ 35 gobpf https://github.com/iovisor/gobpf github.com/iovisor/gobpf/bcc bcc binding (libbcc͕ඞཁ) github.com/iovisor/gobpf/elf elf

    loader (elfόΠφϦ͸ࣗ෼ͰίϯύΠϧ͢Δ)
36. 36.

    ຊ೔આ໌͍ͯ͠ͳ͍͜ͱ 36 uprobe (⇔ kprobe) USDT (⇔ tracepoint) ftrace

37. 37.

    ·ͱΊ ैདྷͷLinux͔Βଘࡏ͢ΔτϨʔγϯάػߏΛBPFͰϓϩάϥϚϥϒϧʹ ར༻͢Δ͜ͱ͕Ͱ͖·͢ bccΛ࢖͏ͱBPFͰͷτϨʔεϓϩάϥϜ࡞੒͕͙ͬͱָʹͳΓ·͢ bccʹΑͬͯ(BPFͷ͜ͱΛԿ΋஌Βͳͯ͘΋)؆୯ʹBPFʹΑΔτϨʔγϯά ͕࣮ߦͰ͖·͢ Let’s try! 37

38. 38.

    38