Upgrade to Pro — share decks privately, control downloads, hide ads and more …

BPF/bccによる トレーシング入門 (仮)

mmisono
November 05, 2018

BPF/bccによる トレーシング入門 (仮)

2018/11/5 OSSセキュリティ技術の会 第四回勉強会
https://secureoss-sig.connpass.com/event/103763/

mmisono

November 05, 2018
Tweet

More Decks by mmisono

Other Decks in Programming

Transcript

  1. #1'CDDʹΑΔ
    τϨʔγϯάೖ໳ Ծ

    2018/11/5 OSS


    View full-size slide

  2. ͜ͷൃදͷ಺༰
    2
    BPFʹΑΔτϨʔγϯάͷ಺෦ಈ࡞ͷઆ໌͕ओʹͳΓ·͢ɽ
    ۩ମతͳπʔϧͷ࢖͍ํͷઆ໌΍ɼτϨʔγϯάͷηΦϦʔɾఆੴͳͲͷ
    ࿩͸͋·Γ͋Γ·ͤΜɽ
    ͜ͷࢿྉͷBPF = eBPFͰ͢

    View full-size slide

  3. 01
    02
    03
    ΞδΣϯμ
    3
    Linux Tracing ͷ֓ཁ
    BPFʹΑΔτϨʔγϯά
    bccʹΑΔτϨʔγϯά

    View full-size slide

  4. 1
    Linux Tracing System
    4

    View full-size slide

  5. ͸͡Ίʹ
    5
    BPFͰͷτϨʔγϯά =
    º ׬શʹ৽͍͠τϨʔγϯάϑϨʔϜϫʔΫ
    ̋ طଘͷτϨʔγϯάϑϨʔϜϫʔΫΛิ͏΋ͷ

    View full-size slide

  6. -JOVY5SBDJOH4ZTUFN$PNQPOFOU
    6
    Performance Counter
    (PMU)
    Tracepoint
    (Static Tracing)
    Kprobe
    (Dynamic Tracing)
    perf_event ftrace Lttng
    SystemTap
    Mcount
    (gprof)
    perf
    tracefs
    (debugfs)
    trace-cmd SystemTap Lttng


    *O,FSOFM
    'SBNFXPSL
    6TFSMBOE5PPM
    %BUBTPVSDF

    View full-size slide

  7. -JOVY5SBDJOH4ZTUFN$PNQPOFOU
    7
    Tracepoint
    (Static Tracing)
    Kprobe
    (Dynamic Tracing)
    perf_event ftrace Lttng
    SystemTap
    Mcount
    (gprof)
    perf
    tracefs
    (debugfs)
    trace-cmd SystemTap Lttng


    *O,FSOFM
    'SBNFXPSL
    6TFSMBOE5PPM
    %BUBTPVSDF
    Performance Counter
    (PMU)
    zzzzzz
    zzzzzz

    View full-size slide

  8. σʔλιʔε
    8
    ɾ$16ݻ༗ͷػೳ
    ɾ.43ܦ༝Ͱ৘ใΛऔಘ
    ɾ*1$ Ωϟογϡώοτ཰ ʜ
    Performance Counter (PMU)
    ɾ4UBUJD5SBDJOH
    ɾΧʔωϧ಺ʹຒΊࠐ·Ε͍ͯΔ
    ɾ$BMMCBDLؔ਺Λొ࿥Ͱ͖Δ
    Tracepoint
    ɾ%ZOBNJD5SBDJOH
    ɾCSFBLQPJOUʹΑΔ
    ಈతϑοΫ
    ɾ$BMMCBDLؔ਺Λొ࿥Ͱ͖Δ
    Kprobe
    1 2 3

    View full-size slide

  9. 1FSGPSNBODF$PVOUFS 1.6

    9
    ɾ$16ݻ༗ͷػೳ
    ɾαΠΫϧ਺ *1$ Ωϟογϡώοτ཰ ෼ذ༧ଌώοτ཰ ʜ
    ɾ*OUFMͷ৔߹
    ɾ.43 .PEFM4QFDJGJD3FHJTUFS
    ͔Βऔಘ
    ɾΞʔΩςΫνϟʹΑͬͯdݸఔ౓
    ɾͲͷ৘ใΛಘ͍͔ͨ.43Ͱઃఆ͢Δ
    ɾಛఆͷ஋ʹୡͨ͠৔߹ׂΓࠐΈΛൃੜ͢Δػೳ͋Γ
    ɾ.43ݸ਺Ҏ্ͷ৘ใΛऔಘ͍ͨ͠৔߹͏·࣌͘෼ׂ͢Δඞཁ͕͋Δ

    View full-size slide

  10. 5SBDFQPJOU
    10
    ɾΧʔωϧιʔεதʹ௚઀ఆٛ
    ) ) (
    ((
    ɾUSBDF@
    ͱ͍͏໊લͷఆ͕ٛ͋Ε͹
    ͍͍ͩͨ5SBDFQPJOUͷఆٛ
    ɾΧʔωϧόʔδϣϯ͕ҟͳͬͯ΋Πϯ
    λϑΣʔεతͳޓ׵ੑ͕͋Δʢ͸ͣʣ
    https://github.com/torvalds/linux/blob/v4.18/fs/exec.c#L1697

    View full-size slide

  11. ,QSPCF
    11
    Insn Break point
    pre handler
    post handler
    Insn
    ( )
    ɾϒϨʔΫϙΠϯτΛར༻ͨ͠
    ಈతϑοΫ
    ɾΧʔωϧ಺ͷେ෦෼͕ϑοΫՄೳ
    ɾΧʔωϧόʔδϣϯʹґଘ
    #

    View full-size slide

  12. ɾ-JOVYඪ४૷උͷϓϩϑΝΠϥ
    ɾΧʔωϧ಺ϑϨʔϜϫʔΫ ( ) ( ) )
    ɾϢʔβπεϖʔεπʔϧ (
    ɾQFSGͰͰ͖Δ͜ͱ
    ɾΠϕϯτͷൃੜճ਺ͷΧ΢ϯτ (
    ɾ)BSEXBSF&WFOU 1FSGPSNBODF$PVOUFS

    ɾ5SBDFQPJOU &WFOU 5SBDFQPJOU ,QSPCF

    ɾ4PGUXBSF&WFOU QFSGಠࣗͷΠϕϯτ

    ɾαϯϓϦϯά (
    ɾ1.6ͷׂΓࠐΈΛར༻ͨ͠αϯϓϦϯά
    ҰൠʹαΠΫϧ਺Λར༻

    1FSG
    DGQFSGGUSBDFͷ࢓૊Έ IUUQNNJIBUFOBCMPHDPNFOUSZ
    12
    kprobe
    Performance
    Counter
    perf_event
    tracepoint
    perf_event_open(2)
    Hardware Tracepoint
    perf
    mmaped
    ring buffer
    Software

    View full-size slide

  13. 2
    Tracing with BPF

    View full-size slide

  14. 5SBDJOHXJUI#1'
    Tracepoint
    Kporbe
    Perf software event
    Perf hardware event
    Event
    Call
    BPF Program
    Helper Function
    pid
    uid


    eBPF Map
    perf buffer

    View full-size slide

  15. 15
    bpf(2) system call
    Create BPF map
    Kernel
    Userland
    BPF map
    User Program

    View full-size slide

  16. 16
    bpf(2) system call
    Verifier
    C source
    BPF
    Program
    JIT
    (Optional)
    Load BPF Program
    Kernel
    Userland
    BPF map
    LLVM/Clang
    User Program
    Event
    Attach
    BPF bytecode
    Tracepoint
    Kporbe
    Performane counter

    View full-size slide

  17. 17
    bpf(2) system call
    C source
    BPF
    Program
    Load BPF Program
    Kernel
    Userland
    BPF map
    LLVM/Clang
    User Program
    Event
    Call
    Return value
    Access
    BPF bytecode
    Tracepoint
    Kporbe
    Performane counter
    Call
    Return value
    Helper Function

    View full-size slide

  18. 18
    bpf(2) system call
    C source
    BPF
    Program
    Load BPF Program
    Kernel
    Userland
    BPF map
    LLVM/Clang
    User Program
    Event
    BPF bytecode
    Tracepoint
    Kporbe
    Performane counter
    Read BPF map

    View full-size slide

  19. #1'ϓϩάϥϜͷྫ
    19
    ɾF#1' NBQ͔Β͜Ε·Ͱͷܭ਺݁ՌΛऔಘ
    ɾ݁ՌʹΛ଍ͯ͠NBQʹॻ͖໭͢
    Πϕϯτൃੜճ਺ͷܭ਺
    ɾϖΞͱͳΔؔ਺Λݟ͚ͭΔ FH BMMPDGSFF

    ɾQSPMPHVFͷؔ਺Ͱ࣌ࠁΛऔಘɼNBQʹ֨ೲ
    ɾFQJMPHVFͷؔ਺ͰNBQʹ֨ೲͨ࣌͠ࠁͱͷࠩΛܭࢉ
    ϨΠςϯγͷଌఆ

    View full-size slide

  20. #1'ϓϩάϥϜྫ
    20
    https://github.com/torvalds/linux/blob/v4.18/samples/bpf/tracex3_kern.c
    ɾCMLJP MBUFODZͷଌఆ
    * , ( ( ( ( (
    * , * ,( * ( ( ( *) ( * (

    View full-size slide

  21. Χʔωϧαϙʔτঢ়گ
    21
    ػೳ -JOVY7FSTJPO #1'1SPHSBN5ZQF
    ,QSPCF


    6QSPCF


    5SBDFQPJOU


    1FSGTPGUXBSF
    IBSEXBSFFWFOU


    https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md

    View full-size slide

  22. #1'ͷ࣮ࡍͷར༻ํ๏
    22
    Linux Sample
    https://github.com/torvalds/linux/tree/master/samples/bpf
    bpf(2)
    http://man7.org/linux/man-pages/man2/bpf.2.html
    pef_event_open(2)
    http://man7.org/linux/man-pages/man2/perf_event_open.2.html

    View full-size slide

  23. 3
    Tracing with bcc

    View full-size slide

  24. #1'ϓϩάϥϜ࡞੒Ͱେมͳ఺
    • υΩϡϝϯτෆ଍ɼγεςϜίʔϧͷཧղ͕େม
    • CQG
    QFSG@FWFOU@PQFO
    ͱ͍͏ڧఢ
    • $ݴޠͷจ๏ͱͯ͠ؾΛ͚ͭΔ͜ͱ͕ଟʑ͋Δ
    • FH จࣈྻఆ਺͸ελοΫʹ഑ஔ͢Δ
    • #1'NBQͷऔΓѻ͍
    • #1'ϓϩάϥϜʹ#1'NBQͷGJMFEFTDSJQUPSΛຒΊࠐΉඞཁ͕͋Δ
    • Ұํ$MBOHͰ࡞੒ͨ͠#1'ϓϩάϥϜ͸&-'όΠφϦ
    • &-'όΠφϦΛద੾ʹϩʔυ͢Δϩʔμʔ͕ඞཁ
    • -JOVYͷαϯϓϧʹଘࡏ͢Δ͕ɼҰൠͷΞϓϦέʔγϣϯ͔Β͸࢖͍ʹ͍͘

    View full-size slide

  25. CDD #1'$PNQJMFS$PMMFDUJPO

    25
    ɾIUUQTHJUIVCDPNJPWJTPSCDD
    ɾ#1'ϓϩάϥϜ࡞੒Λαϙʔτ͢ΔͨΊͷϥΠϒϥϦ
    ஫τϨʔγϯάʹݶఆ͢Δ΋ͷͰ͸ͳ͍

    ɾ#1'༻NPEJGJFE$ίϯύΠϥ ϩʔμ
    ɾଞݴޠόΠϯσΟϯά -VB 1ZUIPO (P
    ˞τϨʔγϯάίʔυࣗମ͸$Ͱهड़
    ɾCDDΛ༻͍ͨτϨʔγϯάπʔϧ܈

    View full-size slide

  26. CDDͰͷϓϩάϥϜྫ
    26


    finish_task_switch() kprobe
    Pythonmap
    !
    https://github.com/iovisor/bcc/blob/master/examples/tracing/task_switch.py

    View full-size slide

  27. .PEJGJFE$
    27
    https://github.com/iovisor/bcc/blob/master/examples/tracing/vfsreadlat.c
    &bcc (5BPF.4, modified C
    &BPF map206&-3
    ' 7+19
    &…
    &Clang$%%)5:%#*
    AST

    & eBPF map/8

    &"!%
    https://github.com/iovisor/bcc/blob/master/docs/reference_g
    uide.md

    View full-size slide

  28. CDDDPNQJMFS
    28
    )
    : :
    (

    View full-size slide

  29. πʔϧͱͯ͠ͷCDD
    29
    CDDͷϦϙδτϦʹɼCDDΛར༻ͨ͠τϨʔγϯάπʔϧؚ͕·Ε͍ͯΔ
    IUUQTHJUIVCDPNJPWJTPSCDDUSFFNBTUFSUPPMT
    ओཁEJTUSPʹQBDLBHF͕ଘࡏ
    ɾ6CVOUV
    ɾ'FEPSB
    ɾ"SDI
    ɾ(FOUPP
    ɾPQFO464&
    ɾ3)&-
    IUUQTHJUIVCDPNJPWJTPSCDDCMPCNBTUFS*/45"--NE

    View full-size slide

  30. πʔϧͱͯ͠ͷCDD
    30

    View full-size slide

  31. πʔϧͱͯ͠ͷCDD
    31

    View full-size slide

  32. CDDͷܽ఺ ࢓༷

    32
    ࣮ߦ࣌͝ͱʹίϯύΠϧ͕ൃੜ
    ͨͩ͠ɼBPFʹ͸جຊతʹҾ਺ͷ֓೦͕ͳ͍ͨΊಈతίϯύΠϧ͕ඞཁͳ৔໘͸ଟʑ͋Δ
    ґଘؔ܎͕૿Ճ͢Δ
    ݱঢ়Python3ରԠ͕͍·͍ͪ

    View full-size slide

  33. Complementary
    33

    View full-size slide

  34. ΍ͬͺΓ$Ҏ֎ͰτϨʔγϯά͍ͨ͠
    34
    bpftrace (Dtrace-like ⇨ LLVM ⇨ eBPF)
    https://github.com/iovisor/bpftrace
    ply (Dtrace-like ⇨ eBPF)
    https://github.com/iovisor/ply
    py2bpf (Python byte code ⇨ eBPF)
    https://github.com/facebookresearch/py2bpf

    View full-size slide

  35. (P͔Βͷར༻
    35
    gobpf
    https://github.com/iovisor/gobpf
    github.com/iovisor/gobpf/bcc
    bcc binding (libbcc͕ඞཁ)
    github.com/iovisor/gobpf/elf
    elf loader (elfόΠφϦ͸ࣗ෼ͰίϯύΠϧ͢Δ)

    View full-size slide

  36. ຊ೔આ໌͍ͯ͠ͳ͍͜ͱ
    36
    uprobe (⇔ kprobe)
    USDT (⇔ tracepoint)
    ftrace

    View full-size slide

  37. ·ͱΊ
    ैདྷͷLinux͔Βଘࡏ͢ΔτϨʔγϯάػߏΛBPFͰϓϩάϥϚϥϒϧʹ
    ར༻͢Δ͜ͱ͕Ͱ͖·͢
    bccΛ࢖͏ͱBPFͰͷτϨʔεϓϩάϥϜ࡞੒͕͙ͬͱָʹͳΓ·͢
    bccʹΑͬͯ(BPFͷ͜ͱΛԿ΋஌Βͳͯ͘΋)؆୯ʹBPFʹΑΔτϨʔγϯά
    ͕࣮ߦͰ͖·͢
    Let’s try!
    37

    View full-size slide