Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
BPF/bccによるトレーシング入門 (仮)
Search
mmisono
November 05, 2018
Programming
5
2.1k
BPF/bccによる トレーシング入門 (仮)
2018/11/5 OSSセキュリティ技術の会 第四回勉強会
https://secureoss-sig.connpass.com/event/103763/
mmisono
November 05, 2018
Tweet
Share
More Decks by mmisono
See All by mmisono
Tracing BitVisor with bpftrace
mmisono
0
700
vIOMMU implementation in BitVisor
mmisono
0
530
bitvisor.ko : BitVisor as a module
mmisono
0
770
BPFを利用したBitVisor内部でのパケットフィルタリング (+α)
mmisono
2
980
Other Decks in Programming
See All in Programming
CSC305 Lecture 26
javiergs
PRO
0
140
Stackless и stackful? Корутины и асинхронность в Go
lamodatech
0
700
PHPとAPI Platformで作る本格的なWeb APIアプリケーション(入門編) / phpcon 2024 Intro to API Platform
ttskch
0
160
Итераторы в Go 1.23: зачем они нужны, как использовать, и насколько они быстрые?
lamodatech
0
710
Recoilを剥がしている話
kirik
5
6.6k
talk-with-local-llm-with-web-streams-api
kbaba1001
0
180
LLM Supervised Fine-tuningの理論と実践
datanalyticslabo
3
1.1k
Security_for_introducing_eBPF
kentatada
0
110
htmxって知っていますか?次世代のHTML
hiro_ghap1
0
330
数十万行のプロジェクトを Scala 2から3に完全移行した
xuwei_k
0
270
CSC509 Lecture 14
javiergs
PRO
0
140
SymfonyCon Vienna 2025: Twig, still relevant in 2025?
fabpot
3
1.2k
Featured
See All Featured
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
How to Think Like a Performance Engineer
csswizardry
22
1.2k
Designing Experiences People Love
moore
138
23k
Reflections from 52 weeks, 52 projects
jeffersonlam
347
20k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
28
2.1k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
8
1.2k
Adopting Sorbet at Scale
ufuk
73
9.1k
Building an army of robots
kneath
302
44k
Product Roadmaps are Hard
iamctodd
PRO
49
11k
Site-Speed That Sticks
csswizardry
2
190
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
38
1.9k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
Transcript
#1'CDDʹΑΔ τϨʔγϯάೖ Ծ 2018/11/5 OSS
͜ͷൃදͷ༰ 2 BPFʹΑΔτϨʔγϯάͷ෦ಈ࡞ͷઆ໌͕ओʹͳΓ·͢ɽ ۩ମతͳπʔϧͷ͍ํͷઆ໌ɼτϨʔγϯάͷηΦϦʔɾఆੴͳͲͷ ͋·Γ͋Γ·ͤΜɽ ͜ͷࢿྉͷBPF = eBPFͰ͢
01 02 03 ΞδΣϯμ 3 Linux Tracing ͷ֓ཁ BPFʹΑΔτϨʔγϯά bccʹΑΔτϨʔγϯά
1 Linux Tracing System 4
͡Ίʹ 5 BPFͰͷτϨʔγϯά = º શʹ৽͍͠τϨʔγϯάϑϨʔϜϫʔΫ ̋ طଘͷτϨʔγϯάϑϨʔϜϫʔΫΛิ͏ͷ
-JOVY5SBDJOH4ZTUFN$PNQPOFOU 6 Performance Counter (PMU) Tracepoint (Static Tracing) Kprobe (Dynamic
Tracing) perf_event ftrace Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng *O,FSOFM 'SBNFXPSL 6TFSMBOE5PPM %BUBTPVSDF
-JOVY5SBDJOH4ZTUFN$PNQPOFOU 7 Tracepoint (Static Tracing) Kprobe (Dynamic Tracing) perf_event ftrace
Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng *O,FSOFM 'SBNFXPSL 6TFSMBOE5PPM %BUBTPVSDF Performance Counter (PMU) zzzzzz zzzzzz
σʔλιʔε 8 ɾ$16ݻ༗ͷػೳ ɾ.43ܦ༝ͰใΛऔಘ ɾ*1$ Ωϟογϡώοτ ʜ Performance Counter (PMU)
ɾ4UBUJD5SBDJOH ɾΧʔωϧʹຒΊࠐ·Ε͍ͯΔ ɾ$BMMCBDLؔΛొͰ͖Δ Tracepoint ɾ%ZOBNJD5SBDJOH ɾCSFBLQPJOUʹΑΔ ಈతϑοΫ ɾ$BMMCBDLؔΛొͰ͖Δ Kprobe 1 2 3
1FSGPSNBODF$PVOUFS 1.6 9 ɾ$16ݻ༗ͷػೳ ɾαΠΫϧ *1$ Ωϟογϡώοτ ذ༧ଌώοτ ʜ ɾ*OUFMͷ߹
ɾ.43 .PEFM4QFDJGJD3FHJTUFS ͔Βऔಘ ɾΞʔΩςΫνϟʹΑͬͯdݸఔ ɾͲͷใΛಘ͍͔ͨ.43Ͱઃఆ͢Δ ɾಛఆͷʹୡͨ͠߹ׂΓࠐΈΛൃੜ͢Δػೳ͋Γ ɾ.43ݸҎ্ͷใΛऔಘ͍ͨ͠߹͏·ׂ࣌͘͢Δඞཁ͕͋Δ
5SBDFQPJOU 10 ɾΧʔωϧιʔεதʹఆٛ ) ) ( (( ɾUSBDF@ ͱ͍͏໊લͷఆ͕ٛ͋Ε ͍͍ͩͨ5SBDFQPJOUͷఆٛ
ɾΧʔωϧόʔδϣϯ͕ҟͳͬͯΠϯ λϑΣʔεతͳޓੑ͕͋Δʢͣʣ https://github.com/torvalds/linux/blob/v4.18/fs/exec.c#L1697
,QSPCF 11 Insn Break point pre handler post handler Insn
( ) ɾϒϨʔΫϙΠϯτΛར༻ͨ͠ ಈతϑοΫ ɾΧʔωϧͷେ෦͕ϑοΫՄೳ ɾΧʔωϧόʔδϣϯʹґଘ #
ɾ-JOVYඪ४උͷϓϩϑΝΠϥ ɾΧʔωϧϑϨʔϜϫʔΫ ( ) ( ) ) ɾϢʔβπεϖʔεπʔϧ ( ɾQFSGͰͰ͖Δ͜ͱ
ɾΠϕϯτͷൃੜճͷΧϯτ ( ɾ)BSEXBSF&WFOU 1FSGPSNBODF$PVOUFS ɾ5SBDFQPJOU &WFOU 5SBDFQPJOU ,QSPCF ɾ4PGUXBSF&WFOU QFSGಠࣗͷΠϕϯτ ɾαϯϓϦϯά ( ɾ1.6ͷׂΓࠐΈΛར༻ͨ͠αϯϓϦϯά ҰൠʹαΠΫϧΛར༻ 1FSG DGQFSGGUSBDFͷΈ IUUQNNJIBUFOBCMPHDPNFOUSZ 12 kprobe Performance Counter perf_event tracepoint perf_event_open(2) Hardware Tracepoint perf mmaped ring buffer Software
2 Tracing with BPF
5SBDJOHXJUI#1' Tracepoint Kporbe Perf software event Perf hardware event Event
Call BPF Program Helper Function pid uid … eBPF Map perf buffer
15 bpf(2) system call Create BPF map Kernel Userland BPF
map User Program
16 bpf(2) system call Verifier C source BPF Program JIT
(Optional) Load BPF Program Kernel Userland BPF map LLVM/Clang User Program Event Attach BPF bytecode Tracepoint Kporbe Performane counter
17 bpf(2) system call C source BPF Program Load BPF
Program Kernel Userland BPF map LLVM/Clang User Program Event Call Return value Access BPF bytecode Tracepoint Kporbe Performane counter Call Return value Helper Function
18 bpf(2) system call C source BPF Program Load BPF
Program Kernel Userland BPF map LLVM/Clang User Program Event BPF bytecode Tracepoint Kporbe Performane counter Read BPF map
#1'ϓϩάϥϜͷྫ 19 ɾF#1' NBQ͔Β͜Ε·Ͱͷܭ݁ՌΛऔಘ ɾ݁ՌʹΛͯ͠NBQʹॻ͖͢ Πϕϯτൃੜճͷܭ ɾϖΞͱͳΔؔΛݟ͚ͭΔ FH BMMPDGSFF ɾQSPMPHVFͷؔͰ࣌ࠁΛऔಘɼNBQʹ֨ೲ
ɾFQJMPHVFͷؔͰNBQʹ֨ೲͨ࣌͠ࠁͱͷࠩΛܭࢉ ϨΠςϯγͷଌఆ
#1'ϓϩάϥϜྫ 20 https://github.com/torvalds/linux/blob/v4.18/samples/bpf/tracex3_kern.c ɾCMLJP MBUFODZͷଌఆ * , ( ( (
( ( * , * ,( * ( ( ( *) ( * (
Χʔωϧαϙʔτঢ়گ 21 ػೳ -JOVY7FSTJPO #1'1SPHSBN5ZQF ,QSPCF
6QSPCF 5SBDFQPJOU 1FSGTPGUXBSF IBSEXBSFFWFOU https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md
#1'ͷ࣮ࡍͷར༻ํ๏ 22 Linux Sample https://github.com/torvalds/linux/tree/master/samples/bpf bpf(2) http://man7.org/linux/man-pages/man2/bpf.2.html pef_event_open(2) http://man7.org/linux/man-pages/man2/perf_event_open.2.html
3 Tracing with bcc
#1'ϓϩάϥϜ࡞Ͱେมͳ • υΩϡϝϯτෆɼγεςϜίʔϧͷཧղ͕େม • CQG QFSG@FWFOU@PQFO ͱ͍͏ڧఢ •
$ݴޠͷจ๏ͱͯ͠ؾΛ͚ͭΔ͜ͱ͕ଟʑ͋Δ • FH จࣈྻఆελοΫʹஔ͢Δ • #1'NBQͷऔΓѻ͍ • #1'ϓϩάϥϜʹ#1'NBQͷGJMFEFTDSJQUPSΛຒΊࠐΉඞཁ͕͋Δ • Ұํ$MBOHͰ࡞ͨ͠#1'ϓϩάϥϜ&-'όΠφϦ • &-'όΠφϦΛదʹϩʔυ͢Δϩʔμʔ͕ඞཁ • -JOVYͷαϯϓϧʹଘࡏ͢Δ͕ɼҰൠͷΞϓϦέʔγϣϯ͔Β͍ʹ͍͘
CDD #1'$PNQJMFS$PMMFDUJPO 25 ɾIUUQTHJUIVCDPNJPWJTPSCDD ɾ#1'ϓϩάϥϜ࡞Λαϙʔτ͢ΔͨΊͷϥΠϒϥϦ τϨʔγϯάʹݶఆ͢ΔͷͰͳ͍ ɾ#1'༻NPEJGJFE$ίϯύΠϥ ϩʔμ ɾଞݴޠόΠϯσΟϯά -VB
1ZUIPO (P ˞τϨʔγϯάίʔυࣗମ$Ͱهड़ ɾCDDΛ༻͍ͨτϨʔγϯάπʔϧ܈
CDDͰͷϓϩάϥϜྫ 26 finish_task_switch() kprobe Python map
! https://github.com/iovisor/bcc/blob/master/examples/tracing/task_switch.py
.PEJGJFE$ 27 https://github.com/iovisor/bcc/blob/master/examples/tracing/vfsreadlat.c &bcc (5BPF.4, modified C &BPF map206&-3 &#'
7+19 &… &Clang$%%)5 :%#* AST & eBPF map/8 &"!% https://github.com/iovisor/bcc/blob/master/docs/reference_g uide.md
CDDDPNQJMFS 28 ) : : (
πʔϧͱͯ͠ͷCDD 29 CDDͷϦϙδτϦʹɼCDDΛར༻ͨ͠τϨʔγϯάπʔϧؚ͕·Ε͍ͯΔ IUUQTHJUIVCDPNJPWJTPSCDDUSFFNBTUFSUPPMT ओཁEJTUSPʹQBDLBHF͕ଘࡏ ɾ6CVOUV ɾ'FEPSB ɾ"SDI ɾ(FOUPP ɾPQFO464&
ɾ3)&- IUUQTHJUIVCDPNJPWJTPSCDDCMPCNBTUFS*/45"--NE
πʔϧͱͯ͠ͷCDD 30
πʔϧͱͯ͠ͷCDD 31
CDDͷܽ ༷ 32 ࣮ߦ࣌͝ͱʹίϯύΠϧ͕ൃੜ ͨͩ͠ɼBPFʹجຊతʹҾͷ֓೦͕ͳ͍ͨΊಈతίϯύΠϧ͕ඞཁͳ໘ଟʑ͋Δ ґଘ͕ؔ૿Ճ͢Δ ݱঢ়Python3ରԠ͕͍·͍ͪ
Complementary 33
ͬͺΓ$Ҏ֎ͰτϨʔγϯά͍ͨ͠ 34 bpftrace (Dtrace-like ⇨ LLVM ⇨ eBPF) https://github.com/iovisor/bpftrace ply
(Dtrace-like ⇨ eBPF) https://github.com/iovisor/ply py2bpf (Python byte code ⇨ eBPF) https://github.com/facebookresearch/py2bpf
(P͔Βͷར༻ 35 gobpf https://github.com/iovisor/gobpf github.com/iovisor/gobpf/bcc bcc binding (libbcc͕ඞཁ) github.com/iovisor/gobpf/elf elf
loader (elfόΠφϦࣗͰίϯύΠϧ͢Δ)
ຊઆ໌͍ͯ͠ͳ͍͜ͱ 36 uprobe (⇔ kprobe) USDT (⇔ tracepoint) ftrace
·ͱΊ ैདྷͷLinux͔Βଘࡏ͢ΔτϨʔγϯάػߏΛBPFͰϓϩάϥϚϥϒϧʹ ར༻͢Δ͜ͱ͕Ͱ͖·͢ bccΛ͏ͱBPFͰͷτϨʔεϓϩάϥϜ࡞͕͙ͬͱָʹͳΓ·͢ bccʹΑͬͯ(BPFͷ͜ͱΛԿΒͳͯ͘)؆୯ʹBPFʹΑΔτϨʔγϯά ͕࣮ߦͰ͖·͢ Let’s try! 37
38