Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
BPF/bccによるトレーシング入門 (仮)
Search
mmisono
November 05, 2018
Programming
5
2.1k
BPF/bccによる トレーシング入門 (仮)
2018/11/5 OSSセキュリティ技術の会 第四回勉強会
https://secureoss-sig.connpass.com/event/103763/
mmisono
November 05, 2018
Tweet
Share
More Decks by mmisono
See All by mmisono
Tracing BitVisor with bpftrace
mmisono
0
680
vIOMMU implementation in BitVisor
mmisono
0
530
bitvisor.ko : BitVisor as a module
mmisono
0
750
BPFを利用したBitVisor内部でのパケットフィルタリング (+α)
mmisono
2
970
Other Decks in Programming
See All in Programming
Pinia Colada が実現するスマートな非同期処理
naokihaba
4
220
Ethereum_.pdf
nekomatu
0
460
見せてあげますよ、「本物のLaravel批判」ってやつを。
77web
7
7.7k
Better Code Design in PHP
afilina
PRO
0
130
Hotwire or React? ~アフタートーク・本編に含めなかった話~ / Hotwire or React? after talk
harunatsujita
1
120
Jakarta EE meets AI
ivargrimstad
0
170
リアーキテクチャxDDD 1年間の取り組みと進化
hsawaji
1
220
Enabling DevOps and Team Topologies Through Architecture: Architecting for Fast Flow
cer
PRO
0
330
Duckdb-Wasmでローカルダッシュボードを作ってみた
nkforwork
0
130
CSC509 Lecture 13
javiergs
PRO
0
100
エンジニアとして関わる要件と仕様(公開用)
murabayashi
0
290
Tauriでネイティブアプリを作りたい
tsucchinoko
0
370
Featured
See All Featured
Side Projects
sachag
452
42k
Thoughts on Productivity
jonyablonski
67
4.3k
Optimising Largest Contentful Paint
csswizardry
33
2.9k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
The Language of Interfaces
destraynor
154
24k
The Power of CSS Pseudo Elements
geoffreycrofte
73
5.3k
Designing for humans not robots
tammielis
250
25k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Large-scale JavaScript Application Architecture
addyosmani
510
110k
What's new in Ruby 2.0
geeforr
343
31k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
27
840
Transcript
#1'CDDʹΑΔ τϨʔγϯάೖ Ծ 2018/11/5 OSS
͜ͷൃදͷ༰ 2 BPFʹΑΔτϨʔγϯάͷ෦ಈ࡞ͷઆ໌͕ओʹͳΓ·͢ɽ ۩ମతͳπʔϧͷ͍ํͷઆ໌ɼτϨʔγϯάͷηΦϦʔɾఆੴͳͲͷ ͋·Γ͋Γ·ͤΜɽ ͜ͷࢿྉͷBPF = eBPFͰ͢
01 02 03 ΞδΣϯμ 3 Linux Tracing ͷ֓ཁ BPFʹΑΔτϨʔγϯά bccʹΑΔτϨʔγϯά
1 Linux Tracing System 4
͡Ίʹ 5 BPFͰͷτϨʔγϯά = º શʹ৽͍͠τϨʔγϯάϑϨʔϜϫʔΫ ̋ طଘͷτϨʔγϯάϑϨʔϜϫʔΫΛิ͏ͷ
-JOVY5SBDJOH4ZTUFN$PNQPOFOU 6 Performance Counter (PMU) Tracepoint (Static Tracing) Kprobe (Dynamic
Tracing) perf_event ftrace Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng *O,FSOFM 'SBNFXPSL 6TFSMBOE5PPM %BUBTPVSDF
-JOVY5SBDJOH4ZTUFN$PNQPOFOU 7 Tracepoint (Static Tracing) Kprobe (Dynamic Tracing) perf_event ftrace
Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng *O,FSOFM 'SBNFXPSL 6TFSMBOE5PPM %BUBTPVSDF Performance Counter (PMU) zzzzzz zzzzzz
σʔλιʔε 8 ɾ$16ݻ༗ͷػೳ ɾ.43ܦ༝ͰใΛऔಘ ɾ*1$ Ωϟογϡώοτ ʜ Performance Counter (PMU)
ɾ4UBUJD5SBDJOH ɾΧʔωϧʹຒΊࠐ·Ε͍ͯΔ ɾ$BMMCBDLؔΛొͰ͖Δ Tracepoint ɾ%ZOBNJD5SBDJOH ɾCSFBLQPJOUʹΑΔ ಈతϑοΫ ɾ$BMMCBDLؔΛొͰ͖Δ Kprobe 1 2 3
1FSGPSNBODF$PVOUFS 1.6 9 ɾ$16ݻ༗ͷػೳ ɾαΠΫϧ *1$ Ωϟογϡώοτ ذ༧ଌώοτ ʜ ɾ*OUFMͷ߹
ɾ.43 .PEFM4QFDJGJD3FHJTUFS ͔Βऔಘ ɾΞʔΩςΫνϟʹΑͬͯdݸఔ ɾͲͷใΛಘ͍͔ͨ.43Ͱઃఆ͢Δ ɾಛఆͷʹୡͨ͠߹ׂΓࠐΈΛൃੜ͢Δػೳ͋Γ ɾ.43ݸҎ্ͷใΛऔಘ͍ͨ͠߹͏·ׂ࣌͘͢Δඞཁ͕͋Δ
5SBDFQPJOU 10 ɾΧʔωϧιʔεதʹఆٛ ) ) ( (( ɾUSBDF@ ͱ͍͏໊લͷఆ͕ٛ͋Ε ͍͍ͩͨ5SBDFQPJOUͷఆٛ
ɾΧʔωϧόʔδϣϯ͕ҟͳͬͯΠϯ λϑΣʔεతͳޓੑ͕͋Δʢͣʣ https://github.com/torvalds/linux/blob/v4.18/fs/exec.c#L1697
,QSPCF 11 Insn Break point pre handler post handler Insn
( ) ɾϒϨʔΫϙΠϯτΛར༻ͨ͠ ಈతϑοΫ ɾΧʔωϧͷେ෦͕ϑοΫՄೳ ɾΧʔωϧόʔδϣϯʹґଘ #
ɾ-JOVYඪ४උͷϓϩϑΝΠϥ ɾΧʔωϧϑϨʔϜϫʔΫ ( ) ( ) ) ɾϢʔβπεϖʔεπʔϧ ( ɾQFSGͰͰ͖Δ͜ͱ
ɾΠϕϯτͷൃੜճͷΧϯτ ( ɾ)BSEXBSF&WFOU 1FSGPSNBODF$PVOUFS ɾ5SBDFQPJOU &WFOU 5SBDFQPJOU ,QSPCF ɾ4PGUXBSF&WFOU QFSGಠࣗͷΠϕϯτ ɾαϯϓϦϯά ( ɾ1.6ͷׂΓࠐΈΛར༻ͨ͠αϯϓϦϯά ҰൠʹαΠΫϧΛར༻ 1FSG DGQFSGGUSBDFͷΈ IUUQNNJIBUFOBCMPHDPNFOUSZ 12 kprobe Performance Counter perf_event tracepoint perf_event_open(2) Hardware Tracepoint perf mmaped ring buffer Software
2 Tracing with BPF
5SBDJOHXJUI#1' Tracepoint Kporbe Perf software event Perf hardware event Event
Call BPF Program Helper Function pid uid … eBPF Map perf buffer
15 bpf(2) system call Create BPF map Kernel Userland BPF
map User Program
16 bpf(2) system call Verifier C source BPF Program JIT
(Optional) Load BPF Program Kernel Userland BPF map LLVM/Clang User Program Event Attach BPF bytecode Tracepoint Kporbe Performane counter
17 bpf(2) system call C source BPF Program Load BPF
Program Kernel Userland BPF map LLVM/Clang User Program Event Call Return value Access BPF bytecode Tracepoint Kporbe Performane counter Call Return value Helper Function
18 bpf(2) system call C source BPF Program Load BPF
Program Kernel Userland BPF map LLVM/Clang User Program Event BPF bytecode Tracepoint Kporbe Performane counter Read BPF map
#1'ϓϩάϥϜͷྫ 19 ɾF#1' NBQ͔Β͜Ε·Ͱͷܭ݁ՌΛऔಘ ɾ݁ՌʹΛͯ͠NBQʹॻ͖͢ Πϕϯτൃੜճͷܭ ɾϖΞͱͳΔؔΛݟ͚ͭΔ FH BMMPDGSFF ɾQSPMPHVFͷؔͰ࣌ࠁΛऔಘɼNBQʹ֨ೲ
ɾFQJMPHVFͷؔͰNBQʹ֨ೲͨ࣌͠ࠁͱͷࠩΛܭࢉ ϨΠςϯγͷଌఆ
#1'ϓϩάϥϜྫ 20 https://github.com/torvalds/linux/blob/v4.18/samples/bpf/tracex3_kern.c ɾCMLJP MBUFODZͷଌఆ * , ( ( (
( ( * , * ,( * ( ( ( *) ( * (
Χʔωϧαϙʔτঢ়گ 21 ػೳ -JOVY7FSTJPO #1'1SPHSBN5ZQF ,QSPCF
6QSPCF 5SBDFQPJOU 1FSGTPGUXBSF IBSEXBSFFWFOU https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md
#1'ͷ࣮ࡍͷར༻ํ๏ 22 Linux Sample https://github.com/torvalds/linux/tree/master/samples/bpf bpf(2) http://man7.org/linux/man-pages/man2/bpf.2.html pef_event_open(2) http://man7.org/linux/man-pages/man2/perf_event_open.2.html
3 Tracing with bcc
#1'ϓϩάϥϜ࡞Ͱେมͳ • υΩϡϝϯτෆɼγεςϜίʔϧͷཧղ͕େม • CQG QFSG@FWFOU@PQFO ͱ͍͏ڧఢ •
$ݴޠͷจ๏ͱͯ͠ؾΛ͚ͭΔ͜ͱ͕ଟʑ͋Δ • FH จࣈྻఆελοΫʹஔ͢Δ • #1'NBQͷऔΓѻ͍ • #1'ϓϩάϥϜʹ#1'NBQͷGJMFEFTDSJQUPSΛຒΊࠐΉඞཁ͕͋Δ • Ұํ$MBOHͰ࡞ͨ͠#1'ϓϩάϥϜ&-'όΠφϦ • &-'όΠφϦΛదʹϩʔυ͢Δϩʔμʔ͕ඞཁ • -JOVYͷαϯϓϧʹଘࡏ͢Δ͕ɼҰൠͷΞϓϦέʔγϣϯ͔Β͍ʹ͍͘
CDD #1'$PNQJMFS$PMMFDUJPO 25 ɾIUUQTHJUIVCDPNJPWJTPSCDD ɾ#1'ϓϩάϥϜ࡞Λαϙʔτ͢ΔͨΊͷϥΠϒϥϦ τϨʔγϯάʹݶఆ͢ΔͷͰͳ͍ ɾ#1'༻NPEJGJFE$ίϯύΠϥ ϩʔμ ɾଞݴޠόΠϯσΟϯά -VB
1ZUIPO (P ˞τϨʔγϯάίʔυࣗମ$Ͱهड़ ɾCDDΛ༻͍ͨτϨʔγϯάπʔϧ܈
CDDͰͷϓϩάϥϜྫ 26 finish_task_switch() kprobe Python map
! https://github.com/iovisor/bcc/blob/master/examples/tracing/task_switch.py
.PEJGJFE$ 27 https://github.com/iovisor/bcc/blob/master/examples/tracing/vfsreadlat.c &bcc (5BPF.4, modified C &BPF map206&-3 &#'
7+19 &… &Clang$%%)5 :%#* AST & eBPF map/8 &"!% https://github.com/iovisor/bcc/blob/master/docs/reference_g uide.md
CDDDPNQJMFS 28 ) : : (
πʔϧͱͯ͠ͷCDD 29 CDDͷϦϙδτϦʹɼCDDΛར༻ͨ͠τϨʔγϯάπʔϧؚ͕·Ε͍ͯΔ IUUQTHJUIVCDPNJPWJTPSCDDUSFFNBTUFSUPPMT ओཁEJTUSPʹQBDLBHF͕ଘࡏ ɾ6CVOUV ɾ'FEPSB ɾ"SDI ɾ(FOUPP ɾPQFO464&
ɾ3)&- IUUQTHJUIVCDPNJPWJTPSCDDCMPCNBTUFS*/45"--NE
πʔϧͱͯ͠ͷCDD 30
πʔϧͱͯ͠ͷCDD 31
CDDͷܽ ༷ 32 ࣮ߦ࣌͝ͱʹίϯύΠϧ͕ൃੜ ͨͩ͠ɼBPFʹجຊతʹҾͷ֓೦͕ͳ͍ͨΊಈతίϯύΠϧ͕ඞཁͳ໘ଟʑ͋Δ ґଘ͕ؔ૿Ճ͢Δ ݱঢ়Python3ରԠ͕͍·͍ͪ
Complementary 33
ͬͺΓ$Ҏ֎ͰτϨʔγϯά͍ͨ͠ 34 bpftrace (Dtrace-like ⇨ LLVM ⇨ eBPF) https://github.com/iovisor/bpftrace ply
(Dtrace-like ⇨ eBPF) https://github.com/iovisor/ply py2bpf (Python byte code ⇨ eBPF) https://github.com/facebookresearch/py2bpf
(P͔Βͷར༻ 35 gobpf https://github.com/iovisor/gobpf github.com/iovisor/gobpf/bcc bcc binding (libbcc͕ඞཁ) github.com/iovisor/gobpf/elf elf
loader (elfόΠφϦࣗͰίϯύΠϧ͢Δ)
ຊઆ໌͍ͯ͠ͳ͍͜ͱ 36 uprobe (⇔ kprobe) USDT (⇔ tracepoint) ftrace
·ͱΊ ैདྷͷLinux͔Βଘࡏ͢ΔτϨʔγϯάػߏΛBPFͰϓϩάϥϚϥϒϧʹ ར༻͢Δ͜ͱ͕Ͱ͖·͢ bccΛ͏ͱBPFͰͷτϨʔεϓϩάϥϜ࡞͕͙ͬͱָʹͳΓ·͢ bccʹΑͬͯ(BPFͷ͜ͱΛԿΒͳͯ͘)؆୯ʹBPFʹΑΔτϨʔγϯά ͕࣮ߦͰ͖·͢ Let’s try! 37
38