Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
BPF/bccによるトレーシング入門 (仮)
Search
mmisono
November 05, 2018
Programming
5
2.1k
BPF/bccによる トレーシング入門 (仮)
2018/11/5 OSSセキュリティ技術の会 第四回勉強会
https://secureoss-sig.connpass.com/event/103763/
mmisono
November 05, 2018
Tweet
Share
More Decks by mmisono
See All by mmisono
Tracing BitVisor with bpftrace
mmisono
0
680
vIOMMU implementation in BitVisor
mmisono
0
520
bitvisor.ko : BitVisor as a module
mmisono
0
740
BPFを利用したBitVisor内部でのパケットフィルタリング (+α)
mmisono
2
960
Other Decks in Programming
See All in Programming
Piniaの現状と今後
waka292
5
1.4k
Vitest Browser Mode への期待 / Vitest Browser Mode
odanado
PRO
2
1.7k
Mastering Dependencies in Kotlin Multiplatform
tomifabian
0
110
Modern Angular: Renovation for Your Applications
manfredsteyer
PRO
0
200
Synchronizationを支える技術
s_shimotori
1
150
Kaigi on Rails 2024 - Rails APIモードのためのシンプルで効果的なCSRF対策 / kaigionrails-2024-csrf
corocn
5
3.3k
現場で役立つモデリング 超入門
masuda220
PRO
12
2.9k
プロジェクト新規参入者のリードタイム短縮の観点から見る、品質の高いコードとアーキテクチャを保つメリット
d_endo
1
1k
OpenTelemetryでRailsのパフォーマンス分析を始めてみよう(KoR2024)
ymtdzzz
4
1.5k
飲食業界向けマルチプロダクトを実現させる開発体制とリアルな現状
hiroya0601
1
390
約9000個の自動テストの 時間を50分->10分に短縮 Flakyテストを1%以下に抑えた話
hatsu38
23
10k
カラム追加で増えるActiveRecordのメモリサイズ イメージできますか?
asayamakk
4
1.5k
Featured
See All Featured
Testing 201, or: Great Expectations
jmmastey
38
7k
Designing for Performance
lara
604
68k
The World Runs on Bad Software
bkeepers
PRO
65
11k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
250
21k
It's Worth the Effort
3n
183
27k
Rebuilding a faster, lazier Slack
samanthasiow
79
8.6k
Practical Orchestrator
shlominoach
186
10k
5 minutes of I Can Smell Your CMS
philhawksworth
202
19k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
167
49k
No one is an island. Learnings from fostering a developers community.
thoeni
19
3k
VelocityConf: Rendering Performance Case Studies
addyosmani
325
24k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
41
2.1k
Transcript
#1'CDDʹΑΔ τϨʔγϯάೖ Ծ 2018/11/5 OSS
͜ͷൃදͷ༰ 2 BPFʹΑΔτϨʔγϯάͷ෦ಈ࡞ͷઆ໌͕ओʹͳΓ·͢ɽ ۩ମతͳπʔϧͷ͍ํͷઆ໌ɼτϨʔγϯάͷηΦϦʔɾఆੴͳͲͷ ͋·Γ͋Γ·ͤΜɽ ͜ͷࢿྉͷBPF = eBPFͰ͢
01 02 03 ΞδΣϯμ 3 Linux Tracing ͷ֓ཁ BPFʹΑΔτϨʔγϯά bccʹΑΔτϨʔγϯά
1 Linux Tracing System 4
͡Ίʹ 5 BPFͰͷτϨʔγϯά = º શʹ৽͍͠τϨʔγϯάϑϨʔϜϫʔΫ ̋ طଘͷτϨʔγϯάϑϨʔϜϫʔΫΛิ͏ͷ
-JOVY5SBDJOH4ZTUFN$PNQPOFOU 6 Performance Counter (PMU) Tracepoint (Static Tracing) Kprobe (Dynamic
Tracing) perf_event ftrace Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng *O,FSOFM 'SBNFXPSL 6TFSMBOE5PPM %BUBTPVSDF
-JOVY5SBDJOH4ZTUFN$PNQPOFOU 7 Tracepoint (Static Tracing) Kprobe (Dynamic Tracing) perf_event ftrace
Lttng SystemTap Mcount (gprof) perf tracefs (debugfs) trace-cmd SystemTap Lttng *O,FSOFM 'SBNFXPSL 6TFSMBOE5PPM %BUBTPVSDF Performance Counter (PMU) zzzzzz zzzzzz
σʔλιʔε 8 ɾ$16ݻ༗ͷػೳ ɾ.43ܦ༝ͰใΛऔಘ ɾ*1$ Ωϟογϡώοτ ʜ Performance Counter (PMU)
ɾ4UBUJD5SBDJOH ɾΧʔωϧʹຒΊࠐ·Ε͍ͯΔ ɾ$BMMCBDLؔΛొͰ͖Δ Tracepoint ɾ%ZOBNJD5SBDJOH ɾCSFBLQPJOUʹΑΔ ಈతϑοΫ ɾ$BMMCBDLؔΛొͰ͖Δ Kprobe 1 2 3
1FSGPSNBODF$PVOUFS 1.6 9 ɾ$16ݻ༗ͷػೳ ɾαΠΫϧ *1$ Ωϟογϡώοτ ذ༧ଌώοτ ʜ ɾ*OUFMͷ߹
ɾ.43 .PEFM4QFDJGJD3FHJTUFS ͔Βऔಘ ɾΞʔΩςΫνϟʹΑͬͯdݸఔ ɾͲͷใΛಘ͍͔ͨ.43Ͱઃఆ͢Δ ɾಛఆͷʹୡͨ͠߹ׂΓࠐΈΛൃੜ͢Δػೳ͋Γ ɾ.43ݸҎ্ͷใΛऔಘ͍ͨ͠߹͏·ׂ࣌͘͢Δඞཁ͕͋Δ
5SBDFQPJOU 10 ɾΧʔωϧιʔεதʹఆٛ ) ) ( (( ɾUSBDF@ ͱ͍͏໊લͷఆ͕ٛ͋Ε ͍͍ͩͨ5SBDFQPJOUͷఆٛ
ɾΧʔωϧόʔδϣϯ͕ҟͳͬͯΠϯ λϑΣʔεతͳޓੑ͕͋Δʢͣʣ https://github.com/torvalds/linux/blob/v4.18/fs/exec.c#L1697
,QSPCF 11 Insn Break point pre handler post handler Insn
( ) ɾϒϨʔΫϙΠϯτΛར༻ͨ͠ ಈతϑοΫ ɾΧʔωϧͷେ෦͕ϑοΫՄೳ ɾΧʔωϧόʔδϣϯʹґଘ #
ɾ-JOVYඪ४උͷϓϩϑΝΠϥ ɾΧʔωϧϑϨʔϜϫʔΫ ( ) ( ) ) ɾϢʔβπεϖʔεπʔϧ ( ɾQFSGͰͰ͖Δ͜ͱ
ɾΠϕϯτͷൃੜճͷΧϯτ ( ɾ)BSEXBSF&WFOU 1FSGPSNBODF$PVOUFS ɾ5SBDFQPJOU &WFOU 5SBDFQPJOU ,QSPCF ɾ4PGUXBSF&WFOU QFSGಠࣗͷΠϕϯτ ɾαϯϓϦϯά ( ɾ1.6ͷׂΓࠐΈΛར༻ͨ͠αϯϓϦϯά ҰൠʹαΠΫϧΛར༻ 1FSG DGQFSGGUSBDFͷΈ IUUQNNJIBUFOBCMPHDPNFOUSZ 12 kprobe Performance Counter perf_event tracepoint perf_event_open(2) Hardware Tracepoint perf mmaped ring buffer Software
2 Tracing with BPF
5SBDJOHXJUI#1' Tracepoint Kporbe Perf software event Perf hardware event Event
Call BPF Program Helper Function pid uid … eBPF Map perf buffer
15 bpf(2) system call Create BPF map Kernel Userland BPF
map User Program
16 bpf(2) system call Verifier C source BPF Program JIT
(Optional) Load BPF Program Kernel Userland BPF map LLVM/Clang User Program Event Attach BPF bytecode Tracepoint Kporbe Performane counter
17 bpf(2) system call C source BPF Program Load BPF
Program Kernel Userland BPF map LLVM/Clang User Program Event Call Return value Access BPF bytecode Tracepoint Kporbe Performane counter Call Return value Helper Function
18 bpf(2) system call C source BPF Program Load BPF
Program Kernel Userland BPF map LLVM/Clang User Program Event BPF bytecode Tracepoint Kporbe Performane counter Read BPF map
#1'ϓϩάϥϜͷྫ 19 ɾF#1' NBQ͔Β͜Ε·Ͱͷܭ݁ՌΛऔಘ ɾ݁ՌʹΛͯ͠NBQʹॻ͖͢ Πϕϯτൃੜճͷܭ ɾϖΞͱͳΔؔΛݟ͚ͭΔ FH BMMPDGSFF ɾQSPMPHVFͷؔͰ࣌ࠁΛऔಘɼNBQʹ֨ೲ
ɾFQJMPHVFͷؔͰNBQʹ֨ೲͨ࣌͠ࠁͱͷࠩΛܭࢉ ϨΠςϯγͷଌఆ
#1'ϓϩάϥϜྫ 20 https://github.com/torvalds/linux/blob/v4.18/samples/bpf/tracex3_kern.c ɾCMLJP MBUFODZͷଌఆ * , ( ( (
( ( * , * ,( * ( ( ( *) ( * (
Χʔωϧαϙʔτঢ়گ 21 ػೳ -JOVY7FSTJPO #1'1SPHSBN5ZQF ,QSPCF
6QSPCF 5SBDFQPJOU 1FSGTPGUXBSF IBSEXBSFFWFOU https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md
#1'ͷ࣮ࡍͷར༻ํ๏ 22 Linux Sample https://github.com/torvalds/linux/tree/master/samples/bpf bpf(2) http://man7.org/linux/man-pages/man2/bpf.2.html pef_event_open(2) http://man7.org/linux/man-pages/man2/perf_event_open.2.html
3 Tracing with bcc
#1'ϓϩάϥϜ࡞Ͱେมͳ • υΩϡϝϯτෆɼγεςϜίʔϧͷཧղ͕େม • CQG QFSG@FWFOU@PQFO ͱ͍͏ڧఢ •
$ݴޠͷจ๏ͱͯ͠ؾΛ͚ͭΔ͜ͱ͕ଟʑ͋Δ • FH จࣈྻఆελοΫʹஔ͢Δ • #1'NBQͷऔΓѻ͍ • #1'ϓϩάϥϜʹ#1'NBQͷGJMFEFTDSJQUPSΛຒΊࠐΉඞཁ͕͋Δ • Ұํ$MBOHͰ࡞ͨ͠#1'ϓϩάϥϜ&-'όΠφϦ • &-'όΠφϦΛదʹϩʔυ͢Δϩʔμʔ͕ඞཁ • -JOVYͷαϯϓϧʹଘࡏ͢Δ͕ɼҰൠͷΞϓϦέʔγϣϯ͔Β͍ʹ͍͘
CDD #1'$PNQJMFS$PMMFDUJPO 25 ɾIUUQTHJUIVCDPNJPWJTPSCDD ɾ#1'ϓϩάϥϜ࡞Λαϙʔτ͢ΔͨΊͷϥΠϒϥϦ τϨʔγϯάʹݶఆ͢ΔͷͰͳ͍ ɾ#1'༻NPEJGJFE$ίϯύΠϥ ϩʔμ ɾଞݴޠόΠϯσΟϯά -VB
1ZUIPO (P ˞τϨʔγϯάίʔυࣗମ$Ͱهड़ ɾCDDΛ༻͍ͨτϨʔγϯάπʔϧ܈
CDDͰͷϓϩάϥϜྫ 26 finish_task_switch() kprobe Python map
! https://github.com/iovisor/bcc/blob/master/examples/tracing/task_switch.py
.PEJGJFE$ 27 https://github.com/iovisor/bcc/blob/master/examples/tracing/vfsreadlat.c &bcc (5BPF.4, modified C &BPF map206&-3 &#'
7+19 &… &Clang$%%)5 :%#* AST & eBPF map/8 &"!% https://github.com/iovisor/bcc/blob/master/docs/reference_g uide.md
CDDDPNQJMFS 28 ) : : (
πʔϧͱͯ͠ͷCDD 29 CDDͷϦϙδτϦʹɼCDDΛར༻ͨ͠τϨʔγϯάπʔϧؚ͕·Ε͍ͯΔ IUUQTHJUIVCDPNJPWJTPSCDDUSFFNBTUFSUPPMT ओཁEJTUSPʹQBDLBHF͕ଘࡏ ɾ6CVOUV ɾ'FEPSB ɾ"SDI ɾ(FOUPP ɾPQFO464&
ɾ3)&- IUUQTHJUIVCDPNJPWJTPSCDDCMPCNBTUFS*/45"--NE
πʔϧͱͯ͠ͷCDD 30
πʔϧͱͯ͠ͷCDD 31
CDDͷܽ ༷ 32 ࣮ߦ࣌͝ͱʹίϯύΠϧ͕ൃੜ ͨͩ͠ɼBPFʹجຊతʹҾͷ֓೦͕ͳ͍ͨΊಈతίϯύΠϧ͕ඞཁͳ໘ଟʑ͋Δ ґଘ͕ؔ૿Ճ͢Δ ݱঢ়Python3ରԠ͕͍·͍ͪ
Complementary 33
ͬͺΓ$Ҏ֎ͰτϨʔγϯά͍ͨ͠ 34 bpftrace (Dtrace-like ⇨ LLVM ⇨ eBPF) https://github.com/iovisor/bpftrace ply
(Dtrace-like ⇨ eBPF) https://github.com/iovisor/ply py2bpf (Python byte code ⇨ eBPF) https://github.com/facebookresearch/py2bpf
(P͔Βͷར༻ 35 gobpf https://github.com/iovisor/gobpf github.com/iovisor/gobpf/bcc bcc binding (libbcc͕ඞཁ) github.com/iovisor/gobpf/elf elf
loader (elfόΠφϦࣗͰίϯύΠϧ͢Δ)
ຊઆ໌͍ͯ͠ͳ͍͜ͱ 36 uprobe (⇔ kprobe) USDT (⇔ tracepoint) ftrace
·ͱΊ ैདྷͷLinux͔Βଘࡏ͢ΔτϨʔγϯάػߏΛBPFͰϓϩάϥϚϥϒϧʹ ར༻͢Δ͜ͱ͕Ͱ͖·͢ bccΛ͏ͱBPFͰͷτϨʔεϓϩάϥϜ࡞͕͙ͬͱָʹͳΓ·͢ bccʹΑͬͯ(BPFͷ͜ͱΛԿΒͳͯ͘)؆୯ʹBPFʹΑΔτϨʔγϯά ͕࣮ߦͰ͖·͢ Let’s try! 37
38