SREのためのeBPF活用ステップアップガイド

Transcript

1. SREͷͨΊͷeBPF׆༻ εςοϓΞοϓΨΠυ 2025/07/12 SRE NEXT 2025 Sohei Iwahori (GREE, Inc.)
2. who? » Sohei Iwahori (@egmc) » גࣜձࣾάϦʔ ΠϯϑϥετϥΫνϟ෦ γχΞϦʔυΤϯδχΞ » Πϯϑϥͱ؂ࢹγεςϜ » SRE NEXT 2025 Co-Chair » eBPF Japan MeetupӡӦ
3. ࠓ·ͰͷSRE NEXT
4. ·͓͖͑ » έʔεͷ౎߹ͰʢίϯςφͰ͸ͳ͘ʣVM؀ڥʢUbuntuʣɺPHPͷ࿩͕ଟΊ ʹͳΓ·͢ » RubyɺPythonͳͲ͸େମಉ͡Α͏ʹͰ͖Δͱࢥ͍·͢ » ڞ༗ϥΠϒϥϦΛτϨʔε͢Δ࡞๏͸΄΅ಉ͡ » Go͸ͪΐͬͱؤுΔඞཁ͕͋Γɾɾ » AppendixʹϙΠϯλ͓͖·͢ !
5. ΞδΣϯμ » eBPFͷ֓ཁ » SRE͕௚઀eBPFΛѻ͏ϝϦοτ » ར༻͢Δ্Ͱͷݱ࣮తͳ೉͠͞ » ಋೖεςοϓ » Step1 طଘͷπʔϧΛར༻͢Δ » Step2 bpftraceΛར༻͢Δ » Step3 eBPFϓϩάϥϜΛॻ͍ͯར༻͢Δ » Recap
6. eBPFͷ֓ཁ
7. eBPFͷ֓ཁ(1/2) eBPFͱ͸ʁ1 1 eBPFͱ͸ https://ebpf.io/ja/what-is-ebpf/.
8. eBPFͷ֓ཁ(2/2) eBPFͰ࢖͑ΔProbe Types2 2 bpftrace https://github.com/bpftrace/bpftrace.
9. ࢖ΘΕ͍ͯΔͱ͜ΖʢObservabilityؔ࿈ʣ » Pixie » OpenTelemetry(opentelemetry-go-instrumentationͳͲ) » Pyroscope » Grafana Beyla » ͦͷଞ঎༻੡඼ͳͲ » τϨʔεɺϓϩϑΝΠϥͳͲͷιϦϡʔγϣϯͰར༻͞Ε͍ͯΔ
10. SRE͕௚઀eBPFΛѻ͏ϝϦοτ
11. SRE͕௚઀eBPFΛѻ͏ϝϦοτ » طଘͷπʔϧʢϕϯμ੡඼ɺOSSʣΛར༻ͯ͠՝୊ղܾͰ͖͍ͯΕ͹Ϥγ » طଘͷπʔϧͰख͕ಧ͔ͳ͍෦෼ʹ՝୊͕͋ͬͨ৔߹ » SRE͕eBPFΛ௚઀ѻ͏͜ͱͰΑΓϐϯϙΠϯτͳ՝୊ղܾ͕Ͱ͖Δ » ୹ظతͳ՝୊ʹରͯ͠͸bpftraceͳͲͰΦϯσϚϯυʹ » ௕ظతͳ՝୊ʹରͯ͠͸ઐ༻ͷπʔϧΛ࡞ΔͳͲͷΞϓϩʔνͰ
12. Αͦ͞͏Ͱ͢Ͷʁ
13. ར༻͢Δ্Ͱͷ ݱ࣮తͳ೉͠͞
14. Ή͔ͣ͠͞ » ґଘղܾ໰୊ » ίʔυΛॻ͘೉͠͞ » ৘ใɺར༻ऀͷগͳ͞
15. ґଘղܾ໰୊ » ެࣜͷఏڙύοέʔδ͕ݹ͘BCCπʔϧɺbpftraceͳͲ͕ಈ͔ͳ͍ » BCCΛϏϧυ͠Α͏ͱ͢ΔͱOSόʔδϣϯ͝ͱʹґଘύοέʔδʢClangɺLLVMͳ ͲʣΛࢦఆ͍ͯ͠ΕΔඞཁ͕͋Γେม » ͦͷଞ৔߹ʹΑΓσόοάγϯϘϧͷύοέʔδͳͲ΋ඞཁ » ௕ظతʹ҆ఆͯ͠ಈ͔ͤΔ؀ڥΛOSόʔδϣϯΛ·͍ͨͰҡ࣋͢Δ͚ͩͰ΋େม » 2022͘Β͍͔Βঢ়گ͸Α͘ͳ͖͍ͬͯͯΔͷͱΧʔωϧͷΠϕϯτ͸CO-REΞϓϩʔ νʹΑΓ͋Δఔ౓ҟͳΔΧʔωϧόʔδϣϯͰಈ࡞͢Δπʔϧ͕࡞ΕΔΑ͏ʹͳͬͨ
16. ίʔυΛॻ͘೉͠͞ » eBPFͷίʔυ͸ओʹCݴޠͬΆ͍΋ͷͰॻ͔Ε͍ͯΔ » Χʔωϧ಺Ͱಈ࡞͢ΔͨΊ੍໿΋ଟ͍ʢϧʔϓͷ੍ݶɺϝϞϦΞΫηεͳ Ͳʣ » ϔϧύؔ਺΍ϚΫϩʹΑͬͯ͋Δఔ౓ఆܕԽ͞Ε͍ͯΔ » ޙ൒Ͱσόοάํ๏ʹ͍ͭͯ࿩͠·͢
17. ৘ใɺར༻ऀͷগͳ͞(੩తϓϩʔϒͷ໰୊) » eBPFͰ͸USDTͱ͍͏ϢʔβʔۭؒͰಈ࡞͢ΔϓϩάϥϜͷ੩తͳτϨʔε ϙΠϯτ͕αϙʔτ͞Ε͍ͯ·͢ » Python/Ruby/PHPͳͲͷݴޠϥϯλΠϜɺMySQLͳͲͰ΋ར༻Մೳ » όʔδϣϯΛ·͍ͨͰ௕ظతʹར༻Ͱ͖ΔͷͰʢupbobeʹൺ΂ͯʣπʔϧ΁ ૊ΈࠐΈ΍͍͢ » ͔͠͠όΠφϦ͕ --enable-dtrace ͳͲͰDTraceΛ༗ޮʹϏϧυ͞Ε͍ͯ Δඞཁ͕͋Γɺ͜ͷ͋ͨΓ͸σΟετϦϏϡʔγϣϯґଘ
18. ৘ใɺར༻ऀͷগͳ͞(੩తϓϩʔϒͷ໰୊) » ubuntuͰఏڙ͞ΕΔެࣜͷPHPύοέʔδͰ͸جຊతʹ͸DTrace͸༗ޮԽ͞Ε͍ͯΔ » ͔͠͠24.04(Noble)ͷύοέʔδͰ͸disable͞Ε͍ͯͨ͠ » ౰࣌ݩͷdebianύοέʔδଆͰϏϧυʹࣦഊ͢Δ໰୊͕͋ͬͨΒ͍͠ » bug reportΛ͋͛ͨ݁Ռ 25.04(Plucky)͔Β͸࠶౓༗ޮԽͯ͠΋Β͑ͨ3 » ·ͨɺPHPͰ͸ USE_ZEND_DTRACE=1 ͱ͍͏؀ڥม਺Λର৅ϓϩηεʹηοτ͢Δඞཁ͕͋Δ͕υΩϡϝϯτʹॻ͔Ε͍ͯ ͳ͔ͬͨ4 » ਓʑͷؔ৺͕େࣄ 4 https://github.com/php/doc-en/pull/4456 3 https://bugs.launchpad.net/ubuntu/+source/php8.3/+bug/2088977
19. ৘ใɺར༻ऀͷগͳ͞(AI͸ʁ) » ChatGPT4oར༻࣌Ͱ͸eBPFͷίʔυͷਫ਼౓͸ମײ40%͘Β͍ͩͬͨ » Claude Sonnet 4 / Opus4͸݁ߏॻ͍ͯ͘ΕΔ » ͱ͸͍͑ʮͲ͜·Ͱ͕ఆܕͷίʔυ͔Θ͔Βͳ͍ʯΈ͍ͨͳͱ͜Ζ͸͋Δͱ ࢥ͏ͷͰυΩϡϝϯτͱαϯϓϧ͸͋Δఔ౓ಡΉͱྑ͍
20. ͱ͍͏͜ͱͰɾɾ » ศརͳeBPFͰ͕͢೉͠͞΋͋ΔͷͰ » SRE͕ಋೖ͍ͯͨ͘͠Ίͷஈ֊తͳεςοϓΛɺ۩ମతͳ՝୊ղܾͷྫͱͱ ΋ʹݟ͍͖ͯ·͠ΐ͏
21. ಋೖεςοϓ
22. Step1 طଘͷπʔϧΛར༻͢Δ killsnoopͷྫ ϫʔΧʔಥવࢮ໰୊
23. ՝୊ɿϓϩηεͷಥવࢮ » RubyͷShoryukenΛར༻ͨ͠Ξϥʔτ௨஌γεςϜ » Քಇ8೥ఔ౓ɺʢsystemd؅ཧͰ͸ͳ͘ʣૉ๿ͳ stop/start༻ͷγΣϧεΫϦϓτͰಈ࡞͍ͯͨ͠ » Shoryukenʹ͸Process.daemonΛ࢖ͬͯ σʔϞφΠζ͢ΔΦϓγϣϯ͕͋Δ » ىಈ࣌ʹpidΛه࿥ͯ͠ର৅ʹkill͢ΔΑ͏ͳ εΫϦϓτ » ͋Δ࣌OSόʔδϣϯΞοϓΛߦͬͨΒɺ0࣌ʹಥવ͢ ΂ͯͷϫʔΧʔ͕μ΢ϯ » ޾͍ΦʔτεέʔϧʹΑΓ਺෼ޙʹ͸෮چͨ͠ ͕ɾɾ
24. ՝୊2 » ϩάΛΈΔͱͲ͏΋logrotate͕͋΍͍͠ » logrotate࣌ʹਖ਼ৗʹstop/start͍ͯ͠Δ͕ɺه࿥ΛΈΔͱstart௚ޙʹ SIGTERMΛड͚ͯࡴ͞Ε͍ͯΔ » SIGTERMΛʮ୭͕ʯૹ͖͍ͬͯͯΔͷΛ஌Γ͍͕ͨɺRubyଆͷSignal.trap Ͱ͸ૹ৴ݩͷpidͳͲ͸ͱΕͳ͍ » eBPF ͳΒγεςϜίʔϧͷτϨʔεϙΠϯτ͔ΒύϥϝʔλʹΞΫηε͢Δ ͜ͱ͕Ͱ͖Δ
25. BPFπʔϧͰ΍ͬͯΈΔ » killsnoopɺkillsnoop.bt5ͱ͍͏ͦͷ΋ͷͣ͹Γͳπʔϧ͕͋Δ » killsnoop͸BCC൛ɺkillsnoop.bt͸bpftrace൛ » OSύοέʔδͰఏڙ͞Ε͍ͯΔBCC൛͕ݹ͘ಈ࡞͠ͳ͔ͬͨͷͰkillsnoop.btΛ࢖ͬͨ » ૹ৴ݩϓϩηεͷIDɺγάφϧͳͲΛϦΞϧλΠϜʹදࣔͯ͘͠ΕΔ » τϨʔεϙΠϯτ͸Χʔωϧ಺Ͱఆٛ͞Ε͍ͯͯ6ɺsyscalls.h಺Ͱఆٛ͞Ε͍ͯΔϚΫϩ7Λܦ༝ͯ͠ύϥϝʔλ Λड͚औΕΔτϨʔεϙΠϯτΛࣗಈͰఆٛͯ͘͠ΕΔ 7 https://github.com/torvalds/linux/blob/master/include/linux/syscalls.h#L225C9-L225C24 6 https://github.com/torvalds/linux/blob/66701750d5565c574af42bef0b789ce0203e3071/kernel/signal.c#L3944-L3958 5 https://github.com/bpftrace/bpftrace/blob/master/tools/killsnoop.bt
26. ࣮ߦ݁Ռ » 0࣌෇ۙͰൃੜ͢Δ͜ͱ͸Θ͔ͬ ͍ͯͨͷͰɺࡶʹcronͰಈ͔ͯ͠ ϩάΛϑΝΠϧʹॻ͖ग़ͯ͠Έͨ » pid1ʢsystemdʣ͕kill͍ͯ͠Δ » ࢖ͬͯͳ͍͸͕ͣͩɾɾʁ killsnoop.bt 00:00:01 466297 kill 10 439956 0 00:00:04 1 systemd 15 466335 0 00:00:04 1 systemd 18 466335 0 logrotate 00:00:01 xxx COMMAND=/usr/bin/kill -USR1 439956
27. ͜ΕΛ౿·͑ͯͷ݁࿦ » OSόʔδϣϯΞοϓͰlogrotateͷ࣮ߦ͕cron->systemd.timerͷ࣮ߦʹมΘ͍ͬͯͨ » systemd.timer͸Type=oneshotͷαʔϏεͱͯ͠ىಈ͢Δ » Type=oneshotͷαʔϏε͸࣮ߦऴྃ࣌ʹunitͷcgroupʹॴଐ͢Δ͢΂ͯͷϓϩηε͕ kill͞ΕΔσϑΥϧτϞʔυͰಈ͘ » Process.daemon͸σʔϞφΠζʢϓϩηεάϧʔϓͷ੾Γ཭͠ʣ͸΍ͬͯ͘ΕΔ͕ɺ cgourp͸ಛʹԿ΋͠ͳ͍ » ݁Ռ࠶ىಈ࣌ʹlogrotate unit͔Βىಈ͞ΕͨϫʔΧʔ͸SIGTERMͷૹग़ର৅ʹͳΓKILL ͞Εͨ
28. ղܾ » ૉ๿ͳstart/stopεΫϦϓτΛ΍ΊϫʔΧʔͦͷ΋ͷsystemd؅ཧͱ͢Δ͜ͱ Ͱղܾ » systemdͦͷ΋ͷͷػೳʹΑΓεΫϦϓτͷେ൒ͷػೳ͸࿫͑ΔΑ͏ʹ » xx snoopܥ͸ͦͷଞ৭ʑศརίϚϯυ͕༻ҙ͞Εͯ·͢ » execsnoop » opensnoop
29. Step2 bpftraceΛར༻͢Δ ෛՙࢼݧʹ͓͚ΔMemcachedͷ ߴτϥϑΟοΫݪҼௐࠪ
30. bpftraceʹ͍ͭͯ » awkͬΆ͍ॻ͖ํͰ೚ҙͷτϨʔε͕࢖͑Δ » ϫϯϥΠφʔͰ࢖ͬͨΓɺεΫϦϓτΛϑΝΠϧͰ༻ҙ͓͖࣮ͯ͠ߦ ͨ͠Γʢ.btͳπʔϧ͸͜ͷελΠϧʣ bpftrace is a high-level tracing language for Linux. bpftrace uses LLVM as a backend to compile scripts to eBPF-bytecode and makes use of libbpf and bcc for interacting with the Linux BPF subsystem, as well as existing Linux tracing capabilities: kernel dynamic tracing (kprobes), user-level dynamic tracing (uprobes), tracepoints, etc. The bpftrace language is inspired by awk, C, and predecessor tracers such as DTrace and SystemTap.8 8 https://github.com/bpftrace/bpftrace
31. bpftraceʹ͍ͭͯ » جຊ͸ϑοΫϙΠϯτͷࢦఆͱϓϩάϥϜίʔυͷηοτ » ΧʔωϧͷΠϕϯτҎ֎ʹ΋ɺuprobe/uretprobe/USDTͰϢʔβʔεϖʔεʹ΋ΞλονͰ ͖Δ » ूܭͳͲ΋Ͱ͖ΔͷͰεϙοτͷௐࠪʹ༗༻ $ sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }' Attaching 1 probe... curl /etc/ld.so.cache curl /lib/x86_64-linux-gnu/libcurl.so.4 curl /lib/x86_64-linux-gnu/libz.so.1 curl /lib/x86_64-linux-gnu/libc.so.6 curl /lib/x86_64-linux-gnu/libnghttp2.so.14 curl /lib/x86_64-linux-gnu/libidn2.so.0 curl /lib/x86_64-linux-gnu/librtmp.so.1 curl /lib/x86_64-linux-gnu/libssh.so.4
32. ՝୊ɿෛՙࢼݧʹ͓͚ΔϘτϧωοΫ » ͋ΔҊ݅ͰAWS্ͷ؀ڥͰෛՙࢼݧΛ࣮ࢪ͍ͯͨ͠ͱ͜Ζ͋ΔϙΠϯτͰϨ Πςϯγ͕ѱ͘ͳͬͨ » 2୆த1୆ͷElasticache(Memcached)΁ͷτϥϑΟοΫ͕ଟ͗ͯ͢ଳҬΛ࢖͍ ੾͍ͬͯͨ » Memcached΁ͷΠϯλʔϑΣʔε͸ϑϨʔϜϫʔΫʢFuelPHPʣʹ͓೚ͤ » ෼ࢄΞϧΰϦζϜΛมߋͨ͠ΒτϥϑΟοΫͷภΔϊʔυ͕มΘͬͨ » ͦΕͧΕͷϊʔυίϚϯυͷ਺͸΄΅ಉҰͰҧ͍͸τϥϑΟοΫͷΈ
33. ෛՙࢼݧʹ͓͚ΔϘτϧωοΫௐࠪͰͷར༻
34. ෛՙࢼݧʹ͓͚ΔϘτϧωοΫௐࠪͰͷར༻ » ൃߦ͞ΕΔίϚϯυ਺͸ಉҰͳͷͰ෼ࢄࣗମ͸ग़དྷ͍ͯΔ » ঢ়گతʹ͸ಛఆͷΩʔͰڊେͳΦϒδΣΫτ͕΍ΓͱΓ͞Ε͍ͯΔΑ͏ʹݟ ͑Δ » ཪͱΓΛߦ͏ͨΊʹbpftraceͰlibmemcachedʹuprobeΛ࢓ֻ͚τϨʔεΛ औͬͯΈͨ
35. ෛՙࢼݧʹ͓͚ΔϘτϧωοΫௐࠪͰͷར༻ » ൃߦ͞ΕΔίϚϯυ਺͸ಉҰͳͷͰ෼ࢄࣗମ͸ग़དྷ͍ͯΔ » ঢ়گతʹ͸ಛఆͷΩʔͰڊେͳΦϒδΣΫτ͕΍ΓͱΓ͞Ε͍ͯΔΑ͏ʹݟ ͑Δ » ཪͱΓΛߦ͏ͨΊʹbpftraceͰlibmemcachedʹuprobeΛ࢓ֻ͚τϨʔεΛ औͬͯΈͨ
36. bpftraceʹΑΔௐࠪ $ sudo bpftrace -e 'uprobe:/usr/lib/x86_64-linux-gnu/libmemcached.so.11.0.0:memcached_set { printf("----");time(); printf("key_length: %d\nkey: %s\n", arg2, str(arg1)); printf("val_length: %d\nval: %s\n", arg4, str(arg3) );}'
37. ղܾ » fuelphpͷmemcachedυϥΠό͕memcachedʹετΞ͢ΔࡍʹΦϦδφϧͷ ΩʔΛੜ੒ͨ͠ΩʔʹϚοϐϯά » Ϛοϐϯά৘ใͷΠϯσοΫε͸୯ҰͷΦϒδΣΫτʹ֨ೲ͞Ε͍ͯͨ » ΩʔͷϚοϐϯά৘ใͳͷͰຖճࢀরɺߋ৽͞Ε͍ͯͨ » ౰֘Ҋ݅Ͱ͸Ωʔ͕૿͑΍͍͢܏޲ʹ͋ͬͨͨΊɺΦϒδΣΫτ͕ංେԽͯ͠ ͍ͨ » ࠷ऴతʹϑϨʔϜϫʔΫͷػߏΛ࢖Θͳ͍࣮૷ʹஔ͖׵͑໰୊Λղফ
38. ͓·͚ɿΞϓϦέʔγϣϯଆͰΈͯͳ͍ؔ਺ͷ໭Γ஋ // PHPଆͰ໭Γ஋ΛΈ͍ͯͳ͍͕ apcu_store($cache_key, array('time' => $time, 'data' => $value), 0); // uprobeͰ௚઀C֦ுΛΈΔ͜ͱͰ࣮ࡍͷ໭Γ஋Λ֬ೝͰ͖Δ sudo bpftrace -e 'uretprobe:/usr/lib/php/20190902/apcu.so:apc_cache_store {printf ("%d\n", retval)}' 1 1 1 0 ...
39. Step3 eBPFϓϩάϥϜΛॻ͍ͯར༻͢ Δ ebpf_exporterΛར༻ͨ͠ PHPͷະར༻ίʔυͷ͋ͿΓग़͠ଞ
40. ebpf_exporter » cloudflare/ebpf_exporter9 » ىಈ࣌ʹϩʔυͨ͠eBPFϓϩάϥϜ͔Βऔಘͨ͠ϝτϦΫεΛPrometheusܗࣜͰ export͢Δ » ϝτϦΫεͷσʔλ͸BPF_MAPΛܦ༝ͯ͠Ϣʔβʔεϖʔε΁౉͞ΕΔ » ͋Β͔͡Ί࡞੒ͨ͠eBPFϓϩάϥϜʢELFόΠφϦʣͱରʹͳΔYAMLϑΝΠϧʢϝτϦ Ϋεͷ։ࣔΛ࢓ํΛࢦࣔ͢Δ΋ͷʣΛ࡞੒͢Δ͜ͱͰ೚ҙͷϝτϦΫεΛੜ੒Ͱ͖·͢ 9 https://github.com/cloudflare/ebpf_exporter
41. ebpf_exporterͷྫ » ௨ৗͷϊʔυͷϝτϦΫεͰ͸औΕͳ͍ʮIPΞυϨεɺѼઌϙʔτʯΛϥϕϧʹͯ͠ΑΓৄࡉͳσʔλΛಘΔ
42. ՝୊ɿະར༻ίʔυ » ௕ظӡ༻λΠτϧͰ͸ίʔυ͕ංେԽ » όʔδϣϯΞοϓͷͨͼʹॻ͖׵͕͑ඞཁ » ίʔυͷ૯ྔΛݮΒ͍͕࣮ͨ͠ࡍʹ࢖ΘΕ͍ͯΔ͔Θ͔Βͳ͍
43. ebpf_exporterʹΑΔPHPΞϓϦέʔγϣϯͷ ՄࢹԽ » uprobe/USDTΛར༻ͯ͠PHPͷ಺෦ϝτϦΫεΛऔಘ͢Δ » PHPͷUSDTͰ͸ͨͱ͑͹ϑΝΠϧͷίϯύΠϧɺϦΫΤετͷ։࢝/ऴྃɺΤϥʔͷൃੜͳͲͷϑοΫϙΠϯτ͕༻ ҙ͞Ε͍ͯΔ10 » ͜ΕΒΛϝτϦΫεͱͯ͠ՄࢹԽ͢Δ͜ͱͰSRE/։ൃऀʹ༗༻ͳ৘ใΛఏڙ͢Δ » εςοϓΞοϓΨΠυͳͷͰ࡞ΓํΛݟ͍͖ͯ·͢ » ʮίϯύΠϧͷΠϕϯτΛΈͯະ࢖༻ίʔυͷ͋ͿΓग़͠ʯΛͯ͠Έ·͢ 10 https://www.php.net/manual/ja/features.dtrace.dtrace.php
44. ߏ੒
45. ͲͷΑ͏ʹॻ͖࢝ΊΕ͹Α͍͔ » exmaplesҎԼΛோΊΔ » ໨తʹ͍ۙαϯϓϧΛಡΉ » ఆܕΛ཈͑Δ $ ls examples/ | head -n20 Makefile accept-latency.bpf.c accept-latency.yaml bio-trace.bpf.c bio-trace.png bio-trace.yaml biolatency.bpf.c biolatency.png biolatency.yaml bits.bpf.h bpf-jit.bpf.c bpf-jit.yaml cachestat-pre-kernel-5.16.bpf.c cachestat-pre-kernel-5.16.yaml cachestat.bpf.c cachestat.yaml cephfs-dist.bpf.c cephfs-dist.yaml cfs-throttling-trace.bpf.c cfs-throttling-trace.png
46. ͲͷΑ͏ʹॻ͖࢝ΊΕ͹Α͍͔
47. ͲͷΑ͏ʹॻ͖࢝ΊΕ͹Α͍͔
48. ͍͚ͦ͏Ͱ͢Ͷ
49. ॻ͍ͯΈͨʢൈਮʣ php.bpf.c #define MAX_STR_LEN 256 struct call_t { char filename[MAX_STR_LEN]; }; struct { __uint(type, BPF_MAP_TYPE_LRU_HASH); __uint(max_entries, 65536); __type(key, struct call_t); __type(value, u64); } php_compile_file_total SEC(".maps"); SEC("usdt//usr/lib/apache2/modules/libphp8.1.so:php:compile__file__entry") int BPF_USDT(do_count, char *arg0, char *arg1) { struct call_t call = {}; bpf_probe_read_user_str(&call.filename, sizeof(call.filename), arg1); truncate_string(call.filename, MAX_STR_LEN); static const char fmtstr[] = "compile file entry: %s, %s\n"; bpf_trace_printk(fmtstr, sizeof(fmtstr), arg0, arg1); increment_map(&php_compile_file_total, &call, 1); return 0; } php.yaml metrics: counters: - name: php_compile_file_total help: Number of php:compile__file__entry USDT calls per filepath labels: - name: filename size: 256 decoders: - name: string
50. ։ൃϑϩʔ » eBPFϓϩάϥϜΛϏϧυ͢Δʢexmaples಺ͷMakefile͕͋Δɺத਎͸clangΛ-target bpfͰ ࣮ߦ͍ͯ͠Δʣ » ίϯύΠϧΤϥʔ͕͋Ε͹͜ͷ࣌఺Ͱ஄͔ΕΔ » ebpf_exporterΛىಈ͢Δ » ىಈ࣌ʹϓϩάϥϜΛϩʔυ͢Δ » eBPFͷVerifierʹҾ͔͔ͬΔͱ͜͜Ͱམͱ͞ΕΔ » ىಈͨ͠ΒhttpͰϝτϦΫεΛ֬ೝ͠ͳ͕ΒɺλʔήοτͷΠϕϯτ͕࣮ߦ͢ΔʢPHPεΫ Ϧϓτͷ࣮ߦͳͲ
51. σόοά1 จࣈྻग़ྗ͍ͨ͠ » bpf_trace_printk ϔϧύؔ਺͕͋Δ11 » printfతͳϑΥʔϚοτͰprint debug » /sys/kernel/debug/tracing/trace_pipe Λಡ Ή͜ͱͰจࣈྻͰग़ྗ͕ಘΒΕΔ 11 https://docs.ebpf.io/linux/helper-function/bpftraceprintk/
52. σόοά2 BPF_MAPͷத਎ΛΈ͍ͨ » MAPͷத਎͸ sudo bpftool map ͰJSONͰಘΒΕΔ » bpftool mapͰIDΛௐ΂ͯdump idΛ͢Δ $ sudo bpftool map dump id ` sudo bpftool map |grep php_compile |egrep -o '^[0-9]+'` | jq . [ { "key": { "filename": "/var/www/html/" }, "value": 725 } ]
53. Ͱ͖·ͨ͠
54. Ԡ༻ฤ
55. ศརͰ͢Ͷ
56. Recap » eBPF͸ΦϒβʔόϏϦςΟܥͷπʔϧ಺෦Ͱ΋࢖༻͞Ε͍ͯΔ » SRE͕௚઀ѻ͏͜ͱͰΑΓϐϯϙΠϯτͳ՝୊ղܾʹར༻Ͱ͖Δ » ΞυϗοΫͳௐࠪ » ࣗ૊৫ͷ՝୊ղܾͷͨΊͷઐ༻πʔϧ » طଘπʔϧͷར༻ɺbpftraceͷར༻ɺeBPFϓϩάϥϜͷࣗ࡞ͷॱ͕͓͢͢Ί
57. Thank you for listening
58. Appendix » bpftraceʹΑΔGoΞϓϦέʔγϣϯͷτϨʔε » Real World Debugging with eBPF » https://github.com/egmc/ebpf_exporter/blob/ebpf-php-sample/ examples/php.bpf.c