Share
Embed
Transcript
56 slides
-
who? » Sohei Iwahori (@egmc) » גࣜձࣾάϦʔ ΠϯϑϥετϥΫνϟ෦ γχΞϦʔυΤϯδχΞ »
ΠϯϑϥͱࢹγεςϜ » SRE NEXT 2025 Co-Chair » eBPF Japan MeetupӡӦ -
ΞδΣϯμ » eBPFͷ֓ཁ » SRE͕eBPFΛѻ͏ϝϦοτ » ར༻͢Δ্Ͱͷݱ࣮తͳ͠͞ » ಋೖεςοϓ »
Step1 طଘͷπʔϧΛར༻͢Δ » Step2 bpftraceΛར༻͢Δ » Step3 eBPFϓϩάϥϜΛॻ͍ͯར༻͢Δ » Recap -
ΘΕ͍ͯΔͱ͜ΖʢObservabilityؔ࿈ʣ » Pixie » OpenTelemetry(opentelemetry-go-instrumentationͳͲ) » Pyroscope » Grafana Beyla
» ͦͷଞ༻ͳͲ » τϨʔεɺϓϩϑΝΠϥͳͲͷιϦϡʔγϣϯͰར༻͞Ε͍ͯΔ -
ґଘղܾ » ެࣜͷఏڙύοέʔδ͕ݹ͘BCCπʔϧɺbpftraceͳͲ͕ಈ͔ͳ͍ » BCCΛϏϧυ͠Α͏ͱ͢ΔͱOSόʔδϣϯ͝ͱʹґଘύοέʔδʢClangɺ LLVMͳͲʣΛࢦఆ͍ͯ͠ΕΔඞཁ͕͋Γେม » ͦͷଞ߹ʹΑΓσόοάγϯϘϧͷύοέʔδͳͲඞཁ » ظతʹ҆ఆͯ͠ಈ͔ͤΔڥΛOSόʔδϣϯΛ·͍ͨͰҡ࣋͢Δ͚ͩͰେม
» 2022͘Β͍͔Βঢ়گΑ͘ͳ͖͍ͬͯͯΔͷͱCO-REΞϓϩʔνʹΑΓ͋Δఔ ҟͳΔΧʔωϧόʔδϣϯͰಈ࡞͢Δπʔϧ͕࡞ΕΔΑ͏ʹͳͬͨ -
ใɺར༻ऀͷগͳ͞(੩తϓϩʔϒͷ) » eBPFͰUSDTͱ͍͏ϢʔβʔۭؒͰಈ࡞͢ΔϓϩάϥϜͷ੩తͳτϨʔε ϙΠϯτ͕αϙʔτ͞Ε͍ͯ·͢ » Python/Ruby/PHPͳͲͷݴޠϥϯλΠϜɺMySQLͳͲͰར༻Մೳ » όʔδϣϯΛ·͍ͨͰظతʹར༻Ͱ͖ΔͷͰʢupbobeʹൺͯʣπʔϧ ΈࠐΈ͍͢ »
͔͠͠όΠφϦ͕ --enable-dtrace ͳͲͰDTraceΛ༗ޮʹϏϧυ͞Ε͍ͯ Δඞཁ͕͋Γɺ͜ͷ͋ͨΓσΟετϦϏϡʔγϣϯґଘ -
ใɺར༻ऀͷগͳ͞(੩తϓϩʔϒͷ) » ubuntuͰఏڙ͞ΕΔެࣜύοέʔδͰجຊతʹDTrace༗ޮԽ͞Ε͍ͯΔ » ͔͠͠24.04(Noble)ͷύοέʔδͰdisable͞Ε͍ͯͨ͠ » ࣌ݩͷdebianύοέʔδଆͰϏϧυʹࣦഊ͢Δ͕͋ͬͨΒ͍͠ » bug reportΛ͋͛ͨ݁Ռ
25.04(Plucky)͔Β࠶༗ޮԽͯ͠Β͑ͨ3 » ·ͨɺPHPͰ USE_ZEND_DTRACE=1 ͱ͍͏ڥมΛରϓϩηεʹηοτ͢Δඞཁ͕͋Δ͕υΩϡϝϯτʹॻ͔Ε͍ͯ ͳ͔ͬͨ4 » ਓʑͷؔ৺͕େࣄ 4 https://github.com/php/doc-en/pull/4456 3 https://bugs.launchpad.net/ubuntu/+source/php8.3/+bug/2088977 -
ใɺར༻ऀͷগͳ͞(AIʁ) » ChatGPT4oར༻࣌ͰeBPFͷίʔυͷਫ਼ମײ40%͘Β͍ͩͬͨ » Claude Sonnet 4 / Opus4݁ߏॻ͍ͯ͘ΕΔ »
ͱ͍͑ʮͲ͜·Ͱ͕ఆܕͷίʔυ͔Θ͔Βͳ͍ʯΈ͍ͨͳͱ͜Ζ͋Δͱ ࢥ͏ͷͰυΩϡϝϯτͱαϯϓϧ͋ΔఔಡΉͱྑ͍ -
՝ » RubyͷShoryukenΛར༻ͨ͠Ξϥʔτ௨γεςϜ » Քಇ8ఔɺʢsystemdཧͰͳ͘ʣૉͳstop/start༻ͷγΣϧεΫϦϓτͰ ಈ࡞͍ͯͨ͠ » ShoryukenʹProcess.daemonΛͬͯσʔϞφΠζ͢ΔΦϓγϣϯ͕͋Δ » ىಈ࣌ʹpidΛهͯ͠ରʹkill͢ΔΑ͏ͳεΫϦϓτ
» ͋Δ࣌OSόʔδϣϯΞοϓΛߦͬͨΒɺ0࣌ʹಥવͯ͢ͷϫʔΧʔ͕μϯ » ͍ΦʔτεέʔϧʹΑΓޙʹ෮چ͕ͨ͠ɾɾ -
՝2 » ϩάΛΈΔͱͲ͏logrotate͕͍͋͠ » logrotate࣌ʹਖ਼ৗʹstop/start͍ͯ͠Δ͕ɺهΛΈΔͱstartޙʹ SIGTERMΛड͚ͯࡴ͞Ε͍ͯΔ » SIGTERMΛʮ୭͕ʯૹ͖͍ͬͯͯΔͷΛΓ͍͕ͨɺRubyଆͷSignal.trap Ͱૹ৴ݩͷpidͳͲͱΕͳ͍ »
eBPF ͳΒγεςϜίʔϧͷτϨʔεϙΠϯτ͔ΒύϥϝʔλʹΞΫηε͢Δ ͜ͱ͕Ͱ͖Δ -
BPFπʔϧͰͬͯΈΔ » killsnoopɺkillsnoop.bt5ͱ͍͏ͦͷͷͣΓͳπʔϧ͕͋Δ » killsnoopBCC൛ɺkillsnoop.btbpftrace൛ » OSύοέʔδͰఏڙ͞Ε͍ͯΔBCC൛͕ݹ͘ಈ࡞͠ͳ͔ͬͨͷͰkillsnoop.btΛͬͨ » ૹ৴ݩϓϩηεͷIDɺγάφϧͳͲΛϦΞϧλΠϜʹදࣔͯ͘͠ΕΔ »
τϨʔεϙΠϯτΧʔωϧͰఆٛ͞Ε͍ͯͯ6ɺsyscalls.hͰఆٛ͞Ε͍ͯΔϚΫϩ7Λܦ༝ͯ͠ύϥϝʔλ Λड͚औΕΔτϨʔεϙΠϯτΛࣗಈͰఆٛͯ͘͠ΕΔ 7 https://github.com/torvalds/linux/blob/master/include/linux/syscalls.h#L225C9-L225C24 6 https://github.com/torvalds/linux/blob/66701750d5565c574af42bef0b789ce0203e3071/kernel/signal.c#L3944-L3958 5 https://github.com/bpftrace/bpftrace/blob/master/tools/killsnoop.bt -
࣮ߦ݁Ռ » 0࣌ۙͰൃੜ͢Δ͜ͱΘ͔ͬ ͍ͯͨͷͰɺࡶʹcronͰಈ͔ͯ͠ ϩάΛϑΝΠϧʹॻ͖ग़ͯ͠Έͨ » pid1ʢsystemdʣ͕kill͍ͯ͠Δ » ͬͯͳ͍͕ͣͩɾɾʁ killsnoop.bt
00:00:01 466297 kill 10 439956 0 00:00:04 1 systemd 15 466335 0 00:00:04 1 systemd 18 466335 0 logrotate 00:00:01 xxx COMMAND=/usr/bin/kill -USR1 439956 -
͜ΕΛ౿·͑ͯͷ݁ » OSόʔδϣϯΞοϓͰlogorateͷ࣮ߦ͕cron->systemd.timerͷ࣮ߦʹมΘ͍ͬͯͨ » systemd.timerType=oneshotͷαʔϏεͱͯ͠ىಈ͢Δ » Type=oneshotͷαʔϏε࣮ߦऴྃ࣌ʹunitͷcgroupʹॴଐ͢Δͯ͢ͷϓϩηε͕ kill͞ΕΔσϑΥϧτϞʔυͰಈ͘ » Process.daemonσʔϞφΠζʢϓϩηεάϧʔϓͷΓ͠ʣͬͯ͘ΕΔ͕ɺ
cgourpಛʹԿ͠ͳ͍ » ݁Ռ࠶ىಈ࣌ʹlogorate unit͔Βىಈ͞ΕͨϫʔΧʔSIGTERMͷૹग़ରʹͳΓKILL ͞Εͨ -
bpftraceʹ͍ͭͯ » awkͬΆ͍ॻ͖ํͰҙͷτϨʔε͕͑Δ » ϫϯϥΠφʔͰͬͨΓɺεΫϦϓτΛϑΝΠϧͰ༻ҙ͓͖࣮ͯ͠ߦ ͨ͠Γʢ.btͳπʔϧ͜ͷελΠϧʣ bpftrace is a high-level
tracing language for Linux. bpftrace uses LLVM as a backend to compile scripts to eBPF-bytecode and makes use of libbpf and bcc for interacting with the Linux BPF subsystem, as well as existing Linux tracing capabilities: kernel dynamic tracing (kprobes), user-level dynamic tracing (uprobes), tracepoints, etc. The bpftrace language is inspired by awk, C, and predecessor tracers such as DTrace and SystemTap.8 8 https://github.com/bpftrace/bpftrace -
bpftraceʹ͍ͭͯ » جຊϑοΫϙΠϯτͷࢦఆͱϓϩάϥϜίʔυͷηοτ » ΧʔωϧͷΠϕϯτҎ֎ʹɺuprobe/uretprobe/USDTͰϢʔβʔεϖʔεʹΞλονͰ ͖Δ » ूܭͳͲͰ͖ΔͷͰεϙοτͷௐࠪʹ༗༻ $ sudo
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }' Attaching 1 probe... curl /etc/ld.so.cache curl /lib/x86_64-linux-gnu/libcurl.so.4 curl /lib/x86_64-linux-gnu/libz.so.1 curl /lib/x86_64-linux-gnu/libc.so.6 curl /lib/x86_64-linux-gnu/libnghttp2.so.14 curl /lib/x86_64-linux-gnu/libidn2.so.0 curl /lib/x86_64-linux-gnu/librtmp.so.1 curl /lib/x86_64-linux-gnu/libssh.so.4 -
ෛՙࢼݧʹ͓͚ΔϘτϧωοΫௐࠪͰͷར༻ » ͋ΔҊ݅ͰAWS্ͷڥͰෛՙࢼݧΛ࣮ࢪ͍ͯͨ͠ͱ͜Ζ͋ΔϙΠϯτͰϨ Πςϯγ͕ѱ͘ͳͬͨ » 2த1ͷElasticache(Memcached)ͷτϥϑΟοΫ͕ଟ͗ͯ͢ଳҬΛ͍ ͍ͬͯͨ » MemcachedͷΠϯλʔϑΣʔεϑϨʔϜϫʔΫʢFuelPHPʣʹ͓ͤ »
ࢄΞϧΰϦζϜΛมߋͨ͠ΒτϥϑΟοΫͷภΔϊʔυ͕มΘͬͨ » ͦΕͧΕͷϊʔυίϚϯυͷ΄΅ಉҰͰҧ͍τϥϑΟοΫͷΈ -
bpftraceʹΑΔௐࠪ $ sudo bpftrace -e 'uprobe:/usr/lib/x86_64-linux-gnu/libmemcached.so.11.0.0:memcached_set { printf("----");time(); printf("key_length: %d\nkey:
%s\n", arg2, str(arg1)); printf("val_length: %d\nval: %s\n", arg4, str(arg3) );}' -
͓·͚ɿΞϓϦέʔγϣϯଆͰΈͯͳ͍ؔͷΓ // PHPଆͰΓΛΈ͍ͯͳ͍͕ apcu_store($cache_key, array('time' => $time, 'data' => $value),
0); // uprobeͰC֦ுΛΈΔ͜ͱͰ࣮ࡍͷΓΛ֬ೝͰ͖Δ sudo bpftrace -e 'uretprobe:/usr/lib/php/20190902/apcu.so:apc_cache_store {printf ("%d\n", retval)}' 1 1 1 0 ... -
ebpf_exporter » cloudflare/ebpf_exporter9 » ىಈ࣌ʹϩʔυͨ͠eBPFϓϩάϥϜ͔Βऔಘͨ͠ϝτϦΫεΛPrometheusܗࣜͰ export͢Δ » ϝτϦΫεͷσʔλBPF_MAPΛܦ༝ͯ͠Ϣʔβʔεϖʔε͞ΕΔ » ͋Β͔͡Ί࡞ͨ͠eBPFϓϩάϥϜʢELFόΠφϦʣͱରʹͳΔYAMLϑΝΠϧʢϝτϦ
Ϋεͷ։ࣔΛํΛࢦࣔ͢ΔͷʣΛ࡞͢Δ͜ͱͰҙͷϝτϦΫεΛੜͰ͖·͢ 9 https://github.com/cloudflare/ebpf_exporter -
ebpf_exporterʹΑΔPHPΞϓϦέʔγϣϯͷ ՄࢹԽ » uprobe/USDTΛར༻ͯ͠PHPͷ෦ϝτϦΫεΛऔಘ͢Δ » PHPͷUSDTͰͨͱ͑ϑΝΠϧͷίϯύΠϧɺϦΫΤετͷ։࢝/ऴྃɺΤϥʔͷൃੜͳͲͷϑοΫϙΠϯτ͕༻ ҙ͞Ε͍ͯΔ10 » ͜ΕΒΛϝτϦΫεͱͯ͠ՄࢹԽ͢Δ͜ͱͰSRE/։ൃऀʹ༗༻ͳใΛఏڙ͢Δ »
εςοϓΞοϓΨΠυͳͷͰ࡞ΓํΛݟ͍͖ͯ·͢ » ʮίϯύΠϧͷΠϕϯτΛΈͯະ༻ίʔυͷ͋ͿΓग़͠ʯΛͯ͠Έ·͢ 10 https://www.php.net/manual/ja/features.dtrace.dtrace.php -
ͲͷΑ͏ʹॻ͖࢝ΊΕΑ͍͔ » exmaplesҎԼΛோΊΔ » తʹ͍ۙαϯϓϧΛಡΉ » ఆܕΛ͑Δ $ ls examples/
| head -n20 Makefile accept-latency.bpf.c accept-latency.yaml bio-trace.bpf.c bio-trace.png bio-trace.yaml biolatency.bpf.c biolatency.png biolatency.yaml bits.bpf.h bpf-jit.bpf.c bpf-jit.yaml cachestat-pre-kernel-5.16.bpf.c cachestat-pre-kernel-5.16.yaml cachestat.bpf.c cachestat.yaml cephfs-dist.bpf.c cephfs-dist.yaml cfs-throttling-trace.bpf.c cfs-throttling-trace.png -
ॻ͍ͯΈͨʢൈਮʣ php.bpf.c #define MAX_STR_LEN 256 struct call_t { char filename[MAX_STR_LEN];
}; struct { __uint(type, BPF_MAP_TYPE_LRU_HASH); __uint(max_entries, 65536); __type(key, struct call_t); __type(value, u64); } php_compile_file_total SEC(".maps"); SEC("usdt//usr/lib/apache2/modules/libphp8.1.so:php:compile__file__entry") int BPF_USDT(do_count, char *arg0, char *arg1) { struct call_t call = {}; bpf_probe_read_user_str(&call.filename, sizeof(call.filename), arg1); truncate_string(call.filename, MAX_STR_LEN); static const char fmtstr[] = "compile file entry: %s, %s\n"; bpf_trace_printk(fmtstr, sizeof(fmtstr), arg0, arg1); increment_map(&php_compile_file_total, &call, 1); return 0; } php.yaml metrics: counters: - name: php_compile_file_total help: Number of php:compile__file__entry USDT calls per filepath labels: - name: filename size: 256 decoders: - name: string -
։ൃϑϩʔ » eBPFϓϩάϥϜΛϏϧυ͢ΔʢexmaplesͷMakefile͕͋ΔɺதclangΛ-target bpfͰ ࣮ߦ͍ͯ͠Δʣ » ίϯύΠϧΤϥʔ͕͋Ε͜ͷ࣌Ͱ͔ΕΔ » ebpf_exporterΛىಈ͢Δ »
ىಈ࣌ʹϓϩάϥϜΛϩʔυ͢Δ » eBPFͷVerifierʹҾ͔͔ͬΔͱ͜͜Ͱམͱ͞ΕΔ » ىಈͨ͠ΒhttpͰϝτϦΫεΛ֬ೝ͠ͳ͕ΒɺλʔήοτͷΠϕϯτ͕࣮ߦ͢ΔʢPHPεΫ Ϧϓτͷ࣮ߦͳͲ -
σόοά1 จࣈྻग़ྗ͍ͨ͠ » bpf_trace_printk ϔϧύ͕ؔ͋Δ11 » printfతͳϑΥʔϚοτͰprint debug » /sys/kernel/debug/tracing/trace_pipe
Λಡ Ή͜ͱͰจࣈྻͰग़ྗ͕ಘΒΕΔ 11 https://docs.ebpf.io/linux/helper-function/bpftraceprintk/ -
σόοά2 BPF_MAPͷதΛΈ͍ͨ » MAPͷத sudo bpftool map ͰJSONͰಘΒΕΔ » bpftool
mapͰIDΛௐͯdump idΛ͢Δ $ sudo bpftool map dump id ` sudo bpftool map |grep php_compile |egrep -o '^[0-9]+'` | jq . [ { "key": { "filename": "/var/www/html/" }, "value": 725 } ] -
Recap » eBPFΦϒβʔόϏϦςΟܥͷπʔϧ෦Ͱ༻͞Ε͍ͯΔ » SRE͕ѻ͏͜ͱͰΑΓϐϯϙΠϯτͳ՝ղܾʹར༻Ͱ͖Δ » ΞυϗοΫͳௐࠪ » ࣗ৫ͷ՝ղܾͷͨΊͷઐ༻πʔϧ »
طଘπʔϧͷར༻ɺbpftraceͷར༻ɺeBPFϓϩάϥϜͷࣗ࡞ͷॱ͕͓͢͢Ί