eBPF Can Do It! A Deep Dive into Real-World PHP Issues Solved with eBPF

Transcript

1. eBPF Can Do It! A Deep Dive into Real-World PHP Issues Solved with eBPF 2026/09/11 PHP×Tokyo September 2026 Sohei Iwahori (GREE, Inc.)
2. About Me • Sohei Iwahori • X/bsky/GitHub: @egmc • Senior Lead Engineer at GREE, Inc. • • Leading Monitoring Unit Community • eBPF Japan Meetup
3. Agenda • What is eBPF? • Cases • • #1 Memcached Performance Issue Diagnosis • #2 Dead Code Detection • #3 Inspecting C extension behavior • #4 Measuring the time taken for legacy batch jobs • #5 Long-Term Internal Metrics Recap
4. Agenda • What is eBPF? • Cases • • #1 Memcached Performance Issue Diagnosis • #2 Dead Code Detection • #3 Inspecting C extension behavior • #4 Measuring the time taken for legacy batch jobs • #5 Long-Term Internal Metrics Recap
5. What is eBPF?
6. 1 eBPF Overview 1 What is eBPF? https://ebpf.io/ja/what-is-ebpf/.
7. 1 eBPF Overview 1 What is eBPF? https://ebpf.io/ja/what-is-ebpf/.
8. Observability Tools that use eBPF internally • Pixie • Pyroscope • OpenTelemetry eBPF Instrumentation (Grafana Beyla) • Commercial Products for Observability
9. How to use eBPF? • Use existing tools • Use bpftrace command • Write your own eBPF code (in a C-like language)
10. Cases
11. Dead Code Detection
12. Problem (caused by dead code) • Barriers to version upgrades • Slow deployment process with a huge code base • Extremely hard to tell whether code is actually unused • Not only in web code but also in batch jobs, etc. • Some jobs might run once a month or even less often
13. php-dcr https://github.com/egmc/php-dcr2 2 https://github.com/egmc/php-dcr
14. php-dcr • Hook a DTrace trace point for PHP compile events • Attach to PHP binaries (cli / apache module / php-fpm) • The BPF program runs when compile events happen, and records compile information • Provide JSON-formatted reports via an HTTP API
15. DTrace?
16. DTrace?
17. Enabling DTrace in PHP • To enable DTrace • The PHP binary must be compiled with --enable-dtrace • Already done in most Linux distros • USE_ZEND_DTRACE=1 environment variable must be set • You can add it to a systemd unit file, etc. $ php -i |grep -i dtrace DTrace Support => available, disabled $ USE_ZEND_DTRACE=1 php -i |grep -i dtrace DTrace Support => enabled USE_ZEND_DTRACE => 1
18. php-dcr (eBPF code) #include "vmlinux.h" #include <bpf/bpf_helpers.h> #include <bpf/usdt.bpf.h> #define MAX_STR_LEN 512 char filename[MAX_STR_LEN]; // BPF_MAP macro struct { __uint(type, BPF_MAP_TYPE_LRU_HASH); __uint(max_entries, 65536); __type(key, char[MAX_STR_LEN]); __type(value, u64); } php_compile_file SEC(".maps"); // tracepoint SEC("usdt//usr/lib/apache2/modules/libphp8.1.so:php:compile__file__return") int BPF_USDT(compile_file_return, char *arg0, char *arg1) { // current time u64 ts = bpf_ktime_get_ns(); bpf_probe_read_user_str(&filename, sizeof(filename), arg0); // update BPF_MAP with time, compiled file name bpf_map_update_elem(&php_compile_file, &filename, &ts, BPF_ANY); return 0; } char LICENSE[] SEC("license") = "GPL";
19. php-dcr Overview (Data Flow) PHP Processes Apache (mod_php) Linux Kernel USDT probe php-dcr (Go) eBPF USDT: compile__file__return PHP-FPM PHP CLI USDT probe → Record file path + timestamp BPF map reader BPF Map (polls every 5 seconds) Scan USDT probe HTTP Target directory *.php External client HTTP API :8080 /v1/report, /v1/stats
20. How php-dcr Works (With a demo Laravel app)
21. How php-dcr Works - Sample App
22. How php-dcr Works - Sample App
23. How php-dcr Works - Run php-dcr # php-dcr --target-dir="/var/www/sample/laravel-sampletodo/app" --emit-log-events
24. How php-dcr Works - Access the app (Compile events)
25. How php-dcr Works - Load Model from CLI (Another event)
26. How php-dcr Works - View report
27. How php-dcr Works - View report
28. How php-dcr Works - Stats
29. How php-dcr Works - Otel Logs
30. Measuring the time taken for legacy batch jobs
31. Problem - Added latency during migration Before migration During migration to location B Network location A Network location A App / API App / API Batch jobs Migrating Batch jobs +1.6 ms latency DB Network location B (target) Low latency (co-located) DB
32. Problem - Added latency during migration Before migration During migration to location B Network location A Network location A App / API App / API Batch jobs Batch jobs TC command inject delay Migrating +1.6 ms latency Network location B (target) DB DB Co-located: delay added artificially to simulate the move
33. Web latency is easy, but how do we check batch jobs?
34. bpftrace • eBPF-based tracing tool with an awk-like language • Run as a one-liner or from a script file (.bt) bpftrace is a high-level tracing language for Linux. bpftrace uses LLVM as a backend to compile scripts to eBPF-bytecode and makes use of libbpf and bcc for interacting with the Linux BPF subsystem, as well as existing Linux tracing capabilities: kernel dynamic tracing (kprobes), user-level dynamic tracing (uprobes), tracepoints, etc. The bpftrace language is inspired by awk, C, and predecessor tracers such as DTrace and SystemTap.3 3 https://github.com/bpftrace/bpftrace
35. About bpftrace • Combination of tracepoints and programs • Aside from kernel events, bpftrace can attach to user space events (functions) using uprobe/uretprobe/USDT • Useful built-in functions for showing statistics $ sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }' Attaching 1 probe... curl /etc/ld.so.cache curl /lib/x86_64-linux-gnu/libcurl.so.4 curl /lib/x86_64-linux-gnu/libz.so.1 curl /lib/x86_64-linux-gnu/libc.so.6 curl /lib/x86_64-linux-gnu/libnghttp2.so.14 curl /lib/x86_64-linux-gnu/libidn2.so.0 curl /lib/x86_64-linux-gnu/librtmp.so.1 curl /lib/x86_64-linux-gnu/libssh.so.4
36. bpftrace (script)
37. bpftrace (script)
38. bpftrace (script)
39. bpftrace (script)
40. Result
41. Result
42. Recap
43. Recap • eBPF can be used for solving issues in the PHP world • bpftrace provides a simple interface as an investigation tool • You can write your own tool with eBPF to solve specific problems
44. Recap • Now a bpftrace example is part of the PHP manual • Give it a try!
45. Thank you for listening!