2 About Myself • Masanori Misono (Shinagawa Laboratory, The University of Tokyo) • Github: @mmisono • A Committer of bpftrace (130+commits) • Attending BitVisor Summit from 2016- • 2017 : BPFΛར༻ͨ͠BitVisor෦ͰͷύέοτϑΟϧλϦϯά (+α) (Packet filtering (+α) in BitVisor with BPF) • 2018 : bitvisor.ko : BitVisor as a module • 2019 : vIOMMU implementation in BitVisor
3 Motivation • Performance evaluation is always of critical importance • Performance evaluation of BitVisor itself is not so easy • VMEXIT count, latency • When shadow driver working • … • How can we get this? • We want to tracing tool for BitVisor!
8 What does it do? • Use bcc to compile BPF program • Implement a hypercall (vmcall/vmmcall) to load a BPF program • Implement a hypercall to get a BPF map • Statically instrument tracing points (like Linux’s tracepoint)
9 It works! But… • The safety problem • The limited verifier (vs. Linux verifier is ~10k) • BPF (user codes) runs in VMM root-mode ring0 VMX ROOT mode ring 0 What if the program has bug?
10 It works! But… (cont’d) • The implementation is somewhat specific to BitVisor • We modified BCC to generate dedicated BPF code for BitVisor • BCC (and other tools) are actively developed • Can we use reduce the modification of userland program?
12 BPF is more and more popular! https://ebpf.io/summit-2020/ http://www.brendangregg.com/bpf-performance-tools-book.html https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine https://gihyo.jp/magazine/SD/archive/2020/202010 Software Design 2020年10月号 November 6, 2019 August 20, 2020
19 Goal (※ basically same as 2017’s) bpftrace BPF VM BPF map Retrieve data when necessary Guest OS BitVisor Load BPF program an event call BPF call back store/retrieve data Load BPF Helper functions
22 Safe Execution • Implementing or Porting Linux’s verifier is very hard • Our approach • Safe execution by running BPF program In a VMX root ring3 (a.k.a protection domain) VMX ROOT mode Ring0 Ring3 BPF VM BPF map BitVisor Main Thread Protection Domain Helper Functions an event
23 Comparison with 2017’s Ring0 Ring3 BPF VM BPF map BitVisor Main Thread Protection Domain Helper Functions an event Ring0 BPF VM BPF map BitVisor Main Thread Helper Functions an event 2017 2020 Messaging overhead, but gives the safety
25 Implement the same helper function as Linux • BPF program can call external functions by BPF CALL instruction • Implement the same helper function • CALL 1 : BPF_MAP_LOOKUP_ELEM • CALL 2 : BPF_MAP_UPDETE_ELEM • CALL 3 : BPF_MAP_DELETE_ELEM • …. BPF VM BPF map Helper Functions Other functions
26 Implementation • Port ubpf to BitVisor (the same as the before) • ubpf is an userland eBPF VM: https://github.com/iovisor/ubpf • Implement basic BPF helper functions in a protection domain • Modify bpftrace so that it call vmmcall instead of system call when interacting BPF functionality • I do not change any BPF code generation part of bpftrace!
28 BitVisor Ring0 ó Protection domain • Use msghandler to call BPF VM and pass data • The mechanism to communicate with other threads/processes using callbacks Ring0 Ring3 BPF VM BPF map BitVisor Main Thread Protection Domain Helper Functions an event sendmsg()
29 BitVisor ó bpftrace Communication • bpftrace interacts kernel with bpf(2) system call • Implementing corresponding hypercall (vmcall/vmmcall) for bpf(2) bpftrace BPF map bpf(BPF_PROG_LOAD) bpf(BPF_MAP_LOOKUP_ELEM) BitVisor Guest OS
30 Implementation (cont’d) • Total modification • BitVisor ~1000LOC (excludes ubpf and third-party libraries) • bpftrace ~300LOC • This includes comments, blank lines and debug codes. The actual amount of modification is much smaller
31 Execution Overview ebpf VM ring3 ring0 BitVisor Guest OS vmcall handler 2. register the program ① Loading BPF Program bpftrace 1. compile & load BPF program Helper Functions map BitVisor Main Thread Protection Domain
32 Execution Overview ebpf VM ring3 ring0 notify event an event occurs Run BPF program map ② Event handling bpftrace Helper Functions BitVisor Guest OS BitVisor Main Thread Protection Domain
33 Execution Overview ebpf VM ring3 ring0 vmcall handler 1. Request the map data map 2. Retrieve map 3. return map data ③ Retrieve tracing information Helper Functions bpftrace BitVisor Guest OS BitVisor Main Thread Protection Domain
37 Performance Evaluation • How much is the overhead of a message passing? (ring0 ó ring3) • I developed and experimented everything on VMWare Fusion on macOS with nested virtualization • Therefore, no legitimate evaluation presentation today :( • One day I want to try to do proper evaluation
38 Discussion and Future Work • The current implementation is very preliminarily • Only support an integer key/value pair map • I guess it’s not so hard to extend this • What can we do if this project work enough? • Guest-Host cooperating tracing • BitVisor introspection according to the guest behavior • … • I think there must be a lot of fun things to do!
39 Conclusion • Propose another way to trace BitVisor’s events • By utilizing a protection domain, BPF program run with safety guaranteed in a VMX root mode • Extend bpftrace and users can trace BitVisor with it • Let’s enjoy tracing! !
mmisono
udzura
ganota
tbaldauf
xibuka
nttcom
yoshinagaiwnl
sattosan
takufujii
seike460
drumath2237
iseki
tgalopin
jonyablonski
ufuk
jakevdp
philnash
jennybc
edds
dougneiner
jonrohan
keithpitt
cherdarchuk
panda_program
swwweet