Tracing BitVisor with bpftrace

@BitVisor Summit9 (2020-11-30)
https://bitvisor.connpass.com/event/191785/

Transcript

1. Tracing BitVisor with bpftrace Masanori Misono The University of Tokyo 2020-11-30 BitVisor Summit 9
2. 2 About Myself • Masanori Misono (Shinagawa Laboratory, The University of Tokyo) • Github: @mmisono • A Committer of bpftrace (130+commits) • Attending BitVisor Summit from 2016- • 2017 : BPFΛར༻ͨ͠BitVisor಺෦ͰͷύέοτϑΟϧλϦϯά (+α) (Packet filtering (+α) in BitVisor with BPF) • 2018 : bitvisor.ko : BitVisor as a module • 2019 : vIOMMU implementation in BitVisor
3. 3 Motivation • Performance evaluation is always of critical importance • Performance evaluation of BitVisor itself is not so easy • VMEXIT count, latency • When shadow driver working • … • How can we get this? • We want to tracing tool for BitVisor!
4. 4 From my 2017’s presentation…
5. 5 What is (e)BPF? • Now Linux extend BPF (thus eBPF) and uses it in various ways • tracing, networking, security, …
6. 6 Why (e)BPF is used in Linux?
7. 7 From my 2017’s presentation… (cont’d)
   /&('$,(*"-$,*"+('). * $'#/$, ,*% /,(/"'!   //// / //////////// /  /// /  / /  /////////// / Result I ported basic BPF functionality to BitVisor
8. 8 What does it do? • Use bcc to compile BPF program • Implement a hypercall (vmcall/vmmcall) to load a BPF program • Implement a hypercall to get a BPF map • Statically instrument tracing points (like Linux’s tracepoint)
9. 9 It works! But… • The safety problem • The limited verifier (vs. Linux verifier is ~10k) • BPF (user codes) runs in VMM root-mode ring0 VMX ROOT mode ring 0 What if the program has bug?
10. 10 It works! But… (cont’d) • The implementation is somewhat specific to BitVisor • We modified BCC to generate dedicated BPF code for BitVisor • BCC (and other tools) are actively developed • Can we use reduce the modification of userland program?
11. 3 years later…
12. 12 BPF is more and more popular! https://ebpf.io/summit-2020/ http://www.brendangregg.com/bpf-performance-tools-book.html https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine https://gihyo.jp/magazine/SD/archive/2020/202010 Software Design 2020年10月号 November 6, 2019 August 20, 2020
13. It’s time to revisit the problem!
14. It’s time to revisit the problem! … BCC is really great, but is there another popular BPF tracing tool now?
16. 16 bpftrace (from bpftrace.org) ※ There are other useful tools, of course Only needing lines of script ※ Unofficial mascot
17. 17 bpftrace (from bpftrace.org) ※ There are other useful tools, of course Then get the result! ※ Unofficial mascot
18. Let’s try to use bpftrace for tracing BitVisor!
19. 19 Goal (※ basically same as 2017’s) bpftrace BPF VM BPF map Retrieve data when necessary Guest OS BitVisor Load BPF program an event call BPF call back store/retrieve data Load BPF Helper functions
20. 20 Challenge 1. Provide safe execution 2. Use BPF code that generate bpftrace as is
21. 21 Challenge 1. Provide safe execution 2. Use BPF code that generate bpftrace as is
22. 22 Safe Execution • Implementing or Porting Linux’s verifier is very hard • Our approach • Safe execution by running BPF program In a VMX root ring3 (a.k.a protection domain) VMX ROOT mode Ring0 Ring3 BPF VM BPF map BitVisor Main Thread Protection Domain Helper Functions an event
23. 23 Comparison with 2017’s Ring0 Ring3 BPF VM BPF map BitVisor Main Thread Protection Domain Helper Functions an event Ring0 BPF VM BPF map BitVisor Main Thread Helper Functions an event 2017 2020 Messaging overhead, but gives the safety
24. 24 Challenge 1. Provide safe execution 2. Use BPF code that generate bpftrace as is
25. 25 Implement the same helper function as Linux • BPF program can call external functions by BPF CALL instruction • Implement the same helper function • CALL 1 : BPF_MAP_LOOKUP_ELEM • CALL 2 : BPF_MAP_UPDETE_ELEM • CALL 3 : BPF_MAP_DELETE_ELEM • …. BPF VM BPF map Helper Functions Other functions
26. 26 Implementation • Port ubpf to BitVisor (the same as the before) • ubpf is an userland eBPF VM: https://github.com/iovisor/ubpf • Implement basic BPF helper functions in a protection domain • Modify bpftrace so that it call vmmcall instead of system call when interacting BPF functionality • I do not change any BPF code generation part of bpftrace!
27. 27 How to notify event? • Statically define events (like Linux’s tracepoint), the same as before • Example Added part an event
28. 28 BitVisor Ring0 ó Protection domain • Use msghandler to call BPF VM and pass data • The mechanism to communicate with other threads/processes using callbacks Ring0 Ring3 BPF VM BPF map BitVisor Main Thread Protection Domain Helper Functions an event sendmsg()
29. 29 BitVisor ó bpftrace Communication • bpftrace interacts kernel with bpf(2) system call • Implementing corresponding hypercall (vmcall/vmmcall) for bpf(2) bpftrace BPF map bpf(BPF_PROG_LOAD) bpf(BPF_MAP_LOOKUP_ELEM) BitVisor Guest OS
30. 30 Implementation (cont’d) • Total modification • BitVisor ~1000LOC (excludes ubpf and third-party libraries) • bpftrace ~300LOC • This includes comments, blank lines and debug codes. The actual amount of modification is much smaller
31. 31 Execution Overview
    ebpf VM ring3 ring0 BitVisor Guest OS vmcall handler 2. register the program ① Loading BPF Program bpftrace 1. compile & load BPF program Helper Functions map BitVisor Main Thread Protection Domain
32. 32 Execution Overview
    ebpf VM ring3 ring0 notify event an event occurs Run BPF program map ② Event handling bpftrace Helper Functions BitVisor Guest OS BitVisor Main Thread Protection Domain
33. 33 Execution Overview
    ebpf VM ring3 ring0 vmcall handler 1. Request the map data map 2. Retrieve map 3. return map data ③ Retrieve tracing information Helper Functions bpftrace BitVisor Guest OS BitVisor Main Thread Protection Domain
34. 34 Demo
35. 35 Trace Script and the Result 1: External Interrupt 7: Interrupt Window 31: RDMSR 18: VMCALL In BitVisor The trace script
36. 36 The Generated Program
37. 37 Performance Evaluation • How much is the overhead of a message passing? (ring0 ó ring3) • I developed and experimented everything on VMWare Fusion on macOS with nested virtualization • Therefore, no legitimate evaluation presentation today :( • One day I want to try to do proper evaluation
38. 38 Discussion and Future Work • The current implementation is very preliminarily • Only support an integer key/value pair map • I guess it’s not so hard to extend this • What can we do if this project work enough? • Guest-Host cooperating tracing • BitVisor introspection according to the guest behavior • … • I think there must be a lot of fun things to do!
39. 39 Conclusion • Propose another way to trace BitVisor’s events • By utilizing a protection domain, BPF program run with safety guaranteed in a VMX root mode • Extend bpftrace and users can trace BitVisor with it • Let’s enjoy tracing! !
41. 41 Demo
42. 42 Demo