Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Tracing BitVisor with bpftrace

734096b490c456ce1e8670d279ac30cf?s=47 mmisono
November 30, 2020

Tracing BitVisor with bpftrace

@BitVisor Summit9 (2020-11-30)
https://bitvisor.connpass.com/event/191785/

734096b490c456ce1e8670d279ac30cf?s=128

mmisono

November 30, 2020
Tweet

Transcript

  1. Tracing BitVisor with bpftrace Masanori Misono The University of Tokyo

    2020-11-30 BitVisor Summit 9
  2. 2 About Myself • Masanori Misono (Shinagawa Laboratory, The University

    of Tokyo) • Github: @mmisono • A Committer of bpftrace (130+commits) • Attending BitVisor Summit from 2016- • 2017 : BPFΛར༻ͨ͠BitVisor಺෦ͰͷύέοτϑΟϧλϦϯά (+α) (Packet filtering (+α) in BitVisor with BPF) • 2018 : bitvisor.ko : BitVisor as a module • 2019 : vIOMMU implementation in BitVisor
  3. 3 Motivation • Performance evaluation is always of critical importance

    • Performance evaluation of BitVisor itself is not so easy • VMEXIT count, latency • When shadow driver working • … • How can we get this? • We want to tracing tool for BitVisor!
  4. 4 From my 2017’s presentation…

  5. 5 What is (e)BPF? • Now Linux extend BPF (thus

    eBPF) and uses it in various ways • tracing, networking, security, …
  6. 6 Why (e)BPF is used in Linux?

  7. 7 From my 2017’s presentation… (cont’d) /&('$,(*"-$,*"+('). * $'#/$, ,*%

    /,(/"'!   //// / //////////// /  /// /  / /  /////////// / Result I ported basic BPF functionality to BitVisor
  8. 8 What does it do? • Use bcc to compile

    BPF program • Implement a hypercall (vmcall/vmmcall) to load a BPF program • Implement a hypercall to get a BPF map • Statically instrument tracing points (like Linux’s tracepoint)
  9. 9 It works! But… • The safety problem • The

    limited verifier (vs. Linux verifier is ~10k) • BPF (user codes) runs in VMM root-mode ring0 VMX ROOT mode ring 0 What if the program has bug?
  10. 10 It works! But… (cont’d) • The implementation is somewhat

    specific to BitVisor • We modified BCC to generate dedicated BPF code for BitVisor • BCC (and other tools) are actively developed • Can we use reduce the modification of userland program?
  11. 3 years later…

  12. 12 BPF is more and more popular! https://ebpf.io/summit-2020/ http://www.brendangregg.com/bpf-performance-tools-book.html https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine

    https://gihyo.jp/magazine/SD/archive/2020/202010 Software Design 2020年10月号 November 6, 2019 August 20, 2020
  13. It’s time to revisit the problem!

  14. It’s time to revisit the problem! … BCC is really

    great, but is there another popular BPF tracing tool now?
  15. None
  16. 16 bpftrace (from bpftrace.org) ※ There are other useful tools,

    of course Only needing lines of script ※ Unofficial mascot
  17. 17 bpftrace (from bpftrace.org) ※ There are other useful tools,

    of course Then get the result! ※ Unofficial mascot
  18. Let’s try to use bpftrace for tracing BitVisor!

  19. 19 Goal (※ basically same as 2017’s) bpftrace BPF VM

    BPF map Retrieve data when necessary Guest OS BitVisor Load BPF program an event call BPF call back store/retrieve data Load BPF Helper functions
  20. 20 Challenge 1. Provide safe execution 2. Use BPF code

    that generate bpftrace as is
  21. 21 Challenge 1. Provide safe execution 2. Use BPF code

    that generate bpftrace as is
  22. 22 Safe Execution • Implementing or Porting Linux’s verifier is

    very hard • Our approach • Safe execution by running BPF program In a VMX root ring3 (a.k.a protection domain) VMX ROOT mode Ring0 Ring3 BPF VM BPF map BitVisor Main Thread Protection Domain Helper Functions an event
  23. 23 Comparison with 2017’s Ring0 Ring3 BPF VM BPF map

    BitVisor Main Thread Protection Domain Helper Functions an event Ring0 BPF VM BPF map BitVisor Main Thread Helper Functions an event 2017 2020 Messaging overhead, but gives the safety
  24. 24 Challenge 1. Provide safe execution 2. Use BPF code

    that generate bpftrace as is
  25. 25 Implement the same helper function as Linux • BPF

    program can call external functions by BPF CALL instruction • Implement the same helper function • CALL 1 : BPF_MAP_LOOKUP_ELEM • CALL 2 : BPF_MAP_UPDETE_ELEM • CALL 3 : BPF_MAP_DELETE_ELEM • …. BPF VM BPF map Helper Functions Other functions
  26. 26 Implementation • Port ubpf to BitVisor (the same as

    the before) • ubpf is an userland eBPF VM: https://github.com/iovisor/ubpf • Implement basic BPF helper functions in a protection domain • Modify bpftrace so that it call vmmcall instead of system call when interacting BPF functionality • I do not change any BPF code generation part of bpftrace!
  27. 27 How to notify event? • Statically define events (like

    Linux’s tracepoint), the same as before • Example Added part an event
  28. 28 BitVisor Ring0 ó Protection domain • Use msghandler to

    call BPF VM and pass data • The mechanism to communicate with other threads/processes using callbacks Ring0 Ring3 BPF VM BPF map BitVisor Main Thread Protection Domain Helper Functions an event sendmsg()
  29. 29 BitVisor ó bpftrace Communication • bpftrace interacts kernel with

    bpf(2) system call • Implementing corresponding hypercall (vmcall/vmmcall) for bpf(2) bpftrace BPF map bpf(BPF_PROG_LOAD) bpf(BPF_MAP_LOOKUP_ELEM) BitVisor Guest OS
  30. 30 Implementation (cont’d) • Total modification • BitVisor ~1000LOC (excludes

    ubpf and third-party libraries) • bpftrace ~300LOC • This includes comments, blank lines and debug codes. The actual amount of modification is much smaller
  31. 31 Execution Overview  ebpf VM ring3 ring0 BitVisor Guest

    OS vmcall handler 2. register the program ① Loading BPF Program bpftrace 1. compile & load BPF program Helper Functions map BitVisor Main Thread Protection Domain
  32. 32 Execution Overview  ebpf VM ring3 ring0 notify event

    an event occurs Run BPF program map ② Event handling bpftrace Helper Functions BitVisor Guest OS BitVisor Main Thread Protection Domain
  33. 33 Execution Overview  ebpf VM ring3 ring0 vmcall handler

    1. Request the map data map 2. Retrieve map 3. return map data ③ Retrieve tracing information Helper Functions bpftrace BitVisor Guest OS BitVisor Main Thread Protection Domain
  34. 34 Demo

  35. 35 Trace Script and the Result 1: External Interrupt 7:

    Interrupt Window 31: RDMSR 18: VMCALL In BitVisor The trace script
  36. 36 The Generated Program

  37. 37 Performance Evaluation • How much is the overhead of

    a message passing? (ring0 ó ring3) • I developed and experimented everything on VMWare Fusion on macOS with nested virtualization • Therefore, no legitimate evaluation presentation today :( • One day I want to try to do proper evaluation
  38. 38 Discussion and Future Work • The current implementation is

    very preliminarily • Only support an integer key/value pair map • I guess it’s not so hard to extend this • What can we do if this project work enough? • Guest-Host cooperating tracing • BitVisor introspection according to the guest behavior • … • I think there must be a lot of fun things to do!
  39. 39 Conclusion • Propose another way to trace BitVisor’s events

    • By utilizing a protection domain, BPF program run with safety guaranteed in a VMX root mode • Extend bpftrace and users can trace BitVisor with it • Let’s enjoy tracing! !
  40. None
  41. 41 Demo

  42. 42 Demo