Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Tracing BitVisor with bpftrace

mmisono
November 30, 2020

Tracing BitVisor with bpftrace

@BitVisor Summit9 (2020-11-30)
https://bitvisor.connpass.com/event/191785/

mmisono

November 30, 2020
Tweet

More Decks by mmisono

Other Decks in Technology

Transcript

  1. Tracing BitVisor with bpftrace
    Masanori Misono
    The University of Tokyo
    2020-11-30 BitVisor Summit 9

    View Slide

  2. 2
    About Myself
    • Masanori Misono (Shinagawa Laboratory, The University of Tokyo)
    • Github: @mmisono
    • A Committer of bpftrace (130+commits)
    • Attending BitVisor Summit from 2016-
    • 2017 : BPFΛར༻ͨ͠BitVisor಺෦ͰͷύέοτϑΟϧλϦϯά (+α)
    (Packet filtering (+α) in BitVisor with BPF)
    • 2018 : bitvisor.ko : BitVisor as a module
    • 2019 : vIOMMU implementation in BitVisor

    View Slide

  3. 3
    Motivation
    • Performance evaluation is always of critical importance
    • Performance evaluation of BitVisor itself is not so easy
    • VMEXIT count, latency
    • When shadow driver working
    • …
    • How can we get this?
    • We want to tracing tool for BitVisor!

    View Slide

  4. 4
    From my 2017’s presentation…

    View Slide

  5. 5
    What is (e)BPF?
    • Now Linux extend BPF (thus eBPF) and uses it in various ways
    • tracing, networking, security, …

    View Slide

  6. 6
    Why (e)BPF is used in Linux?

    View Slide

  7. 7
    From my 2017’s presentation… (cont’d)
    /&('$,(*"-$,*"+(').
    * $'#/$, ,*% /,(/"'!

    /////
    /////////////
    ////

    //
    ////////////
    Result
    I ported basic BPF functionality to BitVisor

    View Slide

  8. 8
    What does it do?
    • Use bcc to compile BPF program
    • Implement a hypercall
    (vmcall/vmmcall) to load a BPF
    program
    • Implement a hypercall to get a
    BPF map
    • Statically instrument tracing
    points (like Linux’s tracepoint)

    View Slide

  9. 9
    It works! But…
    • The safety problem
    • The limited verifier (vs. Linux verifier is ~10k)
    • BPF (user codes) runs in VMM root-mode ring0
    VMX ROOT mode
    ring 0
    What if the program has bug?

    View Slide

  10. 10
    It works! But… (cont’d)
    • The implementation is somewhat specific to BitVisor
    • We modified BCC to generate dedicated BPF code for BitVisor
    • BCC (and other tools) are actively developed
    • Can we use reduce the modification of userland program?

    View Slide

  11. 3 years later…

    View Slide

  12. 12
    BPF is more and more popular!
    https://ebpf.io/summit-2020/
    http://www.brendangregg.com/bpf-performance-tools-book.html
    https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine
    https://gihyo.jp/magazine/SD/archive/2020/202010
    Software Design 2020年10月号
    November 6, 2019
    August 20, 2020

    View Slide

  13. It’s time to revisit the problem!

    View Slide

  14. It’s time to revisit the problem!
    … BCC is really great,
    but is there another popular BPF tracing tool now?

    View Slide

  15. View Slide

  16. 16
    bpftrace
    (from bpftrace.org)
    ※ There are other useful tools, of course
    Only needing lines of script
    ※ Unofficial mascot

    View Slide

  17. 17
    bpftrace
    (from bpftrace.org)
    ※ There are other useful tools, of course
    Then get the result!
    ※ Unofficial mascot

    View Slide

  18. Let’s try to use bpftrace for tracing BitVisor!

    View Slide

  19. 19
    Goal
    (※ basically same as 2017’s)
    bpftrace
    BPF VM
    BPF map
    Retrieve data when necessary
    Guest OS
    BitVisor
    Load BPF program
    an event
    call BPF call back
    store/retrieve data
    Load BPF
    Helper functions

    View Slide

  20. 20
    Challenge
    1. Provide safe execution
    2. Use BPF code that generate bpftrace as is

    View Slide

  21. 21
    Challenge
    1. Provide safe execution
    2. Use BPF code that generate bpftrace as is

    View Slide

  22. 22
    Safe Execution
    • Implementing or Porting Linux’s verifier is very hard
    • Our approach
    • Safe execution by running BPF program In a VMX root ring3
    (a.k.a protection domain)
    VMX ROOT
    mode
    Ring0
    Ring3
    BPF VM
    BPF map
    BitVisor Main Thread
    Protection Domain
    Helper Functions
    an event

    View Slide

  23. 23
    Comparison with 2017’s
    Ring0
    Ring3
    BPF VM
    BPF map
    BitVisor Main Thread
    Protection Domain
    Helper Functions
    an event
    Ring0
    BPF VM
    BPF map
    BitVisor Main Thread
    Helper Functions
    an event
    2017
    2020
    Messaging overhead, but gives the safety

    View Slide

  24. 24
    Challenge
    1. Provide safe execution
    2. Use BPF code that generate bpftrace as is

    View Slide

  25. 25
    Implement the same helper function as Linux
    • BPF program can call external functions by BPF CALL instruction
    • Implement the same helper function
    • CALL 1 : BPF_MAP_LOOKUP_ELEM
    • CALL 2 : BPF_MAP_UPDETE_ELEM
    • CALL 3 : BPF_MAP_DELETE_ELEM
    • ….
    BPF VM
    BPF map
    Helper Functions
    Other functions

    View Slide

  26. 26
    Implementation
    • Port ubpf to BitVisor (the same as the before)
    • ubpf is an userland eBPF VM: https://github.com/iovisor/ubpf
    • Implement basic BPF helper functions in a protection domain
    • Modify bpftrace so that it call vmmcall instead of system call
    when interacting BPF functionality
    • I do not change any BPF code generation part of bpftrace!

    View Slide

  27. 27
    How to notify event?
    • Statically define events (like Linux’s tracepoint), the same as before
    • Example
    Added
    part an event

    View Slide

  28. 28
    BitVisor Ring0 ó Protection domain
    • Use msghandler to call BPF VM and pass data
    • The mechanism to communicate with other threads/processes using
    callbacks
    Ring0
    Ring3
    BPF VM
    BPF map
    BitVisor Main Thread
    Protection Domain
    Helper Functions
    an event
    sendmsg()

    View Slide

  29. 29
    BitVisor ó bpftrace Communication
    • bpftrace interacts kernel with bpf(2) system call
    • Implementing corresponding hypercall (vmcall/vmmcall) for bpf(2)
    bpftrace
    BPF map
    bpf(BPF_PROG_LOAD) bpf(BPF_MAP_LOOKUP_ELEM)
    BitVisor
    Guest OS

    View Slide

  30. 30
    Implementation (cont’d)
    • Total modification
    • BitVisor ~1000LOC (excludes ubpf and third-party libraries)
    • bpftrace ~300LOC
    • This includes comments, blank lines and debug codes. The actual
    amount of modification is much smaller

    View Slide

  31. 31
    Execution Overview
    ebpf VM
    ring3 ring0
    BitVisor
    Guest OS
    vmcall
    handler
    2. register
    the program
    ① Loading BPF Program
    bpftrace
    1. compile
    & load BPF
    program
    Helper Functions
    map
    BitVisor Main Thread
    Protection Domain

    View Slide

  32. 32
    Execution Overview
    ebpf VM
    ring3 ring0
    notify event
    an event occurs
    Run BPF program
    map
    ② Event handling
    bpftrace
    Helper Functions
    BitVisor
    Guest OS
    BitVisor Main Thread
    Protection Domain

    View Slide

  33. 33
    Execution Overview
    ebpf VM
    ring3 ring0
    vmcall
    handler
    1. Request
    the map data
    map 2. Retrieve map
    3. return map data
    ③ Retrieve tracing information
    Helper Functions
    bpftrace
    BitVisor
    Guest OS
    BitVisor Main Thread
    Protection Domain

    View Slide

  34. 34
    Demo

    View Slide

  35. 35
    Trace Script and the Result
    1: External Interrupt
    7: Interrupt Window
    31: RDMSR
    18: VMCALL
    In BitVisor
    The trace script

    View Slide

  36. 36
    The Generated Program

    View Slide

  37. 37
    Performance Evaluation
    • How much is the overhead of a message passing? (ring0 ó ring3)
    • I developed and experimented everything on VMWare Fusion on
    macOS with nested virtualization
    • Therefore, no legitimate evaluation presentation today :(
    • One day I want to try to do proper evaluation

    View Slide

  38. 38
    Discussion and Future Work
    • The current implementation is very preliminarily
    • Only support an integer key/value pair map
    • I guess it’s not so hard to extend this
    • What can we do if this project work enough?
    • Guest-Host cooperating tracing
    • BitVisor introspection according to the guest behavior
    • …
    • I think there must be a lot of fun things to do!

    View Slide

  39. 39
    Conclusion
    • Propose another way to trace BitVisor’s events
    • By utilizing a protection domain, BPF program run with safety
    guaranteed in a VMX root mode
    • Extend bpftrace and users can trace BitVisor with it
    • Let’s enjoy tracing! !

    View Slide

  40. View Slide

  41. 41
    Demo

    View Slide

  42. 42
    Demo

    View Slide