More efficient remote debugging with Thin Hypervisor

194b192123e0671dc9f81647d61411a6?s=47 Ren Kimura
November 11, 2017

More efficient remote debugging with Thin Hypervisor

My talk at AVTOKYO2017


Ren Kimura

November 11, 2017


  1. 2.

    Whoami • Ren Kimura (@RKX1209) • Kobe University student (M1)

    • Security hobbyist • Intersted in: Binary Analysis, Exploitation…. • • BTW: Welcome to Japan @pancake! AVTOKYO2017
  2. 3.

    System debugging • Sometime, you need to analyze entire system

    (including kernel) ◦ Analyze rootkit-infected system. ◦ Root cause analysis for system crash. ◦ Debug device driver or some kind of kernel modules. • In such case, you will use VM to run target system • Fortunately, there are many analysis interface on VM. ◦ ex. GDB serial interface for QEMU(KVM), Virtualbox, VMware... AVTOKYO2017
  3. 4.

    Realtime system debugging • Imagine the real world cases. •

    Your system is always running on VM? Really?? • It can be possible that you accidentally encounter some problems on real machine. AVTOKYO2017 Hardware System Problem Hardware System Hardware VM You must save system dump and reproduce problem again on VM
  4. 5.

    Realtime system debugging • Reproducilbility has gone during migration, Target

    System<=>VM ◦ Can you reproduce device driver bug on VM again? => Maybe No you can’t. ◦ Can you produce completely same hardware on VM? => Maybe No. ◦ Can you make rootkit runtime status same to that on real hardware? => No. Anti-sandbox AVTOKYO2017 You are not guranteed to reproduce complete same state on VM. Hardware System Problem Hardware System Hardware VM Problem ?
  5. 6.

    Realtime system debugging • If you always run your machine

    on VM, you can analyze the system anytime. • But, It’s not usual case, Of course. • User feel some uncomfortable because of performance overhead, emulation.. ◦ In other words, This solution is intransparent to user. • That’s why we use “Thin Hypervisor Technology”. ◦ User transparent monitoring. AVTOKYO2017
  6. 7.

    BitVisor • There are some kind of Thin Hypervisor for

    secure monitoring. [*1] ◦ ex. ksm, SimpeVisor, HyperPlatform, Bareflank and BitVisor • “Thin” mean, lightweight. Ex. No hardware emulation, Single Guest OS... • This time, I chose BitVisor as a base platform. AVTOKYO2017 [*1] Hardware System Hardware VM System Hardware Hardware System VM Hypervisor (KVM, Xen…) BitVisor Traditional Hypervisor Thin Hypervisor(BitVisor)
  7. 8.

    Real time gdb debugging with BitVisor • Target system runs

    on BitVisor • When some problem happen on system, you can attach the system via gdb • gdbserver in BitVisor always wait connection from client • Then you can analyze why the problem happen through gdb. AVTOKYO2017 System Hardware BitVisor Problem GDB Local Machine Attach an entire system
  8. 9.

    Implementation • Implement gdbstub on BitVisor. • Use lwip for

    hypervisor-level network communication. • Hardware breakpoint capability. Modify Guest debug register. (i.e. vmcs.dr7) • Stop/Continue the VM when hit the breakpiont and execute “continue” by user. AVTOKYO2017 gdb protocol communication via lwip ( ) System Hardware BitVisor gdbstub GDB Local Machine
  9. 10.

    Supported GDB protocol • ?. Report why the target halted.

    • c, C, Continue the target (possibly with a particular signal). • g and G. Read or write general registers. • qC and H. Report the current thread or set the thread for subsequent operations. • m and M. Read or write main memory. • p and P. Read or write a specific register. • qSupported. Report the features supported by the RSP server. • z and Z. Clear or set breakpoints or watchpoints. AVTOKYO2017 See:
  10. 11.

    DEMO (Linux kernel debug) $ gdb -q (gdb) target remote (gdb) hb *0xffffffff81055ab0 #Set breakpiont to do_page_fault() (gdb) c Continuing. Breakpoint 1, 0xffffffff81055ab0 in ?? () (gdb) x/10i $rip (gdb) i r (gdb) d 1 (gdb) c AVTOKYO2017 bitvisor-gdb: