More efficient remote debugging with Thin Hypervisor Ren Kimura (@RKX1209) AVTOKYO2017

Whoami ● Ren Kimura (@RKX1209) ● Kobe University student (M1) ● Security hobbyist ● Intersted in: Binary Analysis, Exploitation…. ● https://rkx1209.github.io/about/ ● BTW: Welcome to Japan @pancake! AVTOKYO2017

System debugging ● Sometime, you need to analyze entire system (including kernel) ○ Analyze rootkit-infected system. ○ Root cause analysis for system crash. ○ Debug device driver or some kind of kernel modules. ● In such case, you will use VM to run target system ● Fortunately, there are many analysis interface on VM. ○ ex. GDB serial interface for QEMU(KVM), Virtualbox, VMware... AVTOKYO2017

Realtime system debugging ● Imagine the real world cases. ● Your system is always running on VM? Really?? ● It can be possible that you accidentally encounter some problems on real machine. AVTOKYO2017 Hardware System Problem Hardware System Hardware VM You must save system dump and reproduce problem again on VM

Realtime system debugging ● Reproducilbility has gone during migration, Target System<=>VM ○ Can you reproduce device driver bug on VM again? => Maybe No you can’t. ○ Can you produce completely same hardware on VM? => Maybe No. ○ Can you make rootkit runtime status same to that on real hardware? => No. Anti-sandbox AVTOKYO2017 You are not guranteed to reproduce complete same state on VM. Hardware System Problem Hardware System Hardware VM Problem ?

Realtime system debugging ● If you always run your machine on VM, you can analyze the system anytime. ● But, It’s not usual case, Of course. ● User feel some uncomfortable because of performance overhead, emulation.. ○ In other words, This solution is intransparent to user. ● That’s why we use “Thin Hypervisor Technology”. ○ User transparent monitoring. AVTOKYO2017

BitVisor ● There are some kind of Thin Hypervisor for secure monitoring. [*1] ○ ex. ksm, SimpeVisor, HyperPlatform, Bareflank and BitVisor ● “Thin” mean, lightweight. Ex. No hardware emulation, Single Guest OS... ● This time, I chose BitVisor as a base platform. AVTOKYO2017 [*1] https://ntddk.github.io/2017/03/04/thin-hypervisor-2017/ Hardware System Hardware VM System Hardware Hardware System VM Hypervisor (KVM, Xen…) BitVisor Traditional Hypervisor Thin Hypervisor(BitVisor)

Real time gdb debugging with BitVisor ● Target system runs on BitVisor ● When some problem happen on system, you can attach the system via gdb ● gdbserver in BitVisor always wait connection from client ● Then you can analyze why the problem happen through gdb. AVTOKYO2017 System Hardware BitVisor Problem GDB Local Machine Attach an entire system

Implementation ● Implement gdbstub on BitVisor. ● Use lwip for hypervisor-level network communication. ● Hardware breakpoint capability. Modify Guest debug register. (i.e. vmcs.dr7) ● Stop/Continue the VM when hit the breakpiont and execute “continue” by user. AVTOKYO2017 gdb protocol communication via lwip ( https://github.com/RKX1209/bitvisor-gdb ) System Hardware BitVisor gdbstub GDB Local Machine

Supported GDB protocol ● ?. Report why the target halted. ● c, C, Continue the target (possibly with a particular signal). ● g and G. Read or write general registers. ● qC and H. Report the current thread or set the thread for subsequent operations. ● m and M. Read or write main memory. ● p and P. Read or write a specific register. ● qSupported. Report the features supported by the RSP server. ● z and Z. Clear or set breakpoints or watchpoints. AVTOKYO2017 See: https://github.com/RKX1209/bitvisor-gdb/blob/master/debug/gdbstub.c

DEMO (Linux kernel debug) $ gdb -q (gdb) target remote 192.168.0.5:1234 (gdb) hb *0xffffffff81055ab0 #Set breakpiont to do_page_fault() (gdb) c Continuing. Breakpoint 1, 0xffffffff81055ab0 in ?? () (gdb) x/10i $rip (gdb) i r (gdb) d 1 (gdb) c AVTOKYO2017 bitvisor-gdb: https://github.com/RKX1209/bitvisor-gdb