Practical Rust (Hypervisor) Firmware

Slide 1

Slide 1 text

Practical Rust (Hypervisor) Firmware July 10, 2021 Kernel/VM Online Part 3 @retrage

Slide 2

Slide 2 text

Rust Hypervisor Firmware

Slide 3

Slide 3 text

What’s Rust Hypervisor Firmware? • Small f irmware written from scratch in Rust • One of Cloud Hypervisor[1] subproject • Focuses on booting Linux cloud workloads • Ubuntu and Clear Linux • Supported VMMs: • Cloud Hypervisor and QEMU

Slide 4

Slide 4 text

Architecture Overview • Minimal device drivers and f ilesystem support • VirtIO block device • FAT f ilesystem • Linux boot modes: • Native Linux loader • EFI compatible layer VirtIO Block Device FAT Filesystem EFI Compatible Layer Native Linux Loader Rust Hypervisor Firmware

Slide 5

Slide 5 text

EFI Compatible Layer • Small subset of EFI features • Implemented 8 out of 39 EFI Boot Services • Enough to boot customized Ubuntu VM image • Works with typical Linux boot loaders (shim[2]+GRUB) • It was not compatible with the EFI spec. to boot: • Stock Ubuntu cloud images[3] • Other OSs?

Slide 6

Slide 6 text

🤔 Can it boot Windows?

Slide 7

Slide 7 text

Challenge Accepted 😏

Slide 8

Slide 8 text

How to Boot Windows

Slide 9

Slide 9 text

Windows Boot Stages • Windows boot process has three steps[4]: 1. Boot Manager (bootmgfw.e f i) • Determines boot order from EFI variables/BCD 2. OS Loader (winload.e f i) • Setups Windows speci f ic pre-boot process 3. NT OS Kernel (ntoskrnl.exe) • Need to support above chain loading

Slide 10

Slide 10 text

Changes Needed to Boot Windows EFI Variable Support EFI Device Path Decoder Support Chain Loading Support Fix Memory Segments Directory Support for EFI File Protocol EFI Runtime Services Support Fix EFI Memory Map Clock/Delay Support Fix PE Loader

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Implement EFI Runtime Services

Slide 13

Slide 13 text

Boot Services and Runtime Services • Boot Services: available during BDS phase and TSL phase • Runtime Services: must be always available DXE Run Time (RT) Transient System Load (TSL) Boot Device Selection (BDS) EFI Runtime Services Availability EFI Boot Services Availability

Slide 14

Slide 14 text

EFI Boot Process System Control Hand-o ff Flow • Loader loads and starts OS • EFI Boot Services is still available at this point DXE Run Time (RT) Transient System Load (TSL) OS Loader OS Kernel Boot Device Selection (BDS) Boot Dispatcher Boot Manager

Slide 15

Slide 15 text

EFI Boot Process System Control Hand-o ff Flow • The OS requests system control hand-off to the EFI • This is done by calling gBS->ExitBootServices () DXE Run Time (RT) OS Kernel Transient System Load (TSL) OS Loader OS Kernel Boot Device Selection (BDS) Boot Dispatcher Boot Manager ExitBootServices()

Slide 16

Slide 16 text

Paging Problem Regarding EFI Runtime Services • The OS is responsible to manage the system memory • It also remaps the EFI Runtime Services to the new virtual address • The EFI Runtime Services code must be relocated EFI Runtime Code mov $0x175bd0, %rax call *%rax 0x175bd0: ... ... 0x175000: EFI Runtime Code mov $0x175bd0, %rax call *%rax 0xfffc607f046ebd0: ... ... 0xfffc607f046e000: Unrecoverable Page Fault Set new page table

Slide 17

Slide 17 text

EFI Runtime Code Relocation • The OS must call gRT- >SetVirtualAddressMap() • It provides the new virtual address map to the EFI Runtime Services • The EFI Runtime Services do self relocation using given map • How to implement? SetVirtualAddressMap() 0x175000 -> 0xfffc607f046e000 ... EFI Runtime Code mov $0x175bd0, %rax call *%rax 0x175bd0: ... ... 0x175000: Self Relocator Fixup

Slide 18

Slide 18 text

EFI Runtime Services Design • Generate EFI Runtime Services as an independent dynamic ELF binary • The binary is embedded in the static binary using objcopy • Relocator does 1st relocation • Self-relocator does 2nd relocation Boot Services Code Relocator Self Relocator Runtime Services (Dynamic ELF) Rust Hypervisor Firmware (Static ELF)

Slide 19

Slide 19 text

Target Windows Guest Environment • Exactly same environment supported by Cloud Hypervisor[5] • Recent versions of Windows: • Windows 10 20H2 and 21H1 • Windows Server 2019 • Installed 3rd-party VirtIO device drivers[6] • Enabled Emergency Management Services (EMS) [7] to use serial console • Cloud Hypervisor does not support graphics

Slide 20

Slide 20 text

Booting Windows on Cloud Hypervisor Pass the f irmware image from —kernel option

Slide 21

Slide 21 text

Booting Windows on Cloud Hypervisor EMS SAC appeared to tty via UART serial

Slide 22

Slide 22 text

Booting Windows on Cloud Hypervisor Everything is up, cmd.exe is available

Slide 23

Slide 23 text

Porting to Physical Machine

Slide 24

Slide 24 text

Porting to coreboot • coreboot[8]: Open source f irmware • coreboot ROM: • Basic system initialization + static ELF binary payload • Added coreboot table parser to provide memory map info (~170 LoC) System Initialization Code Intel FSP Rust (Hypervisor) Firmware

Slide 25

Slide 25 text

Target Machine • MinnowBoard MAX[9] • Intel Atom (Bay Trail) • Easily access to SPI f lash • Pin header exposed • UART serial available x86 SBC from 2014

Slide 26

Slide 26 text

Booting Rust (Hypervisor) Firmware from coreboot It panicked because no VirtIO block device found

Slide 27

Slide 27 text

Summary • Rust Hypervisor Firmware is a f irmware written in Rust • Added more compatibility to the f irmware to boot Windows • It requires sane implementation of EFI Runtime Services • Ported the f irmware to coreboot with small efforts • It now boots up on a physical machine • It still cannot boot OS due to lack of physical disk device drivers • https://github.com/cloud-hypervisor/rust-hypervisor- f irmware

Slide 28

Slide 28 text

References • [0] https://github.com/cloud-hypervisor/rust-hypervisor- f irmware • [1] https://github.com/cloud-hypervisor/cloud-hypervisor • [2] https://github.com/rhboot/shim • [3] https://cloud-images.ubuntu.com • [4] https://ue f i.org/sites/default/ f iles/resources/UEFI-Plugfest-WindowsBootEnvironment.pdf • [5] https://github.com/cloud-hypervisor/cloud-hypervisor/blob/master/docs/windows.md • [6] https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.185-2/ • [7] https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/bcdedit--ems • [8] https://coreboot.org • [9] https://www.elinux.org/Minnowboard:MinnowMax

Slide 29

Slide 29 text

Appendix

Slide 30

Slide 30 text

Pros & Cons of Writing Firmware in Rust Rust is not a silver bullet • Pros: • no_std-friendly ecosystem • core crate is modular design • Clean compile-time environment • macro/const_fn • C preprocessor is hell • Strong type system • Cons: • Nightly Rust is required • Some of ABIs are unstable • Efforts needed to keep dependencies up-to-date • Cargo is so not f lexible • e.g. oreboot uses GNU Make in addition to Cargo

Slide 31

Slide 31 text

Debugging Issues • Issue #1: WinDbg is not available in early boot phase • winload.e f i (2nd phase) starts WinDbg session before loading the kernel • It cannot be used for debugging bootmgfw.e f i (1st phase) • Issue #2: Dealing with KASLR • KASLR is also applied to EFI runtime services • It is hard to f ind EFI service caller from backtrace

Slide 32

Slide 32 text

Debugging Issue #1 • Issue #1: WinDbg is not available in early boot phase • winload.e f i starts WinDbg session before loading the kernel • It cannot be used for debugging bootmgfw.e f i • -> Ghidra: • Microsoft Symbol Server: • Also provides bootmgfw.e f i and winload.e f i symbols • Ghidra Firmware Utilities: • Adds EFI-speci f ic types and annotates well-known GUIDs

Slide 33

Slide 33 text

Debugging Issue #2 • Issue #2: Dealing with KASLR • winload.e f i loads NT OS kernel at randomized offset • It is also applied to EFI runtime services • It makes debugging EFI runtime services hard • -> Custom GDB Python script • Hooks kernel hand off and calculates KASLR offset • Set breakpoints using the offset value