SREcon20 Give Your PXE Wings!

What is PXE anyway and why does it work?
System bootstrapping is one of the great mysteries in IT. In this talk, we'll break the entire bootstrapping process down into components. That will allow us to discuss ways to make it faster, simpler, and more reliable. Most importantly, we'll show how you can improve the automation options involved in day two operations for anything that boots.

Transcript

1. Give Your PXE Wings! It’s not magic! How booting actually works. Presentation for virtual SREcon 2020 By Rob Hirschfeld, RackN
2. Rob Hirschfeld @zehicle Co-Founder of RackN We created Digital Rebar Bare Metal Provisioning ++ @L8istSh9y Podcast on PXE: http://bit.ly/pxewings
3. O/S In concept, Provisioning is Easy! We’re just installing an operating system on a server or switch! Why is that so hard?! • Bootstrapping • Firmware Limitations • Variation • Networking • Security • Performance • Post configuration O/S Computer
4. In concept, Provisioning is Easy! We’re just installing an operating system on a server or switch! Why is that so hard?! • Bootstrapping • Firmware Limitations • Variation • Networking • Security • Performance • Post configuration Pre- & Post- Config And that’s not even including: • System Inventory • System Validation • Hardware Configuration • Naming & Addressing • Credentials Injection
5. Exploring Provisioning Approaches Netboot (25/40 min) • PXE • iPXE • ONIE • Kickstart • Preseed Esoteric Flavors (5/40 min) • kexec • Secure Boot • BMC Boot Image Deploy (10/40 min) • Packer • Write Boot Part • Cloud Init
6. Exploring Provisioning Approaches All roads lead to a kernel init process Netboot (25/40 min) • PXE • iPXE • ONIE • Kickstart • Preseed Esoteric Flavors (5/40 min) • kexec • Secure Boot • BMC Boot Image Deploy (10/40 min) • Packer • Write Boot Part • Cloud Init
7. PXE
8. PXE
9. Let’s PXE!
10. Server lpxelinux.0 Bootstrapping is a multi-stage process Firmware PXE DHCP NextServer & Options Bootloader Stage 1 Provisioning Service(s) TFTP ipxe.efi Bootloader Stage 2 HTTP(S) O/S ISO HTTP(S) O/S Kernel
11. Server First: Get on the network Firmware PXE DHCP NextServer & Options Provisioning Service(s)
12. Server lpxelinux.0 Then: Download a Bootloader Bootloader Stage 1 Provisioning Service(s) TFTP
13. Server Then get a BETTER Bootloader Provisioning Service(s) ipxe.efi Bootloader Stage 2 HTTP(S)
14. Server Finally load a “real” operating system Provisioning Service(s) O/S ISO HTTP(S) O/S Kernel
15. Server lpxelinux.0 Each stage is actually a NEW O/S Load Firmware PXE DHCP NextServer & Options Bootloader Stage 1 Provisioning Service(s) TFTP ipxe.efi Bootloader Stage 2 HTTP(S) O/S ISO HTTP(S) O/S Kernel DHCP DHCP DHCP DHCP
16. Server And modern servers can skip TFTP! So… technically, no longer PXE Firmware PXE DHCP NextServer & Options Provisioning Service(s) ipxe.efi Bootloader Stage 2 HTTP(S) O/S ISO HTTP(S) O/S Kernel
17. Yay! We’re done, right?
18. Provisioning Service(s) Server Provisioning is more than PXE O/S Kernel iPXE Bootloader O/S ISO Kickstart Installation Post-Config Config Templates Download Packages Access & Apps
19. Provisioning Service(s) Server Hardware varies, so Install must be guided by templates O/S Kernel Kickstart Config Templates
20. Provisioning Service(s) Server ISOs are minimal and stale So they must be updated O/S Kernel Kickstart Installation Download Packages Repo Mirrors
21. Provisioning Service(s) Server And then you can actually connect to start configuring the system! O/S Kernel Kickstart Installation Post-Config Access & Apps
22. Provisioning Service(s) Server Automating Provisioning means Connecting all these steps together O/S Kernel iPXE Bootloader O/S ISO Kickstart Installation Post-Config Config Templates Download Packages Access & Apps
23. Infrastrastructure as Code iPXE Bootloader O/S ISO Provisioning Service(s) Server But wait…. There’s more to consider! O/S Kernel Kickstart Installation Post-Config Config Templates Download Packages Access & Apps Out of Band Management (BMC, IPMI, Redfish, etc)
24. IaC? Show us some templates!
25. Typical PXE Questions Why is this so fragile? What about PXE over Wifi? What about using a VLAN? Can I dockerize this? What about setting BIOS & RAID? How can I make this faster?
26. How can we simplify that?!! At RackN, we’ve been using an in memory operating system, “sledgehammer,” based on CentOS. It’s highly optimized to • Run on nearly any hardware • Load very quickly • Collect deep inventory • Have built-in tools for system tasks like hardware config Provisioning Service(s) Server PXE PXE/iPXE Small Footprint RAM only O/S Informed Installation Guided Installation Inventory reboot
27. Image Based Deployment (10x faster!) Provisioning Service(s) Server PXE PXE/iPXE Small Footprint RAM only O/S Write O/S To Drive(s) O/S Image as Archive Informed Installation reboot Machine Init
28. Container Registries Immutable Provisioning Highly Available Provisioning Service(s) Server PXE PXE/iPXE Minimal Footprint RAM only O/S Config & Attach Disks Machine Initialize Load Apps & Containers
29. And now… Advanced Provisioning!
30. ESXi Provisioning Provisioning Service(s) Server O/S Kernel iPXE Bootloader O/S ISO weasel restricted CLI/python Config Templates Access & Apps Control via VMw APIs VMware Tooling ESXi
31. ONIE: Open Network Install Environment Designed for Embedded Systems where we’re replacing the O/S as a complete image. Does have DHCP options for a startup script. Provisioning Service(s) Switch Current Firmware DHCP HTTP(S) O/S Image New Firmware
32. Server Running O/S kexec (kernel execute) New Kernel Provisioning Service(s) Normal Provision New O/S Download New Kernel Starts Without Rebooting kexec
33. Server Running O/S kexec (kernel execute) New Kernel Provisioning Service(s) Normal Provision New O/S Download New Kernel New Installation kexec kexec Start iPXE Normal Provision Without Rebooting
34. Secure Boot Required SIGNED Bootloaders Server Secured Firmware DHCP NextServer & Options Provisioning Service(s) Signed ipxe.efi Trusted Bootloader HTTP(S) Signed O/S ISO HTTP(S) Trusted O/S Kernel Enabled Verified Verified TPM
35. BMC Boot option 1 Server Firmware Install Media Provisioning Service(s) BMC Attached O/S Install Kickstart Installation Post-Config Download Packages Access & Apps
36. BMC Boot option 2 Server Firmware Install Media Provisioning Service(s) Bootloader Kickstart Installation Post-Config Download Packages Access & Apps BMC Config Templates DHCP
37. Thanks! Contact us: Rob Hirschfeld, RackN.com Digital Rebar Behind the Firewall, Self-Service Infrastructure as Code Self-Trials: rebar.digital