SREcon20 Give Your PXE Wings!

What is PXE anyway and why does it work?
System bootstrapping is one of the great mysteries in IT. In this talk, we'll break the entire bootstrapping process down into components. That will allow us to discuss ways to make it faster, simpler, and more reliable. Most importantly, we'll show how you can improve the automation options involved in day two operations for anything that boots.

Transcript

1. Give Your PXE Wings!
   It’s not magic! How booting actually works.
   Presentation for virtual SREcon 2020
   By Rob Hirschfeld, RackN
   

   View Slide

2. Rob Hirschfeld
   @zehicle
   Co-Founder of RackN
   We created Digital Rebar
   Bare Metal Provisioning ++
   @L8istSh9y Podcast on PXE:
   http://bit.ly/pxewings
   

   View Slide

3. O/S
   In concept,
   Provisioning is Easy!
   We’re just installing an operating
   system on a server or switch!
   Why is that so hard?!
   ● Bootstrapping
   ● Firmware Limitations
   ● Variation
   ● Networking
   ● Security
   ● Performance
   ● Post configuration
   O/S
   Computer
   

   View Slide

4. In concept,
   Provisioning is Easy!
   We’re just installing an operating
   system on a server or switch!
   Why is that so hard?!
   ● Bootstrapping
   ● Firmware Limitations
   ● Variation
   ● Networking
   ● Security
   ● Performance
   ● Post configuration
   Pre- &
   Post-
   Config
   And that’s not even including:
   ● System Inventory
   ● System Validation
   ● Hardware Configuration
   ● Naming & Addressing
   ● Credentials Injection
   

   View Slide

5. Exploring Provisioning Approaches
   Netboot (25/40 min)
   ● PXE
   ● iPXE
   ● ONIE
   ● Kickstart
   ● Preseed
   Esoteric Flavors (5/40 min)
   ● kexec
   ● Secure Boot
   ● BMC Boot
   Image Deploy (10/40 min)
   ● Packer
   ● Write Boot Part
   ● Cloud Init
   

   View Slide

6. Exploring Provisioning Approaches
   All roads lead to a kernel init process
   Netboot (25/40 min)
   ● PXE
   ● iPXE
   ● ONIE
   ● Kickstart
   ● Preseed
   Esoteric Flavors (5/40 min)
   ● kexec
   ● Secure Boot
   ● BMC Boot
   Image Deploy (10/40 min)
   ● Packer
   ● Write Boot Part
   ● Cloud Init
   

   View Slide

7. PXE
   

   View Slide

8. PXE
   

   View Slide

9. Let’s PXE!
   

   View Slide

10. Server
    lpxelinux.0
    Bootstrapping is a multi-stage process
    Firmware
    PXE
    DHCP
    NextServer
    & Options
    Bootloader
    Stage 1
    Provisioning
    Service(s)
    TFTP
    ipxe.efi
    Bootloader
    Stage 2
    HTTP(S)
    O/S ISO
    HTTP(S)
    O/S Kernel
    

    View Slide

11. Server
    First: Get on the network
    Firmware
    PXE
    DHCP
    NextServer
    & Options
    Provisioning
    Service(s)
    

    View Slide

12. Server
    lpxelinux.0
    Then: Download a Bootloader
    Bootloader
    Stage 1
    Provisioning
    Service(s)
    TFTP
    

    View Slide

13. Server
    Then get a BETTER Bootloader
    Provisioning
    Service(s)
    ipxe.efi
    Bootloader
    Stage 2
    HTTP(S)
    

    View Slide

14. Server
    Finally load a “real” operating system
    Provisioning
    Service(s)
    O/S ISO
    HTTP(S)
    O/S Kernel
    

    View Slide

15. Server
    lpxelinux.0
    Each stage is actually a NEW O/S Load
    Firmware
    PXE
    DHCP
    NextServer
    & Options
    Bootloader
    Stage 1
    Provisioning
    Service(s)
    TFTP
    ipxe.efi
    Bootloader
    Stage 2
    HTTP(S)
    O/S ISO
    HTTP(S)
    O/S Kernel
    DHCP
    DHCP
    DHCP
    DHCP
    

    View Slide

16. Server
    And modern servers can skip TFTP!
    So… technically, no longer PXE
    Firmware
    PXE
    DHCP
    NextServer
    & Options
    Provisioning
    Service(s)
    ipxe.efi
    Bootloader
    Stage 2
    HTTP(S)
    O/S ISO
    HTTP(S)
    O/S Kernel
    

    View Slide

17. Yay! We’re done, right?
    

    View Slide

18. Provisioning
    Service(s)
    Server
    Provisioning is more than PXE
    O/S Kernel
    iPXE
    Bootloader
    O/S ISO
    Kickstart
    Installation
    Post-Config
    Config
    Templates
    Download
    Packages
    Access &
    Apps
    

    View Slide

19. Provisioning
    Service(s)
    Server
    Hardware varies, so
    Install must be guided by templates
    O/S Kernel
    Kickstart
    Config
    Templates
    

    View Slide

20. Provisioning
    Service(s)
    Server
    ISOs are minimal and stale
    So they must be updated
    O/S Kernel
    Kickstart
    Installation
    Download
    Packages
    Repo
    Mirrors
    

    View Slide

21. Provisioning
    Service(s)
    Server
    And then you can actually connect
    to start configuring the system!
    O/S Kernel
    Kickstart
    Installation
    Post-Config
    Access &
    Apps
    

    View Slide

22. Provisioning
    Service(s)
    Server
    Automating Provisioning means
    Connecting all these steps together
    O/S Kernel
    iPXE
    Bootloader
    O/S ISO
    Kickstart
    Installation
    Post-Config
    Config
    Templates
    Download
    Packages
    Access &
    Apps
    

    View Slide

23. Infrastrastructure
    as Code
    iPXE
    Bootloader
    O/S ISO
    Provisioning
    Service(s)
    Server
    But wait….
    There’s more to consider!
    O/S Kernel
    Kickstart
    Installation
    Post-Config
    Config
    Templates
    Download
    Packages
    Access &
    Apps
    Out of Band Management
    (BMC, IPMI, Redfish, etc)
    

    View Slide

24. IaC?
    Show us some templates!
    

    View Slide

25. Typical PXE Questions
    Why is this so fragile?
    What about PXE over Wifi?
    What about using a VLAN?
    Can I dockerize this?
    What about setting BIOS & RAID?
    How can I make this faster?
    

    View Slide

26. How can we
    simplify that?!!
    At RackN, we’ve been using an in
    memory operating system,
    “sledgehammer,” based on CentOS.
    It’s highly optimized to
    ● Run on nearly any hardware
    ● Load very quickly
    ● Collect deep inventory
    ● Have built-in tools for system
    tasks like hardware config
    Provisioning
    Service(s)
    Server
    PXE
    PXE/iPXE
    Small
    Footprint
    RAM only
    O/S
    Informed
    Installation
    Guided
    Installation
    Inventory
    reboot
    

    View Slide

27. Image Based Deployment (10x faster!)
    Provisioning
    Service(s)
    Server
    PXE
    PXE/iPXE
    Small
    Footprint
    RAM only
    O/S
    Write O/S
    To Drive(s)
    O/S Image
    as Archive
    Informed
    Installation
    reboot
    Machine
    Init
    

    View Slide

28. Container
    Registries
    Immutable Provisioning
    Highly
    Available
    Provisioning
    Service(s)
    Server
    PXE PXE/iPXE
    Minimal
    Footprint
    RAM only
    O/S
    Config &
    Attach Disks
    Machine
    Initialize
    Load Apps
    & Containers
    

    View Slide

29. And now…
    Advanced Provisioning!
    

    View Slide

30. ESXi Provisioning
    Provisioning
    Service(s)
    Server
    O/S Kernel
    iPXE
    Bootloader
    O/S ISO
    weasel
    restricted
    CLI/python
    Config
    Templates
    Access &
    Apps
    Control via
    VMw APIs
    VMware
    Tooling
    ESXi
    

    View Slide

31. ONIE: Open Network Install Environment
    Designed for Embedded Systems where we’re
    replacing the O/S as a complete image.
    Does have DHCP options for a startup script.
    Provisioning
    Service(s)
    Switch
    Current
    Firmware
    DHCP
    HTTP(S)
    O/S Image
    New
    Firmware
    

    View Slide

32. Server
    Running O/S
    kexec (kernel execute)
    New Kernel
    Provisioning
    Service(s)
    Normal
    Provision
    New O/S
    Download
    New Kernel
    Starts
    Without
    Rebooting
    kexec
    

    View Slide

33. Server
    Running O/S
    kexec (kernel execute)
    New Kernel
    Provisioning
    Service(s)
    Normal
    Provision
    New O/S
    Download
    New Kernel
    New
    Installation
    kexec
    kexec
    Start
    iPXE
    Normal
    Provision
    Without
    Rebooting
    

    View Slide

34. Secure Boot Required SIGNED Bootloaders
    Server
    Secured
    Firmware
    DHCP
    NextServer
    & Options
    Provisioning
    Service(s)
    Signed
    ipxe.efi
    Trusted
    Bootloader
    HTTP(S)
    Signed
    O/S ISO
    HTTP(S)
    Trusted
    O/S Kernel
    Enabled
    Verified
    Verified
    TPM
    
    

    View Slide

35. BMC Boot option 1
    Server
    Firmware
    Install
    Media
    Provisioning
    Service(s)
    BMC
    Attached
    O/S Install
    Kickstart
    Installation
    Post-Config
    Download
    Packages
    Access &
    Apps
    

    View Slide

36. BMC Boot option 2
    Server
    Firmware
    Install
    Media
    Provisioning
    Service(s)
    Bootloader
    Kickstart
    Installation
    Post-Config
    Download
    Packages
    Access &
    Apps
    BMC
    Config
    Templates
    DHCP
    

    View Slide

37. Thanks!
    Contact us:
    Rob Hirschfeld, RackN.com
    Digital Rebar
    Behind the Firewall, Self-Service
    Infrastructure as Code
    Self-Trials: rebar.digital
    

    View Slide