Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SREcon20 Give Your PXE Wings!

SREcon20 Give Your PXE Wings!

What is PXE anyway and why does it work?
System bootstrapping is one of the great mysteries in IT. In this talk, we'll break the entire bootstrapping process down into components. That will allow us to discuss ways to make it faster, simpler, and more reliable. Most importantly, we'll show how you can improve the automation options involved in day two operations for anything that boots.

Rob Hirschfeld

December 08, 2020
Tweet

More Decks by Rob Hirschfeld

Other Decks in Technology

Transcript

  1. Give Your PXE Wings!
    It’s not magic! How booting actually works.
    Presentation for virtual SREcon 2020
    By Rob Hirschfeld, RackN

    View Slide

  2. Rob Hirschfeld
    @zehicle
    Co-Founder of RackN
    We created Digital Rebar
    Bare Metal Provisioning ++
    @L8istSh9y Podcast on PXE:
    http://bit.ly/pxewings

    View Slide

  3. O/S
    In concept,
    Provisioning is Easy!
    We’re just installing an operating
    system on a server or switch!
    Why is that so hard?!
    ● Bootstrapping
    ● Firmware Limitations
    ● Variation
    ● Networking
    ● Security
    ● Performance
    ● Post configuration
    O/S
    Computer

    View Slide

  4. In concept,
    Provisioning is Easy!
    We’re just installing an operating
    system on a server or switch!
    Why is that so hard?!
    ● Bootstrapping
    ● Firmware Limitations
    ● Variation
    ● Networking
    ● Security
    ● Performance
    ● Post configuration
    Pre- &
    Post-
    Config
    And that’s not even including:
    ● System Inventory
    ● System Validation
    ● Hardware Configuration
    ● Naming & Addressing
    ● Credentials Injection

    View Slide

  5. Exploring Provisioning Approaches
    Netboot (25/40 min)
    ● PXE
    ● iPXE
    ● ONIE
    ● Kickstart
    ● Preseed
    Esoteric Flavors (5/40 min)
    ● kexec
    ● Secure Boot
    ● BMC Boot
    Image Deploy (10/40 min)
    ● Packer
    ● Write Boot Part
    ● Cloud Init

    View Slide

  6. Exploring Provisioning Approaches
    All roads lead to a kernel init process
    Netboot (25/40 min)
    ● PXE
    ● iPXE
    ● ONIE
    ● Kickstart
    ● Preseed
    Esoteric Flavors (5/40 min)
    ● kexec
    ● Secure Boot
    ● BMC Boot
    Image Deploy (10/40 min)
    ● Packer
    ● Write Boot Part
    ● Cloud Init

    View Slide

  7. PXE

    View Slide

  8. PXE

    View Slide

  9. Let’s PXE!

    View Slide

  10. Server
    lpxelinux.0
    Bootstrapping is a multi-stage process
    Firmware
    PXE
    DHCP
    NextServer
    & Options
    Bootloader
    Stage 1
    Provisioning
    Service(s)
    TFTP
    ipxe.efi
    Bootloader
    Stage 2
    HTTP(S)
    O/S ISO
    HTTP(S)
    O/S Kernel

    View Slide

  11. Server
    First: Get on the network
    Firmware
    PXE
    DHCP
    NextServer
    & Options
    Provisioning
    Service(s)

    View Slide

  12. Server
    lpxelinux.0
    Then: Download a Bootloader
    Bootloader
    Stage 1
    Provisioning
    Service(s)
    TFTP

    View Slide

  13. Server
    Then get a BETTER Bootloader
    Provisioning
    Service(s)
    ipxe.efi
    Bootloader
    Stage 2
    HTTP(S)

    View Slide

  14. Server
    Finally load a “real” operating system
    Provisioning
    Service(s)
    O/S ISO
    HTTP(S)
    O/S Kernel

    View Slide

  15. Server
    lpxelinux.0
    Each stage is actually a NEW O/S Load
    Firmware
    PXE
    DHCP
    NextServer
    & Options
    Bootloader
    Stage 1
    Provisioning
    Service(s)
    TFTP
    ipxe.efi
    Bootloader
    Stage 2
    HTTP(S)
    O/S ISO
    HTTP(S)
    O/S Kernel
    DHCP
    DHCP
    DHCP
    DHCP

    View Slide

  16. Server
    And modern servers can skip TFTP!
    So… technically, no longer PXE
    Firmware
    PXE
    DHCP
    NextServer
    & Options
    Provisioning
    Service(s)
    ipxe.efi
    Bootloader
    Stage 2
    HTTP(S)
    O/S ISO
    HTTP(S)
    O/S Kernel

    View Slide

  17. Yay! We’re done, right?

    View Slide

  18. Provisioning
    Service(s)
    Server
    Provisioning is more than PXE
    O/S Kernel
    iPXE
    Bootloader
    O/S ISO
    Kickstart
    Installation
    Post-Config
    Config
    Templates
    Download
    Packages
    Access &
    Apps

    View Slide

  19. Provisioning
    Service(s)
    Server
    Hardware varies, so
    Install must be guided by templates
    O/S Kernel
    Kickstart
    Config
    Templates

    View Slide

  20. Provisioning
    Service(s)
    Server
    ISOs are minimal and stale
    So they must be updated
    O/S Kernel
    Kickstart
    Installation
    Download
    Packages
    Repo
    Mirrors

    View Slide

  21. Provisioning
    Service(s)
    Server
    And then you can actually connect
    to start configuring the system!
    O/S Kernel
    Kickstart
    Installation
    Post-Config
    Access &
    Apps

    View Slide

  22. Provisioning
    Service(s)
    Server
    Automating Provisioning means
    Connecting all these steps together
    O/S Kernel
    iPXE
    Bootloader
    O/S ISO
    Kickstart
    Installation
    Post-Config
    Config
    Templates
    Download
    Packages
    Access &
    Apps

    View Slide

  23. Infrastrastructure
    as Code
    iPXE
    Bootloader
    O/S ISO
    Provisioning
    Service(s)
    Server
    But wait….
    There’s more to consider!
    O/S Kernel
    Kickstart
    Installation
    Post-Config
    Config
    Templates
    Download
    Packages
    Access &
    Apps
    Out of Band Management
    (BMC, IPMI, Redfish, etc)

    View Slide

  24. IaC?
    Show us some templates!

    View Slide

  25. Typical PXE Questions
    Why is this so fragile?
    What about PXE over Wifi?
    What about using a VLAN?
    Can I dockerize this?
    What about setting BIOS & RAID?
    How can I make this faster?

    View Slide

  26. How can we
    simplify that?!!
    At RackN, we’ve been using an in
    memory operating system,
    “sledgehammer,” based on CentOS.
    It’s highly optimized to
    ● Run on nearly any hardware
    ● Load very quickly
    ● Collect deep inventory
    ● Have built-in tools for system
    tasks like hardware config
    Provisioning
    Service(s)
    Server
    PXE
    PXE/iPXE
    Small
    Footprint
    RAM only
    O/S
    Informed
    Installation
    Guided
    Installation
    Inventory
    reboot

    View Slide

  27. Image Based Deployment (10x faster!)
    Provisioning
    Service(s)
    Server
    PXE
    PXE/iPXE
    Small
    Footprint
    RAM only
    O/S
    Write O/S
    To Drive(s)
    O/S Image
    as Archive
    Informed
    Installation
    reboot
    Machine
    Init

    View Slide

  28. Container
    Registries
    Immutable Provisioning
    Highly
    Available
    Provisioning
    Service(s)
    Server
    PXE PXE/iPXE
    Minimal
    Footprint
    RAM only
    O/S
    Config &
    Attach Disks
    Machine
    Initialize
    Load Apps
    & Containers

    View Slide

  29. And now…
    Advanced Provisioning!

    View Slide

  30. ESXi Provisioning
    Provisioning
    Service(s)
    Server
    O/S Kernel
    iPXE
    Bootloader
    O/S ISO
    weasel
    restricted
    CLI/python
    Config
    Templates
    Access &
    Apps
    Control via
    VMw APIs
    VMware
    Tooling
    ESXi

    View Slide

  31. ONIE: Open Network Install Environment
    Designed for Embedded Systems where we’re
    replacing the O/S as a complete image.
    Does have DHCP options for a startup script.
    Provisioning
    Service(s)
    Switch
    Current
    Firmware
    DHCP
    HTTP(S)
    O/S Image
    New
    Firmware

    View Slide

  32. Server
    Running O/S
    kexec (kernel execute)
    New Kernel
    Provisioning
    Service(s)
    Normal
    Provision
    New O/S
    Download
    New Kernel
    Starts
    Without
    Rebooting
    kexec

    View Slide

  33. Server
    Running O/S
    kexec (kernel execute)
    New Kernel
    Provisioning
    Service(s)
    Normal
    Provision
    New O/S
    Download
    New Kernel
    New
    Installation
    kexec
    kexec
    Start
    iPXE
    Normal
    Provision
    Without
    Rebooting

    View Slide

  34. Secure Boot Required SIGNED Bootloaders
    Server
    Secured
    Firmware
    DHCP
    NextServer
    & Options
    Provisioning
    Service(s)
    Signed
    ipxe.efi
    Trusted
    Bootloader
    HTTP(S)
    Signed
    O/S ISO
    HTTP(S)
    Trusted
    O/S Kernel
    Enabled
    Verified
    Verified
    TPM

    View Slide

  35. BMC Boot option 1
    Server
    Firmware
    Install
    Media
    Provisioning
    Service(s)
    BMC
    Attached
    O/S Install
    Kickstart
    Installation
    Post-Config
    Download
    Packages
    Access &
    Apps

    View Slide

  36. BMC Boot option 2
    Server
    Firmware
    Install
    Media
    Provisioning
    Service(s)
    Bootloader
    Kickstart
    Installation
    Post-Config
    Download
    Packages
    Access &
    Apps
    BMC
    Config
    Templates
    DHCP

    View Slide

  37. Thanks!
    Contact us:
    Rob Hirschfeld, RackN.com
    Digital Rebar
    Behind the Firewall, Self-Service
    Infrastructure as Code
    Self-Trials: rebar.digital

    View Slide