PremDay #3 - Server provisioning at Proton

Proton explains how they build images that use a fully end-to-end signed bootchain

Transcript

1. Server provisioning at Proton a.k.a. Building images that use a fully end to end signed bootchain
2. Me - anthony.callegaro@proton.ch Head of Infrastructure-Systems team at Proton Taking care of platforms (Baremetal, VM, K8s provisioning) Base infrastructure (AAA, DNS, DHCP, NTP...) Base infrastructure tooling (SoT, configuration management, automation platform...) OS and security update 2
3. Scope of the talk Will only talk of Bare Metal and VMs Most of our apps are running on K8s (which we are already moving to Talos) But we still have a lot of requests for Bare Metal (Databases, Load Balancers...), VMs (mails and abuse, DNS, DHCP...) This is an ongoing POC 3
4. History - Provisioning - 5 years ago Manual PXE Scripts that generate grub and kickstart from Source of Truth (SoT) Network configured by scripts ran from SoT (netbox) data with manually inputed data Tons of manual intevention to configure HW RAID, upgrade FW and fix broken puppet 4
5. History - Configuration management - 7 years ago Puppet inherited from our friends at CERN Puppet Enforce state Ansible deployment for complex payload 5
6. Now - Provisioning Using Rackn Rebar for Discovery/HW Testing/FW upgrade/Provisioning Fully integrated with netbox Automation to configure network Still need an operator Still install OS and run puppet 6
7. Now - Configuration Management Puppet Enforce state ? Agent is disabled on prod Most host with agent have various issues Everything is done through manual interaction Very few teams are actually using puppet correctly Currently thousand of servers with ten of thousand of changes to apply 7
8. Our new plaform Goals Fully automated provisioning Scale like an hyperscaler KISS ! Fully declarative 8
9. Our New platform paradigm Build same base images for BareMetal, VM, public cloud, containing AAA Security tooling Monitoring Base OS and Services Different OSes and architecture Able to update the base without client intervention 9
10. Our New platform paradigm Immutable RO /root A/B partitioning for update Live patching through overlay Tamper proof fully signed end to end bootchain 10
11. Our New platform paradigm Users/client can use base image or build custom profile on top of it They can use whatever configuration management or tool to customise it CI that will build constantly updated base images and custom images Customisation on boot pulled from SoT. Teams can also customise their install there. 11
12. Solution mkosi for image building distro agnostic systemd based easy secureboot integration with auto-enrollment UKI boot images (kernel + initrd + kernel cmdline) dm-verity for partition signing Tinkerbell for provisioning (user redfish or openbmc) 12
13. Solution's process Tested with canary Fixed schedule every X month for release Team's responsibility to test before roll out 13