Zespre (Chih-Hsin) Chang, NTUST, 2024-08-03 Towards a Robust FreeBSD-based Cloud: Porting OpenStack Components COSCUP 2024 

About Me • Zespre (Chih-Hsin) Chang • Email: [email protected] • Blog: https://blog.zespre.com • Roles • Senior Software Engineer at SUSE • Consultant at FreeBSD Foundation • Projects I’m currently working on • OpenStack on FreeBSD (today’s topic) • Harvester HCI • KubeVirtBMC 2 

Agenda • Introduction • Background • Current Status • Challenges • Roadmap • Conclusion 3 

OpenStack on FreeBSD Project origin • CHERI (Capability Hardware Enhanced RISC Instructions) • Managing a set of Morello evaluation boards with OpenStack Ironic • The OpenStack on FreeBSD project • Started in January 2022 • Chih-Hsin Chang & Li-Wen Hsu (lwhsu@) • Initially targeting on porting OpenStack Ironic • Pivot to VM- fi rst 4 

OpenStack on FreeBSD Open what? • A cloud infrastructure for virtual machines, containers, and bare metals • Consist of a stack of open-source software component 5 

https://docs.openstack.org/install-guide/get-started-logical-architecture.html 

OpenStack Essentials Keystone • Authentication and authorization • Support LDAP backend • Service discovery 7 

OpenStack Essentials Glance & Placement • Glance • Serve VM images and their metadata • Support various storage backends: RBD, Swift, Cinder, … 8 • Placement • Track cloud resource inventory and usage • Help other services, e.g. Nova, make the decision about resource allocation 

OpenStack Essentials Neutron • API server • Agents • `neutron--agent`: L2 network connectivity to OpenStack resources • `neutron-l3-agent`: virtual routers and fl oating IPs • `neutron-dhcp-agent`: IP address issuance • `neutron-metadata-agent`: cloud-init metadata and user data • ML2 (Modular Layer 2) plug-ins • Type drivers: fl at, Geneve, GRE, VLAN, and VXLAN • Mechanism drivers: Open vSwitch, Linux bridge, OVN, SRIOV, MacVTap and L2 population 9 

OpenStack Essentials Nova • API server • Scheduler • Collect resource usage from compute nodes/Placement API • Decide what node to run the instance • Conductor • Prepare instance information based on DB entries 10 

OpenStack Essentials Nova • Compute • Manage instance lifecycle through hypervisor on each compute node • Hypervisor manager • Serial proxy • Provide access to instance console over WebSocket 11 

OpenStack Essentials Ironic • Manages bare-metals in contrast to typical Nova usage • Deployment models • Stand-alone mode • Keystone + Ironic • As a Nova virt driver 12 

Project Status Dev environment • In-house development environment • Processors: 2 x Intel Xeon E5-2680 v4 • Motherboard: Supermicro X10DRL-i • Memory: 64 GB • Storage: 1 TB SSD • Single-node, all-in-one cluster • Remote PoC site: openstack[012] • Under construction • Targeting OpenStack 2024.1 Caracal + FreeBSD 15.0-CURRENT • Three-node HA cluster 13 

14 So, how does it look right now? • Install from source • Install dependencies • Pull down the (modi fi ed) source code • Build and install OpenStack packages • Each component runs in its own Python virtual environment Project Status 

Project Status So, how does it look right now? • Keystone: source code unmodi fi ed • Glance: source code unmodi fi ed • Placement: source unmodi fi ed • Neutron • Code patches • Con fi guration: fl at network + Open vSwitch • Nova • Code patches • Con fi guration: libvirt + bhyve 15 

Project Status So, how does it look right now? • Limitations • No tenant network isolation • Need external DHCP service • No fl oating IPs 16 

Events Gaining exposure • AsiaBSDCon 2023 (FreeBSD Developer Summit) • AsiaBSDCon 2024 • BSDCan 2024 17 

Demo Create a new instance 

Project Status The OpenStack on FreeBSD GitHub organization • Step-by-step build and installation guide can be found at openstack-on- freebsd/docs • Issue management is centralized in openstack-on-freebsd/admin • Source code • openstack-on-freebsd/neutron • openstack-on-freebsd/nova 19 https://github.com/openstack-on-freebsd 

Project Status The OpenStack on FreeBSD GitHub organization • Ports • openstack-on-freebsd/openstack (kudos to chuck@) • Custom solutions • openstack-on-freebsd/socat-manager • openstack-on-freebsd/novaconsole 20 https://github.com/openstack-on-freebsd 

Challenges Computing • Nova abstracts the operations against the underlying hypervisors • Nova virtualization driver • Well-de fi ned interfaces • Per-compute node con fi guration • Currently supported drivers • `libvirt.LibvirtDriver` • `fake.FakeDriver` • `ironic.IronicDriver` • `vmwareapi.VMwareVCDriver` • `zvm.zvmDriver` 21 virt driver 

Acts Leveraging the libvirt Driver • libvirt • Only implement a limited set of functions for FreeBSD/bhyve • libvirt virt driver • Some operations speci fi c to bhyve are not covered by libvirt • Require a new virtualization type - bhyve 22 libvirt drv. bhyve 

Challenges Networking • L2 agents • Lack of Linux bridge and iptables • L3 agent • Lack of iptables • DHCP agent • Lack of Linux network namespace and veth pairs • The current choice of ML2 drivers for FreeBSD • Type driver: `flat` • Mechanism driver: `openvswitch` 23 

24 VLAN + Open vSwitch (Linux host) Flat + Open vSwitch (FreeBSD host) 

Acts Open vSwitch on FreeBSD • Open vSwitch `datapath_type=netdev`, without DPDK • No `openvswitch` kernel module • The combination is considered experimental (not tested thoroughly) • Performance issue • TODOs • Enable DPDK for better performance • Develop the native FreeBSD bridge agent • Replace Linux network namespace + veth(4) with vnet(9) + epair(4) for DHCP functionality • Replace Linux network namespace + iptables(8) with vnet(9) + pf(4) for virtual router functionality • Replace iptables(8) with pf(4) for security group functionality 25 

Challenges IP Address Mismatch 1. `neutron-dhcp-agent` issue 2. Disable DHCP for the fl at network 3. Set up a DHCP server external to the cluster 4. VMs get IP addresses from the external DHCP server 5. Source IP address does not match the one Neutron allocated • Policy enforced by the fl ow rules in the underlying Open vSwitch 6. Packets originating from VMs get dropped silently 26 

Acts IP Address Mismatch • Workaround • Con fi gure VM’s IP address via console • Formal solution • Make `neutron-dhcp-agent` function normally on FreeBSD • Due to this workaround, another challenge emerged… 27 

Challenges VM consoles • cu(1) - call Unix • Required access to the compute host • Impractical and insecure 28 

Acts Double proxying • Introduce socat-manager • Listen on Unix socket • Maintain TCP port to nmdm(4) mappings • Manage socat(1) processes • libvirt hook script • Take domain XML as input • Call socat-manager with parameters (port and nmdm(4) device name) 29 $ /usr/local/bin/socat \ file:/dev/nmdm21B,ispeed=9600,ospeed=9600,raw,echo=0 \ tcp-listen:10014,bind=0.0.0.0,reuseaddr,fork 

30 On Linux hosts On FreeBSD hosts 

Challenges Privilege • Principle of least privilege • Run with reduced/no privilege • Escalate when absolutely required • Operations will be translated into commands and run on the OS 31 

Challenges The evolution of privilege mechanism in OpenStack • sudo(8) • One-shot • All or nothing • `oslo.rootwrap` • Allow advanced fi lters • Support one-shot or daemon mode • Performance penalty • Does not allow long-lived/streaming commands 32 $ sudo command $ sudo nova—rootwrap \ /etc/nova/rootwrap.conf \ command 

Challenges The evolution of privilege mechanism in OpenStack • `oslo.privsep` • Leverage Linux capabilities(7) • Drop root superpowers but only keep what is required • Two-process model (unprivileged and privileged) • Connected over a local communication channel • Share the same fate 33 

Acts Fallback • Linux capabilities(7) are not available on FreeBSD • Workaround • Fallback using rootwrap • Formal solution • Leverage FreeBSD’s own privilege management mechanism • mac(9) framework 34 

Future Work • Develop native drivers for components that have plug-ins architecture • Neutron • Nova • Port additional OpenStack components to FreeBSD • Ironic • Horizon • Create corresponding FreeBSD ports • Catch up on newer OpenStack releases • 2024.01 Caracal! • Continuous engagement and knowledge sharing • Performance and scalability improvements 35 

Conclusion • Bring Linux- fi rst design to FreeBSD • Follow the Windows path (?) • Use cases are very limited • We dropped too many things to make it viable • There are many topics/issues that need expertise • Explore network implementation equivalents • Finding a suitable privilege separation implementation • Need to formalize the changes 36 

Thank you Got questions? -> [email protected]