Massive Bare-Metal Operating System Provisioning Improvement Date Huang date_huang@edge-core.com OpenInfra Day Taiwan 2019/11/12

About me ● Date Huang 黃宇強 ● DozenCloud ARMv8 Cloud ● Edgecore Networks Sr. Engineer ○ SDN Solution for Datacenter, Cloud

Nowadays Cloud ● Cloud Techs change FAST ● Infrastructure-as-Code (IaC) for System Provisioning and Service Deployment ● Software-Defined Datacenter (SDDC) for System Provisioning and Service Deployment ● Re-deploy Operating System to Bare-Metal Servers ● Bare-Metal System Provisioning toolkit

Nowadays Cloud ● Cloud Techs change FAST ● Infrastructure-as-Code (IaC) for System Provisioning and Service Deployment ● Software-Defined Datacenter (SDDC) for System Provisioning and Service Deployment ● Re-deploy Operating System to Bare-Metal Servers ● Bare-Metal System Provisioning toolkit

Legacy Approach ● For Massive System Provisioning ● Use Unicast / Multicast to transfer OS image to server disks ● Save temporary image in RAM or another partition or disk ● Unitcast ○ Metal as a Service (MaaS) ○ OpenStack Ironic ● Multicast ○ Clonezilla Server Edition (Made in Taiwan) ○ Symantec Ghost Solution Suite

No content

Problems need to be sloved ● Cost lots of time and Remaining Time is not predictable ● Every issue will cause System Provisioning process stuck ● Targets will share server’s bandwidth via Unicast (Scale Difficult, Slow down) ● Targets aren’t always stable. Need to re-transfer data blocks via Multicast (Slow down) ● Targets need to join deployment group and start at the same time via Multicast

Problems need to be sloved cont’d ● Unicast is HARD to scaling up / out ○ Server Bandwidth isn’t enough ● Mutlicast / Broadcast is NOT always available ○ Enterprise Network / Public Cloud disallow Multicast / Broadcast ○ L2-less Datacenter (Clos Network)

Bittorrent ● Bittorrent-based System Provisioning ● Avoid Multicast / Broadcast ● Async Data Transfer ○ Don’t need to join deployment group simultaneously ● Reduce Image Provider Network Loading (Server-side)

No content

Challenge in Bittorrent ● For Peer-to-Peer file sharing, Temporary Storage is Required ○ Can’t assume getting data order in Bittorrent ○ Need to store whole disk image ● For Bare-Metal Provisioning, the available storage is RAM ● Image size will be Limited by RAM size ○ RAM is normally not big enough to store images and be ready to convert / restore to disk

Brand New Mechanism - EZIO ● Compatible with regular Bittorrent Protocol ○ Using Libtorrent-rasterbar ● Seed / Proxy Image with regular Bittorrent Client ○ e.g qBittorrent, transmission ● Read/Write image from/into Disk Directly ○ No need any temporary storage to store image for converting/restoring to disk ● https://github.com/tjjh89017/ezio

EZIO Design - Image Format ● Design BT-compatible image format ● Continuous block into one file, save offset in ‘hex’ into name

EZIO Design - Torrent ● Torrent will record all continuous block ○ offset and length ● Calculate SHA1 hash to verify

EZIO Design - Partition Offset ● ‘Piece’ is minimum unit in Bittorrent ● Cut continuous block into piece (Info is saved in torrent)

EZIO Design - Partition Offset cont’d ● When recv ‘Piece 1’, we will know data is combined with 2 blocks ○ 0x1000000 - 0x10a2000 ○ 0x8000000 - 0x8f5e000 ● Directly read/write from disk

EZIO Compared with Legacy Approach ● Case ○ Massive OS Deployment to multiple Servers in Data Center ○ OS Image is larger than 50GB ● Experimental environment ○ Network: Cisco 3560G 1Gbps ○ Server: Dell T1700 with Intel Xeon E3-1226, 16G ram, 1TB hard disk ○ PC Client: 32 Client, same as Server ○ Image: Ubuntu Linux with 50GB data in disk. Multicast Image is compressed by pzstd. BT Image is raw file

EZIO Compared with Legacy Approach cont’d ● Legacy Approach (e.g. CloneZilla Multicast Mode) ○ Slow, 0.5~2 GiB/min (8~34 MB/s) ○ If one server stuck, ALL servers stuck ○ CANNOT be used in L2-less environment (Broadcast and Multicast are not available)

EZIO Compared with Legacy Approach cont’d ● Bittorrent Deployment (EZIO) ○ Fast, 2.5~6 GiB/min (42~100 MB/s) ○ Performance, Reliability and Flexibility Improvements in Heterogeneous Environment (e.g. 100Mbps, 1Gbps, HDD, SSD, etc) ○ If one server stuck, others continue and no impact at all ○ Can be used in L2-less environment

Number of client Time (Unicast) Time (EZIO) Time (Multicast) Ratio (BT/Multicast) 1 474 675 390 1.731 2 948 1273 474 2.686 4 1896 1331 638 2.086 8 3792 1412 980 1.441 16 7584 1005 1454 0.691 24 11376 1048 1992 0.526 32 15168 1143 2203 0.519

Number of client Time (Unicast) Time (EZIO) Time (Multicast) Ratio (BT/Multicast) 1 474 675 390 1.731 2 948 1273 474 2.686 4 1896 1331 638 2.086 8 3792 1412 980 1.441 16 7584 1005 1454 0.691 24 11376 1048 1992 0.526 32 15168 1143 2203 0.519 Incresing Incresing Flat

Easy way to try EZIO ● Clonezilla Lite Server Mode ○ https://clonezilla.org/ ○ https://clonezilla.org/show-live-doc-content.p hp?topic=clonezilla-live/doc/12_lite_server_BT _from_dev

Demo ● CloneZilla Lite Server with EZIO support ● https://clonezilla.org/

Still Working on ● gRPC Support to control EZIO daemon ● Working on FAST deployment in DataCenter ○ Patch OpenStack Ironic to support BT Provisioning ○ Patch MaaS Curtin to support BT Provisioning

Any Questions? ● https://github.com/tjjh89017/ezio ● tjjh89017@hotmail.com ● date_huang@edge-core.com

Reference ● https://www.mdpi.com/2076-3417/9/2/296 ● https://github.com/tjjh89017/ezio ● https://www.libtorrent.org