Massive Bare-Metal Operating System Provisioning Improvement - OpenInfra Day Taiwan 2019/11/12

Transcript

1. Massive Bare-Metal Operating System Provisioning Improvement Date Huang date_huang@edge-core.com OpenInfra Day Taiwan 2019/11/12
2. About me • Date Huang 黃宇強 • DozenCloud ARMv8 Cloud • Edgecore Networks Sr. Engineer ◦ SDN Solution for Datacenter, Cloud
3. Nowadays Cloud • Cloud Techs change FAST • Infrastructure-as-Code (IaC) for System Provisioning and Service Deployment • Software-Defined Datacenter (SDDC) for System Provisioning and Service Deployment • Re-deploy Operating System to Bare-Metal Servers • Bare-Metal System Provisioning toolkit
4. Nowadays Cloud • Cloud Techs change FAST • Infrastructure-as-Code (IaC) for System Provisioning and Service Deployment • Software-Defined Datacenter (SDDC) for System Provisioning and Service Deployment • Re-deploy Operating System to Bare-Metal Servers • Bare-Metal System Provisioning toolkit
5. Legacy Approach • For Massive System Provisioning • Use Unicast / Multicast to transfer OS image to server disks • Save temporary image in RAM or another partition or disk • Unitcast ◦ Metal as a Service (MaaS) ◦ OpenStack Ironic • Multicast ◦ Clonezilla Server Edition (Made in Taiwan) ◦ Symantec Ghost Solution Suite
7. Problems need to be sloved • Cost lots of time and Remaining Time is not predictable • Every issue will cause System Provisioning process stuck • Targets will share server’s bandwidth via Unicast (Scale Difficult, Slow down) • Targets aren’t always stable. Need to re-transfer data blocks via Multicast (Slow down) • Targets need to join deployment group and start at the same time via Multicast
8. Problems need to be sloved cont’d • Unicast is HARD to scaling up / out ◦ Server Bandwidth isn’t enough • Mutlicast / Broadcast is NOT always available ◦ Enterprise Network / Public Cloud disallow Multicast / Broadcast ◦ L2-less Datacenter (Clos Network)
9. Bittorrent • Bittorrent-based System Provisioning • Avoid Multicast / Broadcast • Async Data Transfer ◦ Don’t need to join deployment group simultaneously • Reduce Image Provider Network Loading (Server-side)
11. Challenge in Bittorrent • For Peer-to-Peer file sharing, Temporary Storage is Required ◦ Can’t assume getting data order in Bittorrent ◦ Need to store whole disk image • For Bare-Metal Provisioning, the available storage is RAM • Image size will be Limited by RAM size ◦ RAM is normally not big enough to store images and be ready to convert / restore to disk
12. Brand New Mechanism - EZIO • Compatible with regular Bittorrent Protocol ◦ Using Libtorrent-rasterbar • Seed / Proxy Image with regular Bittorrent Client ◦ e.g qBittorrent, transmission • Read/Write image from/into Disk Directly ◦ No need any temporary storage to store image for converting/restoring to disk • https://github.com/tjjh89017/ezio
13. EZIO Design - Image Format • Design BT-compatible image format • Continuous block into one file, save offset in ‘hex’ into name
14. EZIO Design - Torrent • Torrent will record all continuous block ◦ offset and length • Calculate SHA1 hash to verify
15. EZIO Design - Partition Offset • ‘Piece’ is minimum unit in Bittorrent • Cut continuous block into piece (Info is saved in torrent)
16. EZIO Design - Partition Offset cont’d • When recv ‘Piece 1’, we will know data is combined with 2 blocks ◦ 0x1000000 - 0x10a2000 ◦ 0x8000000 - 0x8f5e000 • Directly read/write from disk
17. EZIO Compared with Legacy Approach • Case ◦ Massive OS Deployment to multiple Servers in Data Center ◦ OS Image is larger than 50GB • Experimental environment ◦ Network: Cisco 3560G 1Gbps ◦ Server: Dell T1700 with Intel Xeon E3-1226, 16G ram, 1TB hard disk ◦ PC Client: 32 Client, same as Server ◦ Image: Ubuntu Linux with 50GB data in disk. Multicast Image is compressed by pzstd. BT Image is raw file
18. EZIO Compared with Legacy Approach cont’d • Legacy Approach (e.g. CloneZilla Multicast Mode) ◦ Slow, 0.5~2 GiB/min (8~34 MB/s) ◦ If one server stuck, ALL servers stuck ◦ CANNOT be used in L2-less environment (Broadcast and Multicast are not available)
19. EZIO Compared with Legacy Approach cont’d • Bittorrent Deployment (EZIO) ◦ Fast, 2.5~6 GiB/min (42~100 MB/s) ◦ Performance, Reliability and Flexibility Improvements in Heterogeneous Environment (e.g. 100Mbps, 1Gbps, HDD, SSD, etc) ◦ If one server stuck, others continue and no impact at all ◦ Can be used in L2-less environment
20. Number of client Time (Unicast) Time (EZIO) Time (Multicast) Ratio (BT/Multicast) 1 474 675 390 1.731 2 948 1273 474 2.686 4 1896 1331 638 2.086 8 3792 1412 980 1.441 16 7584 1005 1454 0.691 24 11376 1048 1992 0.526 32 15168 1143 2203 0.519
21. Number of client Time (Unicast) Time (EZIO) Time (Multicast) Ratio (BT/Multicast) 1 474 675 390 1.731 2 948 1273 474 2.686 4 1896 1331 638 2.086 8 3792 1412 980 1.441 16 7584 1005 1454 0.691 24 11376 1048 1992 0.526 32 15168 1143 2203 0.519 Incresing Incresing Flat
22. Easy way to try EZIO • Clonezilla Lite Server Mode ◦ https://clonezilla.org/ ◦ https://clonezilla.org/show-live-doc-content.p hp?topic=clonezilla-live/doc/12_lite_server_BT _from_dev
23. Demo • CloneZilla Lite Server with EZIO support • https://clonezilla.org/
24. Still Working on • gRPC Support to control EZIO daemon • Working on FAST deployment in DataCenter ◦ Patch OpenStack Ironic to support BT Provisioning ◦ Patch MaaS Curtin to support BT Provisioning
25. Any Questions? • https://github.com/tjjh89017/ezio • tjjh89017@hotmail.com • date_huang@edge-core.com
26. Reference • https://www.mdpi.com/2076-3417/9/2/296 • https://github.com/tjjh89017/ezio • https://www.libtorrent.org