Slide 1

Slide 1 text

Massive Bare-Metal Operating System Provisioning Improvement Date Huang [email protected] OpenInfra Day Taiwan 2019/11/12

Slide 2

Slide 2 text

About me ● Date Huang 黃宇強 ● DozenCloud ARMv8 Cloud ● Edgecore Networks Sr. Engineer ○ SDN Solution for Datacenter, Cloud

Slide 3

Slide 3 text

Nowadays Cloud ● Cloud Techs change FAST ● Infrastructure-as-Code (IaC) for System Provisioning and Service Deployment ● Software-Defined Datacenter (SDDC) for System Provisioning and Service Deployment ● Re-deploy Operating System to Bare-Metal Servers ● Bare-Metal System Provisioning toolkit

Slide 4

Slide 4 text

Nowadays Cloud ● Cloud Techs change FAST ● Infrastructure-as-Code (IaC) for System Provisioning and Service Deployment ● Software-Defined Datacenter (SDDC) for System Provisioning and Service Deployment ● Re-deploy Operating System to Bare-Metal Servers ● Bare-Metal System Provisioning toolkit

Slide 5

Slide 5 text

Legacy Approach ● For Massive System Provisioning ● Use Unicast / Multicast to transfer OS image to server disks ● Save temporary image in RAM or another partition or disk ● Unitcast ○ Metal as a Service (MaaS) ○ OpenStack Ironic ● Multicast ○ Clonezilla Server Edition (Made in Taiwan) ○ Symantec Ghost Solution Suite

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Problems need to be sloved ● Cost lots of time and Remaining Time is not predictable ● Every issue will cause System Provisioning process stuck ● Targets will share server’s bandwidth via Unicast (Scale Difficult, Slow down) ● Targets aren’t always stable. Need to re-transfer data blocks via Multicast (Slow down) ● Targets need to join deployment group and start at the same time via Multicast

Slide 8

Slide 8 text

Problems need to be sloved cont’d ● Unicast is HARD to scaling up / out ○ Server Bandwidth isn’t enough ● Mutlicast / Broadcast is NOT always available ○ Enterprise Network / Public Cloud disallow Multicast / Broadcast ○ L2-less Datacenter (Clos Network)

Slide 9

Slide 9 text

Bittorrent ● Bittorrent-based System Provisioning ● Avoid Multicast / Broadcast ● Async Data Transfer ○ Don’t need to join deployment group simultaneously ● Reduce Image Provider Network Loading (Server-side)

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

Challenge in Bittorrent ● For Peer-to-Peer file sharing, Temporary Storage is Required ○ Can’t assume getting data order in Bittorrent ○ Need to store whole disk image ● For Bare-Metal Provisioning, the available storage is RAM ● Image size will be Limited by RAM size ○ RAM is normally not big enough to store images and be ready to convert / restore to disk

Slide 12

Slide 12 text

Brand New Mechanism - EZIO ● Compatible with regular Bittorrent Protocol ○ Using Libtorrent-rasterbar ● Seed / Proxy Image with regular Bittorrent Client ○ e.g qBittorrent, transmission ● Read/Write image from/into Disk Directly ○ No need any temporary storage to store image for converting/restoring to disk ● https://github.com/tjjh89017/ezio

Slide 13

Slide 13 text

EZIO Design - Image Format ● Design BT-compatible image format ● Continuous block into one file, save offset in ‘hex’ into name

Slide 14

Slide 14 text

EZIO Design - Torrent ● Torrent will record all continuous block ○ offset and length ● Calculate SHA1 hash to verify

Slide 15

Slide 15 text

EZIO Design - Partition Offset ● ‘Piece’ is minimum unit in Bittorrent ● Cut continuous block into piece (Info is saved in torrent)

Slide 16

Slide 16 text

EZIO Design - Partition Offset cont’d ● When recv ‘Piece 1’, we will know data is combined with 2 blocks ○ 0x1000000 - 0x10a2000 ○ 0x8000000 - 0x8f5e000 ● Directly read/write from disk

Slide 17

Slide 17 text

EZIO Compared with Legacy Approach ● Case ○ Massive OS Deployment to multiple Servers in Data Center ○ OS Image is larger than 50GB ● Experimental environment ○ Network: Cisco 3560G 1Gbps ○ Server: Dell T1700 with Intel Xeon E3-1226, 16G ram, 1TB hard disk ○ PC Client: 32 Client, same as Server ○ Image: Ubuntu Linux with 50GB data in disk. Multicast Image is compressed by pzstd. BT Image is raw file

Slide 18

Slide 18 text

EZIO Compared with Legacy Approach cont’d ● Legacy Approach (e.g. CloneZilla Multicast Mode) ○ Slow, 0.5~2 GiB/min (8~34 MB/s) ○ If one server stuck, ALL servers stuck ○ CANNOT be used in L2-less environment (Broadcast and Multicast are not available)

Slide 19

Slide 19 text

EZIO Compared with Legacy Approach cont’d ● Bittorrent Deployment (EZIO) ○ Fast, 2.5~6 GiB/min (42~100 MB/s) ○ Performance, Reliability and Flexibility Improvements in Heterogeneous Environment (e.g. 100Mbps, 1Gbps, HDD, SSD, etc) ○ If one server stuck, others continue and no impact at all ○ Can be used in L2-less environment

Slide 20

Slide 20 text

Number of client Time (Unicast) Time (EZIO) Time (Multicast) Ratio (BT/Multicast) 1 474 675 390 1.731 2 948 1273 474 2.686 4 1896 1331 638 2.086 8 3792 1412 980 1.441 16 7584 1005 1454 0.691 24 11376 1048 1992 0.526 32 15168 1143 2203 0.519

Slide 21

Slide 21 text

Number of client Time (Unicast) Time (EZIO) Time (Multicast) Ratio (BT/Multicast) 1 474 675 390 1.731 2 948 1273 474 2.686 4 1896 1331 638 2.086 8 3792 1412 980 1.441 16 7584 1005 1454 0.691 24 11376 1048 1992 0.526 32 15168 1143 2203 0.519 Incresing Incresing Flat

Slide 22

Slide 22 text

Easy way to try EZIO ● Clonezilla Lite Server Mode ○ https://clonezilla.org/ ○ https://clonezilla.org/show-live-doc-content.p hp?topic=clonezilla-live/doc/12_lite_server_BT _from_dev

Slide 23

Slide 23 text

Demo ● CloneZilla Lite Server with EZIO support ● https://clonezilla.org/

Slide 24

Slide 24 text

Still Working on ● gRPC Support to control EZIO daemon ● Working on FAST deployment in DataCenter ○ Patch OpenStack Ironic to support BT Provisioning ○ Patch MaaS Curtin to support BT Provisioning

Slide 25

Slide 25 text

Any Questions? ● https://github.com/tjjh89017/ezio ● [email protected][email protected]

Slide 26

Slide 26 text

Reference ● https://www.mdpi.com/2076-3417/9/2/296 ● https://github.com/tjjh89017/ezio ● https://www.libtorrent.org