OpenStack環境での Cumulusスイッチ利用事例

OpenStackに採用したWhiteBoxSwitch（CumulusLInux)の運用事例発表スライド( ホワイトボックススイッチユーザ会 / Dec 08 2016)

Transcript

1. ϗϫΠτϘοΫεεΠονϢʔβձ / Dec 08 2016 גࣜձࣾΞΫγε ٶࠊɹߦੜ OpenStack؀ڥͰ CumulusεΠονΛ࢖ͬͯΈͯײͨ͜͡ͱ
2. ൃදऀ঺հ
3. ൃදऀ঺հ ٶࠊ ߦੜ ΞϓϦέʔγϣϯɾωοτϫʔΫΤϯδχΞͱͯ͠ɺ جװܥγεςϜ͔ΒWebαʔϏεߏஙɾεϚϗΞϓ Ϧ։ൃ·Ͱɺ༷ʑͳۀ຿෼໺ͷ։ൃɾߏஙΛܦݧɻ ͦͷޙɺΠϯϑϥߏஙɾӡ༻ΤϯδχΞΛϝΠϯۀ ຿ͱͯ͠ܦݧɻ ݱࡏ͸OpenstackϕʔεͷϓϥΠϕʔτΫϥ΢υͷߏ ஙɾӡ༻ΛϝΠϯۀ຿ʹैࣄɻ ύϑΥʔϚϯεɾηΩϡϦςΟʹ഑ྀͭͭ͠ɺTCO ʹ഑ྀͨ͠ΠϯϑϥΛ೔ʑ௥͍͔͚͍ͯ·͢ɻ Miyagoshi Yukio @AXIS OpenStack SDN Ceph Security Python C OSS DB Linux rasp lxd PublicCloud
4. ձࣾ঺հ Company introduction ITΠϯϑϥߏஙɾӡ༻ αʔόɾετϨʔδͷԾ૝Խ΍VDI ؀ڥߏங౳ͷαʔϏεΛ͸͡Ίɺω οτϫʔΫͷاըɾίϯαϧςΟϯ ά͔Βɺϋʔυ΢ΣΞɾιϑτ΢Σ ΞͷಋೖɺωοτϫʔΫߏஙɾҠߦ WebApplication։ൃ CRMɾσʔλϕʔεߏஙɾECαΠτɾ اۀαΠτɾϥϯσΟϯάϖʔδɾ SNSɾϚονϯάαΠτͳͲେن໛α Πτ͔Βখن໛αΠτ·Ͱ෯޿͘Web αΠτΛ։ൃɺίϯαϧςΟϯά Ϣʔβʔαϙʔτࣄۀ ϕςϥϯελοϑ͕ϑϩʔΛ੔ཧ͠ ͓ͯ٬༷͔Βͷ͓໰͍߹ΘͤʹରԠ ͠ɺΫϥΠΞϯτ༷΁ใࠂ͍ͤͯ͞ ͖ͨͩ·͢ɻ·ͨɺϝϧϚΨ഑৴ͷ ୅ߦɾΩϟϯϖʔϯӡӦ΍ܠ඼ͷൃ ૹۀ຿౳΋αϙʔτ
5. ΞδΣϯμ ಋೖɾӡ༻ϑΣʔζʢύϑΥʔϚϯεɾઃఆ஫ҙ఺ɾӡ༻ঢ়گʣ ΦʔτϝʔγϣϯʢServer Deployʹ࿈ಈͨ͠Switch ConfigurationҊʣ ϓϥΠϕʔτΫϥ΢υͷߏஙɹʢOpenStack؀ڥཁ݅ʹ߹ͬͨεΠονͷબ୒ʣ LinuxBoxͱͯ͠ͷ༻్Λݕূ UbuntuΛಈ͔͢ʢ೔ຊॳʂʣ
6. ϓϥΠϕʔτΫϥ΢υͷߏங OpenStack؀ڥཁ݅ʹ߹ͬͨεΠονͷબ୒
7. എܠ ϓϥΠϕʔτΫϥ΢υͷߏங ։ൃɾݕূɾຊ൪؀ڥͰɺ౷Ұͷ؀ڥͰ؅ཧ͍ͨ͠ɻHyperVisor΋౷Ұɻ OpenStack APIΛར༻ͯ͠ɺInfrastructure as a CodeΛਪਐ͍ͨ͠ طଘԾ૝؀ڥͷػثϦϓϨʔε࣌ظ ※2014ޙ൒
8. ϓϥΠϕʔτΫϥ΢υͷߏங ωοτϫʔΫ؀ڥཁ݅ ComputeNode͕εέʔϧͨ͠ࡍʹɺαʔόؒωοτϫʔΫ઀ଓ΍ςφϯτؒωο τϫʔΫ઀ଓ͕ܦ࿏্ͰϘτϧωοΫʹͳΓʹ͍͘ߏ੒ʹ͍ͨ͠ σʔληϯλʔؒͷL3઀ଓ΍Ϋϥ΢υͱ ͷ઀ଓʢAWS-DirectConnect౳ʣɺ෺ཧε τϨʔδɾαʔό͕ଘࡏ͢ΔͨΊɺςφ ϯτωοτϫʔΫͱ෺ཧVLANΛ઀ଓ͢Δ ඞཁ͕͋ΔɻɹVXLANͱVLANؒͷ௨৴ͷ ଎౓ɾ৑௕ੑΛ୲อ͍ͨ͠ɻ network node Compute Node Compute Node Compute Node Compute Node ಉҰϗετ಺Ͱ΋αϒωοτؒ௨৴͸શͯ/FUXPSL/PEFܦ༝ Compute Node Compute Node Compute Node Compute Node router %73ඇαϙʔτ compute vm vm compute vm vm compute vm vm VNI=20 VNI=10 VNI=30 SWITCH AWS VLAN=100 VLAN=200 DB VXLAN Transport Network ※2014ޙ൒
9. ϓϥΠϕʔτΫϥ΢υͷߏங ωοτϫʔΫߏ੒ͷݕ౼ MidonetΛར༻͢ΔͱɺωοτϫʔΫϊʔυͷूத؅ཧ͔Β։์͞ΕΔ ※2014ޙ൒ IUUQTFSWFSBTDPEFDPNBZFBSXJUINJEPLVSBNJEPOFUBOEPQFOTUBDLIUNM
10. ϓϥΠϕʔτΫϥ΢υͷߏங ωοτϫʔΫߏ੒ͷݕ౼ Midonet+CumulusͰɺHardware VTEP͕γʔϜϨεʹ࣮ݱͰ͖ΔΒ͍͠ ※2014ޙ൒ IUUQTEPDTDVNVMVTOFUXPSLTDPNEJTQMBZ$-*OUFHSBUJOH )BSEXBSF 75&1T XJUI .JEPLVSB .JEP/FU BOE 0QFO4UBDL
11. ϓϥΠϕʔτΫϥ΢υͷߏங POC؀ڥͰͷݕূ݁Ռ Cassandra / ZookeeperΛ̏୆ηοτͰߏங͠ɺMidonet ؀ڥΛOpenStackʹઃఆ CumulusεΠον্ͷopenvswitch-vtepΛ༗ޮʹ͠ɺMidonetͱOVS઀ଓ midonet> list vtep name cumulus-sw1 description cumulus-sw1 management-ip 10.77.92.20 management-port 6632 tunnel-zone tzone0 connection-state CONNECTED VMαʔόͷMACΞυϨε͕ɺCumulusεΠονͷVXLANϒϦοδ্ͰҾ͚Δ͜ͱΛ֬ೝ bridge fdb show br-vxln10000 fa:16:3e:d4:ee:22 dev vxln10000 vlan 0 master br-vxln10000 ʻʹԾ૝αʔόͷMACΞυϨε VXLAN্ͷVM͔ΒVLAN্ͷ෺ཧαʔόʹɺ૬ޓ઀ଓͰ͖Δ͜ͱΛ֬ೝɻ ɾɾɾɾɾɾ)"͸ʁ ※2014ޙ൒
12. ϓϥΠϕʔτΫϥ΢υͷߏங Midonet VTEP HA ※ݱࡏ͸ɺMidonet+CumulusͰͷHardwareVTEP͸ɺHAΛαϙʔτ͍ͯ͠·͢ɻ IUUQTHJUIVCDPNNJEPOFUNJEPOFUCMPCFBGEFDGBDCGFFEPDTWYHX@TFSWJDFNE ݕূ࣌఺ʢ2014ޙ൒࣌఺ʣͰ͸ɺHA͸αϙʔτ͞Ε͍ͯͳ͍
13. ϓϥΠϕʔτΫϥ΢υͷߏங Neutron DVRͷબ୒ OpenStack Juno ͷϦϦʔεͰɺDVR͕αϙʔτ͞Ε͍ͯΔ ※2014ޙ൒ IUUQTXJLJPQFOTUBDLPSHXJLJ3FMFBTF/PUFT+VOP αϒωοτؒ઀ଓͷϘτϧωοΫ՝୊͸ɺNeutron-DVRΛར༻͢Δ ͜ͱͰղܾɻ ※FIP͕ແ͍SNAT௨৴ʹؔͯ͠͸ɺωοτϫʔΫϊʔυܦ༝͸มΘΒͣɻ Hardware VTEPΛݕ౼͍͕ͯͨ͠ɺ֤ComputeNodeͷVLAN Bridgeͱ VTEP BridgeΛ࣋ͨͤΔ͜ͱͰɺNetworkϊʔυʹूதͤͣ෼ࢄՄೳɻ ※ੑೳཁ݅ʹ߹͏଎౓͕ͰΔ͜ͱΛ֬ೝ
14. OpenstackωοτϫʔΫߏ੒(Compute) IUUQEPDTPQFOTUBDLPSHNJUBLBKBOFUXPSLJOHHVJEFTDFOBSJPEWSPWTIUNM ϓϥΠϕʔτΫϥ΢υͷߏங
15. OpenstackωοτϫʔΫߏ੒(Network) IUUQEPDTPQFOTUBDLPSHNJUBLBKBOFUXPSLJOHHVJEFTDFOBSJPEWSPWTIUNM ϓϥΠϕʔτΫϥ΢υͷߏங
16. αʔόΤϯδχΞʢLinux؅ཧऀʣ͕ѻ͍΍͍͢ খن໛ͳνʔϜ͸ωοτϫʔΫͷΈ୲౰ͷΤϯδχΞͱ͍͏͜ͱ͸ͳ͘ɺαʔόߏங·Ͱ୲౰ ͢Δࣄ͕ଟ͍ɻҰൠతͳίϚϯυͰτϥϒϧγϡʔςΟϯά͕ՄೳͳͨΊɺֶशίετ͕௿͘ ͳΔɻʢ͔ͱݴͬͯɺωοτϫʔΫ஌͕ࣝෆཁʹͳΔΘ͚Ͱ͸ͳ͍ͷͰɺ஫ҙ͕ඞཁʣ ※ݸਓతʹ͸ɺ؀ڥίϯςΩετεΠονͷඞཁ͕ͳ͍ͷ͕େ͖͍ϝϦοτ ܦࡁੑ baremetal switch͸ɺଞεΠονʹൺ΂Δͱ௿Ձ֨ͰೖखՄೳɻιϑτ΢ΣΞɾαϙʔτϥΠ ηϯεΛؚΊͯ΋ɺίετϝϦοτ͸ߴ͍ɻ Cold StandbyଆͷCumulusLinuxϥΠηϯε͸ෆཁɻ طଘͷΦʔτϝʔγϣϯπʔϧΛར༻Մೳ Ansible౳ͷطଘͷσϓϩΠπʔϧ౳͕ɺͦͷ··ར༻Մೳɻ ՄೳੑɾϞνϕʔγϣϯ “It’s Just Linux, you can do whatever you want!” Spine-Leafߏ੒ͰɺԣʹεΠονΛ௥Ճ͠қ͍ CumulusSwitchͷબ୒ ϓϥΠϕʔτΫϥ΢υͷߏங
17. CumulusLinux Networkߏ੒ IUUQTDVNVMVTOFUXPSLTDPNNFEJBDVNVMVTQEGUFDIOJDBMWBMJEBUFEEFTJHOHVJEFT0QFO4UBDL$VNVMVT-JOVY7BMJEBUFE%FTJHO(VJEFQEG ϓϥΠϕʔτΫϥ΢υͷߏங
18. ϓϥΠϕʔτΫϥ΢υͷߏங ※ิ଍ MidonetΛར༻͢ΔͱɺNeutronDVRͰ͸ղܾͰ͖ͳ͍໰୊΋ղফ͞Ε·͢ɻ IUUQTUIJOLJUDPKQTUPSZ ̍ʣNetworkʢGWʣϊʔυͷ৑௕Խ ̎ʣNetworkϊʔυ͕Active-ActiveͰӡ༻ՄೳɻεέʔϧΞ΢τ͕Մೳ ※ฐࣾ؀ڥ͸ɺLinuxHAΛJuju-charmܦ༝Ͱߏங͠ରԠ ͦͷଞ৭ʑػೳ͕͋Γ·͕͢ɺԼهαΠτ͕ൺֱͰΘ͔Γ΍͍͢Ͱ͢ɻ
19. ಋೖɾӡ༻ϑΣʔζ ύϑΥʔϚϯεɾઃఆ஫ҙ఺ɾӡ༻ঢ়گ
20. ಋೖϑΣʔζ ύϑΥʔϚϯεݕূʢ̍ʣ CPUߴෛՙ࣌ͷ௨৴΁ͷӨڹ༗ແ֬ೝ /FU1FSG
    $MJFOU ϦΫΤετ /FU1FSG
    4FSWFS ϦΫΤετ Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.00 9306.58 ACLͳ͠௨৴ Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.00 9292.66 ACL͋Γ௨৴
21. ಋೖϑΣʔζ ύϑΥʔϚϯεݕূʢ̎ʣ pingฏۉԠ౴࣌ؒ֬ೝ 64 bytes from 10.10.10.2: icmp_seq=1 ttl=63 time=0.322 ms 64 bytes from 10.10.10.2: icmp_seq=2 ttl=63 time=0.347 ms 64 bytes from 10.10.10.2: icmp_seq=3 ttl=63 time=0.284 ms 64 bytes from 10.10.10.2: icmp_seq=4 ttl=63 time=0.281 ms 64 bytes from 10.10.10.2: icmp_seq=5 ttl=63 time=0.249 ms 64 bytes from 10.10.10.2: icmp_seq=6 ttl=63 time=0.182 ms 64 bytes from 10.10.10.2: icmp_seq=7 ttl=63 time=0.263 ms VXLAN->VLAN 64 bytes from 10.10.11.2: icmp_seq=1 ttl=64 time=3.32 ms 64 bytes from 10.10.11.2: icmp_seq=2 ttl=64 time=0.410 ms 64 bytes from 10.10.11.2: icmp_seq=3 ttl=64 time=0.423 ms 64 bytes from 10.10.11.2: icmp_seq=4 ttl=64 time=0.410 ms 64 bytes from 10.10.11.2: icmp_seq=5 ttl=64 time=0.560 ms 64 bytes from 10.10.11.2: icmp_seq=6 ttl=64 time=0.579 ms 64 bytes from 10.10.11.2: icmp_seq=7 ttl=64 time=0.688 ms VXLAN->VXLAN ಉҰηάϝϯτ(ผϗετ) 64 bytes from 10.10.12.2: icmp_seq=2 ttl=64 time=0.503 ms 64 bytes from 10.10.12.2: icmp_seq=3 ttl=64 time=0.584 ms 64 bytes from 10.10.12.2: icmp_seq=4 ttl=64 time=0.479 ms 64 bytes from 10.10.12.2: icmp_seq=5 ttl=64 time=0.469 ms 64 bytes from 10.10.12.2: icmp_seq=6 ttl=64 time=0.481 ms 64 bytes from 10.10.12.2: icmp_seq=7 ttl=64 time=0.576 ms 64 bytes from 10.10.12.2: icmp_seq=8 ttl=64 time=0.564 ms VXLAN->VXLAN ผηάϝϯτ(ผϗετ) 64 bytes from 10.10.13.2: icmp_seq=2 ttl=64 time=0.134 ms 64 bytes from 10.10.13.2: icmp_seq=3 ttl=64 time=0.112 ms 64 bytes from 10.10.13.2: icmp_seq=4 ttl=64 time=0.130 ms 64 bytes from 10.10.13.2: icmp_seq=5 ttl=64 time=0.092 ms 64 bytes from 10.10.13.2: icmp_seq=6 ttl=64 time=0.111 ms 64 bytes from 10.10.13.2: icmp_seq=7 ttl=64 time=0.117 ms 64 bytes from 10.10.13.2: icmp_seq=8 ttl=64 time=0.092 ms ෺ཧαʔόؒ ಉҰηάϝϯτ
22. ಋೖϑΣʔζ ACLݕূ ௨ৗ࣌ ʮip route add 10.10.10.0/24 via 10.50.0.101ʯΛ࣮ߦ # cl-resource-query IPv4 entries: 15, 0% of maximum value 32668 IPv4 Routes: 15 Total Routes: 22, 0% of maximum value 32768 ௥Ճલ # cl-resource-query IPv4 entries: 16, 0% of maximum value 32668 IPv4 Routes: 16 Total Routes: 23, 0% of maximum value 32768 ௥Ճޙ ʮ-A INPUT -s 10.10.10.1/32 -p tcp --dport 22 -j ACCEPTʯ౳ͷϧʔϧΛ̐ͭ௥Ճ # cl-resource-query Egress ACL entries: 24, 9% of maximum value 256 Egress ACL counters: 48, 4% of maximum value 1024 ௥Ճલ # cl-resource-query Egress ACL entries: 28, 10% of maximum value 256 Egress ACL counters: 52, 5% of maximum value 1024 ௥Ճޙ ʮ-A FORWARD -s 10.10.10.1/32 -p tcp --dport 22 -j ACCEPTʯ౳ͷϧʔϧΛ200Ҏ্௥Ճ # cl-resource-query Ingress ACL entries: 60, 4% of maximum value 1280 Ingress ACL counters: 79, 6% of maximum value 1280 ௥Ճલ # cl-resource-query Ingress ACL entries: 292, 28% of maximum value 1024 Ingress ACL counters: 311, 30% of maximum value 1024 ௥Ճޙ
23. ಋೖϑΣʔζ ACLݕূ ҟৗ࣌ ར༻͍ͯ͠ΔεΠονͷϋʔυ΢ΣΞϦϛοτ ˞IUUQTEPDTDVNVMVTOFUXPSLTDPNEJTQMBZ%0$4/FUpMUFS  "$-T
24. ಋೖϑΣʔζ ACLݕূ ҟৗ࣌ ಉҰϧʔϧͷॏෳ࣌Τϥʔ # cl-acltool -i Reading rule file /etc/cumulus/acl/policy.d/00control_plane.rules ... Processing rules in file /etc/cumulus/acl/policy.d/00control_plane.rules ... Reading rule file /etc/cumulus/acl/policy.d/10_test.rules ... Processing rules in file /etc/cumulus/acl/policy.d/10_test.rules ... error: duplicate rule at lineno 6 No acl policies to install, ... aborting ߏจΤϥʔ # cl-acltool -i Error occurred at line: 103 Try `iptables-restore -h' or 'iptables-restore --help' for more information.) -------------------------------------------------------------------------------- Τϥʔ಺༰ʢলུ) -------------------------------------------------------------------------------- Rolling back .. failed. ᮢ஋Λ௒͑ͨ৔߹ # cl-acltool -i Reading rule file /etc/cumulus/acl/policy.d/00control_plane.rules ... Processing rules in file /etc/cumulus/acl/policy.d/ 00control_plane.rules ... Reading rule file /etc/cumulus/acl/policy.d/10_test.rules ... Processing rules in file /etc/cumulus/acl/policy.d/10_test.rules ... Reading rule file /etc/cumulus/acl/policy.d/ 99control_plane_catch_all.rules ... Processing rules in file /etc/cumulus/acl/policy.d/ 99control_plane_catch_all.rules ... Installing acl policy error: hw sync failed (sync_acl hardware installation failed) Rolling back .. failed.
25. ಋೖϑΣʔζ ACLݕূ ҟৗ࣌ ಉҰϧʔϧͷॏෳ࣌Τϥʔ # cl-acltool -i Reading rule file /etc/cumulus/acl/policy.d/00control_plane.rules ... Processing rules in file /etc/cumulus/acl/policy.d/00control_plane.rules ... Reading rule file /etc/cumulus/acl/policy.d/10_test.rules ... Processing rules in file /etc/cumulus/acl/policy.d/10_test.rules ... error: duplicate rule at lineno 6 No acl policies to install, ... aborting ߏจΤϥʔ # cl-acltool -i Error occurred at line: 103 Try `iptables-restore -h' or 'iptables-restore --help' for more information.) -------------------------------------------------------------------------------- Τϥʔ಺༰ʢলུ) -------------------------------------------------------------------------------- Rolling back .. failed. ᮢ஋Λ௒͑ͨ৔߹ # cl-acltool -i Reading rule file /etc/cumulus/acl/policy.d/00control_plane.rules ... Processing rules in file /etc/cumulus/acl/policy.d/ 00control_plane.rules ... Reading rule file /etc/cumulus/acl/policy.d/10_test.rules ... Processing rules in file /etc/cumulus/acl/policy.d/10_test.rules ... Reading rule file /etc/cumulus/acl/policy.d/ 99control_plane_catch_all.rules ... Processing rules in file /etc/cumulus/acl/policy.d/ 99control_plane_catch_all.rules ... Installing acl policy error: hw sync failed (sync_acl hardware installation failed) Rolling back .. failed.
26. ӡ༻ϑΣʔζ ઃఆ஫ҙ఺ BridgeʹVLAN-aware modeͷར༻ ifreload -aͷར༻ /etc/init.d/networking restart ΍ɺ systemctl restart networking.service Λ࣮ߦ͢ΔͱɺωοτϫʔΫ ͕Ϧηοτ͞ΕΔͨΊɺifreload -aͰ /etc/network/interfacesͷઃఆΛ൓өͤ͞ΔΑ͏ʹ͠·͢ɻ ·ͨɺIFʹ௥Ճͨ͠ΞυϨεͷ࡟আ΍Routingͷ࡟আ͸ɺifreload -a Ͱ͸൓ө͞Εͳ͍͜ͱ͕͋ΔͨΊɺip ίϚ ϯυͰ࡟আ͕࣮֬ɻ※ifreload -a -d σόοΫදࣔ͢ΔͱྲྀΕ͕෼͔Δɻ ※ Vlan-Bridge୯ҐͰɺSTP/RSTPΛ࣮૷͠ͳ͚Ε͹͍͚ͳ͍৔߹΍ VXLAN-bridge(CL3.1 ҎલʣΛར༻͍ͨ͠৔߹͸ɺtraditional modeΛར༻ɻ IUUQTTVQQPSUDVNVMVTOFUXPSLTDPNIDFOVTBSUJDMFT Traditional mode͸ɺ200 VLANͷಉ࣌ར༻ͷΈαϙʔτ VLAN-aware mode͸ɺ2000 VLANͷಉ࣌ར༻Λαϙʔτ Traditional modeΑΓ΋ɺVLAN-awareϞʔυ͸ઃఆ͕γϯ ϓϧʹهࡌͰ͖ɺ؅ཧ্ͷϝϦοτ΋͋Δ
27. ӡ༻ϑΣʔζ ӡ༻ঢ়گ LoadAverage Memory Usage CPU Usages Cephͷself-healingͰߴτϥϑΟοΫ͕ൃੜ͠ ͨ͜ͱ͕͋Δ͕ɺεΠονOSଆͷӨڹ͸ͳ͠ Cumulus LinuxͰͷProduction؀ڥͰɺࠓͷͱ ͜Ζτϥϒϧ͸ͳ͠
28. ӡ༻ϑΣʔζ τϥϒϧγϡʔςΟϯά ىಈ͠ͳ͘ͳͬͨΠϝʔδͷम෮ iptablesίϚϯυͰforwardͷΈਖ਼ৗʹ௥Ճ͞Εͳ͍ʁ ϔϧενΣοΫ࣌ʹར༻͍ͯ͠ΔίϚϯυ lldpctl mstpctl lldpcli cl-netstat ethtool cl-resource-query sensors smonctl brctl ifquery --running ifquery --check tcpdump traceroute /proc/net/഑Լͷ֬ೝ FORWARDϧʔϧ͸ɺiptablesίϚϯυͰదԠͰ͖ͳ͍ INPUT / NAT ςʔϒϧ͸ɺiptablesͰਖ਼ৗʹϋʔυ΢ΣΞʹొ࿥͞ΕΔ cl-acltool -i Λ࣮ߦ͠ɺϧʔϧదԠ͢Δ͜ͱͰରԠՄೳʢACLͷϑΝΠϧฤूʣ ※FW 2.5.10Ͱͷಈ࡞ݕূ # lvs LV VG Attr LSize PERSIST CUMULUS -wi-ao-- 64.00m SYSROOT1 CUMULUS -wi-ao-- 14.69g SYSROOT2 CUMULUS -wi-a--- 14.69g boot࣌ʹผΠϝʔδΛબ୒ͯ͠ɺىಈ LVMίϚϯυͰLogicalVolume͕ϦετΞοϓ͞ΕΔ ͷͰɺϚ΢ϯτͯ͠ฤू͢Δ͜ͱͰम෮Մೳ
29. LinuxBoxͱͯ͠ͷ༻్Λݕূ ̍ʣLoadBalancer/KeyValueσʔλϕʔε౳
30. LinuxBoxͱͯ͠ͷ༻్Λݕূ ෛՙςετΫϥΠΞϯτ 8FC4FSWFS --PBE#BMBODFS IUUQ
    SFRVFTUT IUUQ
    SFRVFTUT 8FC4FSWFS IUUQ
    SFRVFTUT 1 Cumulus্ʹNginxΛઃஔ͠ɺL7ͷ LoadBalancerͱͯ͠ઃஔ 2 10GBͰ֤ϊʔυΛ઀ଓɻશͯͷαʔ ό͸෺ཧϊʔυ 3 wrk(※)πʔϧΛར༻͠ɺϚϧνεϨου Ͱಉ࣌ΞΫηεΛγϛϡϨʔτ ˞
    IUUQTHJUIVCDPNXHXSL L7 LoadBalancer NginxΛઃஔ͠ɺVIPͰtcp/80ΛListen ϦόʔεϓϩΩγ
31. L7 LoadBalancer ෛՙςετ࣌ͷSwitch-CPUར༻཰ WEBαʔό௚઀ΞΫηε࣌ # ./wrk -t 30 -c 300 -d 30 http://10.50.10.130/test.html Running 30s test @ http://10.50.10.130/test.html 30 threads and 300 connections Thread Stats Avg Stdev Max +/- Stdev Latency 10.71ms 18.88ms 291.48ms 87.24% Req/Sec 5.34k 1.10k 14.02k 75.75% 4724371 requests in 30.10s, 1.08GB read Requests/sec: 156965.43 Transfer/sec: 36.82MB Cumulus-Nginxܦ༝ΞΫηε࣌ ./wrk -t 30 -c 300 -d 30 http://10.50.0.101/test.html Running 30s test @ http://10.50.0.101/test.html 30 threads and 300 connections Thread Stats Avg Stdev Max +/- Stdev Latency 1.01s 565.54ms 2.00s 46.87% Req/Sec 7.12 6.38 30.00 75.56% 758 requests in 30.06s, 174.70KB read Socket errors: connect 0, read 0, write 0, timeout 231 Requests/sec: 25.22 Transfer/sec: 5.81KB Nginx΁ϦΫΤετ͕౸ୡ͢ΔલʹɺϘτϧ ωοΫʹͳ͍ͬͯΔ෦෼͕͋Δ LinuxBoxͱͯ͠ͷ༻్Λݕূ
32. L7 LoadBalancer */(3&44@*/5'
    
    TXQ
    */(3&44@$)"*/
    
    */165
    <JQUBCMFT>
    "
    */(3&44@$)"*/
    JOJOUFSGBDF
    */(3&44@*/5'
    N
    BEESUZQF
    ETUUZQF
    -0$"-
    K
    10-*$&
    
    
    
    
    
    
    
    TFUNPEF
    QLU
    TFUSBUF
    
    TFUCVSTU
    
    TFUDMBTT
    
    "
    */(3&44@$)"*/
    JOJOUFSGBDF
    */(3&44@*/5'
    N
    BEESUZQF
    ETUUZQF
    *13065&3
    K
    10-*$&
    
    
    
    
    
    
    
    TFUNPEF
    QLU
    TFUSBUF
    
    TFUCVSTU
    
    TFUDMBTT
    
    "
    */(3&44@$)"*/
    JOJOUFSGBDF
    */(3&44@*/5'
    K
    4&5$-"44
    DMBTT
     ACLઃఆͷiptablesϧʔϧʹɺσϑΥϧτͰrate/burstͷlimitઃఆ͕͞Ε͍ͯΔ FUD
    
    
    cDVNVMVT
    
    
    
    
    
    
    
    c
    BDM
    
    
    
    
    
    
    
    
    
    
    
    c
    BDMDPOG
    
    
    
    
    
    
    
    
    
    
    
    c
    QPMJDZDPOG
    
    
    
    
    
    
    
    
    
    
    
    c
    QPMJDZE
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    c
    DPOUSPM@QMBOFSVMFT
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    A
    DPOUSPM@QMBOF@DBUDI@BMMSVMFT LinuxBoxͱͯ͠ͷ༻్Λݕূ
33. L7 LoadBalancer ࠶ܭଌ݁Ռ Cumulus-Nginxܦ༝ΞΫηε࣌ ./wrk -t 30 -c 300 -d 30 http://10.50.0.101/test.html Running 30s test @ http://10.50.0.101/test.html 30 threads and 300 connections Thread Stats Avg Stdev Max +/- Stdev Latency 195.41ms 157.30ms 1.66s 92.51% Req/Sec 58.36 19.70 101.00 67.04% 52218 requests in 30.10s, 11.75MB read Socket errors: connect 0, read 0, write 0, timeout 8 Requests/sec: 1734.95 Transfer/sec: 399.80KB ෛՙςετ࣌ͷSwitch-CPUར༻཰ SwitchdͷCPUར༻཰͕ுΓ෇͍ ͍ͯΔͨΊɺϘτϧωοΫͱؔ ܎͕͋Γͦ͏ LinuxBoxͱͯ͠ͷ༻్Λݕূ
34. 1 Cumulus্ʹHaproxyΛઃஔ͠ɺL4ͷ LoadBalancerͱͯ͠ઃஔ 2 10GBͰ֤ϊʔυΛ઀ଓɻશͯͷαʔ ό͸෺ཧϊʔυ 3 wrk(※)πʔϧΛར༻͠ɺϚϧνεϨου Ͱಉ࣌ΞΫηεΛγϛϡϨʔτ ˞
    IUUQTHJUIVCDPNXHXSL L4 LoadBalancer HaproxyΛઃஔ͠ɺVIPͰtcp/80ΛListen ϦόʔεϓϩΩγ ෛՙςετΫϥΠΞϯτ 8FC4FSWFS --PBE#BMBODFS IUUQ
    SFRVFTUT IUUQ
    SFRVFTUT IUUQ
    SFRVFTUT NginxҎ֎ͷόϥϯαʔͰɺ࠶ܭଌΛ࣮ࢪ LinuxBoxͱͯ͠ͷ༻్Λݕূ
35. L4 LoadBalancer ෛՙςετ࣌ͷSwitch-CPUར༻཰ WEBαʔό௚઀ΞΫηε࣌ # ./wrk -t 30 -c 300 -d 30 http://10.50.10.130/test.html Running 30s test @ http://10.50.10.130/test.html 30 threads and 300 connections Thread Stats Avg Stdev Max +/- Stdev Latency 10.71ms 18.88ms 291.48ms 87.24% Req/Sec 5.34k 1.10k 14.02k 75.75% 4724371 requests in 30.10s, 1.08GB read Requests/sec: 156965.43 Transfer/sec: 36.82MB Cumulus-HAProxyܦ༝ΞΫηε࣌ ./wrk -t 30 -c 300 -d 30 http://10.50.0.101/test.html Running 30s test @ http://10.50.0.101/test.html 30 threads and 300 connections Thread Stats Avg Stdev Max +/- Stdev Latency 74.66ms 22.02ms 258.39ms 82.96% Req/Sec 135.65 39.43 292.00 74.48% 120556 requests in 30.10s, 28.28MB read Requests/sec: 4005.19 Transfer/sec: 0.94MB LinuxBoxͱͯ͠ͷ༻్Λݕূ
36. 1 Cumulus্ͷiptablesʹɺvip:80ϙʔτΛ webserver:80ͱͯ͠DNATઃఆ 2 10GBͰ֤ϊʔυΛ઀ଓɻશͯͷαʔ ό͸෺ཧϊʔυ 3 wrk(※)πʔϧΛར༻͠ɺϚϧνεϨου Ͱಉ࣌ΞΫηεΛγϛϡϨʔτ ˞
    IUUQTHJUIVCDPNXHXSL iptables-DNAT HTTPड෇༻ͷVIPΛ௥Ճ iptables DNAT Policy௥Ճ ϦόʔεϓϩΩγΛར༻ͤͣɺiptablesͷDNATઃఆͰ࠶ܭଌ ෛՙςετΫϥΠΞϯτ IUUQ
    SFRVFTUT 8FC4FSWFS IUUQ
    SFRVFTUT DNAT Policy LinuxBoxͱͯ͠ͷ༻్Λݕূ
37. iptables-DNAT IUUQTEPDTDVNVMVTOFUXPSLTDPNEJTQMBZ%0$4/FUpMUFS  "$-T iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 10.50.10.130:80 ௥Ճϧʔϧ LinuxBoxͱͯ͠ͷ༻్Λݕূ
38. iptables-DNAT ෛՙςετ࣌ͷSwitch-CPUར༻཰ Cumulus-DNATܦ༝ΞΫηε࣌ ./wrk -t 30 -c 300 -d 30 http://10.50.0.101/test.html Running 30s test @ http://10.50.0.101/test.html 30 threads and 300 connections Thread Stats Avg Stdev Max +/- Stdev Latency 74.32ms 27.17ms 909.53ms 95.86% Req/Sec 135.48 37.73 230.00 76.25% 121095 requests in 30.10s, 28.40MB read Requests/sec: 4023.43 Transfer/sec: 0.94MB Cumulus-HAProxyܦ༝ΞΫηε࣌ ./wrk -t 30 -c 300 -d 30 http://10.50.0.101/test.html Running 30s test @ http://10.50.0.101/test.html 30 threads and 300 connections Thread Stats Avg Stdev Max +/- Stdev Latency 74.66ms 22.02ms 258.39ms 82.96% Req/Sec 135.65 39.43 292.00 74.48% 120556 requests in 30.10s, 28.28MB read Requests/sec: 4005.19 Transfer/sec: 0.94MB LinuxBoxͱͯ͠ͷ༻్Λݕূ
39. 1 Cumulus্ʹRedisServerΛઃஔ͠ɺKey/Value σʔλϕʔεͱͯ͠ઃఆ 2 redisͷdiskॻࠐλΠϛϯά͸σϑΥϧτઃఆ 3 redis-benchmarkπʔϧΛར༻͠ɺผ ϊʔυ͔ΒෛՙςετΛ࣮ࢪ Redis Server RedisΛઃஔ͠ɺVIPͰtcp/6379ΛListen Key/Value Database Server ෛՙςετΫϥΠΞϯτ 3FEJT%#
    4FSWFS QVUHFU
    SFRVFTUT LinuxBoxͱͯ͠ͷ༻్Λݕূ
40. Redis Server ෛՙςετ࣌ͷSwitch-CPUར༻཰ $MJFOU QJOH TFU HFU JODS MQVTI SQVTI SQPQ ϩʔΧϧ        ϦϞʔτ        redis-benchmark -h 10.50.10.101 -p 6379 benchmarkίϚϯυ LinuxBoxͱͯ͠ͷ༻్Λݕূ
41. LinuxBoxͱͯ͠ͷ༻్ ̎ʣΫϥελ౤ථϊʔυ
42. LinuxBoxͱͯ͠ͷ༻్ MongoDB-Arbiter .POHP%# .POHP%# .POHP%# Primary Secondary Arbiter IFBUCFBU SFQMJDBUJPO MongoDBͷΫϥελߏ੒͸ɺPrimary/ Secondary + Arbiterϊʔυ Prmary/Secondaryϊʔυ͸ߴεϖοΫ͕ཁٻ ͞ΕΔ͕ɺArbiterϊʔυ͸௿εϖοΫͰӡ༻ Մೳ Ϋϥελ౤ථϊʔυ͸ɺωοτϫʔΫɾ෺ཧ ߏ੒తʹ΋؂ࢹର৅ͱ͸ผ͕޷·͍͜͠ͱ΋ ͋ΓɺεΠον্ͷಈ࡞͸ߏ੒্΋๬·͍͠ɻ Arbiterϊʔυ௥Ճ͕ඞཁͳ͍
43. LinuxBoxͱͯ͠ͷ༻్ MariaDB-Galera arbiter Master Master galera-arbiter DMVTUFSHSPVQ ಉظ galera clusterߏ੒͸ɺϚϧνϚελͰμ΢ϯ ࣌ͷϚελঢ֨͸QuorumํࣜΛ࠾༻ɻ ࠷খߏ੒Ͱ͸ɺgalera-arbiterϓϩηεΛผ ϊʔυʹ্ཱͪ͛Δ͜ͱͰɺQuorumํࣜͷ ౤ථ཰ΛϚελ̎୆Ͱ΋50%Ҏ্ҡ࣋͢Δࣄ ͕Մೳ Ϋϥελ౤ථϊʔυ͸ɺωοτϫʔΫɾ෺ཧ ߏ੒తʹ΋؂ࢹର৅ͱ͸ผ͕޷·͍͜͠ͱ΋ ͋ΓɺεΠον্ͷಈ࡞͸ߏ੒্΋๬·͍͠ɻ Arbiterϊʔυ௥Ճ͕ඞཁͳ͍ galera-arbiterϊʔυ͸௿εϖοΫͰӡ༻Մೳ
44. LinuxBoxͱͯ͠ͷ༻్ Redis-Sentinel Server RedisCluster͸ɺ؂ࢹαʔό༻ͷSentinelϊʔ υΛQuoramํࣜͰ̏୆Ҏ্ʢح਺ʣ഑ஔ͢ Δඞཁ͕͋Δ 3FEJT 3FEJT Master Slave sentinel sentinel sentinel SFQMJDBUJPO ؂ࢹ ؂ࢹ Sentinelϊʔυ͸௿εϖοΫͰӡ༻Մೳ Ϋϥελ౤ථϊʔυ͸ɺωοτϫʔΫɾ෺ཧ ߏ੒తʹ΋؂ࢹର৅ͱ͸ผ͕޷·͍͜͠ͱ΋ ͋ΓɺεΠον্ͷಈ࡞͸ߏ੒্΋๬·͍͠ɻ sentinelϊʔυΛclagϖΞͷεΠον̎୆ʹઃ ஔ͢Δ͜ͱͰɺԾ૝ɾ෺ཧαʔόΛ࡟ݮ
45. conclusion Key/Valueσʔλϕʔε΋σʔλྔ͕ݶΒΕͨϝλ৘ใ؅ཧ΍ɺγʔέϯε ؅ཧ౳Ͱ͋Ε͹ɺར༻Մೳ ༻్ʹΑͬͯ͸ɺLoadBalancer΋ར༻Մೳɻʢ՝୊͋Γʣ ΞϓϦέʔγϣϯଆͷػೳΛεΠονଆ͕୲͏͜ͱʹͳΓɺີ݁߹ ʹΑΔσϝϦοτ΋ߟྀ͢Δඞཁ͕͋Δ ݕ౼ඞཁ ௿εϖοΫͰ࣮૷ՄೳͳΫϥελ౤ථϊʔυ͸ɺҰ࣌తʹμ΢ϯͯ͠΋Ө ڹ͕গͳ͍ϊʔυͰ΋͋Γɺ࣮૷͠΍͍͢ʢϊʔυ࡟ݮͷޮՌ΋͋Γʣ LinuxBoxͱͯ͠ͷ༻్ ʢSwitch্Ͱlxc͕ಈ͘ͱɺΞϓϦέʔγϣϯΛσϓϩΠ͠΍ͦ͢͏ʣ
46. Φʔτϝʔγϣϯ Server Deployʹ࿈ಈͨ͠Switch ConfigurationҊ
47. Φʔτϝʔγϣϯ juju/maas ෺ཧαʔό ෺ཧϚγϯΛ؅ཧ ΞϓϦέʔγϣϯ͕ར༻͢Δαʔ όΛ.""4ܦ༝Ͱ֫ಘ ΤʔδΣϯτɾΞϓϦέʔγ ϣϯΛΠϯετʔϧ OpenStackDeploymentπʔϧʹɺCanonicalͷJuju/MaasΛར༻ ˞IUUQTKVKVDIBSNTDPN ˞IUUQNBBTJP ฐࣾͷσϓϩΠ؀ڥ
48. Φʔτϝʔγϣϯ Maasͷిݯ؅ཧ MAAS͸αʔόͷిݯ؅ཧ͸ɺҰ௨Γαϙʔτ͞Ε͍ͯΔ
49. Φʔτϝʔγϣϯ MaasͷωοτϫʔΫߏ੒ MAASͰ͸ɺαʔόͷNWߏ੒Λ༧Ίࢦఆͯ͠ɺσϓϩΠ͕Մೳ
50. Φʔτϝʔγϣϯ ݱঢ়ͷMaasʹΑΔ؅ཧ ిݯ0/ DPNNJTTJPO IPMI౳ͷαʔόిݯ؅ ཧʹԊͬͨϓϩτίϧ 1YF
    CPPU %)$1*1ͷ഑෍ αʔόߏ੒৘ใͷొ࿥ -JWF*NBHFͷ഑෍ ిݯ0'' ର৅αʔόΛ௥Ճ ిݯ0/ EFQMPZ IPMI౳ͷαʔόిݯ؅ ཧʹԊͬͨϓϩτίϧ 1YF
    CPPU %)$1*1ͷ഑෍ 04Πϯετʔϧ .""4ʹొ࿥ͨ͠ αʔόߏ੒৘ใΛదԠ બ୒ͨ͠04Πϝʔδ͕ Πϯετʔϧ͞Ε·͢ %JTLύʔςΟγϣϯ৘ ใɾωοτϫʔΫߏ੒ ৘ใ͕దԠ͞Ε·͢ɻ TUBUVTΛEFQPUࡁΈʹมߋ Ϛγϯ௥Ճ࣌ͷγʔέϯε OSσϓϩΠ࣌ͷγʔέϯε
51. Φʔτϝʔγϣϯ Switch؅ཧ·ͰؚΊͨཧ૝తͳ؅ཧҊ εΠονͷ."$"EESFTT Λొ࿥ MAASଆ͕ONIEରԠΛ ߦ͏ඞཁ͕͋Δ %)$1
    *1Λ഑෍ εΠονߏ੒৘ใͷऔಘ εΠονΛ؅ཧԼʹ௥Ճ DPNNJTTJPO 0/*&ܦ༝Ͱ04ͷΠϯετʔϧ εΠον௥Ճͷγʔέϯε ిݯ0/ EFQMPZ 1YF
    CPPU %)$1*1ͷ഑෍ 04Πϯετʔϧ .""4ʹొ࿥ͨ͠ αʔόߏ੒৘ใΛదԠ TUBUVTΛEFQPUࡁΈʹมߋ EFQMPZϞʔυΛ௨஌ εΠονଆ͸MJOL
    VQEPXOΛ ͢ΔϙʔτΛ8BUDI ϙʔτΛݕ஌ γάφϧύέοτΛ/*$୯Ґʹૹ৴ ର৅αʔόͷશϙʔτΛ؂ࢹ 04*'໊ʹରԠ͢Δ εΠονϙʔτΛݕ஌ ରԠ৘ใΛૹ৴ 7-"/CPOEJOH౳ͷઃఆΛϗε τ৘ใʹ߹ΘͤͯεΠονʹదԠ ϚγϯσϓϩΠʗεΠον࿈ܞͷγʔέϯε
52. Φʔτϝʔγϣϯ conclusion αʔόߏ੒ͷΈҙࣝ͢ΔͷΈͰɺεΠονΛҙࣝ͢Δඞཁ͕ͳ͍ εΠονOSͷΠϯετʔϧ΋ɺαʔόͱಉ͡؀ڥͰ౷Ұ ϝϦοτ Juju/Maasͱ͍͏͜ͱ΋͋Γɺ਌࿨ੑ͸SwitchOS͸Ubuntu core͕ྑ ͍ʁ εΠονΠϯετʔϧ·Ͱ͸ࣗಈͰͰ͖Δ͕ɺॳظߏ੒͸ࣗಈԽε ΫϦϓτ౳͕ඞཁ ՝୊ কདྷLinuxBoxͱͯ͠ར༻͢Δ͜ͱΛલఏʹߟ͑ΔͱɺjujuͷσϓϩΠ ઌͱͯ͠ར༻Ͱ͖ΔՄೳੑ͕͋Δ
53. UbuntuΛಈ͔͢ Ubuntu 16.04(Xenial) + ICOS 3.2
54. UbuntuΛಈ͔͢ Ubuntu 16.04(Xenial) + ICOS 3.2 ࣌ؒʹ༨༟͕͋Ε͹ɺσϞ͠·͢ɻ
55. ͝੩ௌ͋Γ͕ͱ͏͟͝·ͨ͠ɻ The End