Dynamic VM Scheduling in OpenStack

CNDT2022で話した内容です。

Transcript

1. Dynamic VM Scheduling 
 in OpenStack ʙ͜ͷલɺθΫγΟಡΜ͚ͩͲɺRFCΑΓ೉͔ͬͨ͠ฤʙ
2. ࢁԼ
   ࿨඙
   
   !QZBNB
   (.0ϖύϘ
   ٕज़ج൫νʔϜ
   
   
   γχΞɾϓϦϯγύϧ
   
   
   
   UFOTOBQPODPN
   
   
   
   QZBNBGVO
   
   
   
   TUOTKQ
3. 45/4
   -JOVY
   /44
   4FSWFS
   TUOTKQ
4. 45/4
6. CNDT ٱ͠ͿΓʹΦϑϥΠϯͰ 
 ࠷ߴ!!1 
 Ͱ΋͜Ε͸࿥ը͚ͩͲͶ
7. ϖύϘͷΠϯϑϥߏ੒ CPU StealʹىҼͨ͠ɺྡਓ໰୊ εʔύʔˑϥΠϒϚΠάϨʔγϣϯ ·ͱΊͱɺ݁ࠗʹ͓͚Δେࣄͳ͜ͱ42બ ࠓ೔࿩͢͜ͱ
8. ϖύϘͷΠϯϑϥߏ੒ 0QFO4UBDL
   #BSFNFUBM
   "84 IFSPLV
   ($1 ෺ཧαʔό
   ୆
   7.ن໛
   ϗεςΟϯάܥͷαʔϏε΍ɺ֤छج൫αʔϏεͰར༻ &,4 3%4 &MBTUJDYYYܥͷϚωʔδυαʔϏεΛத৺ʹར༻ (,& #JH2VFSZΛத৺ʹ"84ͷ%3ɺσʔλΤϯδχΞϦϯάͷج൫ͱͯ͠ར༻
9. 0QFO4UBDL 0QFO4UBDLͰ؅ཧ͢Δ7.ͷ্ʹ 
 ࣗࣾ։ൃͷΤϯδϯΛར༻ͯ͠
   ,VCFSOFUFTΛσϓϩΠ 0QFO4UBDL #BSFNFUBM #BSFNFUBM #BSFNFUBM 7. 7. 7. ,VCFSOFUFT $POUBJOFS $POUBJOFS $POUBJOFS ,VCFSOFUFT
10. CPU StealʹىҼͨ͠ɺ ྡਓ໰୊
11. ॠؒɺ৺ɺॏͶͯ ೔ຊ࣌ؒͰαʔϏε΍ͬͯΔ͔Β ϐʔΫλΠϜ͕ॏෳ 9:00 12:00 20:00 ϐʔΫλΠϜ ϐʔΫλΠϜ ϐʔΫλΠϜ ϐʔΫλΠϜ ϐʔΫλΠϜ ϐʔΫλΠϜ
12. CPU Steal Ϧιʔε͕ୣ͍߹͍ʹͳΓɺCPU Steal͕ൃੜ
13. CPU Stealͷ֓೦ Hypervisor VM1 CPU࢖͍͍ͨʂ 10ඵ͋͛Δ VM2 CPU࢖͍͍ͨʂ 10ඵ͋͛Δ
14. VM؀ڥʹ͓͚ΔCPU Stealͷ֓೦ Hypervisor VM1 CPU࢖͍͍ͨʂ 10ඵ͋͛Δ VM2 CPU࢖͍͍ͨʂ 10ඵ͋͛Δ ࣮ࡍ͸8ඵ͔͋͛͠ΒΕͳ͔ͬͨ ࠩ෼ͷ2ඵ͕CPU Steal
15. CPU Steal͕ى͖Δͱʁ αʔόॲཧೳྗͷ௿Լ αʔόϝτϦΫεͷѱԽ HTTPͷϨΠςϯγͷѱԽ ઀ଓ਺ͷ૿Ճ ϩʔυΞϕϨʔδ஋ͷ্ঢ
16. εʔύʔˑϥΠϒϚΠάϨʔγϣϯ
17. ϥΠϒϚΠάϨʔγϣϯ Hypervisor VM1 VM2 10ඵ CPU 2ඵ CPU Hypervisor VM1 10ඵ CPU VMΛఀࢭ͢Δ͜ͱͳ͘HypervisorؒΛҠಈ
18. ϥΠϒϚΠάϨʔγϣϯ Hypervisor Hypervisor VM1 Disk Memory VM1 Disk STEP 1 σΟεΫΛಉظ ※ڞ༗ετϨʔδΛར༻͍ͯ͠ͳ͍৔߹
19. ϥΠϒϚΠάϨʔγϣϯ Hypervisor Hypervisor VM1 Disk Memory VM1 Disk STEP 2 ϝϞϦΛಉظͯ͠੩ࢭ఺Λ଴ͭ ※ڞ༗ετϨʔδΛར༻͍ͯ͠ͳ͍৔߹ Memory
20. OpenStackͷϥΠϒϚΠάϨʔγϣϯ https://docs.openstack.org/nova/pike/admin/con fi guring-migrations.html
21. ϥΠϒϚΠάϨʔγϣϯ Hypervisor Hypervisor VM1 Disk Memory VM1 Disk Memory STEP 3 ੩ࢭ఺͕औΕͨΒ৽͍͠ΠϯελϯεΛ
 ༗ޮʹͯ͠ɺωοτϫʔΫܦ࿏Λ੾Γସ͑Δ ͓લ͕ࠓ೔͔ΒVM1ͩ
22. εʔύʔˑϥΠϒϚΠάϨʔγϣϯ
23. εʔύʔˑϥΠϒϚΠάϨʔγϣϯ 1. VMʹ͓͍ͯStealͷൃੜΛݕ஌͍ͨ͠ 2.Stealͷൃੜ͍ͯ͠ͳ͍HypervisorʹεέδϡʔϦϯά͍ͨ͠ 3.μ΢ϯλΠϜΛൃੜͤͨ͘͞ͳ͍ 4.ࣗಈͰ৸ͯΔؒʹ͢΂ͯऴΘͬͯ΄͍͠ ͱ͍͏ͷΛɺRubyͰ࣮૷ͨ͠
24. VMͷStealൃੜݕ஌ Load5͕͖͍͠஋Ҏ্͔ͭɺSteal͕ൃੜ͍ͯ͠Δ͜ͱ Steal͕ൃੜͯ͠΋ɺॲཧ͕஗Ԇ͍ͯ͠ͳ͍ͳΒ͹໰୊ͳ͍
25. εέδϡʔϦϯά HypervisorͷϝτϦΫεͰSteal͕͖͍͠஋ҎԼͰɺΑΓۭ͖͕͋Δ ಉ͡ϩʔϧͷVM͕ಉҰαʔόʹ͖͍͠஋Ҏ্ऩ༰͞Ε͍ͯͳ͍ ಉ͡AZͷαʔόͰ͋Δ
26. Availability Zone LB AZ-1 VM1 AZ-2 VM2 LB AZ-1 VM1 VM2 AZ-2 AZΛ·͍ͨͰεέδϡʔϦϯά͢Δͱ ଱ো֐ੑ͕ഁ໓!!1
27. μ΢ϯλΠϜΛൃੜͤ͞ͳ͍ ΦʔέετϨʔλʔͰ޻෉͢Δ $ kubectl drain example.local --force --ignore-daemonsets —delete-emptydir-data # after migrated $ kubectl uncordon example.local $ consul maint -enable # after migrated $ consul maint -disable Kubernetes Consul
28. kubectl drain node-1 pod-1 Scheduling Disabled node-2 pod-2 pod-1 node΁ͷPodͷׂ౰Λఀࢭͯ͠ɺ௥͍ग़͢
29. consul node-1 
 (10.0.0.1) nginx node-2 
 (10.0.0.2) nginx node-3 (10.0.0.3) nginx $ dig web.service.consul a 10.0.0.1 10.0.0.2 10.0.0.3 ϔϧενΣοΫͳͲΛར༻ͯ͠ɺαʔϏεσΟεΧόϦ͕Ͱ͖Δ DNSͷଞʹ΋ɺConsul TemplateΛར༻ͨ͠ ߏ੒ϑΝΠϧͷϨϯμϦϯάͳͲͰ׆༻͍ͯ͠Δ
30. consul maint -enable node-1 
 (10.0.0.1) nginx node-2 
 (10.0.0.2) nginx node-3 (10.0.0.3) nginx $ dig web.service.consul a 10.0.0.1 10.0.0.2 Ϋϥελ͔Β੾Γ཭͠ consul maint -enable
31. ॲཧܥ·ͱΊ Stealͷൃੜ͔ͭ ϩʔυΞϕϨʔδͷ্ঢ ࠷దͳ฼؋ͷબఆ Ϋϥελ͔Βͷ੾Γ཭͠ ϥΠϒϚΠάϨʔγϣϯ Ϋϥελ΁ͷ૊ΈࠐΈ ࣮ߦ؀ڥͱͯ͠͸OpenStackͷ্Ͱಈ͘ Kubernetesͷதͷpodͱͯ͠ɺ Reconciliation loopͬΆ͘ಈ͍͍ͯΔ
32. ຊ౰ʹ͋ͬͨා͍࿩
33. ʮεʔύʔˑϥΠϒϚΠάϨʔγϣϯ 
 ಈ͍ͯΔΜͰ͚͢ͲɺϨΠςϯγ 
 ߴ͍ΜͰ͢ΑͶʯ
34. ϥΠϒϚΠάϨʔγϣϯ͕ଳҬΛ৯͍੾Δ ͓ன͔Βഁ໓!!1
35. ϥΠϒϚΠάϨʔγϣϯ(࠶ܝ) Hypervisor Hypervisor VM1 Disk Memory VM1 Disk STEP 1 σΟεΫΛಉظ ※ڞ༗ετϨʔδΛར༻͍ͯ͠ͳ͍৔߹ ʊਓਓਓਓʊ ʼɹ500Gɹʻ ʉY^Y^Y^Y^ʉ
38. ·ͱΊ ྡਓ໰୊Λࣗ෼Ͱεέδϡʔϥʔॻ͘͜ͱͰղܾͨ͠ OSSʹ͢ΔͱdeschedulerͷΑ͏ͳந৅౓ʹͳΔͷͰɺ
 ࠓճ͸ެ։͠ͳ͔ͬͨ ϥΠϒϚΠάϨʔγϣϯ͸ҙ֎ͱଳҬ৯͏ͷͰɺωοτϫʔΫΛ෼཭ͯ͠ ϨʔτϦϛοτೖΕͨ΄͏͕͍͍
39. ଟ෼ɺձ৔ʹ͍ΔͷͰɺ టʹͳΓ·͠ΐ͏ʂʂʂ
41. ͓͠·͍ ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU