$30 off During Our Annual Pro Sale. View Details »

Dynamic VM Scheduling
 in OpenStack

Dynamic VM Scheduling
 in OpenStack

CNDT2022で話した内容です。

Kazuhiko Yamashita

November 25, 2022
Tweet

More Decks by Kazuhiko Yamashita

Other Decks in Programming

Transcript

  1. Dynamic VM Scheduling

    in OpenStack
    ʙ͜ͷલɺθΫγΟಡΜ͚ͩͲɺRFCΑΓ೉͔ͬͨ͠ฤʙ

    View Slide

  2. ࢁԼ࿨඙!QZBNB
    (.0ϖύϘٕज़ج൫νʔϜ
    γχΞɾϓϦϯγύϧ
    UFOTOBQPODPN
    QZBNBGVO
    TUOTKQ

    View Slide

  3. 45/4
    -JOVY/444FSWFS
    TUOTKQ

    View Slide

  4. 45/4

    View Slide

  5. View Slide

  6. CNDT ٱ͠ͿΓʹΦϑϥΠϯͰ

    ࠷ߴ!!1

    Ͱ΋͜Ε͸࿥ը͚ͩͲͶ

    View Slide

  7. ϖύϘͷΠϯϑϥߏ੒

    CPU StealʹىҼͨ͠ɺྡਓ໰୊

    εʔύʔˑϥΠϒϚΠάϨʔγϣϯ

    ·ͱΊͱɺ݁ࠗʹ͓͚Δେࣄͳ͜ͱ42બ
    ࠓ೔࿩͢͜ͱ

    View Slide

  8. ϖύϘͷΠϯϑϥߏ੒
    0QFO4UBDL
    #BSFNFUBM
    "84 IFSPLV

    ($1
    ෺ཧαʔό୆7.ن໛
    ϗεςΟϯάܥͷαʔϏε΍ɺ֤छج൫αʔϏεͰར༻
    &,4 3%4 &MBTUJDYYYܥͷϚωʔδυαʔϏεΛத৺ʹར༻
    (,& #JH2VFSZΛத৺ʹ"84ͷ%3ɺσʔλΤϯδχΞϦϯάͷج൫ͱͯ͠ར༻

    View Slide

  9. 0QFO4UBDL
    0QFO4UBDLͰ؅ཧ͢Δ7.ͷ্ʹ

    ࣗࣾ։ൃͷΤϯδϯΛར༻ͯ͠
    ,VCFSOFUFTΛσϓϩΠ 0QFO4UBDL
    #BSFNFUBM #BSFNFUBM #BSFNFUBM
    7. 7. 7.
    ,VCFSOFUFT
    $POUBJOFS $POUBJOFS $POUBJOFS
    ,VCFSOFUFT

    View Slide

  10. CPU StealʹىҼͨ͠ɺ


    ྡਓ໰୊

    View Slide

  11. ॠؒɺ৺ɺॏͶͯ
    ೔ຊ࣌ؒͰαʔϏε΍ͬͯΔ͔Β


    ϐʔΫλΠϜ͕ॏෳ
    9:00 12:00 20:00
    ϐʔΫλΠϜ
    ϐʔΫλΠϜ
    ϐʔΫλΠϜ
    ϐʔΫλΠϜ
    ϐʔΫλΠϜ
    ϐʔΫλΠϜ

    View Slide

  12. CPU Steal
    Ϧιʔε͕ୣ͍߹͍ʹͳΓɺCPU Steal͕ൃੜ

    View Slide

  13. CPU Stealͷ֓೦
    Hypervisor
    VM1
    CPU࢖͍͍ͨʂ 10ඵ͋͛Δ
    VM2
    CPU࢖͍͍ͨʂ 10ඵ͋͛Δ

    View Slide

  14. VM؀ڥʹ͓͚ΔCPU Stealͷ֓೦
    Hypervisor
    VM1
    CPU࢖͍͍ͨʂ 10ඵ͋͛Δ
    VM2
    CPU࢖͍͍ͨʂ 10ඵ͋͛Δ
    ࣮ࡍ͸8ඵ͔͋͛͠ΒΕͳ͔ͬͨ
    ࠩ෼ͷ2ඵ͕CPU Steal

    View Slide

  15. CPU Steal͕ى͖Δͱʁ
    αʔόॲཧೳྗͷ௿Լ
    αʔόϝτϦΫεͷѱԽ
    HTTPͷϨΠςϯγͷѱԽ

    ઀ଓ਺ͷ૿Ճ
    ϩʔυΞϕϨʔδ஋ͷ্ঢ

    View Slide

  16. εʔύʔˑϥΠϒϚΠάϨʔγϣϯ

    View Slide

  17. ϥΠϒϚΠάϨʔγϣϯ
    Hypervisor
    VM1 VM2
    10ඵ
    CPU
    2ඵ
    CPU
    Hypervisor
    VM1
    10ඵ
    CPU
    VMΛఀࢭ͢Δ͜ͱͳ͘HypervisorؒΛҠಈ

    View Slide

  18. ϥΠϒϚΠάϨʔγϣϯ
    Hypervisor Hypervisor
    VM1
    Disk
    Memory
    VM1
    Disk
    STEP 1 σΟεΫΛಉظ
    ※ڞ༗ετϨʔδΛར༻͍ͯ͠ͳ͍৔߹

    View Slide

  19. ϥΠϒϚΠάϨʔγϣϯ
    Hypervisor Hypervisor
    VM1
    Disk
    Memory
    VM1
    Disk
    STEP 2 ϝϞϦΛಉظͯ͠੩ࢭ఺Λ଴ͭ
    ※ڞ༗ετϨʔδΛར༻͍ͯ͠ͳ͍৔߹
    Memory

    View Slide

  20. OpenStackͷϥΠϒϚΠάϨʔγϣϯ
    https://docs.openstack.org/nova/pike/admin/con
    fi
    guring-migrations.html

    View Slide

  21. ϥΠϒϚΠάϨʔγϣϯ
    Hypervisor Hypervisor
    VM1
    Disk
    Memory
    VM1
    Disk
    Memory
    STEP 3 ੩ࢭ఺͕औΕͨΒ৽͍͠ΠϯελϯεΛ

    ༗ޮʹͯ͠ɺωοτϫʔΫܦ࿏Λ੾Γସ͑Δ
    ͓લ͕ࠓ೔͔ΒVM1ͩ

    View Slide

  22. εʔύʔˑϥΠϒϚΠάϨʔγϣϯ

    View Slide

  23. εʔύʔˑϥΠϒϚΠάϨʔγϣϯ
    1. VMʹ͓͍ͯStealͷൃੜΛݕ஌͍ͨ͠

    2.Stealͷൃੜ͍ͯ͠ͳ͍HypervisorʹεέδϡʔϦϯά͍ͨ͠

    3.μ΢ϯλΠϜΛൃੜͤͨ͘͞ͳ͍

    4.ࣗಈͰ৸ͯΔؒʹ͢΂ͯऴΘͬͯ΄͍͠
    ͱ͍͏ͷΛɺRubyͰ࣮૷ͨ͠

    View Slide

  24. VMͷStealൃੜݕ஌
    Load5͕͖͍͠஋Ҏ্͔ͭɺSteal͕ൃੜ͍ͯ͠Δ͜ͱ
    Steal͕ൃੜͯ͠΋ɺॲཧ͕஗Ԇ͍ͯ͠ͳ͍ͳΒ͹໰୊ͳ͍

    View Slide

  25. εέδϡʔϦϯά
    HypervisorͷϝτϦΫεͰSteal͕͖͍͠஋ҎԼͰɺΑΓۭ͖͕͋Δ

    ಉ͡ϩʔϧͷVM͕ಉҰαʔόʹ͖͍͠஋Ҏ্ऩ༰͞Ε͍ͯͳ͍

    ಉ͡AZͷαʔόͰ͋Δ

    View Slide

  26. Availability Zone
    LB
    AZ-1
    VM1
    AZ-2
    VM2
    LB
    AZ-1
    VM1
    VM2
    AZ-2
    AZΛ·͍ͨͰεέδϡʔϦϯά͢Δͱ
    ଱ো֐ੑ͕ഁ໓!!1

    View Slide

  27. μ΢ϯλΠϜΛൃੜͤ͞ͳ͍
    ΦʔέετϨʔλʔͰ޻෉͢Δ
    $ kubectl drain example.local --force --ignore-daemonsets —delete-emptydir-data

    # after migrated

    $ kubectl uncordon example.local
    $ consul maint -enable

    # after migrated

    $ consul maint -disable
    Kubernetes
    Consul

    View Slide

  28. kubectl drain
    node-1
    pod-1
    Scheduling Disabled
    node-2
    pod-2
    pod-1
    node΁ͷPodͷׂ౰Λఀࢭͯ͠ɺ௥͍ग़͢

    View Slide

  29. consul
    node-1

    (10.0.0.1)
    nginx
    node-2

    (10.0.0.2)
    nginx
    node-3


    (10.0.0.3)
    nginx
    $ dig web.service.consul a

    10.0.0.1

    10.0.0.2

    10.0.0.3

    ϔϧενΣοΫͳͲΛར༻ͯ͠ɺαʔϏεσΟεΧόϦ͕Ͱ͖Δ
    DNSͷଞʹ΋ɺConsul TemplateΛར༻ͨ͠

    ߏ੒ϑΝΠϧͷϨϯμϦϯάͳͲͰ׆༻͍ͯ͠Δ

    View Slide

  30. consul maint -enable
    node-1

    (10.0.0.1)
    nginx
    node-2

    (10.0.0.2)
    nginx
    node-3


    (10.0.0.3)
    nginx
    $ dig web.service.consul a

    10.0.0.1

    10.0.0.2

    Ϋϥελ͔Β੾Γ཭͠
    consul maint -enable

    View Slide

  31. ॲཧܥ·ͱΊ
    Stealͷൃੜ͔ͭ


    ϩʔυΞϕϨʔδͷ্ঢ
    ࠷దͳ฼؋ͷબఆ
    Ϋϥελ͔Βͷ੾Γ཭͠
    ϥΠϒϚΠάϨʔγϣϯ
    Ϋϥελ΁ͷ૊ΈࠐΈ
    ࣮ߦ؀ڥͱͯ͠͸OpenStackͷ্Ͱಈ͘
    Kubernetesͷதͷpodͱͯ͠ɺ
    Reconciliation loopͬΆ͘ಈ͍͍ͯΔ

    View Slide

  32. ຊ౰ʹ͋ͬͨා͍࿩

    View Slide

  33. ʮεʔύʔˑϥΠϒϚΠάϨʔγϣϯ

    ಈ͍ͯΔΜͰ͚͢ͲɺϨΠςϯγ

    ߴ͍ΜͰ͢ΑͶʯ

    View Slide

  34. ϥΠϒϚΠάϨʔγϣϯ͕ଳҬΛ৯͍੾Δ
    ͓ன͔Βഁ໓!!1

    View Slide

  35. ϥΠϒϚΠάϨʔγϣϯ(࠶ܝ)
    Hypervisor Hypervisor
    VM1
    Disk
    Memory
    VM1
    Disk
    STEP 1 σΟεΫΛಉظ
    ※ڞ༗ετϨʔδΛར༻͍ͯ͠ͳ͍৔߹
    ʊਓਓਓਓʊ


    ʼɹ500Gɹʻ


    ʉY^Y^Y^Y^ʉ


    View Slide

  36. View Slide

  37. View Slide

  38. ·ͱΊ
    ྡਓ໰୊Λࣗ෼Ͱεέδϡʔϥʔॻ͘͜ͱͰղܾͨ͠

    OSSʹ͢ΔͱdeschedulerͷΑ͏ͳந৅౓ʹͳΔͷͰɺ

    ࠓճ͸ެ։͠ͳ͔ͬͨ

    ϥΠϒϚΠάϨʔγϣϯ͸ҙ֎ͱଳҬ৯͏ͷͰɺωοτϫʔΫΛ෼཭ͯ͠
    ϨʔτϦϛοτೖΕͨ΄͏͕͍͍

    View Slide

  39. ଟ෼ɺձ৔ʹ͍ΔͷͰɺ


    టʹͳΓ·͠ΐ͏ʂʂʂ

    View Slide

  40. View Slide

  41. ͓͠·͍
    ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU

    View Slide