Upgrade to Pro — share decks privately, control downloads, hide ads and more …

仮想化環境での利用者公平性

Takuya ASADA
November 20, 2012

 仮想化環境での利用者公平性

Takuya ASADA

November 20, 2012
Tweet

More Decks by Takuya ASADA

Other Decks in Technology

Transcript

  1. Ծ૝Խ؀ڥͰͷ
    ར༻ऀެฏੑ
    ઙా ୓໵ @syuu1228
    12೥11݄20೔Ր༵೔

    View Slide

  2. Ծ૝Խ؀ڥͷެฏੑʁ
    • KVMͷήετΛͨ͘͞Μ্ཱͪ͛ͨ࣌ɺϊʔυؒ
    ͷ௨৴ੑೳ΍CPUར༻࣌ؒͷެฏੑ͸Ͳͷఔ౓อ
    ͨΕ͍ͯΔͷ͔
    • CPUར༻࣌ؒ
    • ௨৴ྔ
    • ϨΠςϯγ
    • VMลΓͷෛՙͷ͹Β͖ͭ
    12೥11݄20೔Ր༵೔

    View Slide

  3. ଌͬͯΈΑ͏
    • ϨΠςϯγɾ௨৴ྔɾCPU࢖༻཰ͳͲͷ໘Ͱ
    Ͳͷఔ౓ެฏͳεέδϡʔϧ͕ߦΘΕΔͷ͔
    • VM਺Λ࣮CPU਺Ҏ্ʹ૿΍ͨ࣌͠ʹԿ͕ى
    ͖Δ͔
    • Ծ૝NICͱSR−IOVͰ͸ੑೳಛੑ͕ͲͷΑ͏
    ʹҟͳΔ͔
    12೥11݄20೔Ր༵೔

    View Slide

  4. ੑೳଌఆ؀ڥ
    • ̎୆ͷLinuxػɾ10G NIC
    • 1ʙ64୆ͷKVMήετɾ128ϓϩηεͷ
    netperf
    1VM→VM͋ͨΓ128ϑϩʔ
    2VM→VM͋ͨΓ64ϑϩʔ
    4VM→VM͋ͨΓ32ϑϩʔ…
    64VM→VM͋ͨΓ2ϑϩʔ
    • TCP Request/ResponseϞʔυ
    1byteͷύέοτΛϐϯϙϯ
    VMϗετ
    ήετ
    ςετػ
    10G NIC
    10G NIC
    ήετ
    netperf
    netserver
    netperf
    12೥11݄20೔Ր༵೔

    View Slide

  5. ϋʔυʗιϑτͷεϖοΫ
    Distribution Ubuntu Server 12.10
    Linux Kernel 3.5.0-18-generic
    QEMU-KVM 1.2.0
    Netperf 2.5.0
    CPU(VMϗετ) Intel Core i7 980 (3.33GHz)
    Memory(VMϗετ) 24GB
    CPU(ςετػ) Intel Core i7 860 (2.8GHz)
    Memory(ςετػ) 8GB
    NIC Intel 82599(ixgbe)
    ෺ཧ6ίΞɺ࿦ཧ12ίΞ
    12೥11݄20೔Ր༵೔

    View Slide

  6. ήετϚγϯͷઃఆ
    • Ծ૝CPUɿ1ͭ
    • Ծ૝NICͷઃఆ
    • macvtapͰbridge / vhost-net
    • SR-IOV
    • 512MBͷϝϞϦׂΓ౰ͯ
    12೥11݄20೔Ր༵೔

    View Slide

  7. ͓͞Β͍ɿSR-IOV
    • ෺ཧNIC͕VMʹରͯ͠Ծ૝NICΛ௚઀ఏڙɹIO͸ϋΠύʔόΠβʔΛհࡏͤͣʹߦΘΕΔ
    • ׂࠐΈ͚ͩ͸Ծ૝Խग़དྷ͍ͯͳ͍ͷͰKVMΛ௨ͯ͡సૹ͞ΕΔ
    • ࠷΋ੑೳ͕ߴ͍͕ϋʔυରԠ͕ඞཁʗήετͷύέοτΛϑΟϧλʔɾվม͢Δࣄ͸೉͍͠
    NIC
    Χʔωϧ
    Χʔωϧ
    ήετ
    Ϣʔβ
    PF
    PFυϥΠό
    qemu
    kvm
    VF
    υϥΠό
    TCP/IP
    ελοΫ
    NIC2
    ϑΥϫʔυ
    ׂΓࠐΈ
    ෺ཧׂΓࠐΈ
    ׂΓࠐΈ
    ׂࠐΈ
    ϋϯυϥ
    VF
    %."ύεεϧʔ
    ϋΠύʔόΠβ
    12೥11݄20೔Ր༵೔

    View Slide

  8. macvtap




    slot='0x07' function='0x0'/>

    12೥11݄20೔Ր༵೔

    View Slide

  9. SR-IOV


    slot='0x10' function='0x2'/>

    slot='0x04' function='0x0'/>

    12೥11݄20೔Ր༵೔

    View Slide

  10. ൺֱର৅ͱͯ͠ͷ
    ࣮ػੑೳଌఆ
    • ̎୆ͷLinuxػɾ10G NIC
    • KVMήετͰnetserver࣮ߦ
    • 128ϓϩηεͷnetperf
    • TCP Request/ResponseϞʔυ
    1byteͷύέοτΛϐϯϙϯ
    VMϗετ
    ςετػ
    10G NIC
    10G NIC
    netperf
    netserver
    netperf
    netserver
    12೥11݄20೔Ր༵೔

    View Slide

  11. VM਺Λ1-64·Ͱ૿Ճ
    vhost-netͱSR-IOVൺֱ
    12೥11݄20೔Ր༵೔

    View Slide

  12. ඵؒτϥϯβΫγϣϯ਺
    0
    1375.00
    2750.00
    4125.00
    5500.00
    1 2 4 8 16 32 64
    vhost-net SR-IOV
    baremetal
    VM਺
    transaction/sec
    ෺ཧίΞ਺ʹ͍ۙลΓͰ
    ੑೳ࠷ߴ
    12೥11݄20೔Ր༵೔

    View Slide

  13. ϨΠςϯγ
    0
    750.00
    1500.00
    2250.00
    3000.00
    1 2 4 8 16 32 64
    vhost-net SR-IOV
    baremetal
    VM਺
    ϨΠςϯγ஋
    ෺ཧίΞ਺ʹ͍ۙลΓͰ
    ϨΠςϯγ࠷ద
    12೥11݄20೔Ր༵೔

    View Slide

  14. CPUෛՙ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32 64
    vhost-net SR-IOV
    baremetal
    VM਺
    ύʔηϯςʔδ
    ෛՙΛVM਺෼ͷίΞ਺ʹ
    ͔͠෼ࢄͰ͖ͯͳ͍
    12೥11݄20೔Ր༵೔

    View Slide

  15. VMؒͷภΓ
    ʢ ඵؒτϥϯβΫγϣϯ਺ʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32 64
    vhost-net SR-IOV
    VM਺
    ύʔηϯςʔδ
    vhost-net < 10%
    SR-IOV < 30%
    12೥11݄20೔Ր༵೔

    View Slide

  16. VMؒͷภΓ
    ʢ ϨΠςϯγʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32 64
    vhost-net SR-IOV
    VM਺
    ύʔηϯςʔδ
    vhost-net < 10%
    SR-IOV < 30%
    12೥11݄20೔Ր༵೔

    View Slide

  17. VMؒͷภΓ
    ʢ CPUෛՙʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32 64
    vhost-net SR-IOV
    VM਺
    ύʔηϯςʔδ
    vhost-net < 10%
    SR-IOV < 20%
    12೥11݄20೔Ր༵೔

    View Slide

  18. ൺֱ݁Ռ
    • ෺ཧίΞ਺ʹ͍ۙVM਺ͩͱੑೳ͕Ұ൪ྑ͍Α͏ʹ
    ݟ͑Δ
    • ෺ཧίΞ਺ͷ10ഒͷVMΛ࣮ߦͯ͠΋ɺ͜ͷෛՙύλ
    ʔϯͰ͸ۃ୺ͳϨΠςϯγͷѱԽ΍ෛՙͷภΓͳͲ
    ͸ݟΒΕͳ͔ͬͨ
    • ෺ཧ؀ڥͱൺ΂͔ͯͳΓੑೳ͕ѱ͍
    →ෛՙΛෳ਺ίΞ΁෼ࢄग़དྷ͍ͯͳ͍ࣄ͕ݪҼͷީ
    ิͱͯ͠ߟ͑ΒΕΔ
    12೥11݄20೔Ր༵೔

    View Slide

  19. Ծ૝؀ڥ্Ͱ
    ෳ਺ίΞ΁ωοτϫʔΫIO
    ෛՙ෼ࢄͤͨ࣌͞ͷੑೳ
    12೥11݄20೔Ր༵೔

    View Slide

  20. Ծ૝؀ڥ্ͰίΞ෼ࢄʁ
    • Ծ૝CPU਺Λ1-16·Ͱ૿΍ͯ͠ΈΔ
    • ίΞΛ૿΍͚ͨͩ͠Ͱ͸ωοτϫʔΫε
    λοΫ͕ฒྻͰ૸Βͳ͍
    • ✕ RSSɿԾ૝NICɾVF͕ະରԠ
    • ○ RPSɿήετଆLinuxͷઃఆͰ༗ޮԽ͠
    ͯΈΔ
    12೥11݄20೔Ր༵೔

    View Slide

  21. RPS
    $ echo "f" > /sys/class/net/eth1/queues/rx-0/
    rps_cpus
    $ echo 4096 > /sys/class/net/eth1/queues/
    rx-0/rps_flow_cnt
    $ echo 32768 > /proc/sys/net/core/
    rps_sock_flow_entries
    12೥11݄20೔Ր༵೔

    View Slide

  22. vCPU
    16
    12೥11݄20೔Ր༵೔

    View Slide

  23. ඵؒτϥϯβΫγϣϯ਺
    ʢvhost-netʣ
    0
    1375.00
    2750.00
    4125.00
    5500.00
    1 2 4 8 16 32 64
    cpu1 cpu2 cpu4
    cpu8 cpu16 baremetal
    VM਺
    transaction/sec
    VM =< 4ͳΒ΍΍վળ
    12೥11݄20೔Ր༵೔

    View Slide

  24. ඵؒτϥϯβΫγϣϯ਺
    ʢvhost-net, RPSʣ
    0
    1375.00
    2750.00
    4125.00
    5500.00
    1 2 4 8 16 32 64
    cpu1 cpu2 cpu4
    cpu8 cpu16 baremetal
    VM਺
    transaction/sec
    VM =< 4ͳΒRPSͰ
    ੑೳվળ
    12೥11݄20೔Ր༵೔

    View Slide

  25. ඵؒτϥϯβΫγϣϯ਺
    ʢSR-IOVʣ
    0
    1375.00
    2750.00
    4125.00
    5500.00
    1 2 4 8 16 32
    cpu1 cpu2 cpu4
    cpu8 cpu16 baremetal
    VM਺
    transaction/sec
    VM =< 4ͳΒ΍΍վળ
    12೥11݄20೔Ր༵೔

    View Slide

  26. ඵؒτϥϯβΫγϣϯ਺
    ʢSR-IOV, RPSʣ
    0
    1375.00
    2750.00
    4125.00
    5500.00
    1 2 4 8 16 32
    cpu1 cpu2 cpu4
    cpu8 cpu16 baremetal
    VM਺
    transaction/sec
    VM =< 4ͳΒRPSͰ
    ੑೳվળ
    12೥11݄20೔Ր༵೔

    View Slide

  27. ϨΠςϯγ
    ʢvhost-netʣ
    0
    750.00
    1500.00
    2250.00
    3000.00
    1 2 4 8 16 32 64
    cpu1 cpu2 cpu4
    cpu8 cpu16 baremetal
    VM਺
    ϨΠςϯγ஋
    tpsͱ͍͍ͩͨಉ͡܏޲
    12೥11݄20೔Ր༵೔

    View Slide

  28. ϨΠςϯγ
    ʢvhost-net, RPSʣ
    0
    750.00
    1500.00
    2250.00
    3000.00
    1 2 4 8 16 32 64
    cpu1 cpu2 cpu4
    cpu8 cpu16 baremetal
    VM਺
    ϨΠςϯγ஋
    tpsͱ͍͍ͩͨಉ͡܏޲
    12೥11݄20೔Ր༵೔

    View Slide

  29. ϨΠςϯγ
    ʢSR-IOVʣ
    0
    750.00
    1500.00
    2250.00
    3000.00
    1 2 4 8 16 32
    cpu1 cpu2 cpu4
    cpu8 cpu16 baremetal
    VM਺
    ϨΠςϯγ஋
    tpsͱ͍͍ͩͨಉ͡܏޲
    12೥11݄20೔Ր༵೔

    View Slide

  30. ϨΠςϯγ
    ʢSR-IOV, RPSʣ
    0
    750.00
    1500.00
    2250.00
    3000.00
    1 2 4 8 16 32
    cpu1 cpu2 cpu4
    cpu8 cpu16 baremetal
    VM਺
    ϨΠςϯγ஋
    tpsͱ͍͍ͩͨಉ͡܏޲
    12೥11݄20೔Ր༵೔

    View Slide

  31. CPUෛՙ
    ʢvhost-netʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32 64
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    VM =< 4ͳΒɺcpu = 1
    ͷ࣌ΑΓෛՙΛ෼ࢄͰ
    ͖͍ͯΔʁ
    12೥11݄20೔Ր༵೔

    View Slide

  32. CPUෛՙ
    ʢvhost-net, RPSʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32 64
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    RPSͰΑΓෛՙΛ෼ࢄ
    12೥11݄20೔Ր༵೔

    View Slide

  33. CPUෛՙ
    ʢSR-IOVʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    12೥11݄20೔Ր༵೔

    View Slide

  34. CPUෛՙ
    ʢSR-IOV, RPSʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    12೥11݄20೔Ր༵೔

    View Slide

  35. VMؒͷภΓ
    ʢ ඵؒτϥϯβΫγϣϯ਺ʗvhost-netʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32 64
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    cpu >= 8, VM >=16ͷ࣌ʹ
    15%Λ௒͑ΔภΓ
    12೥11݄20೔Ր༵೔

    View Slide

  36. VMؒͷภΓ
    ʢ ඵؒτϥϯβΫγϣϯ਺/vhost-net, RPSʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32 64
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    cpu >= 8, VM >=16ͷ࣌ʹ
    15%Λ௒͑ΔภΓ
    12೥11݄20೔Ր༵೔

    View Slide

  37. VMؒͷภΓ
    ʢ ඵؒτϥϯβΫγϣϯ਺ʗSR-IOVʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    ૯εϨου਺͕෺ཧCPU਺Λ
    ௒͑ͨลΓͰେ͖ͳภΓ
    12೥11݄20೔Ր༵೔

    View Slide

  38. VMؒͷภΓ
    ʢ ඵؒτϥϯβΫγϣϯ਺ʗSR-IOV, RPSʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    ૯εϨου਺͕෺ཧCPU਺Λ
    ௒͑ͨลΓͰେ͖ͳภΓ
    12೥11݄20೔Ր༵೔

    View Slide

  39. VMؒͷภΓ
    ʢ ϨΠςϯγʗvhost-netʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32 64
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    tpsͱ͍͍ͩͨಉ͡܏޲
    12೥11݄20೔Ր༵೔

    View Slide

  40. VMؒͷภΓ
    ʢ ϨΠςϯγʗvhost-net, RPSʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32 64
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    12೥11݄20೔Ր༵೔

    View Slide

  41. VMؒͷภΓ
    ʢ ϨΠςϯγʗSR-IOVʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    ૯εϨου਺͕෺ཧCPU਺Λ
    ௒͑ͨลΓͰେ͖ͳภΓ
    12೥11݄20೔Ր༵೔

    View Slide

  42. VMؒͷภΓ
    ʢ ϨΠςϯγʗSR-IOV, RPSʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    ૯εϨου਺͕෺ཧCPU਺Λ
    ௒͑ͨลΓͰେ͖ͳภΓ
    12೥11݄20೔Ր༵೔

    View Slide

  43. VMؒͷภΓ
    ʢ CPUෛՙʗvhost-netʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32 64
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    ͋·Γେ͖ͳภ
    Γ͸ݟΒΕͳ͍
    12೥11݄20೔Ր༵೔

    View Slide

  44. VMؒͷภΓ
    ʢ CPUෛՙʗvhost-net, RPSʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32 64
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    ͋·Γେ͖ͳภ
    Γ͸ݟΒΕͳ͍
    12೥11݄20೔Ր༵೔

    View Slide

  45. VMؒͷภΓ
    ʢ CPUෛՙʗSR-IOVʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    VM਺ͷ্ঢͰ
    एׯ૿Ճ
    12೥11݄20೔Ր༵೔

    View Slide

  46. VMؒͷภΓ
    ʢ CPUෛՙʗSR-IOV, RPSʣ
    0
    25.00
    50.00
    75.00
    100.00
    1 2 4 8 16 32
    cpu1 cpu2 cpu4
    cpu8 cpu16
    VM਺
    ύʔηϯςʔδ
    VM਺ͷ্ঢͰ
    एׯ૿Ճ
    12೥11݄20೔Ր༵೔

    View Slide

  47. ൺֱ݁Ռ
    • Ծ૝ϚγϯͰ࣮ػੑೳͷ൒෼Ҏ্Λग़ͤͨͷ͸VM਺͕͘͝গͳ͘
    vCPU਺͕͔ͳΓଟ͍࣌ͷΈ → ͜ͷ৚݅Ͱ͔͠RPSͰੑೳ͕ग़ͳ͍
    • ૯vCPU਺Λଟͯ͘͠΋ҙ֎ͱੑೳ͸Լ͕Βͳ͍
    • SR-IOVͷ৔߹ʹVF਺͕෺ཧίΞ਺Λ௒͑ͨลΓͰ͔ͳΓੑೳʹภ
    Γ͕ͰΔ
    • NIC্ͷΩϡʔʹରͯ͠ड৴ॲཧʹׂΓ౰ͯΔίΞ͕ෆ଍ͯ͠
    ͜ͷΑ͏ͳঢ়گʹͳΔͱਪଌ
    • vhost-netͰ͸ͦͷΑ͏ͳݱ৅͕ݟΒΕͣɺ૯vCPU਺Λ૿΍ͯ͠΋
    ൺֱతެฏʹॲཧ͞Ε͍ͯΔΑ͏ʹݟ͑Δ
    12೥11݄20೔Ր༵೔

    View Slide

  48. ϚϧνϢʔβ࣌ͷެ
    ฏੑ
    12೥11݄20೔Ր༵೔

    View Slide

  49. ϚϧνϢʔβ࣌ͷެฏੑ
    • σϑΥϧτεέδϡʔϦϯάͰ͸ɺશͯͷKVM
    ήετϓϩηε͕ެฏʹCPU࣌ؒΛׂΓ౰ͯΒ
    ΕΔ
    →ෳ਺ͷϢʔβ͕ෳ਺ͷVMΛىಈ͍ͯ͠Δ৔
    ߹ɺϢʔβลΓͷϦιʔεׂ౰͕ෆެฏʹͳΔ
    • ϢʔβؒͰެฏʹεέδϡʔϦϯάͯ͠ཉ͍͠
    12೥11݄20೔Ր༵೔

    View Slide

  50. cgroup
    • ϓϩηεάϧʔϓʹରͯ͠Ϧιʔε
    ʢCPUɺϝϞϦɺσΟεΫI/Oɺωοτ
    ϫʔΫI/Oʣͷར༻Λ੍ݶɾִ཭͢Δҝ
    ͷLinuxͷػೳ
    12೥11݄20೔Ր༵೔

    View Slide

  51. cgroup tool install
    $ apt-get install cgroup-bin
    12೥11݄20೔Ր༵೔

    View Slide

  52. $ ls /sys/fs/cgroup
    blkioɹϒϩοΫI/Oͷ੍ݶ
    cpuɹCPUϦιʔεͷ੍ݶ
    cpuacctɹCPUͷ૸ߦ࣌ؒͳͲͷ৘ใऩू
    cpusetɹCPUͷׂΓ෇͚ʹؔ͢Δ੍ݶ
    devicesɹσόΠε΁ͷΞΫηεڐՄɾڋ൱
    freezerɹάϧʔϓʹଐ͢ΔϓϩηεͷҰ࣌ఀࢭɾ࠶։
    memoryɹϝϞϦϦιʔεͷ੍ݶ
    net_clsɹૹग़ύέοτʹtc༻ͷλάΛઃఆ
    nsɹωʔϜεϖʔεʢίϯςφ༻ʣ
    12೥11݄20೔Ր༵೔

    View Slide

  53. cd /sys/fs/cgroup/cpu
    $ mkdir grp_a
    $ echo ‘12254’ > grp_a/tasks
    $ echo ‘2184’ > grp_a/tasks
    $ echo ‘512’ > grp_a/cpu.shares
    $ mkdir grp_b
    $ echo ‘9012’ > grp_b/tasks
    $ echo ‘1024’ > grp_b/cpu.shares
    άϧʔϓ࡞੒
    άϧʔϓॴଐϓϩη
    εΛ௥ՃʢPIDʣ
    άϧʔϓ಺ͷλεΫͰ࢖༻ग़དྷ
    ΔCPU࣌ؒͷ૬ରత഑෼஋
    12೥11݄20೔Ր༵೔

    View Slide

  54. libvirtdͱcgroup
    • /sys/fs/cgroup/cpu/libvirt/qemu/ʹ
    VMؔ࿈ϓϩηε͕ࣗಈొ࿥͞ΕΔ
    • /sys/fs/cgroup/cpu/libvirt/qemu//
    vcpuNʹvcpuϓϩηε͕ࣗಈొ࿥͞ΕΔ
    • libvirtܦ༝ͰcgroupͷϦιʔε੍ݶઃఆ
    ͕ग़དྷΔ
    12೥11݄20೔Ր༵೔

    View Slide

  55. virshͱcgroup
    $ virsh schedinfo vm0
    Scheduler : posix
    cpu_shares : 1024
    vcpu_period : 100000
    vcpu_quota : -1
    $ virsh schedinfo --set cpu_shares=512 vm0
    12೥11݄20೔Ր༵೔

    View Slide

  56. ̎ͭͷάϧʔϓʹެฏʹϦ
    ιʔεΛׂΓ౰ͯͯΈΔ
    • /sys/fs/cgroup/grp_a
    • VMΠϯελϯεɿ̒ݸ
    • cpu.sharesɿσϑΥϧτʢ̍̌̎̐ʣ
    • /sys/fs/cgroup/grp_b
    • VMΠϯελϯεɿ̍̌ݸ
    • cpu.sharesɿσϑΥϧτʢ̍̌̎̐ʣ
    12೥11݄20೔Ր༵೔

    View Slide

  57. άϧʔϓͷ࡞Γํ
    $ virsh start
    $ mkdir /sys/fs/cgroup/cpu/grp_a
    $ cat /sys/fs/cgroup/cpu/libvirt/qemu//tasks
    > tmp
    $ cat /sys/fs/cgroup/cpu/libvirt/qemu//
    vcpu*/tasks >> tmp
    $ cat tmp > /sys/fs/cgroup/cpu/grp_a/tasks
    12೥11݄20೔Ր༵೔

    View Slide

  58. vhost-net/16VM࣌ͷCPU࣮ߦ࣌ؒ
    ʢcgroupະ࢖༻࣌ʣ
    0
    625000000
    1250000000
    1875000000
    2500000000
    ࣮ߦ࣌ؒ
    12೥11݄20೔Ր༵೔

    View Slide

  59. vhost-net/16VM࣌ͷCPU࣮ߦ࣌ؒ
    ʢcgroup࢖༻࣌ʣ
    0
    625000000
    1250000000
    1875000000
    2500000000
    ࣮ߦ࣌ؒ
    grp_a grp_b
    12೥11݄20೔Ր༵೔

    View Slide

  60. vhost-net/16VM࣌ͷωοτϫʔΫIO
    ʢcgroupະ࢖༻࣌ʣ
    0
    1375000
    2750000
    4125000
    5500000
    σʔλྔ
    12೥11݄20೔Ր༵೔

    View Slide

  61. vhost-net/16VM࣌ͷωοτϫʔΫIO
    ʢcgroup࢖༻࣌ʣ
    0
    1375000
    2750000
    4125000
    5500000
    σʔλྔ
    grp_a grp_b
    12೥11݄20೔Ր༵೔

    View Slide

  62. άϧʔϓ͝ͱͷ߹ܭ஋ൺֱ
    ʢcgroup࢖༻࣌ʣ
    0
    0.50
    1.00
    1.50
    2.00
    CPU Net
    grp_a grp_b
    1.10
    1.14
    12೥11݄20೔Ր༵೔

    View Slide

  63. ʦࢀߟʧάϧʔϓ͝ͱͷ߹ܭ஋ൺֱ
    ʢcgroupະ࢖༻࣌ʣ
    0
    0.50
    1.00
    1.50
    2.00
    CPU Net
    grp_a grp_b
    1.64
    1.65
    12೥11݄20೔Ր༵೔

    View Slide

  64. ൺֱ݁Ռ
    • άϧʔϓʹ෼͚Δ͚ͩͰσϑΥϧτͰάϧʔϓεέ
    δϡʔϦϯά͕ߦΘΕɺ࣮ଌ஋Ͱ΋άϧʔϓʹର͠
    ͯެฏʹCPU͕഑෼͞Ε͍ͯΔࣄ͕֬ೝͰ͖Δ
    • CPUΛߜΔࣄͰ݁ՌతʹωοτϫʔΫIOΛߜΔࣄ
    ΋͋Δఔ౓ग़དྷ͍ͯΔ
    • Ͱ΋͋·Γݫີ͡Όͳ͍ʢ࣮ଌ14%͘Β͍ͷޡࠩʣ
    12೥11݄20೔Ր༵೔

    View Slide

  65. CFS bandwidth control
    • ͜ͷάϧʔϓ͸cpu.cfs_period_usͷ୯Ґ࣌
    ؒதʹ࠷େcpu.cfs_quota_us͚࣮ͩޮग़དྷΔ
    • άϧʔϓ಺ͷϓϩηε͸period୯Ґ࣌ؒ
    தɺಉ࣌ʹ࣮ߦ͞Ε͏ΔͷͰquota͸period
    Λ௒͑͏Δʢ2CPUͰΊ͍͍ͬͺ͍૸ͬͨ
    Βquota = period * 2ʣ
    12೥11݄20೔Ր༵೔

    View Slide

  66. vhost-net/16VM࣌ͷCPU࣮ߦ࣌ؒ
    ʢquota=200000,period=50000ʣ
    0
    625000000
    1250000000
    1875000000
    2500000000
    ࣮ߦ࣌ؒ
    grp_a grp_b
    12೥11݄20೔Ր༵೔

    View Slide

  67. vhost-net/16VM࣌ͷωοτϫʔΫIO
    ʢquota=200000,period=50000ʣ
    0
    1375000
    2750000
    4125000
    5500000
    σʔλྔ
    grp_a grp_b
    12೥11݄20೔Ր༵೔

    View Slide

  68. άϧʔϓ͝ͱͷ߹ܭ஋ൺֱ
    ʢquota=200000,period=50000ʣ
    0
    0.50
    1.00
    1.50
    2.00
    CPU Net
    grp_a grp_b
    1.01
    0.63
    12೥11݄20೔Ր༵೔

    View Slide

  69. ݕূ݁Ռ
    • ͔֬ʹάϥϑ্͸៉ྷʹͳΔͷ͕ͩɺCPU͕ͲΜͳʹ
    ۭ͍͍ͯͯ΋quota·ͰߦͬͨΒઈର࣮ߦ͞Εͳ͍
    • ͔ͱ͍࣮ͬͯߦ͖͠Εͳ͍ఔͷquotaΛ༩͑Δͱઃఆ͠
    ͳ͍৔߹ͱมΘΒͳ͘ͳΔ
    • CPU΋ωοτϫʔΫIO΋ͪΐ͏ͲΑ͘ͽͬͨΓ1.0ͱ͍
    ͏༁ʹ͸͍͔ͳ͔ͬͨ
    • period͕௕͗͢ΔͱVM͕௕͍࣮ؒߦ͞Εͳͯ͘ྑ͘ͳ
    ͍͜ͱ͕ى͖ͦ͏
    12೥11݄20೔Ր༵೔

    View Slide

  70. net_cls
    $ tc class add dev virbr0 parent 10: classid
    10:1 htb rate 24Mbit
    $ echo 0x100001 > /sys/fs/cgroup/net_cls/
    grp_a/net_cls.classid
    ϓϩηε͔Βૹ৴͞ΕΔύέοτʹtcͷ
    λάΛ෇͚ΔࣄʹΑΓଳҬ੍ݶ͕ग़དྷΔ
    12೥11݄20೔Ր༵೔

    View Slide

  71. ·ͱΊ
    • KVMʹΑΔԾ૝Խ؀ڥͷϦιʔεׂΓ౰ͯͷެฏੑʹ͍ͭͯ
    ݕূͨ͠
    • ࠓճͷ࣮ݧͷ৚݅ԼͰ͸ɺ࣮CPU਺Λେ͖͘௒͑ΔVM਺Ͱ
    ΋ۃ୺ͳϨΠςϯγͷѱԽ΍ෛՙͷภΓͳͲ͸ݟΒΕͳ͔
    ͬͨ
    • ୠ͠ɺSR-IOVͷ৔߹ʹVF਺͕෺ཧίΞ਺Λ௒͑ͨลΓͰ͔
    ͳΓੑೳʹภΓ͕ൃੜͨ͠
    • cgroupΛ༻͍ͯϚϧνϢʔβ࣌ʹϢʔβؒͷCPUׂΓ౰ͯΛ
    ެฏʹग़དྷΔࣄΛ֬ೝͨ͠
    12೥11݄20೔Ր༵೔

    View Slide