ԾԽڥͰͷར༻ऀެฏੑઙా @syuu12281211݄20Ր༵
View Slide
ԾԽڥͷެฏੑʁ• KVMͷήετΛͨ͘͞Μ্ཱͪ͛ͨ࣌ɺϊʔυؒͷ௨৴ੑೳCPUར༻࣌ؒͷެฏੑͲͷఔอͨΕ͍ͯΔͷ͔• CPUར༻࣌ؒ• ௨৴ྔ• ϨΠςϯγ• VMลΓͷෛՙͷΒ͖ͭ1211݄20Ր༵
ଌͬͯΈΑ͏• ϨΠςϯγɾ௨৴ྔɾCPU༻ͳͲͷ໘ͰͲͷఔެฏͳεέδϡʔϧ͕ߦΘΕΔͷ͔• VMΛ࣮CPUҎ্ʹ૿ͨ࣌͠ʹԿ͕ى͖Δ͔• ԾNICͱSR−IOVͰੑೳಛੑ͕ͲͷΑ͏ʹҟͳΔ͔1211݄20Ր༵
ੑೳଌఆڥ• ̎ͷLinuxػɾ10G NIC• 1ʙ64ͷKVMήετɾ128ϓϩηεͷnetperf1VM→VM͋ͨΓ128ϑϩʔ2VM→VM͋ͨΓ64ϑϩʔ4VM→VM͋ͨΓ32ϑϩʔ…64VM→VM͋ͨΓ2ϑϩʔ• TCP Request/ResponseϞʔυ1byteͷύέοτΛϐϯϙϯVMϗετήετςετػ10G NIC10G NICήετnetperfnetservernetperf1211݄20Ր༵
ϋʔυʗιϑτͷεϖοΫDistribution Ubuntu Server 12.10Linux Kernel 3.5.0-18-genericQEMU-KVM 1.2.0Netperf 2.5.0CPU(VMϗετ) Intel Core i7 980 (3.33GHz)Memory(VMϗετ) 24GBCPU(ςετػ) Intel Core i7 860 (2.8GHz)Memory(ςετػ) 8GBNIC Intel 82599(ixgbe)ཧ6ίΞɺཧ12ίΞ1211݄20Ր༵
ήετϚγϯͷઃఆ• ԾCPUɿ1ͭ• ԾNICͷઃఆ• macvtapͰbridge / vhost-net• SR-IOV• 512MBͷϝϞϦׂΓͯ1211݄20Ր༵
͓͞Β͍ɿSR-IOV• ཧNIC͕VMʹରͯ͠ԾNICΛఏڙɹIOϋΠύʔόΠβʔΛհࡏͤͣʹߦΘΕΔ• ׂࠐΈ͚ͩԾԽग़དྷ͍ͯͳ͍ͷͰKVMΛ௨ͯ͡సૹ͞ΕΔ• ࠷ੑೳ͕ߴ͍͕ϋʔυରԠ͕ඞཁʗήετͷύέοτΛϑΟϧλʔɾվม͢Δࣄ͍͠NICΧʔωϧΧʔωϧήετϢʔβPFPFυϥΠόqemukvmVFυϥΠόTCP/IPελοΫNIC2ϑΥϫʔυׂΓࠐΈཧׂΓࠐΈׂΓࠐΈׂࠐΈϋϯυϥVF%."ύεεϧʔϋΠύʔόΠβ1211݄20Ր༵
macvtapslot='0x07' function='0x0'/>1211݄20Ր༵
SR-IOVslot='0x10' function='0x2'/>slot='0x04' function='0x0'/>1211݄20Ր༵
ൺֱରͱͯ͠ͷ࣮ػੑೳଌఆ• ̎ͷLinuxػɾ10G NIC• KVMήετͰnetserver࣮ߦ• 128ϓϩηεͷnetperf• TCP Request/ResponseϞʔυ1byteͷύέοτΛϐϯϙϯVMϗετςετػ10G NIC10G NICnetperfnetservernetperfnetserver1211݄20Ր༵
VMΛ1-64·Ͱ૿Ճvhost-netͱSR-IOVൺֱ1211݄20Ր༵
ඵؒτϥϯβΫγϣϯ01375.002750.004125.005500.001 2 4 8 16 32 64vhost-net SR-IOVbaremetalVMtransaction/secཧίΞʹ͍ۙลΓͰੑೳ࠷ߴ1211݄20Ր༵
ϨΠςϯγ0750.001500.002250.003000.001 2 4 8 16 32 64vhost-net SR-IOVbaremetalVMϨΠςϯγཧίΞʹ͍ۙลΓͰϨΠςϯγ࠷ద1211݄20Ր༵
CPUෛՙ025.0050.0075.00100.001 2 4 8 16 32 64vhost-net SR-IOVbaremetalVMύʔηϯςʔδෛՙΛVMͷίΞʹ͔͠ࢄͰ͖ͯͳ͍1211݄20Ր༵
VMؒͷภΓʢ ඵؒτϥϯβΫγϣϯʣ025.0050.0075.00100.001 2 4 8 16 32 64vhost-net SR-IOVVMύʔηϯςʔδvhost-net < 10%SR-IOV < 30%1211݄20Ր༵
VMؒͷภΓʢ ϨΠςϯγʣ025.0050.0075.00100.001 2 4 8 16 32 64vhost-net SR-IOVVMύʔηϯςʔδvhost-net < 10%SR-IOV < 30%1211݄20Ր༵
VMؒͷภΓʢ CPUෛՙʣ025.0050.0075.00100.001 2 4 8 16 32 64vhost-net SR-IOVVMύʔηϯςʔδvhost-net < 10%SR-IOV < 20%1211݄20Ր༵
ൺֱ݁Ռ• ཧίΞʹ͍ۙVMͩͱੑೳ͕Ұ൪ྑ͍Α͏ʹݟ͑Δ• ཧίΞͷ10ഒͷVMΛ࣮ߦͯ͠ɺ͜ͷෛՙύλʔϯͰۃͳϨΠςϯγͷѱԽෛՙͷภΓͳͲݟΒΕͳ͔ͬͨ• ཧڥͱൺ͔ͯͳΓੑೳ͕ѱ͍→ෛՙΛෳίΞࢄग़དྷ͍ͯͳ͍ࣄ͕ݪҼͷީิͱͯ͠ߟ͑ΒΕΔ1211݄20Ր༵
Ծڥ্ͰෳίΞωοτϫʔΫIOෛՙࢄͤͨ࣌͞ͷੑೳ1211݄20Ր༵
Ծڥ্ͰίΞࢄʁ• ԾCPUΛ1-16·Ͱ૿ͯ͠ΈΔ• ίΞΛ૿͚ͨͩ͠ͰωοτϫʔΫελοΫ͕ฒྻͰΒͳ͍• ✕ RSSɿԾNICɾVF͕ະରԠ• ○ RPSɿήετଆLinuxͷઃఆͰ༗ޮԽͯ͠ΈΔ1211݄20Ր༵
RPS$ echo "f" > /sys/class/net/eth1/queues/rx-0/rps_cpus$ echo 4096 > /sys/class/net/eth1/queues/rx-0/rps_flow_cnt$ echo 32768 > /proc/sys/net/core/rps_sock_flow_entries1211݄20Ր༵
vCPU161211݄20Ր༵
ඵؒτϥϯβΫγϣϯʢvhost-netʣ01375.002750.004125.005500.001 2 4 8 16 32 64cpu1 cpu2 cpu4cpu8 cpu16 baremetalVMtransaction/secVM =< 4ͳΒվળ1211݄20Ր༵
ඵؒτϥϯβΫγϣϯʢvhost-net, RPSʣ01375.002750.004125.005500.001 2 4 8 16 32 64cpu1 cpu2 cpu4cpu8 cpu16 baremetalVMtransaction/secVM =< 4ͳΒRPSͰੑೳվળ1211݄20Ր༵
ඵؒτϥϯβΫγϣϯʢSR-IOVʣ01375.002750.004125.005500.001 2 4 8 16 32cpu1 cpu2 cpu4cpu8 cpu16 baremetalVMtransaction/secVM =< 4ͳΒվળ1211݄20Ր༵
ඵؒτϥϯβΫγϣϯʢSR-IOV, RPSʣ01375.002750.004125.005500.001 2 4 8 16 32cpu1 cpu2 cpu4cpu8 cpu16 baremetalVMtransaction/secVM =< 4ͳΒRPSͰੑೳվળ1211݄20Ր༵
ϨΠςϯγʢvhost-netʣ0750.001500.002250.003000.001 2 4 8 16 32 64cpu1 cpu2 cpu4cpu8 cpu16 baremetalVMϨΠςϯγtpsͱ͍͍ͩͨಉ͡1211݄20Ր༵
ϨΠςϯγʢvhost-net, RPSʣ0750.001500.002250.003000.001 2 4 8 16 32 64cpu1 cpu2 cpu4cpu8 cpu16 baremetalVMϨΠςϯγtpsͱ͍͍ͩͨಉ͡1211݄20Ր༵
ϨΠςϯγʢSR-IOVʣ0750.001500.002250.003000.001 2 4 8 16 32cpu1 cpu2 cpu4cpu8 cpu16 baremetalVMϨΠςϯγtpsͱ͍͍ͩͨಉ͡1211݄20Ր༵
ϨΠςϯγʢSR-IOV, RPSʣ0750.001500.002250.003000.001 2 4 8 16 32cpu1 cpu2 cpu4cpu8 cpu16 baremetalVMϨΠςϯγtpsͱ͍͍ͩͨಉ͡1211݄20Ր༵
CPUෛՙʢvhost-netʣ025.0050.0075.00100.001 2 4 8 16 32 64cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδVM =< 4ͳΒɺcpu = 1ͷ࣌ΑΓෛՙΛࢄͰ͖͍ͯΔʁ1211݄20Ր༵
CPUෛՙʢvhost-net, RPSʣ025.0050.0075.00100.001 2 4 8 16 32 64cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδRPSͰΑΓෛՙΛࢄ1211݄20Ր༵
CPUෛՙʢSR-IOVʣ025.0050.0075.00100.001 2 4 8 16 32cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδ1211݄20Ր༵
CPUෛՙʢSR-IOV, RPSʣ025.0050.0075.00100.001 2 4 8 16 32cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδ1211݄20Ր༵
VMؒͷภΓʢ ඵؒτϥϯβΫγϣϯʗvhost-netʣ025.0050.0075.00100.001 2 4 8 16 32 64cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδcpu >= 8, VM >=16ͷ࣌ʹ15%Λ͑ΔภΓ1211݄20Ր༵
VMؒͷภΓʢ ඵؒτϥϯβΫγϣϯ/vhost-net, RPSʣ025.0050.0075.00100.001 2 4 8 16 32 64cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδcpu >= 8, VM >=16ͷ࣌ʹ15%Λ͑ΔภΓ1211݄20Ր༵
VMؒͷภΓʢ ඵؒτϥϯβΫγϣϯʗSR-IOVʣ025.0050.0075.00100.001 2 4 8 16 32cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδ૯εϨου͕ཧCPUΛ͑ͨลΓͰେ͖ͳภΓ1211݄20Ր༵
VMؒͷภΓʢ ඵؒτϥϯβΫγϣϯʗSR-IOV, RPSʣ025.0050.0075.00100.001 2 4 8 16 32cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδ૯εϨου͕ཧCPUΛ͑ͨลΓͰେ͖ͳภΓ1211݄20Ր༵
VMؒͷภΓʢ ϨΠςϯγʗvhost-netʣ025.0050.0075.00100.001 2 4 8 16 32 64cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδtpsͱ͍͍ͩͨಉ͡1211݄20Ր༵
VMؒͷภΓʢ ϨΠςϯγʗvhost-net, RPSʣ025.0050.0075.00100.001 2 4 8 16 32 64cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδ1211݄20Ր༵
VMؒͷภΓʢ ϨΠςϯγʗSR-IOVʣ025.0050.0075.00100.001 2 4 8 16 32cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδ૯εϨου͕ཧCPUΛ͑ͨลΓͰେ͖ͳภΓ1211݄20Ր༵
VMؒͷภΓʢ ϨΠςϯγʗSR-IOV, RPSʣ025.0050.0075.00100.001 2 4 8 16 32cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδ૯εϨου͕ཧCPUΛ͑ͨลΓͰେ͖ͳภΓ1211݄20Ր༵
VMؒͷภΓʢ CPUෛՙʗvhost-netʣ025.0050.0075.00100.001 2 4 8 16 32 64cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδ͋·Γେ͖ͳภΓݟΒΕͳ͍1211݄20Ր༵
VMؒͷภΓʢ CPUෛՙʗvhost-net, RPSʣ025.0050.0075.00100.001 2 4 8 16 32 64cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδ͋·Γେ͖ͳภΓݟΒΕͳ͍1211݄20Ր༵
VMؒͷภΓʢ CPUෛՙʗSR-IOVʣ025.0050.0075.00100.001 2 4 8 16 32cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδVMͷ্ঢͰएׯ૿Ճ1211݄20Ր༵
VMؒͷภΓʢ CPUෛՙʗSR-IOV, RPSʣ025.0050.0075.00100.001 2 4 8 16 32cpu1 cpu2 cpu4cpu8 cpu16VMύʔηϯςʔδVMͷ্ঢͰएׯ૿Ճ1211݄20Ր༵
ൺֱ݁Ռ• ԾϚγϯͰ࣮ػੑೳͷҎ্Λग़ͤͨͷVM͕͘͝গͳ͘vCPU͕͔ͳΓଟ͍࣌ͷΈ → ͜ͷ݅Ͱ͔͠RPSͰੑೳ͕ग़ͳ͍• ૯vCPUΛଟͯ͘͠ҙ֎ͱੑೳԼ͕Βͳ͍• SR-IOVͷ߹ʹVF͕ཧίΞΛ͑ͨลΓͰ͔ͳΓੑೳʹภΓ͕ͰΔ• NIC্ͷΩϡʔʹରͯ͠ड৴ॲཧʹׂΓͯΔίΞ͕ෆͯ͜͠ͷΑ͏ͳঢ়گʹͳΔͱਪଌ• vhost-netͰͦͷΑ͏ͳݱ͕ݟΒΕͣɺ૯vCPUΛ૿ͯ͠ൺֱతެฏʹॲཧ͞Ε͍ͯΔΑ͏ʹݟ͑Δ1211݄20Ր༵
ϚϧνϢʔβ࣌ͷެฏੑ1211݄20Ր༵
ϚϧνϢʔβ࣌ͷެฏੑ• σϑΥϧτεέδϡʔϦϯάͰɺશͯͷKVMήετϓϩηε͕ެฏʹCPU࣌ؒΛׂΓͯΒΕΔ→ෳͷϢʔβ͕ෳͷVMΛىಈ͍ͯ͠Δ߹ɺϢʔβลΓͷϦιʔεׂ͕ෆެฏʹͳΔ• ϢʔβؒͰެฏʹεέδϡʔϦϯάͯ͠ཉ͍͠1211݄20Ր༵
cgroup• ϓϩηεάϧʔϓʹରͯ͠ϦιʔεʢCPUɺϝϞϦɺσΟεΫI/OɺωοτϫʔΫI/Oʣͷར༻Λ੍ݶɾִ͢ΔҝͷLinuxͷػೳ1211݄20Ր༵
cgroup tool install$ apt-get install cgroup-bin1211݄20Ր༵
$ ls /sys/fs/cgroupblkioɹϒϩοΫI/Oͷ੍ݶcpuɹCPUϦιʔεͷ੍ݶcpuacctɹCPUͷߦ࣌ؒͳͲͷใऩूcpusetɹCPUͷׂΓ͚ʹؔ͢Δ੍ݶdevicesɹσόΠεͷΞΫηεڐՄɾڋ൱freezerɹάϧʔϓʹଐ͢ΔϓϩηεͷҰ࣌ఀࢭɾ࠶։memoryɹϝϞϦϦιʔεͷ੍ݶnet_clsɹૹग़ύέοτʹtc༻ͷλάΛઃఆnsɹωʔϜεϖʔεʢίϯςφ༻ʣ1211݄20Ր༵
cd /sys/fs/cgroup/cpu$ mkdir grp_a$ echo ‘12254’ > grp_a/tasks$ echo ‘2184’ > grp_a/tasks$ echo ‘512’ > grp_a/cpu.shares$ mkdir grp_b$ echo ‘9012’ > grp_b/tasks$ echo ‘1024’ > grp_b/cpu.sharesάϧʔϓ࡞άϧʔϓॴଐϓϩηεΛՃʢPIDʣάϧʔϓͷλεΫͰ༻ग़དྷΔCPU࣌ؒͷ૬ରత1211݄20Ր༵
libvirtdͱcgroup• /sys/fs/cgroup/cpu/libvirt/qemu/ʹVMؔ࿈ϓϩηε͕ࣗಈొ͞ΕΔ• /sys/fs/cgroup/cpu/libvirt/qemu//vcpuNʹvcpuϓϩηε͕ࣗಈొ͞ΕΔ• libvirtܦ༝ͰcgroupͷϦιʔε੍ݶઃఆ͕ग़དྷΔ1211݄20Ր༵
virshͱcgroup$ virsh schedinfo vm0Scheduler : posixcpu_shares : 1024vcpu_period : 100000vcpu_quota : -1$ virsh schedinfo --set cpu_shares=512 vm01211݄20Ր༵
̎ͭͷάϧʔϓʹެฏʹϦιʔεΛׂΓͯͯΈΔ• /sys/fs/cgroup/grp_a• VMΠϯελϯεɿ̒ݸ• cpu.sharesɿσϑΥϧτʢ̍̌̎̐ʣ• /sys/fs/cgroup/grp_b• VMΠϯελϯεɿ̍̌ݸ• cpu.sharesɿσϑΥϧτʢ̍̌̎̐ʣ1211݄20Ր༵
άϧʔϓͷ࡞Γํ$ virsh start $ mkdir /sys/fs/cgroup/cpu/grp_a$ cat /sys/fs/cgroup/cpu/libvirt/qemu//tasks> tmp$ cat /sys/fs/cgroup/cpu/libvirt/qemu//vcpu*/tasks >> tmp$ cat tmp > /sys/fs/cgroup/cpu/grp_a/tasks1211݄20Ր༵
vhost-net/16VM࣌ͷCPU࣮ߦ࣌ؒʢcgroupະ༻࣌ʣ0625000000125000000018750000002500000000࣮ߦ࣌ؒ1211݄20Ր༵
vhost-net/16VM࣌ͷCPU࣮ߦ࣌ؒʢcgroup༻࣌ʣ0625000000125000000018750000002500000000࣮ߦ࣌ؒgrp_a grp_b1211݄20Ր༵
vhost-net/16VM࣌ͷωοτϫʔΫIOʢcgroupະ༻࣌ʣ01375000275000041250005500000σʔλྔ1211݄20Ր༵
vhost-net/16VM࣌ͷωοτϫʔΫIOʢcgroup༻࣌ʣ01375000275000041250005500000σʔλྔgrp_a grp_b1211݄20Ր༵
άϧʔϓ͝ͱͷ߹ܭൺֱʢcgroup༻࣌ʣ00.501.001.502.00CPU Netgrp_a grp_b1.101.141211݄20Ր༵
ʦࢀߟʧάϧʔϓ͝ͱͷ߹ܭൺֱʢcgroupະ༻࣌ʣ00.501.001.502.00CPU Netgrp_a grp_b1.641.651211݄20Ր༵
ൺֱ݁Ռ• άϧʔϓʹ͚Δ͚ͩͰσϑΥϧτͰάϧʔϓεέδϡʔϦϯά͕ߦΘΕɺ࣮ଌͰάϧʔϓʹରͯ͠ެฏʹCPU͕͞Ε͍ͯΔࣄ͕֬ೝͰ͖Δ• CPUΛߜΔࣄͰ݁ՌతʹωοτϫʔΫIOΛߜΔࣄ͋Δఔग़དྷ͍ͯΔ• Ͱ͋·Γݫີ͡Όͳ͍ʢ࣮ଌ14%͘Β͍ͷޡࠩʣ1211݄20Ր༵
CFS bandwidth control• ͜ͷάϧʔϓcpu.cfs_period_usͷ୯Ґ࣌ؒதʹ࠷େcpu.cfs_quota_us͚࣮ͩޮग़དྷΔ• άϧʔϓͷϓϩηεperiod୯Ґ࣌ؒதɺಉ࣌ʹ࣮ߦ͞Ε͏ΔͷͰquotaperiodΛ͑͏Δʢ2CPUͰΊ͍͍ͬͺ͍ͬͨΒquota = period * 2ʣ1211݄20Ր༵
vhost-net/16VM࣌ͷCPU࣮ߦ࣌ؒʢquota=200000,period=50000ʣ0625000000125000000018750000002500000000࣮ߦ࣌ؒgrp_a grp_b1211݄20Ր༵
vhost-net/16VM࣌ͷωοτϫʔΫIOʢquota=200000,period=50000ʣ01375000275000041250005500000σʔλྔgrp_a grp_b1211݄20Ր༵
άϧʔϓ͝ͱͷ߹ܭൺֱʢquota=200000,period=50000ʣ00.501.001.502.00CPU Netgrp_a grp_b1.010.631211݄20Ր༵
ݕূ݁Ռ• ͔֬ʹάϥϑ্៉ྷʹͳΔͷ͕ͩɺCPU͕ͲΜͳʹۭ͍͍ͯͯquota·ͰߦͬͨΒઈର࣮ߦ͞Εͳ͍• ͔ͱ͍࣮ͬͯߦ͖͠Εͳ͍ఔͷquotaΛ༩͑Δͱઃఆ͠ͳ͍߹ͱมΘΒͳ͘ͳΔ• CPUωοτϫʔΫIOͪΐ͏ͲΑ͘ͽͬͨΓ1.0ͱ͍͏༁ʹ͍͔ͳ͔ͬͨ• period͕͗͢ΔͱVM͕͍࣮ؒߦ͞Εͳͯ͘ྑ͘ͳ͍͜ͱ͕ى͖ͦ͏1211݄20Ր༵
net_cls$ tc class add dev virbr0 parent 10: classid10:1 htb rate 24Mbit$ echo 0x100001 > /sys/fs/cgroup/net_cls/grp_a/net_cls.classidϓϩηε͔Βૹ৴͞ΕΔύέοτʹtcͷλάΛ͚ΔࣄʹΑΓଳҬ੍ݶ͕ग़དྷΔ1211݄20Ր༵
·ͱΊ• KVMʹΑΔԾԽڥͷϦιʔεׂΓͯͷެฏੑʹ͍ͭͯݕূͨ͠• ࠓճͷ࣮ݧͷ݅ԼͰɺ࣮CPUΛେ͖͑͘ΔVMͰۃͳϨΠςϯγͷѱԽෛՙͷภΓͳͲݟΒΕͳ͔ͬͨ• ୠ͠ɺSR-IOVͷ߹ʹVF͕ཧίΞΛ͑ͨลΓͰ͔ͳΓੑೳʹภΓ͕ൃੜͨ͠• cgroupΛ༻͍ͯϚϧνϢʔβ࣌ʹϢʔβؒͷCPUׂΓͯΛެฏʹग़དྷΔࣄΛ֬ೝͨ͠1211݄20Ր༵
syuu1228
iwamot
bnasslahsen
shimap_sampo
yumayabe
ryohatanmonolog
whywaita
bitkey
kawaguti
victorrentea
hosimesi11
iselegant
soracom
carmenintech
zenorocha
kastner
sferik
csswizardry
smashingmag
phodgson
kneath
zakiyama
erikaheidi
eileencodes
3n