CgroupʹΑΔϦιʔεִ཭ೖ໳ Docker Meetup Tokyo #4 Ճ౻ହจ 2015-01-17 Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 1 / 29

୭? Ճ౻ହจ ϑΝʔεταʔόגࣜձࣾɹج൫։ൃ෦ http://www.ten-forward.ws/ @ten forward http://gplus.to/tenforward https://github.com/tenforward http://d.hatena.ne.jp/defiant/ Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 2 / 29

୭? 2010 ೥ࠒʹ Cgroup ʹڵຯΛ࣋ͬͯௐࠪΛ࢝Ίͷ͕͖͔͚ͬ ͰίϯςφपลΛ৭ʑ͓͔͚͍ͬͯ·͢ Docker ৄ͋͘͠Γ·ͤΜ Docker Meetup Tokyo Ͱ͓࿩͢Δͷ͸ 2 ౓໨ LXC ΁ͷίϛοτ ೔ຊޠ man pages ࠷ۙ͸গ͠ίʔυ΋ linuxcontainers.org ຋༁ Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 3 / 29

୭? Plamo Linux ϝϯςφ LXC ͰֶͿίϯςφೖ໳ɹʔܰྔԾ૝Խ؀ڥΛ࣮ݱ͢Δٕज़ gihyo.jp Ͱ࿈ࡌ ৽य़ಛผاըɹ 2015 ೥ͷ Linux ͷίϯςφٕज़ (gihyo.jp) ʲվగ৽൛ʳLinux ΤϯδχΞཆ੒ಡຊ (ٕज़ධ࿦ࣾ) Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 4 / 29

ษڧձ ίϯςφܕԾ૝Խͷ৘ใަ׵ձ ৭ʑͳίϯςφٕज़ΛऔΓѻ͏ ࣮૷͔ΒԠ༻·Ͱίϯςφ͕ؔ܎͍ͯ͠Ε͹ԿͰ΋ѻ͏ ౦ژͱେࡕͰަޓʹ։࠵ http://ct-study.connpass.com/ Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 5 / 29

ࠓ೔ͷ໨ඪ Cgroup Λ࢖ͬͯ Docker ίϯςφͷϦιʔεִ཭ɾ੍ݶΛߦ͏ํ ๏Λઆ໌͢Δ Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 6 / 29

ࠓ೔ͷ಺༰ Linux Χʔωϧͱίϯςφ Cgroup ͱ͸ ੍ݶΛઃఆ͢Δ (1) ʙ Docker ͔Β ੍ݶΛઃఆ͢Δ (2) ʙ cgroupfs ௚઀ ੍ݶΛઃఆ͢Δ (3) ʙ systemd Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 7 / 29

LinuxΧʔωϧͱίϯςφ ʰίϯςφʱΛ࣮ݱ͢ΔͨΊͷ Linux Χʔωϧͷओཁͳػೳ Namespace (໊લۭؒ) OS/ΧʔωϧϦιʔεΛִ཭ Ϛ΢ϯτɺϗετ໊ɺPIDɺIPC ΦϒδΣΫτɺUID/GIDɺωοτϫʔΫ Cgroup ίϯϐϡʔλ͕࣋ͭ෺ཧϦιʔεͷ੍ݶ CPUɺϝϞϦɺσόΠεɺωοτϫʔΫɺetc Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 8 / 29

ࠓ೔ͷ಺༰ Linux Χʔωϧͱίϯςφ Cgroup ͱ͸ ੍ݶΛઃఆ͢Δ (1) ʙ Docker ͔Β ੍ݶΛઃఆ͢Δ (2) ʙ cgroupfs ௚઀ ੍ݶΛઃఆ͢Δ (3) ʙ systemd Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 9 / 29

Cgroupͱ͸ ϓϩηε (λεΫ) ΛάϧʔϓԽ (= Cgroup) άϧʔϓ (Cgroup) ಺ͷϓϩηε (λεΫ) ʹରͯ͠·ͱΊͯ Ϧιʔε੍ݶ Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 10 / 29

Cgroupͱ͸ ʙ cgroupfs(1) Cgroup ͸ cgroupfs ͱ͍͏ಛघͳϑΝΠϧγεςϜΛϚ΢ϯτ͠ ͯར༻ cgroupfs ͷϚ΢ϯτྫ # mount -t cgroup cgroup /sys/fs/cgroup Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 11 / 29

Cgroupͱ͸ ʙ cgroupfs(2) άϧʔϓ (Cgroup) Λ࡞੒͢Δʹ͸σΟϨΫτϦΛ࡞੒ άϧʔϓͷ࡞੒ # mkdir /sys/fs/cgroup/group01 Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 12 / 29

Cgroupͱ͸ ʙ cgroupfs(3) Ϧιʔε੍ݶΛઃఆ͢Δʹ͸άϧʔϓ (σΟϨΫτϦ) ҎԼͷϑΝ Πϧʹ஋Λॻ͖ࠐΉ Ϧιʔε੍ݶͷྫ # echo $$ > /sys/fs/cgroup/group01/tasks (ϓϩηεΛάϧʔϓʹొ࿥) # echo 100M > /sys/fs/cgroup/group01/memory.limit_in_bytes (ϝϞϦͷ࢖༻Λ 100MB ʹ੍ݶ) ϓϩηεՔಇத΋ಈతʹมߋՄೳ Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 13 / 29

Cgroupͱ͸ ʙ αϒγεςϜ ੍ݶ͢ΔϦιʔε͝ͱʹʮαϒγεςϜʯ·ͨ͸ʮίϯτϩʔϥʯ ͱݺ͹ΕΔෳ਺ͷػೳ͕ଘࡏ CPU ؔ࿈ (cpu, cpuacct, cpuset) device freezer ϝϞϦؔ࿈ (memory, hugetlb) ωοτϫʔΫ (net cls, net prio) blkio perf event Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 14 / 29

ࠓ೔ͷ಺༰ Linux Χʔωϧͱίϯςφ Cgroup ͱ͸ ੍ݶΛઃఆ͢Δ (1) ʙ Docker ͔Β ੍ݶΛઃఆ͢Δ (2) ʙ cgroupfs ௚઀ ੍ݶΛઃఆ͢Δ (3) ʙ systemd Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 15 / 29

੍ݶΛઃఆ͢Δ(1)ʙDocker͔Β cpu, cpuset, memory αϒγεςϜͷ੍ݶΛ docker ͔Βࢦఆ Մೳ ੍ݶΛࢦఆͯ͠ docker run # docker run -t -i \ --cpu-shares=100 \ --cpuset="0-1" \ --memory="512m" \ ubuntu /bin/bash ͨͩ͠ CPU ͸ ༏ઌ౓ ͷࢦఆͰ૬ରࢦఆ memory ͸ memory.limit in bytes ͱ memory.soft limit in bytes ʹಉ͡஋͕ઃఆ͞ ΕΔɻswap ΛؚΊ੍ͨݶ͕ΧʔωϧͰ༗ޮʹͳ͍ͬͯΔ৔߹͸ memory.memsw.usage in bytes ʹ΋஋͕ઃఆ͞ΕΔ (σϑΥϧτͰઃఆ஋ͷ 2 ഒ) Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 16 / 29

੍ݶΛઃఆ͢Δ(1)ʙDocker͔Β σϞ (https://asciinema.org/a/14923) cpu-shares Λࢦఆͯ͠ 2 ͭͷίϯςφΛىಈ $ docker run -d --cpu-shares=1000 ubuntu sh -c "while :; do true; done" 475eff102bf1981d19567dc64b2c922f4f2adade164656a243de8c8b72330733 $ docker run -d --cpu-shares=100 ubuntu sh -c "while :; do true; done" 744a5282eac4c5faa799bce66e461dfe758e2ea64009e8780bc14a02163e5d05 CPU ͷ࢖༻ঢ়گ PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2597 root 20 0 4444 396 316 R 90.7 0.0 0:09.71 sh 2633 root 20 0 4444 400 316 R 9.0 0.0 0:00.38 sh େମࢦఆ௨Γͷൺ཰ (10:1) Ͱ CPU Λ࢖༻ Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 17 / 29

੍ݶΛઃఆ͢Δ(1)ʙDocker͔Β ઃఆͰ͖Δର৅͕ݶΒΕΔ cpu (૬ରࢦఆ) cpuset (CPU ͷࢦఆ) memory Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 18 / 29

ࠓ೔ͷ಺༰ Linux Χʔωϧͱίϯςφ Cgroup ͱ͸ ੍ݶΛઃఆ͢Δ (1) ʙ Docker ͔Β ੍ݶΛઃఆ͢Δ (2) ʙ cgroupfs ௚઀ ੍ݶΛઃఆ͢Δ (3) ʙ systemd Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 19 / 29

੍ݶΛઃఆ͢Δ(2)ʙcgroupfs௚઀ Cgroup Λ࢖ͬͨϦιʔε੍ݶ͸ϑΝΠϧʹ੍ݶ಺༰Λॻ͖ࠐ Ή͜ͱͰߦ͏ ˠ Docker ίϯςφͷ cgroup Λ୳ͯ͠௚઀ॻ͖ࠐΊ͹ྑ͍!! Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 20 / 29

੍ݶΛઃఆ͢Δ(2)ʙcgroupfs௚઀ Docker ίϯςφͷ cgroup ͷ৔ॴͷྫ Ubuntu /sys/fs/cgroup/(αϒγεςϜ໊)/docker/(ίϯςφ ID) CentOS7 /sys/fs/cgroup/(αϒγεςϜ ໊)/system.slice/(systemd Ϣχοτ໊)/ Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 21 / 29

੍ݶΛઃఆ͢Δ(2)ʙcgroupfs௚઀ σϞ (https://asciinema.org/a/15287) 2 ͭͷίϯςφʹରͯ͠ cpu ΛͦΕͧΕ 10 ˋɺ5 ˋׂΓ౰ͯΔ $ CT1=$(docker run -d ubuntu sh -c "while :; do true; done") $ CT2=$(docker run -d ubuntu sh -c "while :; do true; done") $ cat /sys/fs/cgroup/cpu/docker/cpu.cfs_period_us (୯Ґ࣌ؒͷ֬ೝ) 100000 $ echo 10000 | sudo tee /sys/fs/cgroup/cpu/docker/"$CT1"/cpu.cfs_quota-us (10000/100000 ͚ͩ CPU Λ࢖͏) $ echo 5000 | sudo tee /sys/fs/cgroup/cpu/docker/"$CT2"/cpu.cfs_quota-us (5000/100000 ͚ͩ CPU Λ࢖͏) CPU ͷ࢖༻ঢ়گ PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2230 root 20 0 4444 396 316 R 10.0 0.0 0:27.54 sh 2192 root 20 0 4444 400 316 R 5.0 0.0 0:40.97 sh ઃఆ௨Γ CPU Λ࢖͍ͬͯΔ Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 22 / 29

੍ݶΛઃఆ͢Δ(2)ʙcgroupfs௚઀ cgroup ͷ஌͕ࣝඞཁͱ͔ɺͪΐͬͱ΍΍͍͜͠ Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 23 / 29

ࠓ೔ͷ಺༰ Linux Χʔωϧͱίϯςφ Cgroup ͱ͸ ੍ݶΛઃఆ͢Δ (1) ʙ Docker ͔Β ੍ݶΛઃఆ͢Δ (2) ʙ cgroupfs ௚઀ ੍ݶΛઃఆ͢Δ (3) ʙ systemd Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 24 / 29

੍ݶΛઃఆ͢Δ(3)ʙsystemd systemd ഑ԼͰ docker ͕ಈ͍͍ͯΔ৔߹ɺcgroup ͷ؅ཧ͸ systemd ܦ༝ɻ ϢχοτϑΝΠϧʹ cgroup ͷઃఆΛॻ͍ͯىಈ systemctl ίϚϯυ͔Βಈతʹࢦఆ systemctl ίϚϯυͰಈతʹઃఆ # systemctl set-property (ίϯςφͷϢχοτ໊) CPUShares=512 --runtime Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 25 / 29

੍ݶΛઃఆ͢Δ(3)ʙsystemd systemd ܦ༝ͰઃఆͰ͖Δ cgroup ύϥϝʔλ͕·ͩ·ͩগ ͳ͍ (docker run ͰࢦఆͰ͖Δͷͱ΄΅ಉ౳) (ཧ༝) Χʔωϧͷ cgroup ͷ࣮૷͕ࠓΨϯΨϯมΘ͍ͬͯΔͨ Ίམͪண͘·Ͱ࣮૷Ͱ͖ͳ͍ কདྷతʹ͸ cgroup ͷ؅ཧ͸શͯ systemd ܦ༝Ͱߦ͏͜ͱʹ ͳΔ (͸ͣ) docker run Ͱ cgroup ͷύϥϝʔλΛࢦఆͯ͠΋ཪͰ͸ systemd ܦ༝Ͱઃఆ͞ΕΔ Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 26 / 29

·ͱΊ Linux Χʔωϧͱίϯςφ Cgroup ͱ͸ ੍ݶΛઃఆ͢Δ (1) ʙ Docker ͔Β ੍ݶΛઃఆ͢Δ (2) ʙ cgroupfs ௚઀ ੍ݶΛઃఆ͢Δ (3) ʙ systemd Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 27 / 29

·ͱΊ cgroup Λ࢖ͬͨ Docker ίϯςφͷϦιʔεִ཭ɾ੍ݶͷ ํ๏ docker Ͱࢦఆ cgroupfs ௚઀ systemd cgroupfs Λ௚઀৮Ε͹ Docker ΍ systemd ͰࢦఆͰ͖ͳ͍ύ ϥϝʔλ΋ࢦఆͰ͖Δ Cgroup ͷ಺෦తͳ࣮૷͕·ͩ·ͩվྑɺมԽ͍ͯ͠ΔͷͰ͓ ؾܰʹઃఆͰ͖ΔΑ͏ʹͳΔͷ͸΋͏গ͠ઌ͔΋ Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 28 / 29

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 29 / 29