Cgroupによるリソース隔離入門 (2015-01-17)

Docker Meetup Tokyo #4 の発表資料です。
参考となる情報にはPDF中からリンクをしていますが、資料中のリンクは Speaker Deck 上ではクリックできないので PDF をダウンロードしてご覧ください。

Transcript

1. CgroupʹΑΔϦιʔεִ཭ೖ໳
   Docker Meetup Tokyo #4
   Ճ౻ହจ
   2015-01-17
   Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 1 / 29
   

   View Slide

2. ୭?
   Ճ౻ହจ
   ϑΝʔεταʔόגࣜձࣾɹج൫։ൃ෦
   http://www.ten-forward.ws/
   @ten forward
   http://gplus.to/tenforward
   https://github.com/tenforward
   http://d.hatena.ne.jp/defiant/
   Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 2 / 29
   

   View Slide

3. ୭?
   2010 ೥ࠒʹ Cgroup ʹڵຯΛ࣋ͬͯௐࠪΛ࢝Ίͷ͕͖͔͚ͬ
   ͰίϯςφपลΛ৭ʑ͓͔͚͍ͬͯ·͢
   Docker ৄ͋͘͠Γ·ͤΜ
   Docker Meetup Tokyo Ͱ͓࿩͢Δͷ͸ 2 ౓໨
   LXC ΁ͷίϛοτ
   ೔ຊޠ man pages
   ࠷ۙ͸গ͠ίʔυ΋
   linuxcontainers.org ຋༁
   Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 3 / 29
   

   View Slide

4. ୭?
   Plamo Linux ϝϯςφ
   LXC ͰֶͿίϯςφೖ໳ɹʔܰྔԾ૝Խ؀ڥΛ࣮ݱ͢Δٕज़
   gihyo.jp Ͱ࿈ࡌ
   ৽य़ಛผاըɹ 2015 ೥ͷ Linux ͷίϯςφٕज़ (gihyo.jp)
   ʲվగ৽൛ʳLinux ΤϯδχΞཆ੒ಡຊ (ٕज़ධ࿦ࣾ)
   Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 4 / 29
   

   View Slide

5. ษڧձ
   ίϯςφܕԾ૝Խͷ৘ใަ׵ձ
   ৭ʑͳίϯςφٕज़ΛऔΓѻ͏
   ࣮૷͔ΒԠ༻·Ͱίϯςφ͕ؔ܎͍ͯ͠Ε͹ԿͰ΋ѻ͏
   ౦ژͱେࡕͰަޓʹ։࠵
   http://ct-study.connpass.com/
   Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 5 / 29
   

   View Slide

6. ࠓ೔ͷ໨ඪ
   Cgroup Λ࢖ͬͯ Docker ίϯςφͷϦιʔεִ཭ɾ੍ݶΛߦ͏ํ
   ๏Λઆ໌͢Δ
   Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 6 / 29
   

   View Slide

7. ࠓ೔ͷ಺༰
   Linux Χʔωϧͱίϯςφ
   Cgroup ͱ͸
   ੍ݶΛઃఆ͢Δ (1) ʙ Docker ͔Β
   ੍ݶΛઃఆ͢Δ (2) ʙ cgroupfs ௚઀
   ੍ݶΛઃఆ͢Δ (3) ʙ systemd
   Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 7 / 29
   

   View Slide

8. LinuxΧʔωϧͱίϯςφ
   ʰίϯςφʱΛ࣮ݱ͢ΔͨΊͷ Linux Χʔωϧͷओཁͳػೳ
   Namespace (໊લۭؒ)
   OS/ΧʔωϧϦιʔεΛִ཭
   Ϛ΢ϯτɺϗετ໊ɺPIDɺIPC ΦϒδΣΫτɺUID/GIDɺωοτϫʔΫ
   Cgroup
   ίϯϐϡʔλ͕࣋ͭ෺ཧϦιʔεͷ੍ݶ
   CPUɺϝϞϦɺσόΠεɺωοτϫʔΫɺetc
   Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 8 / 29
   

   View Slide

9. ࠓ೔ͷ಺༰
   Linux Χʔωϧͱίϯςφ
   Cgroup ͱ͸
   ੍ݶΛઃఆ͢Δ (1) ʙ Docker ͔Β
   ੍ݶΛઃఆ͢Δ (2) ʙ cgroupfs ௚઀
   ੍ݶΛઃఆ͢Δ (3) ʙ systemd
   Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 9 / 29
   

   View Slide

10. Cgroupͱ͸
    ϓϩηε (λεΫ) ΛάϧʔϓԽ (= Cgroup)
    άϧʔϓ (Cgroup) ಺ͷϓϩηε (λεΫ) ʹରͯ͠·ͱΊͯ
    Ϧιʔε੍ݶ
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 10 / 29
    

    View Slide

11. Cgroupͱ͸ ʙ cgroupfs(1)
    Cgroup ͸ cgroupfs ͱ͍͏ಛघͳϑΝΠϧγεςϜΛϚ΢ϯτ͠
    ͯར༻
    cgroupfs ͷϚ΢ϯτྫ
    
    # mount -t cgroup cgroup /sys/fs/cgroup
    
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 11 / 29
    

    View Slide

12. Cgroupͱ͸ ʙ cgroupfs(2)
    άϧʔϓ (Cgroup) Λ࡞੒͢Δʹ͸σΟϨΫτϦΛ࡞੒
    άϧʔϓͷ࡞੒
    
    # mkdir /sys/fs/cgroup/group01
    
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 12 / 29
    

    View Slide

13. Cgroupͱ͸ ʙ cgroupfs(3)
    Ϧιʔε੍ݶΛઃఆ͢Δʹ͸άϧʔϓ (σΟϨΫτϦ) ҎԼͷϑΝ
    Πϧʹ஋Λॻ͖ࠐΉ
    Ϧιʔε੍ݶͷྫ
    
    # echo $$ > /sys/fs/cgroup/group01/tasks
    (ϓϩηεΛάϧʔϓʹొ࿥)
    # echo 100M > /sys/fs/cgroup/group01/memory.limit_in_bytes
    (ϝϞϦͷ࢖༻Λ 100MB ʹ੍ݶ)
    
    ϓϩηεՔಇத΋ಈతʹมߋՄೳ
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 13 / 29
    

    View Slide

14. Cgroupͱ͸ ʙ αϒγεςϜ
    ੍ݶ͢ΔϦιʔε͝ͱʹʮαϒγεςϜʯ·ͨ͸ʮίϯτϩʔϥʯ
    ͱݺ͹ΕΔෳ਺ͷػೳ͕ଘࡏ
    CPU ؔ࿈ (cpu, cpuacct, cpuset)
    device
    freezer
    ϝϞϦؔ࿈ (memory, hugetlb)
    ωοτϫʔΫ (net cls, net prio)
    blkio
    perf event
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 14 / 29
    

    View Slide

15. ࠓ೔ͷ಺༰
    Linux Χʔωϧͱίϯςφ
    Cgroup ͱ͸
    ੍ݶΛઃఆ͢Δ (1) ʙ Docker ͔Β
    ੍ݶΛઃఆ͢Δ (2) ʙ cgroupfs ௚઀
    ੍ݶΛઃఆ͢Δ (3) ʙ systemd
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 15 / 29
    

    View Slide

16. ੍ݶΛઃఆ͢Δ(1)ʙDocker͔Β
    cpu, cpuset, memory αϒγεςϜͷ੍ݶΛ docker ͔Βࢦఆ
    Մೳ
    ੍ݶΛࢦఆͯ͠ docker run
    
    # docker run -t -i \
    --cpu-shares=100 \
    --cpuset="0-1" \
    --memory="512m" \
    ubuntu /bin/bash
    
    ͨͩ͠ CPU ͸ ༏ઌ౓ ͷࢦఆͰ૬ରࢦఆ
    memory ͸ memory.limit in bytes ͱ memory.soft limit in bytes ʹಉ͡஋͕ઃఆ͞
    ΕΔɻswap ΛؚΊ੍ͨݶ͕ΧʔωϧͰ༗ޮʹͳ͍ͬͯΔ৔߹͸
    memory.memsw.usage in bytes ʹ΋஋͕ઃఆ͞ΕΔ (σϑΥϧτͰઃఆ஋ͷ 2 ഒ)
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 16 / 29
    

    View Slide

17. ੍ݶΛઃఆ͢Δ(1)ʙDocker͔Β
    σϞ (https://asciinema.org/a/14923)
    cpu-shares Λࢦఆͯ͠ 2 ͭͷίϯςφΛىಈ
    
    $ docker run -d --cpu-shares=1000 ubuntu sh -c "while :; do true; done"
    475eff102bf1981d19567dc64b2c922f4f2adade164656a243de8c8b72330733
    $ docker run -d --cpu-shares=100 ubuntu sh -c "while :; do true; done"
    744a5282eac4c5faa799bce66e461dfe758e2ea64009e8780bc14a02163e5d05
    
    CPU ͷ࢖༻ঢ়گ
    
    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    2597 root 20 0 4444 396 316 R 90.7 0.0 0:09.71 sh
    2633 root 20 0 4444 400 316 R 9.0 0.0 0:00.38 sh
    
    େମࢦఆ௨Γͷൺ཰ (10:1) Ͱ CPU Λ࢖༻
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 17 / 29
    

    View Slide

18. ੍ݶΛઃఆ͢Δ(1)ʙDocker͔Β
    ઃఆͰ͖Δର৅͕ݶΒΕΔ
    cpu (૬ରࢦఆ)
    cpuset (CPU ͷࢦఆ)
    memory
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 18 / 29
    

    View Slide

19. ࠓ೔ͷ಺༰
    Linux Χʔωϧͱίϯςφ
    Cgroup ͱ͸
    ੍ݶΛઃఆ͢Δ (1) ʙ Docker ͔Β
    ੍ݶΛઃఆ͢Δ (2) ʙ cgroupfs ௚઀
    ੍ݶΛઃఆ͢Δ (3) ʙ systemd
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 19 / 29
    

    View Slide

20. ੍ݶΛઃఆ͢Δ(2)ʙcgroupfs௚઀
    Cgroup Λ࢖ͬͨϦιʔε੍ݶ͸ϑΝΠϧʹ੍ݶ಺༰Λॻ͖ࠐ
    Ή͜ͱͰߦ͏
    ˠ Docker ίϯςφͷ cgroup Λ୳ͯ͠௚઀ॻ͖ࠐΊ͹ྑ͍!!
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 20 / 29
    

    View Slide

21. ੍ݶΛઃఆ͢Δ(2)ʙcgroupfs௚઀
    Docker ίϯςφͷ cgroup ͷ৔ॴͷྫ
    Ubuntu
    /sys/fs/cgroup/(αϒγεςϜ໊)/docker/(ίϯςφ ID)
    CentOS7
    /sys/fs/cgroup/(αϒγεςϜ
    ໊)/system.slice/(systemd Ϣχοτ໊)/
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 21 / 29
    

    View Slide

22. ੍ݶΛઃఆ͢Δ(2)ʙcgroupfs௚઀
    σϞ (https://asciinema.org/a/15287)
    2 ͭͷίϯςφʹରͯ͠ cpu ΛͦΕͧΕ 10 ˋɺ5 ˋׂΓ౰ͯΔ
    
    $ CT1=$(docker run -d ubuntu sh -c "while :; do true; done")
    $ CT2=$(docker run -d ubuntu sh -c "while :; do true; done")
    $ cat /sys/fs/cgroup/cpu/docker/cpu.cfs_period_us (୯Ґ࣌ؒͷ֬ೝ)
    100000
    $ echo 10000 | sudo tee /sys/fs/cgroup/cpu/docker/"$CT1"/cpu.cfs_quota-us
    (10000/100000 ͚ͩ CPU Λ࢖͏)
    $ echo 5000 | sudo tee /sys/fs/cgroup/cpu/docker/"$CT2"/cpu.cfs_quota-us
    (5000/100000 ͚ͩ CPU Λ࢖͏)
    
    CPU ͷ࢖༻ঢ়گ
    
    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    2230 root 20 0 4444 396 316 R 10.0 0.0 0:27.54 sh
    2192 root 20 0 4444 400 316 R 5.0 0.0 0:40.97 sh
    
    ઃఆ௨Γ CPU Λ࢖͍ͬͯΔ
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 22 / 29
    

    View Slide

23. ੍ݶΛઃఆ͢Δ(2)ʙcgroupfs௚઀
    cgroup ͷ஌͕ࣝඞཁͱ͔ɺͪΐͬͱ΍΍͍͜͠
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 23 / 29
    

    View Slide

24. ࠓ೔ͷ಺༰
    Linux Χʔωϧͱίϯςφ
    Cgroup ͱ͸
    ੍ݶΛઃఆ͢Δ (1) ʙ Docker ͔Β
    ੍ݶΛઃఆ͢Δ (2) ʙ cgroupfs ௚઀
    ੍ݶΛઃఆ͢Δ (3) ʙ systemd
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 24 / 29
    

    View Slide

25. ੍ݶΛઃఆ͢Δ(3)ʙsystemd
    systemd ഑ԼͰ docker ͕ಈ͍͍ͯΔ৔߹ɺcgroup ͷ؅ཧ͸
    systemd ܦ༝ɻ
    ϢχοτϑΝΠϧʹ cgroup ͷઃఆΛॻ͍ͯىಈ
    systemctl ίϚϯυ͔Βಈతʹࢦఆ
    systemctl ίϚϯυͰಈతʹઃఆ
    
    # systemctl set-property (ίϯςφͷϢχοτ໊) CPUShares=512 --runtime
    
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 25 / 29
    

    View Slide

26. ੍ݶΛઃఆ͢Δ(3)ʙsystemd
    systemd ܦ༝ͰઃఆͰ͖Δ cgroup ύϥϝʔλ͕·ͩ·ͩগ
    ͳ͍ (docker run ͰࢦఆͰ͖Δͷͱ΄΅ಉ౳)
    (ཧ༝) Χʔωϧͷ cgroup ͷ࣮૷͕ࠓΨϯΨϯมΘ͍ͬͯΔͨ
    Ίམͪண͘·Ͱ࣮૷Ͱ͖ͳ͍
    কདྷతʹ͸ cgroup ͷ؅ཧ͸શͯ systemd ܦ༝Ͱߦ͏͜ͱʹ
    ͳΔ (͸ͣ)
    docker run Ͱ cgroup ͷύϥϝʔλΛࢦఆͯ͠΋ཪͰ͸
    systemd ܦ༝Ͱઃఆ͞ΕΔ
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 26 / 29
    

    View Slide

27. ·ͱΊ
    Linux Χʔωϧͱίϯςφ
    Cgroup ͱ͸
    ੍ݶΛઃఆ͢Δ (1) ʙ Docker ͔Β
    ੍ݶΛઃఆ͢Δ (2) ʙ cgroupfs ௚઀
    ੍ݶΛઃఆ͢Δ (3) ʙ systemd
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 27 / 29
    

    View Slide

28. ·ͱΊ
    cgroup Λ࢖ͬͨ Docker ίϯςφͷϦιʔεִ཭ɾ੍ݶͷ
    ํ๏
    docker Ͱࢦఆ
    cgroupfs ௚઀
    systemd
    cgroupfs Λ௚઀৮Ε͹ Docker ΍ systemd ͰࢦఆͰ͖ͳ͍ύ
    ϥϝʔλ΋ࢦఆͰ͖Δ
    Cgroup ͷ಺෦తͳ࣮૷͕·ͩ·ͩվྑɺมԽ͍ͯ͠ΔͷͰ͓
    ؾܰʹઃఆͰ͖ΔΑ͏ʹͳΔͷ͸΋͏গ͠ઌ͔΋
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 28 / 29
    

    View Slide

29. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
    Ճ౻ହจ Docker Meetup Tokyo #4 2015-01-17 29 / 29
    

    View Slide