今さら聞けない Linux コンテナの基礎 (2015-06-20)

by tenforward

Slide 1

Slide 1 text

͍·͞Βฉ͚ͳ͍Linuxίϯςφͷجૅ Ճ౻ହจ 2015-06-20 Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 1 / 108

Slide 2

Slide 2 text

ࣗݾ঺հ Ճ౻ହจ http://www.ten-forward.ws/ @ten forward http://gplus.to/tenforward https://github.com/tenforward http://d.hatena.ne.jp/deﬁant/ (ٕज़ϒϩά) Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 2 / 108

Slide 3

Slide 3 text

ࣗݾ঺հ ϑΝʔεταʔόɹج൫։ൃ෦ɹॴଐ ৽ϒϥϯυͷϗεςΟϯάαʔϏε͸͡Ί·ͨ͠ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 3 / 108

Slide 4

Slide 4 text

ࣗݾ঺հ 2010 ೥ࠒʹ cgroup ͷௐࠪΛ࢝Ίͨͷ͕͖͔͚ͬͰίϯςφ पลΛ৭ʑ͓͔͚ͬͯͨͷ͕ߴͯ͜͡ͷษڧձΛ΍ͬͯ·͢ LXC ΁ͷίϛοτ ೔ຊޠ man pages / ࠷ۙ͸গ͠ίʔυ΋ linuxcontainers.org ຋༁ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 4 / 108

Slide 5

Slide 5 text

ࣗݾ঺հ Plamo Linux ϝϯςφ LXC ͰֶͿίϯςφೖ໳ɹʔܰྔԾ૝Խ؀ڥΛ࣮ݱ͢Δٕज़ gihyo.jp Ͱ࿈ࡌ ʲվగ৽൛ʳLinux ΤϯδχΞཆ੒ಡຊ (ٕज़ධ࿦ࣾ) Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 5 / 108

Slide 6

Slide 6 text

ࠓ೔ͷલ൒ͷ໨ඪ Linux Χʔωϧͷ࣋ͭίϯςφؔ࿈ػೳΛҰ௨Γ঺հ͢Δ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 6 / 108

Slide 7

Slide 7 text

ຊ೔ͷલ൒ͷ಺༰ ίϯςφ֓ཁ Linux ʹ͓͚Δίϯςφͷ࢓૊Έ Namespace Cgroup ωοτϫʔΫػೳ ϧʔτσΟϨΫτϦͷมߋ bind mount CRIU overlayfs Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 7 / 108

Slide 8

Slide 8 text

ίϯςφ֓ཁ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 8 / 108

Slide 9

Slide 9 text

ίϯςφͱ͸ Χʔωϧ͔ΒݟΔͱී௨ʹϓϩηε͕ىಈ͢Δ͚ͩ ىಈ͢Δࡍʹִ཭Λࢦࣔ͢Δ ΧʔωϧͷػೳͰ (ෳ਺ͷ) ಠཱۭͨؒ͠Λ࡞Γग़͠ɼϦιʔ εΛ෼ׂɾ෼഑͢Δ ϓϩηεΛάϧʔϓԽͯ͠ଞͷάϧʔϓͱϦιʔεۭؒΛִ཭ άϧʔϓԽͨ͠ϓϩηεʹର͢ΔϦιʔε੍ݶ OS ϨϕϧͷԾ૝Խ Ծ૝Խͱ͍͏ΑΓ͸ִ཭Խ Ծ૝తͳ OS ؀ڥΛఏڙ͢Δ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 9 / 108

Slide 10

Slide 10 text

ίϯςφͷϝϦοτ ߴີ౓Խ͕Մೳ ىಈ͍ͯ͠Δ OS (Χʔωϧ) ͸Ұͭ Φʔόʔϔου͕খ͍͞ ϋʔυ΢ΣΞͷԾ૝Խ͕ෆཁ ىಈ͕ૣ͍ Ծ૝ϚγϯͷىಈͰ͸ͳ͘ɼϗετ OS ͔ΒݟͨΒ୯ʹϓϩ ηε͕ىಈ͍ͯ͠Δ͚ͩͳͷͰɼී௨ͷϓϩάϥϜ͕ىಈ͢Δ ͷͱ΄ͱΜͲมΘΒͳ͍ ඞͣ͠΋γεςϜΛಈ͔͢ඞཁ͸ͳ͍ (ΞϓϦέʔγϣϯί ϯςφ) ྫ͑͹ίϯςφ಺Ͱ͸ httpd ͷΈ͕ಈ͍͍ͯΔ ίϯςφʹϝϞϦΛݻఆతʹׂΓ౰ͯΔඞཁ͕ͳ͍ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 10 / 108

Slide 11

Slide 11 text

ίϯςφͷσϝϦοτ ҟͳΔ OS ͷγεςϜ / ϓϩάϥϜ͸ಈ͔ͤͳ͍ ୯ʹϗετ OS ্Ͱϓϩηε͕ىಈ͢Δ͚ͩͳͷͰ౰ͨΓલ ΧʔωϧʹؔΘΔૢ࡞͸Ͱ͖ͳ͍ ىಈ͍ͯ͠ΔΧʔωϧ͸มΘΒͳ͍ͷͰ ίϯςφຖʹϩʔυ͢ΔϞδϡʔϧΛม͑ΔͳͲ Χʔωϧͷ࣮૷͸ෳࡶʹͳΔ શͯΧʔωϧͷػೳͱ࣮ͯ͠૷͞Ε͍ͯΔͷͰ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 11 / 108

Slide 12

Slide 12 text

ຊ೔ͷલ൒ͷ಺༰ ίϯςφ֓ཁ Linux ʹ͓͚Δίϯςφͷ࢓૊Έ Namespace Cgroup ωοτϫʔΫػೳ ϧʔτσΟϨΫτϦͷมߋ bind mount CRIU overlayfs Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 12 / 108

Slide 13

Slide 13 text

Linuxʹ͓͚Δίϯςφͷ࢓ ૊Έ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 13 / 108

Slide 14

Slide 14 text

Linux ͰίϯςφΛ࣮ݱ͢ΔͨΊͷػೳ Linux Χʔωϧʹؚ·ΕΔ৭ʑͳػೳΛ૊Έ߹Θͤͯίϯςφ؀ ڥΛ࡞੒͢ΔɻͦΕͧΕͷػೳ͸ίϯςφઐ༻ͷػೳͱ͍͏Θ͚ Ͱ͸ͳ͍ɻ ϓϩηεΛάϧʔϓԽͯ͠ଞͷάϧʔϓͱִ཭ OS Ϧιʔεͷִ཭ ˠ Namespace (໊લۭؒ) άϧʔϓԽͨ͠ϓϩηεʹର͢ΔϦιʔε੍ݶ ϗετͷ෺ཧϦιʔεʹର͢Δ੍ݶ ˠ Cgroup (control group) Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 14 / 108

Slide 15

Slide 15 text

LinuxͰίϯςφΛ࣮ݱ͢ΔͨΊͷػೳ ͦͷଞ ωοτϫʔΫ (veth, macvlan ͳͲ) έʔύϏϦςΟ chroot (pivot root) bind mount Checkpoint/Restore (CRIU) ͳͲͳͲ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 15 / 108

Slide 16

Slide 16 text

ຊ೔ͷલ൒ͷ಺༰ ίϯςφ֓ཁ Linux ʹ͓͚Δίϯςφͷ࢓૊Έ Namespace Cgroup ωοτϫʔΫػೳ ϧʔτσΟϨΫτϦͷมߋ bind mount CRIU overlayfs Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 16 / 108

Slide 17

Slide 17 text

Linuxʹ͓͚Δίϯςφͷ࢓૊Έ Namespace Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 17 / 108

Slide 18

Slide 18 text

Namespace ͷछྨ (1) Mount Namespace: 2.4.19 ϓϩηε͔Βݟ͍͑ͯΔϚ΢ϯτͷू߹ɼૢ࡞Λ෼཭͢Δɽ Namespace ಺ͷ mount, umount ͸ଞͷ Namespace ʹ͸Ө ڹ͠ͳ͍ (ࢀߟ) Ϛ΢ϯτ໊લۭؒΛద༻͢Δ (IBM developerWorks) UTS Namespace: 2.6.19 ϗετ໊ͳͲɼuname(2) ͕ฦ͢஋ͷू߹Λ෼཭ɽ setdomainname(2), sethostname(2) Ͱ Namespace ಺ͷ஋ ͷΈมߋͰ͖Δ PID Namespace: 2.6.24 PID ۭؒͷ෼཭ɽ৽͍͠ PID Namespace Ͱ͸ PID 1 ͔Β࢝ ·Δ PID ׂ͕Γ౰ͯΒΕΔɽ਌͔Βࢠͷ PID Namespace ͸ ݟ͑Δ (਌ͷۭؒͷ PID Λ࣋ͭ) ͕ɼࢠ͔Β਌͸ݟ͑ͳ͍ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 18 / 108

Slide 19

Slide 19 text

Namespace ͷछྨ (2) IPC Namespace: 2.6.19 SysV IPC ΦϒδΣΫτɼPOSIX ϝοηʔδΩϡʔͷִ཭ User Namespace: 2.6.23 ˜ 3.8 ಠཱͨ͠ UID/GID ۭؒͱ֎෦ۭؒͷϚοϐϯά (ྫ͑͹ɼִ ཭ۭؒͰ͸ uid/gid 0/0ɼ֎෦Ͱ͸ 1000/1000 ͱ͔Մೳʹ ͳΔ) Network Namespace: 2.6.26 ωοτϫʔΫϦιʔεͷִ཭ɽωοτϫʔΫσόΠεɼΞυϨ εɼϧʔςΟϯάςʔϒϧɼιέοτɼϑΟϧλϦϯά Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 19 / 108

Slide 20

Slide 20 text

Namespace ͷૢ࡞ (γεςϜίʔϧ) clone(2) Ͱ৽͍͠ϓϩηε Λੜ੒ unshare(2) Ͱ৽͍͠ϓϩηεΛੜ੒ͤͣʹ࣮ߦίϯςΩετ Λ੍ޚ͢Δ setns(2) ͰϓϩηεΛطଘ ͷ Namespace ʹؔ࿈෇͚Δ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 20 / 108

Slide 21

Slide 21 text

ຊ೔ͷલ൒ͷ಺༰ ίϯςφ֓ཁ Linux ʹ͓͚Δίϯςφͷ࢓૊Έ Namespace Cgroup ωοτϫʔΫػೳ ϧʔτσΟϨΫτϦͷมߋ bind mount CRIU overlayfs Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 21 / 108

Slide 22

Slide 22 text

Linuxʹ͓͚Δίϯςφͷ࢓૊Έ Cgroup Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 22 / 108

Slide 23

Slide 23 text

Cgroupͱ͸ ϓϩηεΛάϧʔϓԽ͠ɼάϧʔϓʹରͯ͠Ϧιʔε੍ݶΛߦ͏ɽ ผʹίϯςφઐ༻ͷ࢓૊ΈͰ͸ͳ͍ɽ Cgroup ͷಛ௃ ػೳ͝ͱʹαϒγεςϜʹ෼͔ΕΔ cgroupfs ΛϚ΢ϯτͯ͠σΟϨΫτϦͰάϧʔϓΛද͢ ϓϩηεΛάϧʔϓ಺ͷ tasks ϑΝΠϧʹ௥Ճ͢Δͱؔ࿈͢Δ λεΫ͕εϨου୯ҐͰάϧʔϓʹ௥Ճ͞ΕΔ ෳ਺֊૚ߏ଄ɻվ଄ߏ଄͝ͱʹҟͳΔπϦʔΛ࡞੒Ͱ͖Δɻͨ ͩ͠ɺҰͭͷαϒγεςϜ͕ॴଐͰ͖ΔπϦʔ͸Ұͭ πϦʔͷͲͷϨϕϧͷάϧʔϓʹ΋λεΫ͕ॴଐͰ͖Δ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 23 / 108

Slide 24

Slide 24 text

CgroupͷαϒγεςϜ cpu: 2.6.24 CFS(Completely Fair Scheduler) bandwidth controlɽ୯Ґ ࣌ؒ಺ͷάϧʔϓ಺ͷλεΫ͕࣮ߦͰ͖Δ߹ܭ࣌ؒΛ੍ݶ͢Δ (3.2 Ͱ࣮૷) ૬ର഑෼ɽάϧʔϓؒͷ CPU ࣌ؒͷׂ౰ͷׂ߹Λࢦఆ͢Δɽ ྫ͑͹ GroupA=100,GroupB=50 ͱ͢Δͱ A:B=2:1 cpuacct: 2.6.24 άϧʔϓ಺ͷ CPU ϦιʔεͷϨϙʔτ (CPU ࣌ؒ) cpuset: 2.6.24 ׂΓ౰ͯΔ CPU, ϝϞϦϊʔυͷׂ౰ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 24 / 108

Slide 25

Slide 25 text

CgroupͷαϒγεςϜ device: 2.6.26 σόΠε΁ͷΞΫηεڐՄɼ੍ݶͷࢦఆ freezer: 2.6.28 άϧʔϓ಺ͷϓϩηεΛશͯҰ࣌ఀࢭ͢Δ memory: 2.6.29 ϝϞϦϦιʔεͷ੍ݶ (ϢʔβϝϞϦɼΧʔωϧϝϞϦ) blkio (Block IO): I/O weight controller(2.6.33 Ҏ߱) άϧʔϓͷ༏ઌ౓Λࢦఆ ͢Δ I/O throttling(2.6.37 Ҏ߱) άϧʔϓ಺ͷϓϩηεͷσόΠ εʹର͢Δૢ࡞਺ͷ߹ܭͷࢦఆ (ࢀߟ)Linux2.6.37 ͷ৽ػೳ “I/O throttling” Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 25 / 108

Slide 26

Slide 26 text

CgroupͷαϒγεςϜ hugetlb: 3.6 cgroup ͔Βͷ hugetlb ͷ࢖༻ perf event: 2.6.39 άϧʔϓ୯ҐͰ perf πʔϧͰϞχλϦϯά (ύϑΥʔϚϯε ղੳ) net cls: 2.6.29 ύέοτʹࣝผࢠΛ͚ͭɼτϥϑΟοΫίϯτϩʔϧ (tc) ͱ netﬁlter(3.14 Ҏ߱) ͰίϯτϩʔϧՄೳʹ Linux 3.14 Ͱ net cls cgroup ʹ௥Ճ͞Εͨ netﬁlter ରԠ net prio: 3.3 άϧʔϓؒͰͷωοτϫʔΫͷ༏ઌ౓ΛΠϯλʔϑΣʔεຖʹ ࢦఆ͢Δ Linux 3.3 ͷ৽ػೳ Network priority cgroup Linux 3.3 ͷ৽ػೳ Network priority cgroup (2) Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 26 / 108

Slide 27

Slide 27 text

Cgroupͷ࢖͍ํ Cgroup ͸ίϯςφͱؔ܎ͳ͘࢖༻Մೳ # mount -t tmpfs cgroup_root /sys/fs/cgroup # mkdir /sys/fs/cgroup/memory # mount -t cgroup -o memory cgroup /sys/fs/cgroup/memory (ϝϞϦαϒγεςϜͷ Ϛ΢ϯτ) # mkdir /sys/fs/cgroup/memory/test01 ("test01" ͱ͍͏άϧʔϓͷ࡞੒) # echo $$ > /sys/fs/cgroup/memory/test01/tasks (ϓϩηεΛάϧʔϓʹొ࿥) # cat /sys/fs/cgroup/memory/test01/tasks (άϧʔϓ಺ͷϓϩηεͷ֬ೝ) 2824 2837 # echo 30M > /sys/fs/cgroup/memory/test01/memory.limit_in_bytes (άϧʔϓʹରͯ͠ϝϞϦ্ݶ 30M ͱ͍͏੍ݶΛઃఆ) # cat /sys/fs/cgroup/memory/test01/memory.limit_in_bytes (੍ݶ஋ͷ֬ೝ) 31457280 # cat /sys/fs/cgroup/memory/test01/memory.usage_in_bytes (ݱࡏͷ࢖༻ྔͷ֬ೝ) 565248 Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 27 / 108

Slide 28

Slide 28 text

Cgroupͷ໰୊఺ ΧΦε αϒγεςϜ͝ͱʹҧ͏πϦʔΛ࡞ΕΔ ਌ࢠؔ܎ͷάϧʔϓͷ਌ʹ΋ࢠʹ΋ϓϩηεɾεϨουΛ௥Ճ Ͱ͖Δ ਌ɺࢠͷͦΕͧΕͷࢠϓϩηε΍εϨουͷଘࡏ αϒγεςϜ͝ͱʹҧ͏ಈ͖ ϝϞϦ͸ϓϩηε୯ҐͰ؅ཧɺCPU ͸εϨου୯Ґ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 28 / 108

Slide 29

Slide 29 text

Cgroupͷ໰୊఺ αϒγεςϜؒͷڠௐಈ࡞͕Ͱ͖ͳ͍ ผʑͷπϦʔΛߏ੒Ͱ͖ΔͷͰڠௐΛલఏʹ࣮૷Ͱ͖ͳ͍ ϑΝΠϧೖग़ྗ͸ blkio ͱ memory ʹؔ܎͢Δ͕ڠௐͰ͖ ͳ͍ ෳ਺ͷπϦʔ͸ߏ੒Ͱ͖Δ͕ɺͻͱͭͷπϦʔʹ͔͠ॴଐͰ͖ ͳ͍ɻෳ਺ͷπϦʔΛߏ੒ͨ͠৔߹ɺଞͱڠௐಈ࡞ͨ͠ํ͕ྑ ͍αϒγεςϜͰ΋ͻͱͭͷπϦʔʹ͔͠ॴଐͰ͖ͳ͍ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 29 / 108

Slide 30

Slide 30 text

Cgroup࠶ઃܭ Cgroup ͷ࠶ઃܭɾ࠶࣮૷͕ਐߦதɻ3.16 ͔ΒࢼͤΔɻ ୯Ұ֊૚ߏ଄ ϓϩηε୯ҐͰ؅ཧ ϓϩηε͕ॴଐ͠ͳ͍άϧʔϓ͚͕ͩࢠάϧʔϓΛ࣋ͯΔ (ϧʔτΛআ͘) ֊૚͝ͱʹ༗ޮʹͰ͖ΔαϒγεςϜΛࢦఆͰ͖Δɻͨͩ͠ɺ ਌άϧʔϓͰ༗ޮʹͳ͍ͬͯΔαϒγεςϜͷΈ࢖༻Մೳ άϧʔϓʹଐ͢Δϓϩηε͕ͳ͘ͳͬͨ௨஌Λ poll ͱ [id]notify Ͱड͚औΕΔ (release agent ഇࢭ) ৄ͘͠͸ Linux 3.16 ͔ΒࢼͤΔ cgroup ͷ୯Ұ֊૚ߏ଄ (1) Linux 3.16 ͔ΒࢼͤΔ cgroup ͷ୯Ұ֊૚ߏ଄ (2) Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 30 / 108

Slide 31

Slide 31 text

Cgroup࠶ઃܭ sane behavior Φϓγϣϯ (·ͱ΋ͳৼΔ෣͍!!) 3.16 ͔ΒࢼͤΔ (ͨͩ͋͘͠·Ͱ͓ࢼ͠) mount -t cgroup -o __DEVEL__sane_behavior \ cgroup /path/to/cgroup Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 31 / 108

Slide 32

Slide 32 text

σϞ ʙ Cgroup CPU Throttling CPU Share blkio Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 32 / 108

Slide 33

Slide 33 text

ຊ೔ͷલ൒ͷ಺༰ ίϯςφ֓ཁ Linux ʹ͓͚Δίϯςφͷ࢓૊Έ Namespace Cgroup ωοτϫʔΫػೳ ϧʔτσΟϨΫτϦͷมߋ bind mount CRIU overlayfs Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 33 / 108

Slide 34

Slide 34 text

Linuxʹ͓͚Δίϯςφͷ࢓૊Έ ωοτϫʔΫػೳ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 34 / 108

Slide 35

Slide 35 text

ίϯςφͰ࢖͏ωοτϫʔΫػೳ ʙ veth OpenVZ/Virtuozzo ༝དྷͷػೳ ରͱͳΔΠϯλʔϑΣʔεΛੜ੒͠ɼΠϯλʔϑΣʔεؒͰ ௨৴Λߦ͏ (Layer2 ͷτϯωϧ) ରͷยํΛϗετଆͷϒϦοδʹɼยํΛίϯςφʹ઀ଓ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 35 / 108

Slide 36

Slide 36 text

ίϯςφͰ࢖͏ωοτϫʔΫػೳ ʙ macvlan ෺ཧΠϯλʔϑΣʔεʹผͷ MAC ΞυϨε͕෇͍ͨԾ૝త ͳ৽͍͠ΠϯλʔϑΣʔεΛ࡞੒ɽ͜ͷΠϯλʔϑΣʔεΛ ίϯςφʹׂ౰ Ϟʔυͷઃఆ͕ଘࡏ: private, vepa, bridge Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 36 / 108

Slide 37

Slide 37 text

ຊ೔ͷલ൒ͷ಺༰ ίϯςφ֓ཁ Linux ʹ͓͚Δίϯςφͷ࢓૊Έ Namespace Cgroup ωοτϫʔΫػೳ ϧʔτσΟϨΫτϦͷมߋ bind mount CRIU overlayfs Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 37 / 108

Slide 38

Slide 38 text

Linuxʹ͓͚Δίϯςφͷ࢓૊Έ ϧʔτσΟϨΫτϦͷมߋ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 38 / 108

Slide 39

Slide 39 text

ϧʔτσΟϨΫτϦͷมߋ ϗετͱผͷϑΝΠϧγεςϜΛ࢖༻͢ΔͨΊʹϗετͷ / ͱผ ͷ৔ॴʹίϯςφͷ / ΛҠಈͤ͞Δඞཁ͕͋Δ chroot γεςϜίʔϧ 1979 ೥ Version 7 Unix Ҏདྷ ؆қతͳִ཭؀ڥͱͯ͜͠ΕͰे෼ͳ͜ͱ΋ pivot root γεςϜίʔϧ chroot ͸ൈ͚ग़ͤΔ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 39 / 108

Slide 40

Slide 40 text

ݩ૆؆қίϯςφ؀ڥ ʙ chroot ϧʔτσΟϨΫτϦΛҠಈͤ͞Δ $ lsb_release -d Description: Ubuntu 14.04.2 LTS $ sudo debootstrap --variant=minbase --arch=amd64 vivid \ > /root/vivid http://ftp.jaist.ac.jp/pub/Linux/ubuntu/ I: Retrieving Release I: Retrieving Release.gpg : (ུ) $ sudo chroot /root/vivid # grep DESCRIPTION /etc/lsb-release DISTRIB_DESCRIPTION="Ubuntu 15.04" Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 40 / 108

Slide 41

Slide 41 text

ݩ૆؆қίϯςφ؀ڥ ʙ chroot jailing - chroot jail Λߏஙɾӡ༻͢ΔͨΊͷεΫϦϓτΛॻ ͍ͨ (Kazuho’s Weblog) /usr/bin౳ɺ OS༝དྷͷσΟϨΫτϦΛ chroot؀ڥʹ read- only ͰΤΫεϙʔτͭͭ͠ɺࢦఆ͞ΕͨίϚϯυΛɺͦͷ chroot ؀ڥͰಈ͔͢εΫϦϓτ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 41 / 108

Slide 42

Slide 42 text

ຊ೔ͷલ൒ͷ಺༰ ίϯςφ֓ཁ Linux ʹ͓͚Δίϯςφͷ࢓૊Έ Namespace Cgroup ωοτϫʔΫػೳ ϧʔτσΟϨΫτϦͷมߋ bind mount CRIU overlayfs Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 42 / 108

Slide 43

Slide 43 text

Linuxʹ͓͚Δίϯςφͷ࢓૊Έ bind mount Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 43 / 108

Slide 44

Slide 44 text

bind mount σΟϨΫτϦπϦʔͷҰ෦Λผͷ৔ॴʹϚ΢ϯτ͢Δ $ ls /etc/httpd/ extra/ httpd.conf magic original/ php.ini.dist extra.dist/ httpd.conf.dist mime.types php.ini $ sudo mount --bind /etc/httpd /tmp/bind $ ls /tmp/bind extra/ httpd.conf magic original/ php.ini.dist extra.dist/ httpd.conf.dist mime.types php.ini Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 44 / 108

Slide 45

Slide 45 text

ຊ೔ͷલ൒ͷ಺༰ ίϯςφ֓ཁ Linux ʹ͓͚Δίϯςφͷ࢓૊Έ Namespace Cgroup ωοτϫʔΫػೳ ϧʔτσΟϨΫτϦͷมߋ bind mount CRIU overlayfs Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 45 / 108

Slide 46

Slide 46 text

Linuxʹ͓͚Δίϯςφͷ࢓૊Έ CRIU Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 46 / 108

Slide 47

Slide 47 text

CRIU(1) http://criu.org/ OpenVZ ϓϩδΣΫτͷ Checkpoint/Restore ࣮૷ ΞϓϦέʔγϣϯͷ͋Δ࣌఺ͷঢ়ଶΛอଘ͠ɺ࠶։Ͱ͖Δ Χʔωϧ 3.11 Ҏ߱Ͱ࢖༻Մೳ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 47 / 108

Slide 48

Slide 48 text

CRIU(2) ubuntu@criu1:~$ sudo lxc-start -n ct01 ubuntu@criu1:~$ sudo lxc-checkpoint -v -n ct01 -s -D /tmp/checkpoint (νΣο ΫϙΠϯτॲཧޙʹίϯςφఀࢭ) ubuntu@criu1:~$ sudo lxc-ls --fancy NAME STATE IPV4 IPV6 GROUPS AUTOSTART -------------------------------------------- ct01 STOPPED - - - NO ubuntu@criu1:~$ ls /tmp/criu/ cgroup.img fdinfo-17.img inventory.img pages-15.img core-170.img fdinfo-18.img ipcns-msg-9.img pages-16.img core-176.img fdinfo-2.img ipcns-sem-9.img pages-17.img core-1.img fdinfo-3.img ipcns-shm-9.img pages-1.img core-260.img fdinfo-4.img ipcns-var-9.img pages-2.img core-261.img fdinfo-5.img iptables-8.img pages-3.img : (snip) ubuntu@criu1:~$ sudo rsync -avz --devices --rsync-path="sudo rsync" \ /var/lib/lxc/ct01 [email protected]:/var/lib/lxc ubuntu@criu1:~$ sudo rsync -avz --rsync-path="sudo rsync" \ /tmp/criu [email protected]:/tmp Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 48 / 108

Slide 49

Slide 49 text

CRIU(3) ubuntu@criu2:~$ ls /tmp/criu/ cgroup.img fdinfo-3.img ipcns-sem-9.img pages-3.img core-1.img fdinfo-4.img ipcns-shm-9.img pages-4.img core-255.img fdinfo-5.img ipcns-var-9.img pages-5.img core-260.img fdinfo-6.img iptables-8.img pages-6.img : (snip) ubuntu@criu2:~$ sudo lxc-checkpoint -n ct01 -r -D /tmp/checkpoint -v -d ubuntu@criu2:~$ sudo lxc-ls -f NAME STATE IPV4 IPV6 GROUPS AUTOSTART -------------------------------------------------- ct01 RUNNING 10.0.3.200 - - NO Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 49 / 108

Slide 50

Slide 50 text

ຊ೔ͷલ൒ͷ಺༰ ίϯςφ֓ཁ Linux ʹ͓͚Δίϯςφͷ࢓૊Έ Namespace Cgroup ωοτϫʔΫػೳ ϧʔτσΟϨΫτϦͷมߋ bind mount CRIU overlayfs Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 50 / 108

Slide 51

Slide 51 text

Linuxʹ͓͚Δίϯςφͷ࢓૊Έ overlayfs Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 51 / 108

Slide 52

Slide 52 text

overlayfs Union Filesystem (aufs ͱҰॹ) ίϯςφͱ͸௚઀ؔ܎ͳ͍ 3.18 kernel ͰϚʔδ ίϯςφͷΫϩʔϯΛ࡞੒͢Δͱ͖ͷϑΝΠϧγεςϜͱ͠ ͯ LXC ͔Βར༻Ͱ͖Δ Ubuntu/Plamo ͩͱඇಛݖίϯςφͷΫϩʔϯʹ΋࢖͑Δ Docker Ͱ΋ར༻Ͱ͖Δ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 52 / 108

Slide 53

Slide 53 text

overlayfs࣮ߦྫ # mkdir lower upper overlay work # ls -F lower/ overlay/ upper/ work/ # touch lower/lower # touch upper/upper # mount -n -t overlay \ > -o lowerdir=lower,upperdir=upper,workdir=work \ > overlay overlay # ls overlay/ lower upper # touch overlay/test # ls overlay/ lower test upper # ls upper/ test upper Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 53 / 108

Slide 54

Slide 54 text

overlayfs஫ҙ ৭ʑͳόʔδϣϯͷ overlayfs ͕͋ΔͷͰ஫ҙ Ϟδϡʔϧ໊ workdir Φϓγϣϯ σΟετϦϏϡʔ γϣϯ kernel ΧʔωϧϚʔδલ (v21 Ҏલ) overlayfs ෆཁ Ubuntu 12.04 LTS, 14.04 LTS, SuSE ํ໘ ʙ3.14 ΧʔωϧϚʔδલ (v22 Ҏ߱) overlayfs ඞཁ Plamo 3.15ʙ3.17 ΧʔωϧϚʔδޙ overlay ඞཁ 3.18, 3.19 ෳ਺ lowerdir ରԠ overlay ඞཁ 4.0ʙ ext4 ্Ͱ͔͠ಈ͔ͳ͍ (whiteout ػೳ) upperdir ͱ workdir ͕ ext4 ͷಉ͡ϑΝΠϧγεςϜ্ʹ͋Δ ඞཁ͕͋Δ lowerdir ͸ ext4 Ͱͳͯ͘΋ಈ͘ (͸ͣ) Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 54 / 108

Slide 55

Slide 55 text

લ൒ͷ·ͱΊ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 55 / 108

Slide 56

Slide 56 text

લ൒ͷ·ͱΊ ίϯςφ֓ཁ Linux ʹ͓͚Δίϯςφͷ࢓૊Έ Namespace Cgroup ωοτϫʔΫػೳ veth, macvlan ϧʔτσΟϨΫτϦͷมߋ chroot, pivot root bind mount CRIU overlayfs ίϯςφ͸৭ʑͳΧʔωϧͷػೳΛ࢖ͬͯߏங͞ΕΔɻͦΕͧΕ ͷػೳ͸ίϯςφઐ༻ͱ͍͏Θ͚Ͱ͸ͳ͍ɻ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 56 / 108

Slide 57

Slide 57 text

ٳܜ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 57 / 108

Slide 58

Slide 58 text

ࠓ೔ͷޙ൒ͷ໨ඪ ৭ʑͳίϯςφ࣮૷Λܰ͘঺հ͢Δ Namespace ͷσϞ LXCɺLXD ͷ঺հͱσϞ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 58 / 108

Slide 59

Slide 59 text

ຊ೔ͷޙ൒ͷ಺༰ Linux ʹ͓͚Δίϯςφ࣮૷ Namespace ͷσϞ LXC LXC σϞ LXD LXD σϞ ·ͱΊ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 59 / 108

Slide 60

Slide 60 text

Linuxʹ͓͚Δίϯςφ࣮૷ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 60 / 108

Slide 61

Slide 61 text

Linuxʹ͓͚Δίϯςφ࣮૷ ΧʔωϧʴΧʔωϧʹର͢Δύονʴૢ࡞ίϚϯυ OpenVZ / Parallels Virtuozzo Containers(঎༻) / libct Χʔωϧʴૢ࡞ίϚϯυ Docker / libcontainer LXC systemd rkt libvirt (lxc υϥΠό) vzctl for upstream kernel garden util-linux (unshare, nsenter, taskset, etc.), iproute2(netns) Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 61 / 108

Slide 62

Slide 62 text

util-linux ͝ଘ஌ɺLinux ͷ৭ʑͳϢʔςΟϦςΟϓϩάϥϜΛूΊͨ ΋ͷɻඞͣೖ͍ͬͯΔɻ Ұ൪͓खܰʹίϯςφͬΆ͍؀ڥΛ࡞ΕΔ ݫີʹ͸ Namespace Λ࡞ΕΔ Version 2.24 Ҏ্͕ྑ͍ (2.25 ͳΒ͞Βʹྑ͍) unshare ίϚϯυɾnsenter ίϚϯυ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 62 / 108

Slide 63

Slide 63 text

util-linux util-linux ʹೖ͍ͬͯΔίϚϯυͰ؆୯ʹ Namespace ΛࢼͤΔɻ (util-linux 2.24 Ҏ߱ลΓ͕Φεεϝ) unshare ਌͔Βಠ໊ཱͨ͠લۭؒΛ࡞੒ͯ͠ίϚϯυΛ࣮ߦ nsenter طʹ࡞੒ࡁΈͷ໊લۭؒʹ઀ଓͯ͠ (໊લۭؒͷதʹೖͬͯ) ίϚϯυΛ࣮ߦ util-linux 2.23 Ҏ߱ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 63 / 108

Slide 64

Slide 64 text

unshareίϚϯυ $ unshare -h ࢖͍ํ: unshare [options] [...] Run a program with some namespaces unshared from the parent. Φϓγϣϯ: -m, --mount Ϛ΢ϯτωʔϜεϖʔεͷڞ༗Λղআ͠·͢ -u, --uts UTS ωʔϜεϖʔε (ϗετ໊ͳͲ) ͷڞ༗Λղআ͠·͢ -i, --ipc System V IPC ωʔϜεϖʔεͷڞ༗Λղআ͠·͢ -n, --net ωοτϫʔΫωʔϜεϖʔεͷڞ༗Λղআ͠·͢ -p, --pid PID ωʔϜεϖʔεͷڞ༗Λղআ͠·͢ -U, --user ϢʔβωʔϜεϖʔεͷڞ༗Λղআ͠·͢ -f, --fork fork ͔ͯ͠Β <ϓϩάϥϜ> Λىಈ͠·͢ --mount-proc[=<σΟϨΫτϦ>] proc ϑΝΠϧγεςϜΛ࠷ॳʹϚ΢ϯτ͠·͢ (͜Εʹ͸ --mount ͷҙຯΛؚΈ·͢) -r, --map-root-user map current user to root (implies --user) -s, --setgroups allow|deny control the setgroups syscall in user namespaces :(ུ) Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 64 / 108

Slide 65

Slide 65 text

ຊ೔ͷޙ൒ͷ಺༰ Linux ʹ͓͚Δίϯςφ࣮૷ Namespace ͷσϞ LXC LXC σϞ LXD LXD σϞ ·ͱΊ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 65 / 108

Slide 66

Slide 66 text

NamespaceͷσϞ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 66 / 108

Slide 67

Slide 67 text

σϞ ʙ unshareίϚϯυͰUTS Namespace ମݧ 1 ϗετ্Ͱ hostname ֬ೝ 2 UTS Namespace ࡞੒ͯ͠ϗετ໊ม͑ͯϗετ໊֬ೝ 3 ϗετ্Ͱ hostname มΘ͍ͬͯͳ͍͜ͱΛ֬ೝ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 67 / 108

Slide 68

Slide 68 text

σϞ ʙ unshareίϚϯυͰMount/PID Namespaceମݧ 1 Mount ͱ PID Namespace Λ࡞੒ $ sudo unshare --mount --pid --mount-proc --fork طʹ࣮ߦதͷϓϩηεͷ PID ͸ม͑ΒΕͳ͍ͷͰ fork ͢Δඞ ཁ͋Γ Namespace ಺Ͱ/proc ΛϚ΢ϯτ 2 Namespace ಺ͰϚ΢ϯτ (mount -o bind /usr /mnt) ͠ɺ Ϛ΢ϯτ͞Εͨ͜ͱΛ֬ೝ 3 ϗετͰϚ΢ϯτ͞Εͯͳ͍͜ͱΛ֬ೝ 4 /proc ҎԼΛ֬ೝ͠ɺݶΒΕͨ PID ͔͠ͳ͍͜ͱΛ֬ೝ (ϗ ετͱൺֱ) Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 68 / 108

Slide 69

Slide 69 text

Mount namespace஫ҙ Ϛ΢ϯτͰ͸Ϛ΢ϯτϓϩύήʔγϣϯΛઃఆ͠ɺ Namespace ؒͰϚ΢ϯτ͕ͲͷΑ͏ʹѻΘΕΔ͔ΛઃఆͰ ͖Δɻ σϑΥϧτ͸ “private” Ͱɺ͋Δ Namespace ͰߦΘΕͨϚ΢ ϯτ͸ผͷ໊લۭؒʹ͸఻ΘΒͳ͍ ͋ΔϚ΢ϯτΛ “shared” ͰϚ΢ϯτ͢Δͱɺࢠʹ΋Ϛ΢ϯτ ͕఻ΘΔ systemd ͸’/’(ϧʔτ) Λ”shared” ͰϚ΢ϯτ͢Δ systemd ͕ init ͷ؀ڥͰ͸ɺmount --make-private /ͱ͠ ͔ͯΒͰͳ͍ͱઌͷσϞ௨Γʹ͸ͳΒͳ͍ Ϛ΢ϯτ໊લۭؒΛద༻͢Δ (IBM) mount-setup: change system mount propagation to shared by default Ϛ΢ϯτ໊લۭؒͰͪΐͬͱϋϚͬͨΊ΋ (П (ɾɾ*) ʎ ɹ Χʔωϧͱ͔࿔ͬͨΓͷϝϞ) Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 69 / 108

Slide 70

Slide 70 text

σϞ ʙ ipίϚϯυͰNetwork Namespaceମ ݧ 1 Network Namespace ࡞੒ 2 ϗετͱΠϯλʔϑΣʔεɺϧʔςΟϯάςʔϒϧɺϑΟϧ λϦϯά͕ҧ͏͜ͱΛ֬ೝ 3 veth ϖΞΛ࡞੒ 4 ϖΞͷҰํΛ Namespace ʹॴଐͤ͞Δ 5 ϖΞʹΞυϨεΛ෇༩͠ɺ૬ޓͰ ping ࣮ߦ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 70 / 108

Slide 71

Slide 71 text

σϞ ʙ unshareίϚϯυͰUser Namespace ମݧ 1 User + UTS Namespace ࡞੒͠ɺNamespace Ͱϗετ໊ ͕ม͑ΒΕͳ͍ͷΛ֬ೝ 2 uid/gid ͷ֬ೝͱ uid/gid ͷϚοϐϯάΛ֬ೝ 3 User + UTS Namespace ࡞੒͠ɺϚοϐϯάΛߦ͍ɺ Namespace Ͱϗετ໊͕ม͑ΒΕΔͷΛ֬ೝ 4 uid/gid ͷ֬ೝͱ uid/gid ͷϚοϐϯάΛ֬ೝ (஫ҙ) σϞͷΑ͏ʹ࣮ߦ͢Δʹ͸ util-linux 2.26 Ҏ͕߱ඞཁ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 71 / 108

Slide 72

Slide 72 text

ຊ೔ͷޙ൒ͷ಺༰ Linux ʹ͓͚Δίϯςφ࣮૷ Namespace ͷσϞ LXC LXC σϞ LXD LXD σϞ ·ͱΊ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 72 / 108

Slide 73

Slide 73 text

mincs @mhiramat ͞ΜʹΑΔγΣϧεΫϦϓτͰ؆୯ʹίϯςφ Λ࡞ΔεΫϦϓτ unshare ΍ ip netns ΍ taskset ΍ bind mount ͳͲΛۦ࢖ͯ͠ ͍Δ ίϯςφ؀ڥ͕ͲͷΑ͏ʹ࡞ΒΕΔ͔͕ྑ͘Θ͔Δ https://github.com/mhiramat/mincs Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 73 / 108

Slide 74

Slide 74 text

OpenVZ / Virtuozzo / libct Χʔωϧʹઐ༻ͷύονΛద༻ͨ͠Χʔωϧͱ֤छૢ࡞ίϚ ϯυ͔ΒͳΔ ঎༻൛ͷ Parallels Virtuozzo ContainersɺOSS ൛ͷ OpenVZ RHEL ͷ֤όʔδϣϯʹର͢Δύον/ઐ༻Χʔωϧͱͯ͠Ϧ Ϧʔε 2001 ೥ (!) ʹ Virtuozzo ϦϦʔε (2005 ೥ʹ͸ Windows ൛ ΋!!) ࢀߟࢿྉ OpenVZ - Linux Containersɿୈ 2 ճ ίϯςφܕԾ૝Խͷ৘ ใަ׵ձˏ౦ژ by ւ࿝ᖒ͞Μ OpenVZ Update 2015/02/18 (೔ຊޠ) by ւ࿝ᖒ͞Μ (ୈ 6 ճษڧձ) Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 74 / 108

Slide 75

Slide 75 text

Docker Έͳ͞Μ͝ଘ஌ :-) “Docker” ͸ʮίϯςφʯͰ͸͋Γ·ͤΜɻ Docker ಺ͷίϯςφ࣮૷͸ libcontainer ίϯςφٕज़Λ࢖ͬͯɺίϯςφͷಛ௕Λ׆͔͠ɺΞϓϦ έʔγϣϯΛσϓϩΠˍ࣮ߦ͢Δ ܰྔ ϙʔλϏϦςΟ Πϝʔδͷࠩ෼؅ཧ Πϝʔδͷ഑෍Πϯϑϥ (Docker Hub) Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 75 / 108

Slide 76

Slide 76 text

Docker ࢀߟࢿྉ Docker ΫΠοΫπΞʔ by தҪ͞Μ Vagrant ϢʔβͷͨΊͷ Docker ೖ໳ by shin1x1 ͞Μ (ୈ 3 ճษڧձ) ͍·͞Βฉ͚ͳ͍ Docker - ୈ̑ճίϯςφܕԾ૝Խͷ৘ใަ ׵ձˏେࡕ by ాத๜༟͞Μ άάΕ͹ͨ͘͞Μ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 76 / 108

Slide 77

Slide 77 text

CoreOS Linux Distribution Docker ίϯςφΛ࢖ͬͨΞϓϦέʔγϣϯͷىಈ ΫϥελϦϯά ෼ࢄγεςϜ ηΩϡϦςΟ ҆શͳ OS Ξοϓσʔτ ࠷খݶͷίΞ ࠷௿ݶͷػೳΛ࣋ͭΧʔωϧ ύοέʔδϚωʔδϟͳ͠ Read only ͳ rootfs Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 77 / 108

Slide 78

Slide 78 text

systemd ͝ଘ஌ɺ͜Ε͔Βͷ Linux ʹ͓͚Δ init(ʹཹ·Βͳ͍૖େͳ γεςϜ) systemd-nspawn ͱ͍͏ίϚϯυ͕͋ΓɺίϯςφΛ࡞੒Մೳ Unit ϑΝΠϧͳͲɺsystemd ͷػೳΛ࢖ͬͯίϯςφΛ؅ཧ Մೳ ີ઀ʹ݁ͼ͍֤ͭͨछػೳʹΑΔαϙʔτ ࢀߟࢿྉ systemd in Containers by Lennart Poettering Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 78 / 108

Slide 79

Slide 79 text

rkt CoreOS ʹΑΔ৽͍͠ίϯςφ؀ڥ Docker ͷ࣋ͭ໰୊Λղܾ systemd Λ࢖༻ ࢀߟจݙ Docker ͷॾ໰୊ͱ Rocket ొ৔ͷܦҢ (SOTA) Appc ͱ CoreOS/Rocket (SOTA) Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 79 / 108

Slide 80

Slide 80 text

ຊ೔ͷޙ൒ͷ಺༰ Linux ʹ͓͚Δίϯςφ࣮૷ Namespace ͷσϞ LXC LXC σϞ LXD LXD σϞ ·ͱΊ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 80 / 108

Slide 81

Slide 81 text

LXC Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 81 / 108

Slide 82

Slide 82 text

LXC Linux ΧʔωϧͷػೳΛ࢖ͬͯίϯςφΛ࣮ݱ͢Δπʔϧ܈ ͱϥΠϒϥϦ ݩʑ͸Χʔωϧʹ࣮૷͞ΕͨػೳΛ։ൃऀ͕؆୯ʹࢼͤΔΑ ͏ʹͱ͍͏ͷΛߟ͑ͯ࡞ΒΕͯͨ໛༷ Ubuntu ։ൃऀΛத৺ʹ։ൃ͞Ε͍ͯΔͷͰ Ubuntu ͷίϯ ςφπʔϧΩοτͷཁૉ͕ڧ͍ ࢼ͢ͳΒ·ͣ͸ Ubuntu Ͱ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 82 / 108

Slide 83

Slide 83 text

LXCͷಛ௃ ๛෋ͳίϯςφςϯϓϨʔτ (֤छσΟετϦϏϡʔγϣ ϯ༻) (ίϯςφςϯϓϨʔτΛ࢖Θͣʹ) ϏϧυࡁΠϝʔδΛμ΢ ϯϩʔυͯ͠࢖༻Մೳ API ͱ֤छݴޠόΠϯσΟϯάͷఏڙ Python(2,3) Lua Go Ruby Haskell Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 83 / 108

Slide 84

Slide 84 text

LXCͷಛ௃ ֤छετϨʔδόοΫΤϯυͷαϙʔτ σΟϨΫτϦ btrfs zfs lvm loop aufs overlayfs nbd(1.1 ) Ϋϩʔϯͱεφοϓγϣοτ ηΩϡϦςΟ ҰൠϢʔβͰͷίϯςφىಈ ωετͨ͠ίϯςφ (ίϯςφ಺Ͱͷίϯςφͷىಈ) ೔ຊޠ man pages Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 84 / 108

Slide 85

Slide 85 text

LXCͷόʔδϣϯ ݱ࣌఺ͷ stable ͸ 1.0ɻݱࡏ 1.0.7ɻUbuntu 14.04 LTS ʹ߹ Θͤͯ 5 ೥ؒαϙʔτɻ ϓϩμΫγϣϯ؀ڥͰ࢖͏ͳΒͪ͜Βɻ ࠷৽͸ 1.1 ܥྻͰɺݱࡏ 1.1.2ɻ(2016 ೥ 1 ݄ or 1.2 ͕ϦϦʔ ε͞ΕΔ·Ͱαϙʔτ) ৽͍͠ػೳΛ࢖͏৔߹ɺsystemd ͱ࢖͏৔߹͸ͪ͜Βɻ ͱΓ͋͑ͣࢼ͢ͳΒ... Ubuntu 14.04 LTS / LXC 1.0.7 Ubuntu 15.04 / LXC 1.1.2 Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 85 / 108

Slide 86

Slide 86 text

cgmanager Ubuntu ؀ڥͰ LXC ͱಉ࣌ʹΠϯετʔϧ͞ΕΔ cgroup ؅ ཧσʔϞϯ systemd ͱͷڞଘ ඇಛݖίϯςφ͔Β cgroup Λૢ࡞Ͱ͖ΔΑ͏ʹ ωετͨ͠ίϯςφ͔Β cgroup Λૢ࡞Ͱ͖ΔΑ͏ʹ ϗετͱผͷ mount namespace Ͱ cgroupfs ΛϚ΢ϯτͯ͠ ىಈ (ϗετ͔Βݟ͑ͳ͍) DBus ܦ༝ͰϦΫΤετΛૹड৴ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 86 / 108

Slide 87

Slide 87 text

lxcfs ඇಛݖίϯςφ༻ʹ cgroupfs πϦʔΛఏڙ (/sys/fs/cgroup) cgroup ͰͷϦιʔε੍ݶʹԠͨ͡/proc ҎԼͷϦιʔεؔ࿈ ஋ͷఏڙ cpuinfo meminfo stat uptime ... Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 87 / 108

Slide 88

Slide 88 text

lxcfs ࢖͍ํ $ sudo lxcfs -s -f -o allow_other /var/lib/lxcfs lxcfs $ ls /var/lib/lxcfs/ cgroup proc $ ls /var/lib/lxcfs/proc/ cpuinfo meminfo stat uptime $ ls /var/lib/lxcfs/cgroup/ blkio cpuacct devices hugetlb name=systemd cpu cpuset freezer memory perf_event Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 88 / 108

Slide 89

Slide 89 text

lxcfs lxcfs ͷ proc ҎԼͷϑΝΠϧ͸ͦΕͧΕίϯςφ಺ͷ /proc ҎԼʹόΠϯυϚ΢ϯτ͞ΕΔ ϗετͷ cgroupfs ͷίϯςφʹରԠ͢Δ cgroup ͔Βಘͨ৘ ใΛݩʹίϯςφ಺ͷ֤ϑΝΠϧΛੜ੒ lxcfs ͷ cgroup ҎԼͷσΟϨΫτϦ͸ίϯςφ಺ͷ /sys/fs/cgroup ҎԼʹόΠϯυϚ΢ϯτ͞ΕΔ systemd ͕ίϯςφ಺Ͱಈ͘৔߹ʹඞཁ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 89 / 108

Slide 90

Slide 90 text

lxcfs ͜Ε·Ͱ $ sudo grep cgroup /var/lib/lxc/ct01/config lxc.cgroup.memory.limit_in_bytes = 256M (ϝϞϦ্ݶ 256MB ʹઃఆ) $ sudo lxc-start -n ct01 -d (ίϯςφىಈ) $ grep MemTotal /proc/meminfo (ϗετͷϝϞϦྔ) MemTotal: 1017908 kB $ sudo lxc-attach -n ct01 -- grep MemTotal /proc/meminfo MemTotal: 1017908 kB (ϗετͷϝϞϦྔͦͷ··) lxcfs Λ࢖ͬͨ৔߹ $ sudo grep cgroup /var/lib/lxc/ct01/config lxc.cgroup.memory.limit_in_bytes = 256M (ϝϞϦ্ݶ 256MB ʹઃఆ) $ sudo lxc-start -n ct01 (ίϯςφىಈ) $ grep MemTotal /proc/meminfo (ϗετͷϝϞϦྔ) MemTotal: 1017792 kB $ sudo lxc-attach -n ct01 -- grep MemTotal /proc/meminfo MemTotal: 262144 kB (cgroup Ͱઃఆ੍ͨ͠ݶͷ஋ʹͳ͍ͬͯΔ) Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 90 / 108

Slide 91

Slide 91 text

UbuntuͰͷLXCͷΠϯετʔϧ lxc ύοέʔδΛΠϯετʔϧ͢Δ͚ͩ $ sudo apt-get install lxc Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 91 / 108

Slide 92

Slide 92 text

UbuntuͰͷίϯςφ؀ڥ ίϯςφ༻ͷωοτϫʔΫ͕࡞੒͞ΕΔ (10.0.3.0/24) ϒϦοδ͕࡞੒͞Εɺ෺ཧΠϯλʔϑΣʔεͱͷؒͰ NAT ͕ ઃఆ͞ΕΔ dnsmasq ͰίϯςφʹΞυϨεΛׂΓ౰ͯɺίϯςφ໊͕લ ղܾͰ͖ΔΑ͏ʹ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 92 / 108

Slide 93

Slide 93 text

ίϯςφͷ࡞੒ ςϯϓϨʔτΛࢦఆͯ͠ lxc-create ίϚϯυΛ࣮ߦ͢Δɻ $ ls /usr/share/lxc/templates/ lxc-alpine* lxc-centos* lxc-gentoo* lxc-sshd* lxc-altlinux* lxc-cirros* lxc-openmandriva* lxc-ubuntu* lxc-archlinux* lxc-debian* lxc-opensuse* lxc-ubuntu-cloud* lxc-bind* lxc-download* lxc-oracle* lxc-busybox* lxc-fedora* lxc-plamo* σΟετϦϏϡʔγϣϯςϯϓϨʔτͷ࢖༻ $ sudo lxc-create -n ct01 -t ubuntu μ΢ϯϩʔυςϯϓϨʔτͷ࢖༻ $ sudo lxc-create -n ct01 -t download -- \ > -d ubuntu -r trusty -a amd64 Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 93 / 108

Slide 94

Slide 94 text

ຊ೔ͷޙ൒ͷ಺༰ Linux ʹ͓͚Δίϯςφ࣮૷ Namespace ͷσϞ LXC LXC σϞ LXD LXD σϞ ·ͱΊ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 94 / 108

Slide 95

Slide 95 text

LXCσϞ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 95 / 108

Slide 96

Slide 96 text

σϞ ʙ LXCίϯςφͷىಈ 1 γεςϜίϯςφͷىಈ $ sudo lxc-start -n ct01 -d 2 ΞϓϦέʔγϣϯίϯςφͷىಈ $ sudo lxc-start -d -n ct01 -- \ > /usr/sbin/apache2ctl -D FOREGROUND 3 ίϯςφՔಇͷ֬ೝ $ sudo lxc-ls --fancy 4 Քಇதͷίϯςφ಺ʹೖΔ (ೖΓίϚϯυΛ࣮ߦ) (ssh ࢖Θͳ ͍!!) $ sudo lxc-attach -n ct01 Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 96 / 108

Slide 97

Slide 97 text

σϞ ʙίϯςφͷΫϩʔϯ 1 ී௨ʹΫϩʔϯ $ sudo lxc-clone -o ct01 -n ct02 Created container ct02 as copy of ct01 $ sudo lxc-ls -f NAME STATE IPV4 IPV6 GROUPS AUTOSTART -------------------------------------------- ct01 STOPPED - - - NO ct02 STOPPED - - - NO 2 overlayfs ͰΫϩʔϯ $ sudo lxc-clone -o ct01 -n ct02 -B overlayfs -s Created container ct02 as snapshot of ct01 ίϯςφσΟϨΫτϦͷ֬ೝ Ϋϩʔϯݩ͸ফͤͳ͍ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 97 / 108

Slide 98

Slide 98 text

σϞ ʙLXCͰඇಛݖίϯςφ 1 ઃఆϑΝΠϧͷ֬ೝ /etc/subuid, /etc/subgid, $HOME/.config/lxc/default.conf, /etc/lxc/lxc-usernet 2 ίϯςφ࡞੒ (download ςϯϓϨʔτΛ࢖༻͢Δඞཁ͕ ͋Δ) $ lxc-create -t download -n ct01 -- -d ubuntu -r trusty -a amd64 3 ίϯςφىಈ 4 Ϛοϐϯάͷ֬ೝɺίϯςφ಺֎Ͱͷࠩ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 98 / 108

Slide 99

Slide 99 text

ຊ೔ͷޙ൒ͷ಺༰ Linux ʹ͓͚Δίϯςφ࣮૷ Namespace ͷσϞ LXC LXC σϞ LXD LXD σϞ ·ͱΊ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 99 / 108

Slide 100

Slide 100 text

LXD Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 100 / 108

Slide 101

Slide 101 text

LXD REST API Λఏڙ͢ΔσʔϞϯ (lxd ίϚϯυ) ίϚϯυϥΠϯΫϥΠΞϯτ (lxc ίϚϯυ) OpenStack Nova ϓϥάΠϯ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 101 / 108

Slide 102

Slide 102 text

LXD APIɺίϚϯυϥΠϯπʔϧΛ࢖ͬͯωοτϫʔΫ্ͷίϯ ςφͷ؅ཧ͕Մೳ Πϝʔδϕʔε ϥΠϒϚΠάϨʔγϣϯ LXD ϗετΛ OpenStack ίϯϐϡʔτϊʔυʹ σϑΥϧτ͸ඇಛݖίϯςφ LXC ͷ Go όΠϯσΟϯάΛ࢖ͬͯ liblxc ܦ༝ͰίϯςφΛૢ࡞ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 102 / 108

Slide 103

Slide 103 text

ຊ೔ͷޙ൒ͷ಺༰ Linux ʹ͓͚Δίϯςφ࣮૷ Namespace ͷσϞ LXC LXC σϞ LXD LXD σϞ ·ͱΊ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 103 / 108

Slide 104

Slide 104 text

LXDσϞ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 104 / 108

Slide 105

Slide 105 text

σϞ ʙLXD 1 ίϯςφͷىಈ (ϩʔΧϧɺϦϞʔτ) 2 ίϯςφ্ͷίϚϯυͷ࣮ߦ 3 ϚΠάϨʔγϣϯ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 105 / 108

Slide 106

Slide 106 text

ޙ൒ͷ·ͱΊ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 106 / 108

Slide 107

Slide 107 text

ޙ൒ͷ·ͱΊ Linux ʹ͓͚Δίϯςφ࣮૷ Namespace ͷσϞ LXC LXC σϞ LXD LXD σϞ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 107 / 108

Slide 108

Slide 108 text

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ Ճ౻ହจ ୈ 7 ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ 2015-06-20 108 / 108