Using LXC on Production

Slide 1

Slide 1 text

Using LXC on Production ୈ4ճ ίϯςφܕԾ૝Խͷ৘ใަ׵ձˏ౦ژ 2014.9.6 Isao SHIMIZU @isaoshimizu

Slide 2

Slide 2 text

ࠓճͷݩωλ http://alpha.mixi.co.jp/entry/2014/12171/ ͜ͷΤϯτϦΛগ͠ΞϨϯδ͓ͯ͠࿩͠·͢ɻ

Slide 3

Slide 3 text

ࣗݾ঺հ ਗ਼ਫ ܄ ʢIsao SHIMIZUʣ ! גࣜձࣾϛΫγΟ ϞϯετελδΦॴଐ ! ݱࡏ4೥໨ʢ2011೥ೖࣾʣ mixiͷΠϯϑϥӡ༻ ϞϯελʔετϥΠΫͷΠϯϑϥɾαʔόӡ༻ʢݱࡏʣ ΤϯδχΞϒϩάࣥචʢFedoraɺOpenStackɺLXCͳͲʣ ! લ৬: 2003ʙ2011೥ SIerͰاըɺ։ൃɺΠϯϑϥӡ༻ ૊ΈࠐΈɺWebɺεϚϑΥΞϓϦɺಈը഑৴ͳͲ

Slide 4

Slide 4 text

ࠓ೔ͷൃදͷܦҢ

Slide 5

Slide 5 text

LXCಋೖʹࢸΔ·Ͱ

Slide 6

Slide 6 text

mixiʹ͓͚ΔԾ૝Խ؀ڥ ͍··Ͱ͸͜͏ͩͬͨ

Slide 7

Slide 7 text

KVM Kernel-based Virtual Machine

Slide 8

Slide 8 text

Ծ૝Խ؀ڥͰKVMΛଟ༻͍ͯͨ࣌͠୅ • Ծ૝Խ؀ڥ͸KVM͔͠࢖͍ͬͯͳ͔ͬͨ • ༻్͸։ൃ؀ڥɺεςʔδϯά؀ڥ͕΄ͱΜͲ • ߏங͸ࣗ࡞ͷγΣϧεΫϦϓτͰ • ϒϦοδΠϯλϑΣʔεͷ࡞੒ • Cobblerͱͷ࿈ܞʢϗετ໊ͷ࿈൪Խ΍IPͷॏෳ๷ࢭʣ • virt-install, Kickstart ! • جຊख࡞ۀͰ໘౗͍͘͞

Slide 9

Slide 9 text

KVMͷ͍͍ͱ͜Ζ • ܰ౓ͷར༻Ͱ͸े෼ͳύϑΥʔϚϯε͕ग़Δ • ήετOSʹϚγϯͱಉ͡ѻ͍͕Ͱ͖Δ • ϊ΢ϋ΢ͨ͘͞Μ • Ϋϥ΢υܥͷπʔϧ͕ॆ࣮͍ͯ͠Δ

Slide 10

Slide 10 text

KVMͷͭΒ͍ͱ͜Ζ • Ծ૝ԽʹΑΔϘτϧωοΫ͕େ͖͍ʢͱ͘ʹσΟεΫIOʣ • σΟεΫ༰ྔΛଟ͘ফඅ͢ΔʢOS෼͕େ͖͍ʣ • BIOSઃఆͷґଘʢIntel VTͱ͔AMD-Vͱ͔ʣ

Slide 11

Slide 11 text

OpenStackͷಋೖ

Slide 12

Slide 12 text

OpenStack • 2013೥य़ࠒʹݕূ։࢝ • Version͸Grizzly 2013.01 • ಉ೥ͷՆࠒʹຊ൪ಋೖ • ༻్͸ࣾ಺ϓϩμΫτ޲͚ͷPaaSʢGizmoͱݺ͹ΕΔʣ • ΞϓϦαʔό͸ಠࣗͷσϓϩΠπʔϧΛ࢖ͬͯ • ϛυϧ΢ΣΞͷߏ੒͸ChefͰ • MySQLɺRedisɺJenkinsͳͲ • μογϡϘʔυʢHorizonʣศར • ͷͪʹ։ൃ؀ڥʹ΋ల։

Slide 13

Slide 13 text

mixiʹ͓͚ΔԾ૝Խ؀ڥ ͍·ͷӡ༻

Slide 14

Slide 14 text

LXC Linux Containers

Slide 15

Slide 15 text

LXCΛ࢖͏લͷҹ৅ • ͱʹ͔͍ܰ͘Β͍͠ʢͰ΋Α͘Θ͔ͬͯͳ͍ʣ • KVMͱ͔ͱԿ͕ҧ͏ͷ͔͍·͍ͪΘ͔ͬͯͳ͍ • LXCͷόʔδϣϯ͕͕͖͍͋ͬͯͯͯͦΖͦΖ͍͍ײ͔͡΋ʁ • Kernelগ্͛͠Ε͹࢖͑ͦ͏ • ৽͍ٕ͠ज़ؾʹͳΔɺ࢖ͬͯΈ͍ͨ • ຊ൪Ͱ࢖͍ͬͯΔ࿩͸΄ͱΜͲͳ͍ • ͪΐ͏ͲLinuxCon Japan 2013ͰLXCͷ࿩Λฉ͍ͨ • Ͱ΋ࣾ಺Ͱ͸୭΋৮ͬͯͳ͔ͬͨ

Slide 16

Slide 16 text

LXCʹ͍ͭͯ • KVMͷΑ͏ʹϋʔυ΢ΣΞͳͲͷΤϛϡϨʔγϣϯͷ্ʹԾ ૝ϚγϯΛಈ࡞ͤ͞ΔͷͰ͸ͳ͍ • ϓϩηε΍ωοτϫʔΫɺϢʔβʔۭؒͳͲΛ෼཭ͯ͠ɺԾ ૝తͳ؀ڥΛఏڙ • KernelͷػೳΛ࢖ͬͯ෼཭͞Εͨ؀ڥ • KVMͰى͖͍ͯͨΑ͏ͳɺCPU΍σΟεΫIOͳͲͷύϑΥʔ ϚϯεྼԽ͕جຊతʹൃੜ͠ͳ͍ • ىಈ͕଎͍ʢinitҎ߱ͷىಈ͚ͩʣ • ΋ͪΖΜΦʔϓϯιʔε

Slide 17

Slide 17 text

LXCͷϨϙδτϦ https://github.com/lxc/lxc

Slide 18

Slide 18 text

LXCͷίϛοτਪҠ https://github.com/lxc/lxc

Slide 19

Slide 19 text

KVMͱLXC

Slide 20

Slide 20 text

LXCͷݕূΛ࢝ΊΔ • όʔδϣϯ0.8.0ʢ2012.11.11ϦϦʔεʣ͔Β0.9.0  ʢ2013.4.5ϦϦʔεʣ΁Ξοϓσʔτ͞Ε͍ͯͨ • 1.0.0͸2014೥2݄Λ༧ఆ͍ͯͨ͠ʢ଴ͯͳ͍ʣ • ·ͣ͸ɺ0.9.0Λݕূͯ͠Έ͍ͨ • ·ͣ΍ͬͨ͜ͱ • templatesʹ͋ΔFedoraͷγΣϧεΫϦϓτΛ࢖ͬͯ  LXCͷΠϝʔδ࡞Γ • ͢ΜͳΓಈ͔ͳ͍ • Fedora޲͚ʹϝϯς͞Εͯͳ͍ͷ͔͍Ζ͍Ζमਖ਼ • ͱΓ͋͑ͣࢼߦࡨޡͯ͠ಈ͍ͨʢख࡞ۀίϚϯυϨϕϧʣ

Slide 21

Slide 21 text

ͦΜͳݕূΛ͍ͯ͠Δ͏ͪʹ ΞϨ͕࿩୊ʹ ! ౰࣌2013೥10݄ࠒ

Slide 22

Slide 22 text

Slide 23

Slide 23 text

DockerͷτϨϯυ -9$ͷݕূ࢝Ίͨࠒ

Slide 24

Slide 24 text

ؾʹͳ͍ͬͯͨDockerͷଘࡏ • ౰࣌ͷόʔδϣϯ 0.6.xʢݱࡏ͸1.2.0ʣ • AUFSؾʹͳΔ • Docker Registryศརͦ͏ • GoͷϙʔλϏϦςΟ͢͹Β͍͠ ! • IPϚεΧϨʔυ͸ͪΐͬͱ໘౗͍͘͞ • ίϯςφʹIPΛݸผʹৼͬͯɺԾ૝ϚγϯͷΑ͏ʹѻ͍͍ͨ ʢmacvlan࢖͍͍ͨʣ • taggedVLANͷ؀ڥͰ΋໰୊ͳ͘࢖͍͍ͨ • όʔδϣϯΞοϓ͕ܹ͍͠

Slide 25

Slide 25 text

ಠࣗπʔϧͷ։ൃ΁

Slide 26

Slide 26 text

trailer ʢτϨΠϥʔʣ

Slide 27

Slide 27 text

trailerͱ͸ • Ruby੡ͷࣗࣾͰ։ൃͨ͠πʔϧ • LXCͷϥούʔ • ӡ༻ʹඞཁͳػೳͷΈΛ࣮૷ • IPɺMACΞυϨεͷ࠾൪ʢARMͱݺ͹ΕΔαʔόͱ࿈ܞʣ • ίϯςφΠϝʔδΛμ΢ϯϩʔυͯ͠ల։͢Δ • ىಈதͷίϯςφ͔ΒΠϝʔδΛ࡞Δ • Trailerfileͱݺ͹ΕΔίϯςφఆٛ

Slide 28

Slide 28 text

trailerͷߏ੒

Slide 29

Slide 29 text

trailerΛ࢖ͬͨίϯςφىಈϑϩʔ ᶃ͋Β͔͡Ί࡞ΒΕͨΠϝʔδΛϨϙδτϦαʔό͔Βμ΢ϯϩʔυʢtrailer pullʣ ᶄΠϝʔδΛىಈʢtrailer startʣ (1)ϩʔΧϧʹμ΢ϯϩʔυ͞ΕͨΠϝʔδΛΠϯελϯε༻ͷσΟϨΫτϦʹల։ (2)ARMͱݺ͹ΕΔ಺੡ͷΞυϨε؅ཧπʔϧʹରͯ͠APIΞΫηε͠ɺIPΞυϨεͱ MACΞυϨε͕෷͍ग़͞ΕΔʢARM͸APIΞΫηεՄೳͳDHCPαʔόͷΑ͏ͳ΋ͷʣ (3)औಘͨ͠IPΞυϨεͱMACΛΠϯελϯεʹઃఆ  ʢmacvlan bridgeϞʔυʣͯ͠ɺinitʢsystemdʣΛىಈ (4)trailer start࣮ߦ͔ΒsshͰ઀ଓՄೳʹͳΔ·Ͱʹ͔͔Δ࣌ؒ͸10ඵఔ౓  ʢΠϝʔδαΠζʹΑͬͯଟগͷมಈ͋Γʣ

Slide 30

Slide 30 text

trailerΛ࢖ͬͨίϯςφఀࢭϑϩʔ ᶃఀࢭίϚϯυΛ࣮ߦʢtrailer stopʣ (1)LXCͷϓϩηεͷఀࢭɺσΟϨΫτϦͷ࡟আ

Slide 31

Slide 31 text

trailerΛ࢖ͬͨΠϝʔδͷ࡞੒ͱ ϨϙδτϦαʔό΁ͷΞοϓϩʔυͷϑϩʔ ᶃϕʔεͱͳΔΠϝʔδΛىಈʢtrailer startʣ ᶄΠϯελϯεʹରͯ͠ChefͰϨγϐΛద༻ʢknife-soloΛར༻ʣ ᶅϧʔτϑΝΠϧγεςϜʢσΟϨΫτϦπϦʔʣΛѹॖ͢Δʢtrailer snapshotʣ ᶆΠϝʔδ৘ใ͕ॻ͔ΕͨyamlϑΝΠϧͱrootfs.gzΛtarballʹ͢Δʢtrailer archiveʣ ᶇϨϙδτϦαʔό΁Ξοϓϩʔυʢtrailer pushʣ

Slide 32

Slide 32 text

LXC޲͚ʹ༻ҙͯ͋͠ΔΠϝʔδ •ϕʔεΠϝʔδ •Reverse Proxy (mod_proxy) •Varnish •Q4M (Job Queue) •Application Server (mod_perl) •Tokyo Tyrant •Memcached

Slide 33

Slide 33 text

LXCΛӡ༻͢Δ্ͰؾΛ͚ͭΔ͜ͱ

Slide 34

Slide 34 text

εϨου਺ɺPID਺্ݶ •kernel.threads-max •kernel.pid_max •vm.max_map_count •/etc/security/limits.d/90-nproc.conf Λunlimited ʹ •༻్ʹԠͯ͡File Descriptor਺΍ɺTCP/IPपΓͷKernelύϥϝʔλ ͷௐ੔͕ඞཁɻ •ΠϯελϯεଆͰ͸ઃఆͰ͖ͳ͍Kernelύϥϝʔλ͕͋ͬͨΓ͢ ΔͷͰɺsysctl΍echoͳͲͰઃఆ͢Δࡍʹཁ஫ҙɻ

Slide 35

Slide 35 text

ͦͷଞɺؾΛ͚ͭΔ͜ͱ wར༻Ϧιʔεͷ༧ଌɺݟੵ΋Γ wଞͷίϯςφʹѱӨڹΛٴ΅͞ͳ͍ઃܭ͕ඞཁ wσΟεΫ༰ྔ w༰ྔ੍ݶ͸Ͱ͖ͳ͍ wϞχλϦϯά wάϥϑେࣄ

Slide 36

Slide 36 text

trailerͷσϞ

Slide 37

Slide 37 text

ࢀߟࢿྉ

Slide 38

Slide 38 text

•OpenStackͱLXCΛಋೖͨ͠࿩ - mixi Engineers' Blog •http://alpha.mixi.co.jp/entry/2014/12171/ •LXCͰֶͿίϯςφೖ໳ ʵܰྔԾ૝Խ؀ڥΛ࣮ݱ͢Δٕज़ •http://gihyo.jp/admin/serial/01/linux_containers •Lxc Ͱ࢝ΊΔένένԾ૝Խੜ׆ʁʂ - SlideShare •http://www.slideshare.net/enakai/lxc-8300191 •LXC - Linux Containers •https://linuxcontainers.org/jp/ •LXC(Linux Container) •http://events.linuxfoundation.org/sites/events/files/cojp13_feng.pdf •DockerΛࢧ͑Δٕज़ •http://www.slideshare.net/enakai/docker-34668707 •GitHub - lxc/lxc •https://github.com/lxc/lxc