CoreOS Meetup Tokyo #1 2015-04-09
CoreOSͰӡ༻͢ΔͨΊʹ ߟ͑ͳ͍ͱ͍͚ͳ͍͜ͱHarukasan / MICHII ShunsukeCoreOS Meetup Tokyo #12015-04-09
View Slide
Harukasan / MICHII Shunsuke• 2012ʹ৽ଔͱͯ͠ΠϯϑϥνʔϜʹଐ• ίϯςϯπ৴Λ͡Ίͱͯ͠αʔϏεશମͷج൫Λ୲• 16Gbps͘Β͍ͷը૾৴• αʔϏεϑϩϯτ(nginx)• Fluentd / Kibana / BigQuery• ٕज़ܥࡶࢽͷࣥචͳͲ
beer@harukasanhttp://harukasan.jp/beers• ौ୩ͰϏʔϧΛҿΉΠϕϯτΛ͍ͬͯ·͢• ࠓ݄ͷΠϕϯτ໌༻ҙ͠·͢ʢͨͿΜ23ʁʣ
͡Ίʹ• pixivʹ͓͚ΔCoreOS• ͳʹ͕͍ͨ͠ͷ͔• CoreOSΛ͏ͱԿΛߟ͑ͳ͍ͱ͍͚ͳ͍ͷ͔• ͜͜1ϲ݄͘Β͍΄ͱΜͲ৮ͬͯͳ͍……
CoreOS in pixiv
CoreOS in pixiv• Relase 554͔Β͍࢝Ίͨ• Ұ෦ͷϓϩμΫγϣϯڥͰ༻• IDCFΫϥυ্ʹߏங• ࠷খߏ• ·͍ͩΖ͍Ζͱࢼ͍ͯ͠Δͱ͜Ζ
• pixivͷອը࡞͔ΒΦεεϝ࡞ΛӾཡɺݕࡧͰ͖ΔAndroid͚ΞϓϦ• 20152݄4ϦϦʔε• Build with Play framework / ScalaΦεεϝϚϯΨΛ΄΅ຖߋ৽ʂ pixivϚϯΨ
AppApp App AppLBRPCRPC RPC RPCLBpixiv DBsmanga-app DBCoreOS CoreOS CoreOS CoreOSΦϯϓϨϛεIDCF CloudRequest flowManageManageL3 Private connect
AppApp App AppLBRPCRPC RPC RPCLBManagepixiv DBsmanga-app DBCoreOSManageCoreOS CoreOS CoreOSfleetctlJenkinspullpployDeployment flowΦϯϓϨϛεIDCF CloudL3 Private connect
AppApp App AppLBRPCRPC RPC RPCLBpixiv DBsmanga-app DBCoreOSManageCoreOS CoreOS CoreOSMonitoring flowΦϯϓϨϛεIDCF CloudL3 Private connecttd-agenttd-agenttd-agenttd-agentdd-agentdd-agentdd-agentdd-agentFluentdLogsMuninNagios
ਓͷԹ͔ΈΛײ͡ΔσϓϩΠ• cloud-configͰઃఆ͕ྲྀ͠ࠐ·ΕΔ• σϓϩΠ৬ਓʹΑΔϩʔϦϯάσϓϩΠ• fleetctlͰϦελʔτ͢Δ͜ͱͰσϓϩΠ͢Δ• ࣗಈԽ͍ͨ͠export FLEETCTL_ENDPOINT=http://app-‐1:4001/ NUM=1 fleetctl stop app-‐a@${NUM}.service fleetctl start app-‐a@${NUM}.service
ͳͥCoreOSͳͷ͔
ͳͥCoreOSΛ࠾༻ͨ͠ͷ͔• Play framework / ScalaͷΞϓϦέʔγϣϯͷґଘཧͱ͔ཧ͢Δͷ͕໘ͩͬͨͷͰίϯςφʹด͡ࠐΊ͔ͨͬͨ• ίϯςφ͔͠ಈ͔͞ͳ͍ϗετΛӡ༻͢Δͷ͕໘• ίϯςφΛಈ͔͚ͩͩͬͨ͢ΒCoreOSͰྑ͍ͷͰ
ͳʹ͕͍ͨ͠ͷ͔• ΠϯϑϥνʔϜͱࣗΒͷࣄΛͳ͘͠ଓ͚Δ͜ͱ͕৬ͱͯ͠ཁٻ͞ΕΔࣗݾໃ६ͨ͠ଘࡏͰ͋Δ• ͨͩ͠ؾΛൈ͘ͱࣄ͕૿͑ଓ͚ͯࢮ͵• ΊΜͲ͍͘͜͞ͱͰ͖Δ͚ͩΓͨ͘ͳ͍
ͳΔ͘ঢ়ଶΛؾʹͨ͘͠ͳ͍• όʔδϣϯͷࠩҟ• ੬ऑੑ• ґଘؔ• Ϣʔβੜσʔλ• etc.
ͳΔ͘ঢ়ଶΛؾʹͨ͘͠ͳ͍• Immutable• ίϯςφҎ֎ͷঢ়ଶΛม͑ͳ͍• /usrҎԼͦͦॻ͖͑Ͱ͖ͳ͍• Disposable• ίϯςφΛ͍ͭͰࣺͯΔ͜ͱ͕Ͱ͖Δ• ͳΜ͔͓͔͘͠ͳͬͨΒࣺͯΕྑ͍
CoreOSΛͲ͏ଊ͍͑ͯΔͷ͔• systemd + etcd/fleet/docker• αʔϏεཧͯ͢systemd͕ߦ͏• etcdΛ͔ͭͬͯΫϥελϦϯά͢Δ• fleetΛ͔ͭͬͯαʔϏεΛσϓϩΠ͢Δ• docker/rktΛ͔ͭͬͯίϯςφΠϝʔδΛཧ͢Δ• ͦͷ͏ͪউखʹΞοϓάϨʔυ͞ΕΔ• ͜ΕͰेͩͬͨΒCoreOSબࢶʹͳΔ
CoreOSΛͲ͏ଊ͍͑ͯΔͷ͔• systemd + etcd/fleet/docker• αʔϏεཧͯ͢systemd͕ߦ͏• etcdΛ͔ͭͬͯΫϥελϦϯά͢Δ• fleetΛ͔ͭͬͯαʔϏεΛσϓϩΠ͢Δ• docker/rktΛ͔ͭͬͯίϯςφΠϝʔδΛཧ͢Δ• ͦͷ͏ͪউखʹΞοϓάϨʔυ͞ΕΔ• ͜ΕͰेͩͬͨΒCoreOSબࢶʹͳΔ• ͜ΕҎ্ඞཁͳΒKubernetes͕ඞཁʹͳΔ
CoreOSΛͲ͏ଊ͍͑ͯΔͷ͔• ͭʹSSHͰ͖ͳ͍༁Ͱͳ͍• ύοέʔδཧγεςϜ͍Βͳ͍• ͍͍ͩͨͷ͜ͱsystemd͕Δ• ։ൃ࣌ʹDockerίϯςφΛಈ͔͢Πϯελϯε ͱͯ͠ศར
Container Deployment• όΠφϦʹؔ࿈͢Δͯ͢ͷґଘؔΛίϯςφͷதʹด͡ࠐΊΔ• ίϯςφͷ֎ͷঢ়ଶؾʹ͢Δඞཁ͕ͳ͍/etc/*****.conf…/usr/local/bin/*****…/usr/lib/****…/var/****…Container
Container Deployment• ґଘؔΛ։ൃऀ͕ཧͰ͖Δ• ԾΠϯελϯεΈ͍ͨʹͨ͘͞ΜͷϊʔυΛ1ͭͷཧϚγϯʹಉډͤ͞Δͷͱͪΐͬͱҧ͏• ͋͘·ͰґଘؔΛύοέʔδϯάͨ͠1ϓϩηεͱࢥͬͨΒΘ͔Γ͍͢
Web Application in a Container• ΞϓϦέʔγϣϯϓϩηείϯςφʹ͍͍ͯΔ• 1ϓϩηεʢجຊతʹʣ• ґଘ͢ΔϥΠϒϥϦ͕ଟ͍• ϝϞϦҎ֎ʹঢ়ଶΛ࣋ͨͳ͍• LBͰϦΫΤετΛεΠονͰ͖Δ
CoreOSΛ͏্Ͱߟ͑ͳ͍ͱ͍͚ͳ͍͜ͱ
CoreOSΛ͏্Ͱߟ͑ͳ͍ͱ͍͚ͳ͍͜ͱ• CoreOSͱ͍͏͔ɺͲ͏ͬͯ֎෦ʹґଘ͠ͳ͍ঢ়ଶΛͭ͘Δ͔• ·ͩ͏·͍ํ๏͕ݟ͔ͭͬͯͳ͍͜ͱ͕ଟ͍• γεςϜΦʔέετϨʔγϣϯ• σϓϩΠϝϯτ• ϞχλϦϯά• োରԠ• ࣗಈΞοϓάϨʔυ
etcdʹ͓͚Δ࠷খߏ• ΫϥελΛ࡞Δͱ͖࠷Ͱ4ʹ͓ͯ͘͠• 3ͩͱ1མ͚ͪͨͩͰεϓϦοτϒϨΠϯ
͏1ඞཁ
γεςϜΦʔέετϨʔγϣϯ• fleetͰͰ͖Δͷجຊతʹʮ͜ͷsystemdαʔϏεΛΫϥελͰ͍ͭ͘ಈ͔͔͢ʯͱ͍͏͜ͱ͚ͩ• αʔϏεϝϯόͱ͔Ϛελϊʔυͱ͔Λཧ͢Δػೳͳ͍• ͪΖΜΦʔτεέʔϦϯάͱ͔ͳ͍• etcdΛͬͯࣗͰ͕ΜΔʁ• KubernetesͰΔʁ• LB͘Β͍͏ͪΐͬͱ؆୯ʹΓ͍ͨ
ίϯςφʹର͢Δ ϩʔυόϥϯγϯά• ίϯςφͷstart/stop࣌ʹLBͷAPIΛୟ͘• systemdͷαʔϏεϑΝΠϧʹίϚϯυΛՃ͢Δ https://github.com/coreos/elb-presence• LBͷAPIΛୟ͚ͩ͘ͳͷͰൺֱత؆୯• ϋʔτϏʔτνΣοΫͰ֎ΕΔ͚ͩͰྑ͍
ίϯςφʹର͢Δ ϩʔυόϥϯγϯά• etcdΩʔ͕มߋ͞Εͨͱ͖ʹϑοΫͰ͖Δ• ϑοΫͯ͠LBͷઃఆΛม͑Εྑͦ͞͏• vulcand https://github.com/mailgun/vulcand• confd + nginx https://github.com/kelseyhightower/confd
σϓϩΠϝϯτ• fleetͩͱϩʔϦϯάϦελʔτͰ͖ͳ͍• ϩʔϦϯάϦελʔτ͢ΔΑ͏ͳΈΛͭ͘ΔʹࣗͰεΫϦϓτΛॻ͔͘͠ͳ͍• ଞͷίϯςφ͕ಈ͍͍ͯΔϊʔυͰಈ͔ͳ͍Α͏ʹࢦఆ
[Unit] Description=app After=docker.service Requires=docker.service [Service] Restart=always TimeoutStartSec=0 ExecStartPre=-‐/usr/bin/docker kill app-‐%i ExecStartPre=-‐/usr/bin/docker rm app-‐%i ExecStartPre=/usr/bin/docker pull … ExecStart=/usr/bin/docker run —name app-‐%i … ExecStop=/usr/bin/docker stop app-‐%i [X-‐Fleet] Conflicts=app@*.service [email protected]:
ϞχλϦϯά• ίϯςφ͝ͱʹϞχλϦϯά͠ͳ͍ͱ͍͚ͳ͍• ίϯςφ໊Λ͚ͭͣʹMackerelͰࢹͯͨ͠Β σϓϩΠͷʹࢹର͕૿͔͑ͯͬͨ• DataDogͰͲ͏ʹ͔ͳͬͯΔ
ϞχλϦϯά• ͳʹΛϞχλϦϯά͠ͳ͍ͱ͍͚ͳ͍ͷ͔• αʔϏεϨϕϧ• ϊʔυϨϕϧ• ίϯςφϨϕϧ• ͳΜͰݟͨ͘ͳΔ͚Ͳຊʹશ෦ݟͳ͍ͱ ͍͚ͳ͍ͷ͔ߟ͑Δඞཁ͕͋Δ• 1ϊʔυ=1ίϯςφͰಈ͔͢ͷͰ͋Εࠓ·ͰͲ͓ΓͷϞχλϦϯάͱͦΜͳʹมΘΒͳ͍
ϩάసૹ• ֤ϊʔυʹtd-agentͷίϯςφΛཱ͍ͯͯΔ• cloud-configʹॻ͍ͯىಈͯ͠ྑͦ͞͏
োରԠ• ࠓͷͱ͜Ζϗετʹґଘ͢Δোʹૺ۰ͯ͠ͳ͍• ϋʔτϏʔτ͕ಧ͍ͯΔঢ়گͩͱLB͔ΒखಈͰ ֎ͨ͠Γ͠ͳ͍ͱ͍͚ͳ͍͔ʁ
ࣗಈΞοϓάϨʔυ• ϊʔυ͕མͪΔ͚ͩͱࢥ͑ϊʔυোͱมΘΒͳ͍• ϩοΫΛ͔͚ͳ͕Βͬͯ͘ΕΔͷͰϩʔϦϯάΞοϓάϨʔυ͞ΕΔʢฒྻࢦఆͰ͖Δʣ• fleetͷόʔδϣϯ্͕͕ͬͨͱ͖ʹཧϗετଆͷfleetctl͕ߋ৽͞Εͯͳࣦͯ͘ഊͨ͜͠ͱ͕͋ͬͨ
ࣗಈΞοϓάϨʔυ• ৽͍ͭ͠Λ͍͍͚ͨͲAlphaόάͬͯΔͷͰ Ͳ͏͠Α͏• ࣗͰϏϧυͨ͠Βҙຯͳ͍͠
ωοτϫʔΫ• ಛʹωοτϫʔΫΛΔϞνϕʔγϣϯͳ͍• ϙʔτϑΥϫʔσΟϯάͰͦΕ΄Ͳͳ͍• ϗετωοτϫʔΫͰྑ͍
ηΩϡϦςΟ• CoreOS͕໘Έͯ͘ΕΔͷϗετOS͚ͩ• ίϯςφͷ੬ऑੑؔͳ͍• Dockerͩͱίϯςφʹόʔδϣϯ͕ͳ͍ͷͰ͍ͬͣͭ֬͜ೝ͠ͳ͍ͱ͍͚ͳ͍
·ͱΊ• ݁ہͳΜ͔ΜඞཁʹͳΔ• ͦΕKubernetesͰ(ུ• ͦΕGCPͰ(ུ• ίϯςφ͡Όͳͯ͘AMIͰ…• Ͳ͜·ͰΔඞཁ͕͋Δ͔ߟ͑Δඞཁ͕͋Δ
·ͱΊ• ίϯςφΛಈ͔͚ͩ͢ͷϗετͰ͋ΕCoreOSͪΐ͏Ͳ͍͍େ͖͞• ेʹγϯϓϧ• γϯϓϧ͚ͩʹߟ͑ͳ͍ͱ͍͚ͳ͍͜ͱ͋Δ• ຊʹKubernetesΛࣗͰϚωδϝϯτ͢Δͷʁ
harukasan
shimacos
colopl
ewolff
hirosatogamo
kentosuzuki
tamaiyutaro
grimoh
yamahiro
linedevth
lmi
yrhorita
maggiecrowley
rasmusluckow
quramy
zakiwarfel
bluesmoon
andyhume
brad_frost
jrom
soudai
revolveconf
robhawkes
garrettdimon