CoreOSで運用するために考えないといけないこと / CoreOS in pixiv

by Harukasan

Slide 1

Slide 1 text

CoreOSͰӡ༻͢ΔͨΊʹ  ߟ͑ͳ͍ͱ͍͚ͳ͍͜ͱ Harukasan / MICHII Shunsuke CoreOS Meetup Tokyo #1 2015-04-09

Slide 2

Slide 2 text

Harukasan / MICHII Shunsuke • 2012೥ʹ৽ଔͱͯ͠ΠϯϑϥνʔϜʹ഑ଐ • ίϯςϯπ഑৴Λ͸͡Ίͱͯ͠αʔϏεશମͷج൫Λ୲౰ • 16Gbps͘Β͍ͷը૾഑৴ • αʔϏεϑϩϯτ(nginx) • Fluentd / Kibana / BigQuery • ٕज़ܥࡶࢽͷࣥචͳͲ΋

Slide 3

Slide 3 text

beer@harukasan http://harukasan.jp/beers • ौ୩ͰϏʔϧΛҿΉΠϕϯτΛ΍͍ͬͯ·͢ • ࠓ݄ͷΠϕϯτ͸໌೔༻ҙ͠·͢ʢͨͿΜ23೔ʁʣ

Slide 4

Slide 4 text

͸͡Ίʹ • pixivʹ͓͚ΔCoreOS • ͳʹ͕͍ͨ͠ͷ͔ • CoreOSΛ࢖͏ͱԿΛߟ͑ͳ͍ͱ͍͚ͳ͍ͷ͔ • ͜͜1ϲ݄͘Β͍΄ͱΜͲ৮ͬͯͳ͍……

Slide 5

Slide 5 text

CoreOS in pixiv

Slide 6

Slide 6 text

CoreOS in pixiv • Relase 554͔Β࢖͍࢝Ίͨ • Ұ෦ͷϓϩμΫγϣϯ؀ڥͰ࢖༻ • IDCFΫϥ΢υ্ʹߏங • ࠷খߏ੒ • ·͍ͩΖ͍Ζͱࢼ͍ͯ͠Δͱ͜Ζ

Slide 7

Slide 7 text

• pixivͷອը࡞඼͔ΒΦεεϝ࡞඼Λ ӾཡɺݕࡧͰ͖ΔAndroid޲͚ΞϓϦ • 2015೥2݄4೔ϦϦʔε • Build with Play framework / Scala ΦεεϝϚϯΨΛ΄΅ຖ೔ߋ৽ʂ  pixivϚϯΨ

Slide 8

Slide 8 text

App App App App LB RPC RPC RPC RPC LB pixiv DBs manga-app DB CoreOS CoreOS CoreOS CoreOS ΦϯϓϨϛε IDCF Cloud Request ﬂow Manage Manage L3 Private connect

Slide 9

Slide 9 text

App App App App LB RPC RPC RPC RPC LB Manage pixiv DBs manga-app DB CoreOS Manage CoreOS CoreOS CoreOS ﬂeetctl Jenkins pull pploy Deployment ﬂow ΦϯϓϨϛε IDCF Cloud L3 Private connect

Slide 10

Slide 10 text

App App App App LB RPC RPC RPC RPC LB pixiv DBs manga-app DB CoreOS Manage CoreOS CoreOS CoreOS Monitoring ﬂow ΦϯϓϨϛε IDCF Cloud L3 Private connect td-agent td-agent td-agent td-agent dd-agent dd-agent dd-agent dd-agent Fluentd Logs Munin Nagios

Slide 11

Slide 11 text

ਓͷԹ͔ΈΛײ͡ΔσϓϩΠ • cloud-conﬁgͰઃఆ͕ྲྀ͠ࠐ·ΕΔ • σϓϩΠ৬ਓʹΑΔϩʔϦϯάσϓϩΠ • ﬂeetctlͰϦελʔτ͢Δ͜ͱͰσϓϩΠ͢Δ • ࣗಈԽ͍ͨ͠ export FLEETCTL_ENDPOINT=http://app-‐1:4001/ NUM=1  fleetctl stop app-‐a@${NUM}.service  fleetctl start app-‐a@${NUM}.service

Slide 12

Slide 12 text

ͳͥCoreOSͳͷ͔

Slide 13

Slide 13 text

ͳͥCoreOSΛ࠾༻ͨ͠ͷ͔ • Play framework / ScalaͷΞϓϦέʔγϣϯͷґଘ؅ ཧͱ͔؅ཧ͢Δͷ͕໘౗ͩͬͨͷͰίϯςφʹด͡ ࠐΊ͔ͨͬͨ • ίϯςφ͔͠ಈ͔͞ͳ͍ϗετΛӡ༻͢Δͷ͕໘౗ • ίϯςφΛಈ͔͚ͩͩͬͨ͢ΒCoreOSͰྑ͍ͷͰ͸

Slide 14

Slide 14 text

ͳʹ͕͍ͨ͠ͷ͔ • ΠϯϑϥνʔϜͱ͸ࣗΒͷ࢓ࣄΛͳ͘͠ଓ͚Δ͜ͱ ͕৬຿ͱͯ͠ཁٻ͞ΕΔࣗݾໃ६ͨ͠ଘࡏͰ͋Δ • ͨͩ͠ؾΛൈ͘ͱ࢓ࣄ͕૿͑ଓ͚ͯࢮ͵ • ΊΜͲ͍͘͜͞ͱ͸Ͱ͖Δ͚ͩ΍Γͨ͘ͳ͍

Slide 15

Slide 15 text

ͳΔ΂͘ঢ়ଶΛؾʹͨ͘͠ͳ͍ • όʔδϣϯͷࠩҟ • ੬ऑੑ • ґଘؔ܎ • Ϣʔβੜ੒σʔλ • etc.

Slide 16

Slide 16 text

ͳΔ΂͘ঢ়ଶΛؾʹͨ͘͠ͳ͍ • Immutable • ίϯςφҎ֎ͷঢ়ଶΛม͑ͳ͍ • /usrҎԼ͸ͦ΋ͦ΋ॻ͖׵͑Ͱ͖ͳ͍ • Disposable • ίϯςφΛ͍ͭͰ΋ࣺͯΔ͜ͱ͕Ͱ͖Δ • ͳΜ͔͓͔͘͠ͳͬͨΒࣺͯΕ͹ྑ͍

Slide 17

Slide 17 text

CoreOSΛͲ͏ଊ͍͑ͯΔͷ͔ • systemd + etcd/ﬂeet/docker • αʔϏε؅ཧ͸͢΂ͯsystemd͕ߦ͏ • etcdΛ͔ͭͬͯΫϥελϦϯά͢Δ • ﬂeetΛ͔ͭͬͯαʔϏεΛσϓϩΠ͢Δ • docker/rktΛ͔ͭͬͯίϯςφΠϝʔδΛ؅ཧ͢Δ • ͦͷ͏ͪউखʹΞοϓάϨʔυ͞ΕΔ • ͜ΕͰे෼ͩͬͨΒCoreOS͸બ୒ࢶʹͳΔ

Slide 18

Slide 18 text

Slide 19

Slide 19 text

CoreOSΛͲ͏ଊ͍͑ͯΔͷ͔ • ΂ͭʹSSHͰ͖ͳ͍༁Ͱ͸ͳ͍ • ύοέʔδ؅ཧγεςϜ͸͍Βͳ͍ • ͍͍ͩͨͷ͜ͱ͸systemd͕΍Δ • ։ൃ࣌ʹDockerίϯςφΛಈ͔͢Πϯελϯε  ͱͯ͠ศར

Slide 20

Slide 20 text

Container Deployment • όΠφϦʹؔ࿈͢Δ͢΂ͯͷґଘؔ܎Λίϯςφͷத ʹด͡ࠐΊΔ • ίϯςφͷ֎ͷঢ়ଶ͸ؾʹ͢Δඞཁ͕ͳ͍ /etc/ *****.conf … /usr/local/bin/ ***** … /usr/lib/ **** … /var/ **** … Container

Slide 21

Slide 21 text

Container Deployment • ґଘؔ܎Λ։ൃऀ͕؅ཧͰ͖Δ • Ծ૝ΠϯελϯεΈ͍ͨʹͨ͘͞ΜͷϊʔυΛ1ͭͷ ෺ཧϚγϯʹಉډͤ͞Δͷͱ͸ͪΐͬͱҧ͏ • ͋͘·Ͱ΋ґଘؔ܎Λύοέʔδϯάͨ͠1ϓϩηε ͱࢥͬͨΒΘ͔Γ΍͍͢

Slide 22

Slide 22 text

Web Application in a Container • ΞϓϦέʔγϣϯϓϩηε͸ίϯςφʹ޲͍͍ͯΔ • 1ϓϩηεʢجຊతʹ͸ʣ • ґଘ͢ΔϥΠϒϥϦ͕ଟ͍ • ϝϞϦҎ֎ʹঢ়ଶΛ࣋ͨͳ͍ • LBͰϦΫΤετΛεΠονͰ͖Δ

Slide 23

Slide 23 text

CoreOSΛ࢖͏্Ͱߟ͑ͳ͍ͱ ͍͚ͳ͍͜ͱ

Slide 24

Slide 24 text

CoreOSΛ࢖͏্Ͱߟ͑ͳ͍ͱ ͍͚ͳ͍͜ͱ • CoreOSͱ͍͏͔ɺͲ͏΍ͬͯ֎෦ʹґଘ͠ͳ͍ঢ়ଶ Λͭ͘Δ͔ • ·ͩ͏·͍ํ๏͕ݟ͔ͭͬͯͳ͍͜ͱ͕ଟ͍ • γεςϜΦʔέετϨʔγϣϯ • σϓϩΠϝϯτ • ϞχλϦϯά • ো֐ରԠ • ࣗಈΞοϓάϨʔυ

Slide 25

Slide 25 text

etcdʹ͓͚Δ࠷খߏ੒ • ΫϥελΛ࡞Δͱ͖͸࠷௿Ͱ΋4୆ʹ͓ͯ͘͠ • 3୆ͩͱ1୆མ͚ͪͨͩͰεϓϦοτϒϨΠϯ

Slide 26

Slide 26 text

΋͏1୆ඞཁ

Slide 27

Slide 27 text

γεςϜΦʔέετϨʔγϣϯ • ﬂeetͰͰ͖Δͷ͸جຊతʹʮ͜ͷsystemdαʔϏεΛ Ϋϥελ಺Ͱ͍ͭ͘ಈ͔͔͢ʯͱ͍͏͜ͱ͚ͩ • αʔϏεϝϯόͱ͔Ϛελϊʔυͱ͔Λ؅ཧ͢Δػೳ ͸ͳ͍ • ΋ͪΖΜΦʔτεέʔϦϯάͱ͔ͳ͍ • etcdΛ࢖ͬͯࣗ෼Ͱ͕Μ͹Δʁ • KubernetesͰ΍Δʁ • LB͘Β͍΋͏ͪΐͬͱ؆୯ʹ΍Γ͍ͨ

Slide 28

Slide 28 text

ίϯςφʹର͢Δ  ϩʔυόϥϯγϯά • ίϯςφͷstart/stop࣌ʹLBͷAPIΛୟ͘ • systemdͷαʔϏεϑΝΠϧʹίϚϯυΛ௥Ճ͢Δ  https://github.com/coreos/elb-presence • LBͷAPIΛୟ͚ͩ͘ͳͷͰൺֱత؆୯ • ϋʔτϏʔτνΣοΫͰ֎ΕΔ͚ͩͰ΋ྑ͍

Slide 29

Slide 29 text

ίϯςφʹର͢Δ  ϩʔυόϥϯγϯά • etcd͸Ωʔ͕มߋ͞Εͨͱ͖ʹϑοΫͰ͖Δ • ϑοΫͯ͠LBͷઃఆΛม͑Ε͹ྑͦ͞͏ • vulcand  https://github.com/mailgun/vulcand • confd + nginx  https://github.com/kelseyhightower/confd

Slide 30

Slide 30 text

σϓϩΠϝϯτ • ﬂeetͩͱϩʔϦϯάϦελʔτͰ͖ͳ͍ • ϩʔϦϯάϦελʔτ͢ΔΑ͏ͳ࢓૊ΈΛͭ͘Δʹ ͸ࣗ෼ͰεΫϦϓτΛॻ͔͘͠ͳ͍ • ଞͷίϯςφ͕ಈ͍͍ͯΔϊʔυͰಈ͔ͳ͍Α͏ʹ ࢦఆ

Slide 31

Slide 31 text

[Unit] Description=app After=docker.service Requires=docker.service [Service] Restart=always TimeoutStartSec=0 ExecStartPre=-‐/usr/bin/docker kill app-‐%i ExecStartPre=-‐/usr/bin/docker rm app-‐%i ExecStartPre=/usr/bin/docker pull … ExecStart=/usr/bin/docker run —name app-‐%i … ExecStop=/usr/bin/docker stop app-‐%i [X-‐Fleet] Conflicts=app@*.service [email protected]:

Slide 32

Slide 32 text

ϞχλϦϯά • ίϯςφ͝ͱʹϞχλϦϯά͠ͳ͍ͱ͍͚ͳ͍ • ίϯςφ໊Λ͚ͭͣʹMackerelͰ؂ࢹͯͨ͠Β  σϓϩΠͷ౓ʹ؂ࢹର৅͕૿͑ͯ΍͹͔ͬͨ • DataDogͰͲ͏ʹ͔ͳͬͯΔ

Slide 33

Slide 33 text

ϞχλϦϯά • ͳʹΛϞχλϦϯά͠ͳ͍ͱ͍͚ͳ͍ͷ͔ • αʔϏεϨϕϧ • ϊʔυϨϕϧ • ίϯςφϨϕϧ • ͳΜͰ΋ݟͨ͘ͳΔ͚Ͳຊ౰ʹશ෦ݟͳ͍ͱ  ͍͚ͳ͍ͷ͔ߟ͑Δඞཁ͕͋Δ • 1ϊʔυ=1ίϯςφͰಈ͔͢ͷͰ͋Ε͹ࠓ·ͰͲ͓Γ ͷϞχλϦϯάͱͦΜͳʹมΘΒͳ͍

Slide 34

Slide 34 text

ϩάసૹ • ֤ϊʔυʹtd-agentͷίϯςφΛཱ͍ͯͯΔ • cloud-conﬁgʹॻ͍ͯىಈͯ͠΋ྑͦ͞͏

Slide 35

Slide 35 text

ো֐ରԠ • ࠓͷͱ͜Ζϗετʹґଘ͢Δো֐ʹૺ۰ͯ͠ͳ͍ • ϋʔτϏʔτ͕ಧ͍ͯΔঢ়گͩͱLB͔ΒखಈͰ  ֎ͨ͠Γ͠ͳ͍ͱ͍͚ͳ͍͔΋ʁ

Slide 36

Slide 36 text

ࣗಈΞοϓάϨʔυ • ϊʔυ͕མͪΔ͚ͩͱࢥ͑͹ϊʔυো֐ͱมΘΒͳ͍ • ϩοΫΛ͔͚ͳ͕Β΍ͬͯ͘ΕΔͷͰϩʔϦϯάΞο ϓάϨʔυ͞ΕΔʢฒྻ਺΋ࢦఆͰ͖Δʣ • ﬂeetͷόʔδϣϯ্͕͕ͬͨͱ͖ʹ؅ཧϗετଆͷ ﬂeetctl͕ߋ৽͞Εͯͳࣦͯ͘ഊͨ͜͠ͱ͕͋ͬͨ

Slide 37

Slide 37 text

ࣗಈΞοϓάϨʔυ • ৽͍͠΍ͭΛ࢖͍͍͚ͨͲAlpha͸όάͬͯΔͷͰ  Ͳ͏͠Α͏ • ࣗ෼ͰϏϧυͨ͠Βҙຯͳ͍͠

Slide 38

Slide 38 text

ωοτϫʔΫ • ಛʹωοτϫʔΫΛ੾ΔϞνϕʔγϣϯ͸ͳ͍ • ϙʔτϑΥϫʔσΟϯάͰͦΕ΄Ͳ໰୊ͳ͍ • ϗετωοτϫʔΫͰ΋ྑ͍

Slide 39

Slide 39 text

ηΩϡϦςΟ • CoreOS͕໘౗Έͯ͘ΕΔͷ͸ϗετOS͚ͩ • ίϯςφͷ੬ऑੑ͸ؔ܎ͳ͍ • Dockerͩͱίϯςφʹόʔδϣϯ͕ͳ͍ͷͰ͍ͬͣ͜ ͭ֬ೝ͠ͳ͍ͱ͍͚ͳ͍

Slide 40

Slide 40 text

·ͱΊ • ݁ہͳΜ΍͔Μ΍ඞཁʹͳΔ • ͦΕKubernetesͰ(ུ • ͦΕGCPͰ(ུ • ίϯςφ͡Όͳͯ͘AMIͰ΋… • Ͳ͜·Ͱ΍Δඞཁ͕͋Δ͔ߟ͑Δඞཁ͕͋Δ

Slide 41

Slide 41 text

·ͱΊ • ίϯςφΛಈ͔͚ͩ͢ͷϗετͰ͋Ε͹CoreOS͸ ͪΐ͏Ͳ͍͍େ͖͞ • े෼ʹγϯϓϧ • γϯϓϧ͚ͩʹߟ͑ͳ͍ͱ͍͚ͳ͍͜ͱ͸͋Δ • ຊ౰ʹKubernetesΛࣗ෼ͰϚωδϝϯτ͢Δͷʁ