Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
CoreOSで運用するために考えないといけないこと / CoreOS in pixiv
Search
Harukasan
PRO
April 09, 2015
Technology
8k
23
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
CoreOSで運用するために考えないといけないこと / CoreOS in pixiv
CoreOS Meetup Tokyo #1
2015-04-09
Harukasan
PRO
April 09, 2015
Transcript
41 slides
CoreOSͰӡ༻͢ΔͨΊʹ ߟ͑ͳ͍ͱ͍͚ͳ͍͜ͱ Harukasan / MICHII Shunsuke CoreOS Meetup Tokyo #1
2015-04-09
Harukasan / MICHII Shunsuke • 2012ʹ৽ଔͱͯ͠ΠϯϑϥνʔϜʹଐ • ίϯςϯπ৴Λ͡Ίͱͯ͠αʔϏεશମͷج൫Λ୲ • 16Gbps͘Β͍ͷը૾৴
• αʔϏεϑϩϯτ(nginx) • Fluentd / Kibana / BigQuery • ٕज़ܥࡶࢽͷࣥචͳͲ
beer@harukasan http://harukasan.jp/beers • ौ୩ͰϏʔϧΛҿΉΠϕϯτΛ͍ͬͯ·͢ • ࠓ݄ͷΠϕϯτ໌༻ҙ͠·͢ʢͨͿΜ23ʁʣ
͡Ίʹ • pixivʹ͓͚ΔCoreOS • ͳʹ͕͍ͨ͠ͷ͔ • CoreOSΛ͏ͱԿΛߟ͑ͳ͍ͱ͍͚ͳ͍ͷ͔ • ͜͜1ϲ݄͘Β͍΄ͱΜͲ৮ͬͯͳ͍……
CoreOS in pixiv
CoreOS in pixiv • Relase 554͔Β͍࢝Ίͨ • Ұ෦ͷϓϩμΫγϣϯڥͰ༻ • IDCFΫϥυ্ʹߏங
• ࠷খߏ • ·͍ͩΖ͍Ζͱࢼ͍ͯ͠Δͱ͜Ζ
• pixivͷອը࡞͔ΒΦεεϝ࡞Λ ӾཡɺݕࡧͰ͖ΔAndroid͚ΞϓϦ • 20152݄4ϦϦʔε • Build with Play framework
/ Scala ΦεεϝϚϯΨΛ΄΅ຖߋ৽ʂ pixivϚϯΨ
App App App App LB RPC RPC RPC RPC LB
pixiv DBs manga-app DB CoreOS CoreOS CoreOS CoreOS ΦϯϓϨϛε IDCF Cloud Request flow Manage Manage L3 Private connect
App App App App LB RPC RPC RPC RPC LB
Manage pixiv DBs manga-app DB CoreOS Manage CoreOS CoreOS CoreOS fleetctl Jenkins pull pploy Deployment flow ΦϯϓϨϛε IDCF Cloud L3 Private connect
App App App App LB RPC RPC RPC RPC LB
pixiv DBs manga-app DB CoreOS Manage CoreOS CoreOS CoreOS Monitoring flow ΦϯϓϨϛε IDCF Cloud L3 Private connect td-agent td-agent td-agent td-agent dd-agent dd-agent dd-agent dd-agent Fluentd Logs Munin Nagios
ਓͷԹ͔ΈΛײ͡ΔσϓϩΠ • cloud-configͰઃఆ͕ྲྀ͠ࠐ·ΕΔ • σϓϩΠ৬ਓʹΑΔϩʔϦϯάσϓϩΠ • fleetctlͰϦελʔτ͢Δ͜ͱͰσϓϩΠ͢Δ • ࣗಈԽ͍ͨ͠ export
FLEETCTL_ENDPOINT=http://app-‐1:4001/ NUM=1 fleetctl stop app-‐a@${NUM}.service fleetctl start app-‐a@${NUM}.service
ͳͥCoreOSͳͷ͔
ͳͥCoreOSΛ࠾༻ͨ͠ͷ͔ • Play framework / ScalaͷΞϓϦέʔγϣϯͷґଘ ཧͱ͔ཧ͢Δͷ͕໘ͩͬͨͷͰίϯςφʹด͡ ࠐΊ͔ͨͬͨ • ίϯςφ͔͠ಈ͔͞ͳ͍ϗετΛӡ༻͢Δͷ͕໘
• ίϯςφΛಈ͔͚ͩͩͬͨ͢ΒCoreOSͰྑ͍ͷͰ
ͳʹ͕͍ͨ͠ͷ͔ • ΠϯϑϥνʔϜͱࣗΒͷࣄΛͳ͘͠ଓ͚Δ͜ͱ ͕৬ͱͯ͠ཁٻ͞ΕΔࣗݾໃ६ͨ͠ଘࡏͰ͋Δ • ͨͩ͠ؾΛൈ͘ͱࣄ͕૿͑ଓ͚ͯࢮ͵ • ΊΜͲ͍͘͜͞ͱͰ͖Δ͚ͩΓͨ͘ͳ͍
ͳΔ͘ঢ়ଶΛؾʹͨ͘͠ͳ͍ • όʔδϣϯͷࠩҟ • ੬ऑੑ • ґଘؔ • Ϣʔβੜσʔλ •
etc.
ͳΔ͘ঢ়ଶΛؾʹͨ͘͠ͳ͍ • Immutable • ίϯςφҎ֎ͷঢ়ଶΛม͑ͳ͍ • /usrҎԼͦͦॻ͖͑Ͱ͖ͳ͍ • Disposable •
ίϯςφΛ͍ͭͰࣺͯΔ͜ͱ͕Ͱ͖Δ • ͳΜ͔͓͔͘͠ͳͬͨΒࣺͯΕྑ͍
CoreOSΛͲ͏ଊ͍͑ͯΔͷ͔ • systemd + etcd/fleet/docker • αʔϏεཧͯ͢systemd͕ߦ͏ • etcdΛ͔ͭͬͯΫϥελϦϯά͢Δ •
fleetΛ͔ͭͬͯαʔϏεΛσϓϩΠ͢Δ • docker/rktΛ͔ͭͬͯίϯςφΠϝʔδΛཧ͢Δ • ͦͷ͏ͪউखʹΞοϓάϨʔυ͞ΕΔ • ͜ΕͰेͩͬͨΒCoreOSબࢶʹͳΔ
CoreOSΛͲ͏ଊ͍͑ͯΔͷ͔ • systemd + etcd/fleet/docker • αʔϏεཧͯ͢systemd͕ߦ͏ • etcdΛ͔ͭͬͯΫϥελϦϯά͢Δ •
fleetΛ͔ͭͬͯαʔϏεΛσϓϩΠ͢Δ • docker/rktΛ͔ͭͬͯίϯςφΠϝʔδΛཧ͢Δ • ͦͷ͏ͪউखʹΞοϓάϨʔυ͞ΕΔ • ͜ΕͰेͩͬͨΒCoreOSબࢶʹͳΔ • ͜ΕҎ্ඞཁͳΒKubernetes͕ඞཁʹͳΔ
CoreOSΛͲ͏ଊ͍͑ͯΔͷ͔ • ͭʹSSHͰ͖ͳ͍༁Ͱͳ͍ • ύοέʔδཧγεςϜ͍Βͳ͍ • ͍͍ͩͨͷ͜ͱsystemd͕Δ • ։ൃ࣌ʹDockerίϯςφΛಈ͔͢Πϯελϯε ͱͯ͠ศར
Container Deployment • όΠφϦʹؔ࿈͢Δͯ͢ͷґଘؔΛίϯςφͷத ʹด͡ࠐΊΔ • ίϯςφͷ֎ͷঢ়ଶؾʹ͢Δඞཁ͕ͳ͍ /etc/ *****.conf …
/usr/local/bin/ ***** … /usr/lib/ **** … /var/ **** … Container
Container Deployment • ґଘؔΛ։ൃऀ͕ཧͰ͖Δ • ԾΠϯελϯεΈ͍ͨʹͨ͘͞ΜͷϊʔυΛ1ͭͷ ཧϚγϯʹಉډͤ͞Δͷͱͪΐͬͱҧ͏ • ͋͘·ͰґଘؔΛύοέʔδϯάͨ͠1ϓϩηε ͱࢥͬͨΒΘ͔Γ͍͢
Web Application in a Container • ΞϓϦέʔγϣϯϓϩηείϯςφʹ͍͍ͯΔ • 1ϓϩηεʢجຊతʹʣ •
ґଘ͢ΔϥΠϒϥϦ͕ଟ͍ • ϝϞϦҎ֎ʹঢ়ଶΛ࣋ͨͳ͍ • LBͰϦΫΤετΛεΠονͰ͖Δ
CoreOSΛ͏্Ͱߟ͑ͳ͍ͱ ͍͚ͳ͍͜ͱ
CoreOSΛ͏্Ͱߟ͑ͳ͍ͱ ͍͚ͳ͍͜ͱ • CoreOSͱ͍͏͔ɺͲ͏ͬͯ֎෦ʹґଘ͠ͳ͍ঢ়ଶ Λͭ͘Δ͔ • ·ͩ͏·͍ํ๏͕ݟ͔ͭͬͯͳ͍͜ͱ͕ଟ͍ • γεςϜΦʔέετϨʔγϣϯ •
σϓϩΠϝϯτ • ϞχλϦϯά • োରԠ • ࣗಈΞοϓάϨʔυ
etcdʹ͓͚Δ࠷খߏ • ΫϥελΛ࡞Δͱ͖࠷Ͱ4ʹ͓ͯ͘͠ • 3ͩͱ1མ͚ͪͨͩͰεϓϦοτϒϨΠϯ
͏1ඞཁ
γεςϜΦʔέετϨʔγϣϯ • fleetͰͰ͖Δͷجຊతʹʮ͜ͷsystemdαʔϏεΛ ΫϥελͰ͍ͭ͘ಈ͔͔͢ʯͱ͍͏͜ͱ͚ͩ • αʔϏεϝϯόͱ͔Ϛελϊʔυͱ͔Λཧ͢Δػೳ ͳ͍ • ͪΖΜΦʔτεέʔϦϯάͱ͔ͳ͍ •
etcdΛͬͯࣗͰ͕ΜΔʁ • KubernetesͰΔʁ • LB͘Β͍͏ͪΐͬͱ؆୯ʹΓ͍ͨ
ίϯςφʹର͢Δ ϩʔυόϥϯγϯά • ίϯςφͷstart/stop࣌ʹLBͷAPIΛୟ͘ • systemdͷαʔϏεϑΝΠϧʹίϚϯυΛՃ͢Δ https://github.com/coreos/elb-presence • LBͷAPIΛୟ͚ͩ͘ͳͷͰൺֱత؆୯ •
ϋʔτϏʔτνΣοΫͰ֎ΕΔ͚ͩͰྑ͍
ίϯςφʹର͢Δ ϩʔυόϥϯγϯά • etcdΩʔ͕มߋ͞Εͨͱ͖ʹϑοΫͰ͖Δ • ϑοΫͯ͠LBͷઃఆΛม͑Εྑͦ͞͏ • vulcand https://github.com/mailgun/vulcand •
confd + nginx https://github.com/kelseyhightower/confd
σϓϩΠϝϯτ • fleetͩͱϩʔϦϯάϦελʔτͰ͖ͳ͍ • ϩʔϦϯάϦελʔτ͢ΔΑ͏ͳΈΛͭ͘Δʹ ࣗͰεΫϦϓτΛॻ͔͘͠ͳ͍ • ଞͷίϯςφ͕ಈ͍͍ͯΔϊʔυͰಈ͔ͳ͍Α͏ʹ ࢦఆ
[Unit] Description=app After=docker.service Requires=docker.service [Service]
Restart=always TimeoutStartSec=0 ExecStartPre=-‐/usr/bin/docker kill app-‐%i ExecStartPre=-‐/usr/bin/docker rm app-‐%i ExecStartPre=/usr/bin/docker pull … ExecStart=/usr/bin/docker run —name app-‐%i … ExecStop=/usr/bin/docker stop app-‐%i [X-‐Fleet] Conflicts=app@*.service
[email protected]
:
ϞχλϦϯά • ίϯςφ͝ͱʹϞχλϦϯά͠ͳ͍ͱ͍͚ͳ͍ • ίϯςφ໊Λ͚ͭͣʹMackerelͰࢹͯͨ͠Β σϓϩΠͷʹࢹର͕૿͔͑ͯͬͨ • DataDogͰͲ͏ʹ͔ͳͬͯΔ
ϞχλϦϯά • ͳʹΛϞχλϦϯά͠ͳ͍ͱ͍͚ͳ͍ͷ͔ • αʔϏεϨϕϧ • ϊʔυϨϕϧ • ίϯςφϨϕϧ •
ͳΜͰݟͨ͘ͳΔ͚Ͳຊʹશ෦ݟͳ͍ͱ ͍͚ͳ͍ͷ͔ߟ͑Δඞཁ͕͋Δ • 1ϊʔυ=1ίϯςφͰಈ͔͢ͷͰ͋Εࠓ·ͰͲ͓Γ ͷϞχλϦϯάͱͦΜͳʹมΘΒͳ͍
ϩάసૹ • ֤ϊʔυʹtd-agentͷίϯςφΛཱ͍ͯͯΔ • cloud-configʹॻ͍ͯىಈͯ͠ྑͦ͞͏
োରԠ • ࠓͷͱ͜Ζϗετʹґଘ͢Δোʹૺ۰ͯ͠ͳ͍ • ϋʔτϏʔτ͕ಧ͍ͯΔঢ়گͩͱLB͔ΒखಈͰ ֎ͨ͠Γ͠ͳ͍ͱ͍͚ͳ͍͔ʁ
ࣗಈΞοϓάϨʔυ • ϊʔυ͕མͪΔ͚ͩͱࢥ͑ϊʔυোͱมΘΒͳ͍ • ϩοΫΛ͔͚ͳ͕Βͬͯ͘ΕΔͷͰϩʔϦϯάΞο ϓάϨʔυ͞ΕΔʢฒྻࢦఆͰ͖Δʣ • fleetͷόʔδϣϯ্͕͕ͬͨͱ͖ʹཧϗετଆͷ fleetctl͕ߋ৽͞Εͯͳࣦͯ͘ഊͨ͜͠ͱ͕͋ͬͨ
ࣗಈΞοϓάϨʔυ • ৽͍ͭ͠Λ͍͍͚ͨͲAlphaόάͬͯΔͷͰ Ͳ͏͠Α͏ • ࣗͰϏϧυͨ͠Βҙຯͳ͍͠
ωοτϫʔΫ • ಛʹωοτϫʔΫΛΔϞνϕʔγϣϯͳ͍ • ϙʔτϑΥϫʔσΟϯάͰͦΕ΄Ͳͳ͍ • ϗετωοτϫʔΫͰྑ͍
ηΩϡϦςΟ • CoreOS͕໘Έͯ͘ΕΔͷϗετOS͚ͩ • ίϯςφͷ੬ऑੑؔͳ͍ • Dockerͩͱίϯςφʹόʔδϣϯ͕ͳ͍ͷͰ͍ͬͣ͜ ͭ֬ೝ͠ͳ͍ͱ͍͚ͳ͍
·ͱΊ • ݁ہͳΜ͔ΜඞཁʹͳΔ • ͦΕKubernetesͰ(ུ • ͦΕGCPͰ(ུ • ίϯςφ͡Όͳͯ͘AMIͰ… •
Ͳ͜·ͰΔඞཁ͕͋Δ͔ߟ͑Δඞཁ͕͋Δ
·ͱΊ • ίϯςφΛಈ͔͚ͩ͢ͷϗετͰ͋ΕCoreOS ͪΐ͏Ͳ͍͍େ͖͞ • ेʹγϯϓϧ • γϯϓϧ͚ͩʹߟ͑ͳ͍ͱ͍͚ͳ͍͜ͱ͋Δ • ຊʹKubernetesΛࣗͰϚωδϝϯτ͢Δͷʁ
More Decks by Harukasan
See All by Harukasan
Harucom Follow-up
harukasan
PRO
1
65
Building a Standalone Programming Environment
harukasan
PRO
1
2.2k
Successor to PicoRabbit: Ruby Programming Envorinment / RubyKaigi 2025 follow up
harukasan
PRO
1
1.4k
Write your own mrbgem, Create your own device
harukasan
PRO
1
540
PicoRabbit: a Tiny Presentation Device Powered by Ruby
harukasan
PRO
2
2k
pixivを支える技術 / 技育CAMPアカデミア
harukasan
PRO
3
670
20240401 新卒研修 - ピクシブにおける技術領域
harukasan
PRO
1
1k
ピクシブのコンテンツ配信基盤技術 / pixiv TECH SALON
harukasan
PRO
5
6k
Goにおける画像ファイル処理 / golang.tokyo #19
harukasan
PRO
7
6.9k
Other Decks in Technology
See All in Technology
kubellの最新データ分析環境
kubell_hr
2
210
そのイメージ、どのCIが作った? Artifact Attestationsで来歴を固定する / Which CI system created that image? Pinning provenance with Artifact Attestations
chmikata
0
180
「任されビリティ」で引き受けて「よしなりてぃ」に頼らず任せる / Accept tasks with "makasarebility" and hand them over without relying on "yoshinarily"
pauli
1
870
AIネイティブ組織の作り方。1人のPOとAIから始める、自社での実験
ama_ch
2
1.3k
現地速報! Microsoft Ignite アップデートレポート
kasada
0
390
AWS DevOps Agentの構造から読み解く運用 AI エージェントの設計原則/aws-devops-agent-design-principles
tomoki10
0
190
AI×アジャイルで、石橋を高速で叩きながら渡る~電力会社がゼロから築いた会員サイト改善プロセス~ / 20261112 Ayami Kanazawa & Shigeru Hirai
shift_evolve
PRO
1
110
_マネジメントする人される人ナイト_2026__良いチームを作っていきたい_お気持ち_.pdf
xkxaxkx
0
840
閉域網でのAPI公開の選択肢全部比較!
takano0131
4
670
OCI Artifact による自動運転タクシーに向けた OTA 基盤の構築
biwashi
1
180
人の努力に頼らないセキュリティをエンジニアリングする在り方 / An approach to engineering security that does not rely on human effort
carta_engineering
0
240
RAGの改善を速く回すための評価基盤
yosukeyoshida
6
3.4k
Featured
See All Featured
AI: The stuff that nobody shows you
jnunemaker
PRO
12
1.3k
A brief & incomplete history of UX Design for the World Wide Web: 1989–2019
jct
2
570
Agile Leadership in an Agile Organization
kimpetersen
PRO
0
320
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
360
31k
What does AI have to do with Human Rights?
axbom
PRO
1
2.5k
Stop Working from a Prison Cell
hatefulcrawdad
274
21k
SEO Brein meetup: CTRL+C is not how to scale international SEO
lindahogenes
2
3k
Amusing Abliteration
ianozsvald
1
400
Leading Effective Engineering Teams in the AI Era
addyosmani
9
2.9k
Marketing Yourself as an Engineer | Alaka | Gurzu
gurzu
0
390
B2B Lead Gen: Tactics, Traps & Triumph
marketingsoph
1
310
Code Review Best Practice
trishagee
74
20k