Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
noteのサービスインフラの “式年遷宮” の取り組み #notetechmeetup
Search
varu3
April 22, 2022
Programming
0
240
noteのサービスインフラの “式年遷宮” の取り組み #notetechmeetup
note tech meetup #2 エンジニアLT大会でLTしたスライドです
Youtube:
https://www.youtube.com/watch?v=U5Z23eFyCL0
varu3
April 22, 2022
Tweet
Share
More Decks by varu3
See All by varu3
CNDT2023: noteのEKS移設、ゼンブ見せます
varu3
3
8.2k
大規模障害から見るAWSのバックエンド #awswakaran_tokyo
varu3
6
4.4k
Other Decks in Programming
See All in Programming
Site Reliability Engineering for GMO
pyama86
8
1k
Goのmultiple errorsについて (2024年4月版)
syumai
4
930
Komplexe Oberflächen mit SVG und der Web Animation API
joergneumann
0
670
SIMD Parallel Programming with the Vector API
josepaumard
0
180
見た目から始める生産性向上
ikumatadokoro
7
850
Prepare for Jakarta EE 11 - Performance and Developer Productivity
ivargrimstad
0
820
2 週間で Twitter Bot を作ってみた
contour_gara
0
570
StoreKit2によるiOSのアプリ内課金のリニューアル
kangnux
0
110
DMMプラットフォームがTiDB Cloudを採用した背景
pospome
8
4.1k
GitHub Actionsで泣かないためにやっておきたい設定 / Recommended GHA settings to avoid crying
pinkumohikan
3
540
使ってみよう Azure AI Document Intelligence
kosmosebi
2
320
⼤規模⾔語モデルの拡張(RAG)が 終わったかも知れない件について
nearme_tech
23
15k
Featured
See All Featured
The Invisible Side of Design
smashingmag
294
49k
Stop Working from a Prison Cell
hatefulcrawdad
266
19k
The Cult of Friendly URLs
andyhume
74
5.7k
Music & Morning Musume
bryan
41
5.6k
The Mythical Team-Month
searls
216
42k
Done Done
chrislema
178
15k
Large-scale JavaScript Application Architecture
addyosmani
504
110k
What's in a price? How to price your products and services
michaelherold
237
11k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
9
8.3k
Atom: Resistance is Futile
akmur
259
25k
How STYLIGHT went responsive
nonsquared
92
4.8k
Statistics for Hackers
jakevdp
789
220k
Transcript
noteͷαʔϏεΠϯϑϥͷ “ࣜભٶ” ͷऔΓΈ
2 גࣜձࣾυϦίϜͰ֤छιʔγϟϧήʔϜͷΠϯϑϥΛ୲ͨ͠ޙɺ2020 ΑΓnoteגࣜձࣾʹೖࣾɻೖࣾޙAWSͷίετݮɺnoteͷػೳ͕ ಈ͍͍ͯΔ֤छαʔϏεͷγεςϜઃܭɺϝτϦΫεɾΤϥʔࢹͷඋ σϓϩΠαΠΫϧͷվળͳͲϓϩμΫτ։ൃҎ֎ͷԼճΓ෦ΛΑΓΑ ͘͢Δ͜ͱʹऔΓΜͰ͍·͢ɻ झຯ • ొࢁɾτϨΠϧϥϯχϯά ◦
͜ͳ͍ͩτϨϥϯͷେձʹग़ͯ15kmʢதڃίʔεʣΛͬͯ ͖·ͨ͠ɻΛ௧Ίͯݱࡏɺྍཆத…ɻ͖ͬ͞ࢄาͨ͠Β௧ SRE-T தଜ Ꮺ ʢvaru3ʣ
note inc. noteʹ͓͚ΔγεςϜͷ ”ࣜભٶ” ͱ 3
note inc. noteͷΤϯδχΞνʔϜ͕2022ʹ͚ͯઓ͢Δɺॏཁ՝9બ 4 https://note.jp/n/n39681098a0d2 • ࡢʹൃද͞Εͨ note هࣄ •
ʮίϯςφج൫ͷҠߦΛ࣮ࢪ͠ӡ༻ ޮԽɾলྗԽΛ࣮ݱʯ͕ॏཁ՝ͷ Ұͭͱͯ͠ڍ͛ΒΕͨ • ࡢ͔ΒϓϩδΣΫτ͕ελʔτ ͠ɺSREνʔϜΛத৺ʹঃʑʹҠߦ͠ ͍ͯΔ
note inc. ҰൠతͳίϯςφԽͷϝϦοτ 5 • σϓϩΠαΠΫϧͷվળ ◦ CI/CDΛར༻ͨ͠σϓϩΠͷࣗಈԽ ◦ Blue-GreenσϓϩΠɺCanaryσϓϩΠ
• ։ൃڥͷվળɺӡ༻ͷ༰қԽɺetc… ◦ ։ൃڥΛ༰қʹߏஙͰ͖Δ ◦ KubernetesͷΈΛར༻ͯ͠ӡ༻ͷࣗಈԽ
note inc. ͦΕ͚ͩʁ 6 ϓϩδΣΫτΛਐΊ͍ͯ͘தͰɺͦΕҎ֎ͷϝϦοτݟ͖͑ͯͨ
note inc. ιϑτΣΞࣜભٶͱ͍͏ߟ͑ํ 7 • ࣜભٶ ◦ ҏਆٶͰߦΘΕΔɺ20͓͖ʹશͯͷ఼ࣾΛΓସ ͑Δߦࣄͷ͜ͱ ◦
༻ͷ૿Ճ ٕज़ͷܧঝ ͷͨΊ • γεςϜͰಉ͜͡ͱ͕͍͑Δ ◦ AWS ᴈ໌ظ͔Βӡ༻͞Ε͍ͯΔྺ࢙͋Δ note ͷγες Ϝ ◦ ࣌ͱࠓͰϕετϓϥΫςΟε͕ҟͳΔ෦͋Δ
note inc. ιϑτΣΞࣜભٶͱ͍͏ߟ͑ํ 8 • ʮnote ͬͯͲ͏ͬͯಈ͍͍ͯΔΜ͚ͩͬʁʯ ʮͳΜͰ͜͏ͳ͍ͬͯΔΜ͚ͩͬʁʯΛਂ͘ ΓɺαʔϏεͷશ༰ΛखதʹऩΊΔ͜ͱ͕Ͱ͖Δ •
ͨͩ "γεςϜΛҠߦ͢Δ" ͚ͩͰͳ͘ɺݹ͔͘ Β͋ΔγεςϜશମͷঠѲͱϦϑΝΫλϦϯάΛ ಉ࣌ʹਐΊ͍ͯΔ
note inc. γεςϜશମͷঠѲͱϦϑΝΫλϦϯάͷͨΊʹ͍ͬͯΔ͜ͱ • ωοτϫʔΫߏͷ৽ • Τϥʔ௨ / ϞχλϦϯά /
ϩΪϯάڥͷཧ • ύϑΥʔϚϯεΛܭଌ͠ɺεέʔϧΞτΩϟύγςΟͷݟ͠ • σϓϩΠͱ҆ఆੑͷվળ • rakeλεΫͰ࣮ߦ͞ΕΔόονॲཧͷཧ • ΘΕ͍ͯͳ͍(͔͠Εͳ͍) gem Λ୨Է͠ • ։ൃʹඞཁͳڥΛύοͱ࡞ΕΔΑ͏ʹ • طଘͷCIڥ(Jenkins) ʹґଘ͍ͯ͠Δ෦Λղফ • etc…
note inc. EKSҠઃϓϩδΣΫτ 10
note inc. noteͷEKSҠઃϓϩδΣΫτͷ֓ཁ 11 • noteͷ֤ΞϓϦέʔγϣϯͷ࣮ߦج൫ΛEKS(Kubernetes)ʹҠઃ͢Δ • noteෳαʔϏεʹ͔ΕͯEC2্ͰΞϓϦέʔγϣϯ͕࣮ߦ͞Ε͍ͯΔ ◦ ϑϩϯτΤϯυ:
Nuxt.js ◦ όοΫΤϯυ: Ruby on Rails ◦ όον: Ruby on Rails (rakeλεΫ) ◦ ඇಉظϫʔΧʔ: Sidekiq • ͜ΕΒΛॱ൪ʹEKSҠઃ͍͖ͯ͠ɺ noteͷαʔϏεશମͷӡ༻ޮԽɾলྗԽΛ࣮ݱ͢Δ
note inc. EKSΫϥελͷߏ • ୯ҰΫϥελͷϚϧνςφϯτߏΛ࠾ ༻ ◦ ΞϓϦέʔγϣϯͷׂ͝ͱʹ NodeGroupΛ͚Δ ◦
namespaceͱNodeGroup1:1ͱ͠ɺ nodeselectorͰىಈ͢ΔNodeΛ੍ޚ͢Δ ◦ PodIPΞυϨεͷރׇΛ͙ͨΊɺη ΧϯμϦCIDR(100.XX.XX.XX) Ͱىಈ
note inc. ֤αʔϏε͝ͱͷҠઃॱ όον: Ruby on Rails(rakeλεΫ + cron) όοΫΤϯυ:
Ruby on Rails ϑϩϯτΤϯυ: Nuxt.js Worker: Ruby on Rails(Sidekiq) Ҡߦ қ Ҡߦॱ ※ఆࠁʹॲཧ͕࣮ߦ͢ΔͨΊͷγεςϜ ΠϚίί ߴ
note inc. 202110݄~ ~20226݄ 1. ཁ݅Λຬͨͨ͢Ίͷػೳ ։ൃ 2. ࢹɾΞϥʔτݕͷ ΈΛߏங
3. όονͷςετɾ֬ೝ࡞ۀ • OneshotδϣϒͰͷόονͷ࣮ߦ • betaڥͷσϓϩΠ όονͷҠઃਐḿ • PodͷΞϥʔτݕ (Datadog) • ΞϓϦέʔγϣϯ ͷΞϥʔτݕ (Sentry) • OneshotδϣϒΛ ࣮ߦͰ͖Δڥͷ ։ൃ • cronjob-managerͷ ։ൃ 4. Ҡઃ • ຊ൪ڥͷσϓϩΠ • σϓϩΠޙͷࢹ
note inc. 1. ཁ݅Λຬͨͨ͢Ίͷ ػೳͷ։ൃ 15
note inc. OneshotδϣϒΛ࣮ߦͰ͖Δڥ: k8s-job-executor • Slackbot্͔ΒίϚϯυΛ࣮ߦͰ͖Δػ ೳΛ։ൃͨ͠ • ίϚϯυΛ࣮ߦ͢ΔͱɺPod্ཱ͕͕ͪ ΓίϚϯυΛ࣮ߦ͢Δ
• ग़ྗCloudWatch Logsʹूͯ͠ɺ Managed Service for GrafanaͰࢀর͢Δ • GrafanaͰPod͝ͱɺΞϓϦέʔγϣ ϯ͝ͱʹϩάͷࢀর͕Մೳ
note inc. cronjob-manager(Kubernetes Custom Controller) • KubernetesͰCronJob͝ͱ(1 batch͝ͱʣʹ 1 manifest
Λ࡞ ͢Δඞཁ͕͋Δ • ҰͭͷίϯςφΠϝʔδΛ࡞͢ΔͨͼʹશͯͷCronJobͷΠ ϝʔδΛॻ͖͑Δͷେม…(ҎલγΣϧܳΛۦ͍ͯ͠ ͨʣ • cronjob-managerͱ͍͏Kubernetes Custom ControllerΛ։ൃ • ҰͭͷmanifestͰෳͷCronJobΛ੍ޚ͢Δ͜ͱ͕Ͱ͖Δ • CronJobManagerΛ࡞͢ΔͱCronJob͕ࣗಈͰ࡞ΒΕΔ
note inc. 2. ࢹɾΞϥʔτݕͷ ΈΛߏங 18
note inc. Ξϥʔτݕ • Node, PodϨϕϧͷΞϥʔτͱΞϓϦέʔγϣϯϨϕϧͷΞϥʔτͰݕ ख๏Λ͚Δ • Node, PodϨϕϧͷΞϥʔτ
◦ DatadogΛར༻͢Δ ◦ DatadogAgentΛDaemonsetͱͯ͠σϓϩΠͯ͠ɺϝτϦΫεΛࢹ • ΞϓϦέʔγϣϯϨϕϧͷΞϥʔτ ◦ SentryΛར༻͢Δ ◦ ΞϓϦέʔγϣϯͰग़ͨbacktraceΛৄࡉʹੳͰ͖Δ • Կ͔͕ൃੜͨ͠߹ʹ͍ͣΕ͔ͷΞϥʔτͰݕ͞ΕΔ
note inc. Ξϥʔτݕ: ΞΫγϣϯϨϕϧ • ࠓ·Ͱ1ͭͷchannelʹਨΕྲྀ͠ʹͳ͍ͬͯͨΞϥʔτΛΞΫγϣϯϨϕϧ͝ͱʹཧ • ֤Ϩϕϧຖʹ̏ͭͷSlack channelʹͦΕͧΕ௨͢ΔΑ͏ʹͨ͠ info
• ରԠෆཁͳΞϥʔτ • جຊతʹใΛऩू ͢Δ͚ͩ • োൃੜ࣌ಈ࡞֬ ೝͷͨΊʹར༻͢Δ ใ warn • ཌ·ͰʹରԠ͕ඞ ཁͳΞϥʔτ • ۓٸͷରԠෆཁͩ ͕์ஔ͢ΔͱϢʔ βʔʹӨڹ͕ग़͔Ͷ ͳ͍ͷ critical • ଈ࣌ରԠ͕ඞཁͳΞϥʔτ • ։ൃΤϯδχΞશһ͕ΈΔ • Կ͔͠ΒϢʔβʔʹӨڹ͕ ग़࢝Ί͍ͯΔՄೳੑ͕͋Δ ͷ
note inc. Ξϥʔτݕ: criticalอશ࡞ۀ • critical = Ҭͱͯ͠ɺৗʹ༨ܭͳΞϥʔτ͕ඈͳ͍Α͏ʹҙΛ͏எ·͵ྗ͕ඞཁ • ༨ܭͳΞϥʔτ͕૿͍͑ͯ͘ͱɺΞϥʔτʹରͯ͠ԿΛ͍͍͔ͯ͠Θ͔Βͳ͘ͳΔ
• ͍ͦͯͣ͠Εɺ୭ͦͷνϟϯωϧΛݟͳ͘ͳ͍ͬͯ͘… • ͦ͏ͳΒͳ͍Α͏ʹɺᮢΞϥʔτఆٛΛఆظతʹݟ͠ɺৗʹదͳΞϥʔτͱͳΔΑ͏ ৺͕͚Δ • ͞ΒʹΑ͘͢Δʹʢ՝ʣ ◦ ि࣍ͷϨϙʔςΟϯάͰ͜ΜͳΞϥʔτ͕Կ݅ى͖͍ͯͨͱ͍͏ͷΛαϚϦԽ ◦ ΞϥʔτΛఆ؍ଌ͠ɺΞϥʔτͷ૿ݮΛৗʹҙ͍ࣝͯ͘͠
note inc. 3. όονͷςετɾ֬ೝ࡞ۀ 4. Ҡઃ 22
note inc. όονͷςετɾ֬ೝ࡞ۀ batchͷதΛ֬ೝ k8s-job-executorͰ࣮ ߦ͢Δ on beta Τϥʔ͕ग़͍ͯͳ͍͔ ϩάΛ֬ೝ͢Δ
࣮ߦ࣌ؒͷڍಈΛ֬ ೝ cronjob-managerͰσ ϓϩΠ͢Δ on beta cronjob-managerͰσ ϓϩΠ͢Δ on ຊ൪ ͋Δఔͨ·ͬͨΒ 1 batch͝ͱʹ͜ͷα ΠΫϧΛ܁Γฦ͢ • batchͷશ෦Ͱ300 • SRE-TͱϓϩμΫτ։ൃνʔϜͰ ख͚ͯ͠ɺҰͭͣͭ֬ೝ࡞ۀΛ ࣮ࢪத • priorityͷ͍ͷ͔Βॱ࣍ɺEKS ্Ͱಈ͔͍ͯ͠Δ
note inc. 24 47 / 295
note inc. ·ͱΊ 25
note inc. EKS(Kubernetes)ʹͯ͠Α͔ͬͨͱ͜Ζ Jobͷ࣮ߦཤྺ͕͙֬͢ೝͰ͖Δ • kubectl get jobs ίϚϯυͰۙʹ࣮ߦ ͞ΕͨJobͷཤྺ͕͙͢ʹ֬ೝͰ͖Δ
• աڈͷ࣮ߦϩάͳͲͷৄࡉΛJob͝ͱ ʹݸผʹ֬ೝ͢Δ͜ͱ͕Ͱ͖Δ • ࣦഊͨ͠߹ͷ࠶࣮ߦ༰қ Job͝ͱʹࡉ੍͔͍ޚ͕Մೳ • Job͝ͱʹCPU / Memoryͷ্ݶϦ ΫΤετྔΛࢦఆͰ͖Δ • ࠶࣮ߦճฒྻ࣮ߦͳͲͷࡉ͔͍ ੍ޚ͕Մೳ • ಛఆͷJob͚ͩFargateͰ࣮ߦ͢Δɺͳ Ͳͷ͜ͱՄೳʢࠓޙͷ՝ʣ
note inc. noteͷ”ࣜભٶ”ͰͷϝϦοτ • ӡ༻ͷརศੑ ◦ ΄΅શͯͷಈ࡞ΛSlack͔Βૢ࡞͢Δ͜ͱ͕Ͱ͖ΔΑ͏ʹͨ͜͠ͱͰརศੑ্͕ͨ͠ (ίϯςφͷϏ ϧυɺίϚϯυͷ࣮ߦɺϩάͷࢀরʣ ◦
ϩάͷࢀরݕࡧͳͲ͕Grafanaͷ֎෦πʔϧͰ֬ೝͰ͖ΔΑ͏ʹͳͬͨ • োͷൃݟੑ্͕ ◦ DatadogSentryͷϞχλϦϯάڥΛ͑ͨ͜ͱͰɺCronJobͷಈ࡞ʹ͕͋ͬͨ߹ʹ͙͢ʹݕ ͢Δ͜ͱ͕Ͱ͖ΔΑ͏ʹͳͬͨ ◦ ֤ΞϥʔτΛΞΫγϣϯϨϕϧ͝ͱʹཧ͠ɺΦϯίʔϧڥ͕վળͨ͠ • όονॲཧࣗମͷվળɾݟ͠ ◦ ςετσʔλͷͳ͍ͷɺదͳϩά͕ग़ྗ͞Ε͍ͯͳ͍ͷͳͲΛվળ
note inc. ·ͩ·ͩ͜Ε͔Β͕ ՂڥͰ͕͢ ҆ఆͨ͠αʔϏεΛఏڙͰ͖ΔΑ͏ ʹؤு͍͖ͬͯ·͢ 28