Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
noteのサービスインフラの “式年遷宮” の取り組み #notetechmeetup
Search
varu3
April 22, 2022
Programming
0
350
noteのサービスインフラの “式年遷宮” の取り組み #notetechmeetup
note tech meetup #2 エンジニアLT大会でLTしたスライドです
Youtube:
https://www.youtube.com/watch?v=U5Z23eFyCL0
varu3
April 22, 2022
Tweet
Share
More Decks by varu3
See All by varu3
CNDT2023: noteのEKS移設、ゼンブ見せます
varu3
3
9k
大規模障害から見るAWSのバックエンド #awswakaran_tokyo
varu3
6
4.6k
Other Decks in Programming
See All in Programming
Modern Angular: Renovation for Your Applications
manfredsteyer
PRO
0
240
Boost Performance and Developer Productivity with Jakarta EE 11
ivargrimstad
0
1.5k
イベント駆動で成長して委員会
happymana
1
260
外部システム連携先が10を超えるシステムでのアーキテクチャ設計・実装事例
kiwasaki
1
280
CPython 인터프리터 구조 파헤치기 - PyCon Korea 24
kennethanceyer
0
250
Duckdb-Wasmでローカルダッシュボードを作ってみた
nkforwork
0
100
アジャイルを支えるテストアーキテクチャ設計/Test Architecting for Agile
goyoki
9
3.1k
Nurturing OpenJDK distribution: Eclipse Temurin Success History and plan
ivargrimstad
0
400
Hotwire or React? ~アフタートーク・本編に含めなかった話~ / Hotwire or React? after talk
harunatsujita
1
110
リアーキテクチャxDDD 1年間の取り組みと進化
hsawaji
1
180
僕がつくった48個のWebサービス達
yusukebe
20
17k
NSOutlineView何もわからん:( 前編 / I Don't Understand About NSOutlineView :( Pt. 1
usagimaru
0
270
Featured
See All Featured
Thoughts on Productivity
jonyablonski
67
4.3k
Unsuck your backbone
ammeep
668
57k
Keith and Marios Guide to Fast Websites
keithpitt
409
22k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
26
1.4k
Optimizing for Happiness
mojombo
376
69k
Reflections from 52 weeks, 52 projects
jeffersonlam
346
20k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
231
17k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
15
2k
What's new in Ruby 2.0
geeforr
343
31k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
42
2.2k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
364
24k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
27
4.2k
Transcript
noteͷαʔϏεΠϯϑϥͷ “ࣜભٶ” ͷऔΓΈ
2 גࣜձࣾυϦίϜͰ֤छιʔγϟϧήʔϜͷΠϯϑϥΛ୲ͨ͠ޙɺ2020 ΑΓnoteגࣜձࣾʹೖࣾɻೖࣾޙAWSͷίετݮɺnoteͷػೳ͕ ಈ͍͍ͯΔ֤छαʔϏεͷγεςϜઃܭɺϝτϦΫεɾΤϥʔࢹͷඋ σϓϩΠαΠΫϧͷվળͳͲϓϩμΫτ։ൃҎ֎ͷԼճΓ෦ΛΑΓΑ ͘͢Δ͜ͱʹऔΓΜͰ͍·͢ɻ झຯ • ొࢁɾτϨΠϧϥϯχϯά ◦
͜ͳ͍ͩτϨϥϯͷେձʹग़ͯ15kmʢதڃίʔεʣΛͬͯ ͖·ͨ͠ɻΛ௧Ίͯݱࡏɺྍཆத…ɻ͖ͬ͞ࢄาͨ͠Β௧ SRE-T தଜ Ꮺ ʢvaru3ʣ
note inc. noteʹ͓͚ΔγεςϜͷ ”ࣜભٶ” ͱ 3
note inc. noteͷΤϯδχΞνʔϜ͕2022ʹ͚ͯઓ͢Δɺॏཁ՝9બ 4 https://note.jp/n/n39681098a0d2 • ࡢʹൃද͞Εͨ note هࣄ •
ʮίϯςφج൫ͷҠߦΛ࣮ࢪ͠ӡ༻ ޮԽɾলྗԽΛ࣮ݱʯ͕ॏཁ՝ͷ Ұͭͱͯ͠ڍ͛ΒΕͨ • ࡢ͔ΒϓϩδΣΫτ͕ελʔτ ͠ɺSREνʔϜΛத৺ʹঃʑʹҠߦ͠ ͍ͯΔ
note inc. ҰൠతͳίϯςφԽͷϝϦοτ 5 • σϓϩΠαΠΫϧͷվળ ◦ CI/CDΛར༻ͨ͠σϓϩΠͷࣗಈԽ ◦ Blue-GreenσϓϩΠɺCanaryσϓϩΠ
• ։ൃڥͷվળɺӡ༻ͷ༰қԽɺetc… ◦ ։ൃڥΛ༰қʹߏஙͰ͖Δ ◦ KubernetesͷΈΛར༻ͯ͠ӡ༻ͷࣗಈԽ
note inc. ͦΕ͚ͩʁ 6 ϓϩδΣΫτΛਐΊ͍ͯ͘தͰɺͦΕҎ֎ͷϝϦοτݟ͖͑ͯͨ
note inc. ιϑτΣΞࣜભٶͱ͍͏ߟ͑ํ 7 • ࣜભٶ ◦ ҏਆٶͰߦΘΕΔɺ20͓͖ʹશͯͷ఼ࣾΛΓସ ͑Δߦࣄͷ͜ͱ ◦
༻ͷ૿Ճ ٕज़ͷܧঝ ͷͨΊ • γεςϜͰಉ͜͡ͱ͕͍͑Δ ◦ AWS ᴈ໌ظ͔Βӡ༻͞Ε͍ͯΔྺ࢙͋Δ note ͷγες Ϝ ◦ ࣌ͱࠓͰϕετϓϥΫςΟε͕ҟͳΔ෦͋Δ
note inc. ιϑτΣΞࣜભٶͱ͍͏ߟ͑ํ 8 • ʮnote ͬͯͲ͏ͬͯಈ͍͍ͯΔΜ͚ͩͬʁʯ ʮͳΜͰ͜͏ͳ͍ͬͯΔΜ͚ͩͬʁʯΛਂ͘ ΓɺαʔϏεͷશ༰ΛखதʹऩΊΔ͜ͱ͕Ͱ͖Δ •
ͨͩ "γεςϜΛҠߦ͢Δ" ͚ͩͰͳ͘ɺݹ͔͘ Β͋ΔγεςϜશମͷঠѲͱϦϑΝΫλϦϯάΛ ಉ࣌ʹਐΊ͍ͯΔ
note inc. γεςϜશମͷঠѲͱϦϑΝΫλϦϯάͷͨΊʹ͍ͬͯΔ͜ͱ • ωοτϫʔΫߏͷ৽ • Τϥʔ௨ / ϞχλϦϯά /
ϩΪϯάڥͷཧ • ύϑΥʔϚϯεΛܭଌ͠ɺεέʔϧΞτΩϟύγςΟͷݟ͠ • σϓϩΠͱ҆ఆੑͷվળ • rakeλεΫͰ࣮ߦ͞ΕΔόονॲཧͷཧ • ΘΕ͍ͯͳ͍(͔͠Εͳ͍) gem Λ୨Է͠ • ։ൃʹඞཁͳڥΛύοͱ࡞ΕΔΑ͏ʹ • طଘͷCIڥ(Jenkins) ʹґଘ͍ͯ͠Δ෦Λղফ • etc…
note inc. EKSҠઃϓϩδΣΫτ 10
note inc. noteͷEKSҠઃϓϩδΣΫτͷ֓ཁ 11 • noteͷ֤ΞϓϦέʔγϣϯͷ࣮ߦج൫ΛEKS(Kubernetes)ʹҠઃ͢Δ • noteෳαʔϏεʹ͔ΕͯEC2্ͰΞϓϦέʔγϣϯ͕࣮ߦ͞Ε͍ͯΔ ◦ ϑϩϯτΤϯυ:
Nuxt.js ◦ όοΫΤϯυ: Ruby on Rails ◦ όον: Ruby on Rails (rakeλεΫ) ◦ ඇಉظϫʔΧʔ: Sidekiq • ͜ΕΒΛॱ൪ʹEKSҠઃ͍͖ͯ͠ɺ noteͷαʔϏεશମͷӡ༻ޮԽɾলྗԽΛ࣮ݱ͢Δ
note inc. EKSΫϥελͷߏ • ୯ҰΫϥελͷϚϧνςφϯτߏΛ࠾ ༻ ◦ ΞϓϦέʔγϣϯͷׂ͝ͱʹ NodeGroupΛ͚Δ ◦
namespaceͱNodeGroup1:1ͱ͠ɺ nodeselectorͰىಈ͢ΔNodeΛ੍ޚ͢Δ ◦ PodIPΞυϨεͷރׇΛ͙ͨΊɺη ΧϯμϦCIDR(100.XX.XX.XX) Ͱىಈ
note inc. ֤αʔϏε͝ͱͷҠઃॱ όον: Ruby on Rails(rakeλεΫ + cron) όοΫΤϯυ:
Ruby on Rails ϑϩϯτΤϯυ: Nuxt.js Worker: Ruby on Rails(Sidekiq) Ҡߦ қ Ҡߦॱ ※ఆࠁʹॲཧ͕࣮ߦ͢ΔͨΊͷγεςϜ ΠϚίί ߴ
note inc. 202110݄~ ~20226݄ 1. ཁ݅Λຬͨͨ͢Ίͷػೳ ։ൃ 2. ࢹɾΞϥʔτݕͷ ΈΛߏங
3. όονͷςετɾ֬ೝ࡞ۀ • OneshotδϣϒͰͷόονͷ࣮ߦ • betaڥͷσϓϩΠ όονͷҠઃਐḿ • PodͷΞϥʔτݕ (Datadog) • ΞϓϦέʔγϣϯ ͷΞϥʔτݕ (Sentry) • OneshotδϣϒΛ ࣮ߦͰ͖Δڥͷ ։ൃ • cronjob-managerͷ ։ൃ 4. Ҡઃ • ຊ൪ڥͷσϓϩΠ • σϓϩΠޙͷࢹ
note inc. 1. ཁ݅Λຬͨͨ͢Ίͷ ػೳͷ։ൃ 15
note inc. OneshotδϣϒΛ࣮ߦͰ͖Δڥ: k8s-job-executor • Slackbot্͔ΒίϚϯυΛ࣮ߦͰ͖Δػ ೳΛ։ൃͨ͠ • ίϚϯυΛ࣮ߦ͢ΔͱɺPod্ཱ͕͕ͪ ΓίϚϯυΛ࣮ߦ͢Δ
• ग़ྗCloudWatch Logsʹूͯ͠ɺ Managed Service for GrafanaͰࢀর͢Δ • GrafanaͰPod͝ͱɺΞϓϦέʔγϣ ϯ͝ͱʹϩάͷࢀর͕Մೳ
note inc. cronjob-manager(Kubernetes Custom Controller) • KubernetesͰCronJob͝ͱ(1 batch͝ͱʣʹ 1 manifest
Λ࡞ ͢Δඞཁ͕͋Δ • ҰͭͷίϯςφΠϝʔδΛ࡞͢ΔͨͼʹશͯͷCronJobͷΠ ϝʔδΛॻ͖͑Δͷେม…(ҎલγΣϧܳΛۦ͍ͯ͠ ͨʣ • cronjob-managerͱ͍͏Kubernetes Custom ControllerΛ։ൃ • ҰͭͷmanifestͰෳͷCronJobΛ੍ޚ͢Δ͜ͱ͕Ͱ͖Δ • CronJobManagerΛ࡞͢ΔͱCronJob͕ࣗಈͰ࡞ΒΕΔ
note inc. 2. ࢹɾΞϥʔτݕͷ ΈΛߏங 18
note inc. Ξϥʔτݕ • Node, PodϨϕϧͷΞϥʔτͱΞϓϦέʔγϣϯϨϕϧͷΞϥʔτͰݕ ख๏Λ͚Δ • Node, PodϨϕϧͷΞϥʔτ
◦ DatadogΛར༻͢Δ ◦ DatadogAgentΛDaemonsetͱͯ͠σϓϩΠͯ͠ɺϝτϦΫεΛࢹ • ΞϓϦέʔγϣϯϨϕϧͷΞϥʔτ ◦ SentryΛར༻͢Δ ◦ ΞϓϦέʔγϣϯͰग़ͨbacktraceΛৄࡉʹੳͰ͖Δ • Կ͔͕ൃੜͨ͠߹ʹ͍ͣΕ͔ͷΞϥʔτͰݕ͞ΕΔ
note inc. Ξϥʔτݕ: ΞΫγϣϯϨϕϧ • ࠓ·Ͱ1ͭͷchannelʹਨΕྲྀ͠ʹͳ͍ͬͯͨΞϥʔτΛΞΫγϣϯϨϕϧ͝ͱʹཧ • ֤Ϩϕϧຖʹ̏ͭͷSlack channelʹͦΕͧΕ௨͢ΔΑ͏ʹͨ͠ info
• ରԠෆཁͳΞϥʔτ • جຊతʹใΛऩू ͢Δ͚ͩ • োൃੜ࣌ಈ࡞֬ ೝͷͨΊʹར༻͢Δ ใ warn • ཌ·ͰʹରԠ͕ඞ ཁͳΞϥʔτ • ۓٸͷରԠෆཁͩ ͕์ஔ͢ΔͱϢʔ βʔʹӨڹ͕ग़͔Ͷ ͳ͍ͷ critical • ଈ࣌ରԠ͕ඞཁͳΞϥʔτ • ։ൃΤϯδχΞશһ͕ΈΔ • Կ͔͠ΒϢʔβʔʹӨڹ͕ ग़࢝Ί͍ͯΔՄೳੑ͕͋Δ ͷ
note inc. Ξϥʔτݕ: criticalอશ࡞ۀ • critical = Ҭͱͯ͠ɺৗʹ༨ܭͳΞϥʔτ͕ඈͳ͍Α͏ʹҙΛ͏எ·͵ྗ͕ඞཁ • ༨ܭͳΞϥʔτ͕૿͍͑ͯ͘ͱɺΞϥʔτʹରͯ͠ԿΛ͍͍͔ͯ͠Θ͔Βͳ͘ͳΔ
• ͍ͦͯͣ͠Εɺ୭ͦͷνϟϯωϧΛݟͳ͘ͳ͍ͬͯ͘… • ͦ͏ͳΒͳ͍Α͏ʹɺᮢΞϥʔτఆٛΛఆظతʹݟ͠ɺৗʹదͳΞϥʔτͱͳΔΑ͏ ৺͕͚Δ • ͞ΒʹΑ͘͢Δʹʢ՝ʣ ◦ ि࣍ͷϨϙʔςΟϯάͰ͜ΜͳΞϥʔτ͕Կ݅ى͖͍ͯͨͱ͍͏ͷΛαϚϦԽ ◦ ΞϥʔτΛఆ؍ଌ͠ɺΞϥʔτͷ૿ݮΛৗʹҙ͍ࣝͯ͘͠
note inc. 3. όονͷςετɾ֬ೝ࡞ۀ 4. Ҡઃ 22
note inc. όονͷςετɾ֬ೝ࡞ۀ batchͷதΛ֬ೝ k8s-job-executorͰ࣮ ߦ͢Δ on beta Τϥʔ͕ग़͍ͯͳ͍͔ ϩάΛ֬ೝ͢Δ
࣮ߦ࣌ؒͷڍಈΛ֬ ೝ cronjob-managerͰσ ϓϩΠ͢Δ on beta cronjob-managerͰσ ϓϩΠ͢Δ on ຊ൪ ͋Δఔͨ·ͬͨΒ 1 batch͝ͱʹ͜ͷα ΠΫϧΛ܁Γฦ͢ • batchͷશ෦Ͱ300 • SRE-TͱϓϩμΫτ։ൃνʔϜͰ ख͚ͯ͠ɺҰͭͣͭ֬ೝ࡞ۀΛ ࣮ࢪத • priorityͷ͍ͷ͔Βॱ࣍ɺEKS ্Ͱಈ͔͍ͯ͠Δ
note inc. 24 47 / 295
note inc. ·ͱΊ 25
note inc. EKS(Kubernetes)ʹͯ͠Α͔ͬͨͱ͜Ζ Jobͷ࣮ߦཤྺ͕͙֬͢ೝͰ͖Δ • kubectl get jobs ίϚϯυͰۙʹ࣮ߦ ͞ΕͨJobͷཤྺ͕͙͢ʹ֬ೝͰ͖Δ
• աڈͷ࣮ߦϩάͳͲͷৄࡉΛJob͝ͱ ʹݸผʹ֬ೝ͢Δ͜ͱ͕Ͱ͖Δ • ࣦഊͨ͠߹ͷ࠶࣮ߦ༰қ Job͝ͱʹࡉ੍͔͍ޚ͕Մೳ • Job͝ͱʹCPU / Memoryͷ্ݶϦ ΫΤετྔΛࢦఆͰ͖Δ • ࠶࣮ߦճฒྻ࣮ߦͳͲͷࡉ͔͍ ੍ޚ͕Մೳ • ಛఆͷJob͚ͩFargateͰ࣮ߦ͢Δɺͳ Ͳͷ͜ͱՄೳʢࠓޙͷ՝ʣ
note inc. noteͷ”ࣜભٶ”ͰͷϝϦοτ • ӡ༻ͷརศੑ ◦ ΄΅શͯͷಈ࡞ΛSlack͔Βૢ࡞͢Δ͜ͱ͕Ͱ͖ΔΑ͏ʹͨ͜͠ͱͰརศੑ্͕ͨ͠ (ίϯςφͷϏ ϧυɺίϚϯυͷ࣮ߦɺϩάͷࢀরʣ ◦
ϩάͷࢀরݕࡧͳͲ͕Grafanaͷ֎෦πʔϧͰ֬ೝͰ͖ΔΑ͏ʹͳͬͨ • োͷൃݟੑ্͕ ◦ DatadogSentryͷϞχλϦϯάڥΛ͑ͨ͜ͱͰɺCronJobͷಈ࡞ʹ͕͋ͬͨ߹ʹ͙͢ʹݕ ͢Δ͜ͱ͕Ͱ͖ΔΑ͏ʹͳͬͨ ◦ ֤ΞϥʔτΛΞΫγϣϯϨϕϧ͝ͱʹཧ͠ɺΦϯίʔϧڥ͕վળͨ͠ • όονॲཧࣗମͷվળɾݟ͠ ◦ ςετσʔλͷͳ͍ͷɺదͳϩά͕ग़ྗ͞Ε͍ͯͳ͍ͷͳͲΛվળ
note inc. ·ͩ·ͩ͜Ε͔Β͕ ՂڥͰ͕͢ ҆ఆͨ͠αʔϏεΛఏڙͰ͖ΔΑ͏ ʹؤு͍͖ͬͯ·͢ 28