Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Challenges for Global Service from a Perspective of SRE
Search
Takayuki WATANABE (渡辺 喬之)
February 10, 2018
Technology
3
2.9k
Challenges for Global Service from a Perspective of SRE
Cookpad TechConf 2018:
https://techconf.cookpad.com/2018/
Takayuki WATANABE (渡辺 喬之)
February 10, 2018
Tweet
Share
More Decks by Takayuki WATANABE (渡辺 喬之)
See All by Takayuki WATANABE (渡辺 喬之)
[Money Forward x Shippio] BaySide-Tech-Nite (May 19, 2023)
takanabe
0
86
[Developers Summit 2023] ソフトウェアテスト新時代の幕開け: 機械学習とデータサイエンスで実現するテスト運用の高度化
takanabe
25
10k
SRE NEXT 2022: Sensible Incident Management for Software Startups
takanabe
2
7.5k
SRE NEXT 2020 [C6] Designing fault-tolerant microservices with SRE and circuit breaker centric architecture
takanabe
1
7.1k
Challenges for Global Service from a Perspective of SRE 2nd season
takanabe
3
4.2k
Practical Approaches to Achieve Continuous Deployment with Kubernetes
takanabe
0
210
Building infrastructure on AWS with Ruby
takanabe
0
230
SRE vs ラマダン
takanabe
0
220
Other Decks in Technology
See All in Technology
プッシュ型子育てサービスを、先行プロジェクト実施自治体において開始します
govtechtokyo
0
290
OCI Data Integration技術情報 / ocidi_technical_jp
oracle4engineer
PRO
1
1.5k
Evolutionary Optimization of Model Merging Recipes
fuyu_quant0
3
560
SaaS型Webサービス「カオナビ」のチーム開発でPackage by Featureを取り入れた話/Implementing Package by Feature in kaonavi
kaonavi
0
110
家族アルバム みてねで直面してきた技術的負債 / MIXI KAG 2024
isaoshimizu
17
7.7k
技術イベントはなんとかひねり出す 日経の技術広報の取り組み/techpr3
nishiuma
0
230
Castor - Symfony Live 2024 - Paris
lyrixx
1
110
Kubeflow Pipelines v2 で変わる機械学習パイプライン開発
asei
4
340
データマネジメントを支える武器としてのメタデータ管理
10xinc
2
960
複数の LLM モデルを扱う上で直面した辛みまとめ
kazuyaseki
1
260
任意コード実行の原理
ffri
0
190
初心者が行く!サーバレスWebアプリ開発の道
nagaharutogawa
0
450
Featured
See All Featured
A Philosophy of Restraint
colly
195
15k
The Power of CSS Pseudo Elements
geoffreycrofte
58
4.9k
How STYLIGHT went responsive
nonsquared
92
4.7k
Optimizing for Happiness
mojombo
369
69k
GitHub's CSS Performance
jonrohan
1023
450k
Building Better People: How to give real-time feedback that sticks.
wjessup
350
18k
Building Your Own Lightsaber
phodgson
97
5.6k
Visualization
eitanlees
135
14k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
124
32k
Six Lessons from altMBA
skipperchong
19
2.9k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
153
14k
What the flash - Photography Introduction
edds
64
11k
Transcript
Challenges for Global Service from a Perspective of SRE Takayuki
Watanabe Infrastructure Department SRE Group Cookpad Inc. Feb 10, 2018
Who? 2
ࣗݾհ ɾΠϯϑϥετϥΫνϟʔ෦ SREάϧʔϓ - ΫοΫύουͷάϩʔόϧαʔϏεͷ։ൃʹैࣄ ɾSRE = Site Reliability Engineering
- System & Software EngineeringͰ αΠτͷ৴པੑʹؔΘΔશͯͷ՝ʹऔΓΉ 3 ลڤ೭ (Takayuki Watanabe) twitter: takanabe_w / github: takanabe
ࠓ͢͜ͱ 4 ɾΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ ɾ2017ͷάϩʔόϧαʔϏεͷ ɾάϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝ͱઓ
ΫοΫύουͷ άϩʔόϧαʔϏεͬͯԿʁ 5
ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ 6 ւ֎͚ͷϨγϐαʔϏε
ւ֎͚ͷϨγϐαʔϏε 7 ຊͷΫοΫύουͱ ҟͳΔαʔϏε ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
ւ֎͚ͷϨγϐαʔϏε 8 ݄ؒར༻ऀ3000ສਓҎ্ ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ ※ 2017 1ʙ12݄ฏۉ
ւ֎͚ͷϨγϐαʔϏε 9 22ݴޠ ɾ68ΧࠃͰల։ ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ ※ 2017 12݄࣌
ւ֎͚ͷϨγϐαʔϏε 10 WebɾAndroidɾiOSͰఏڙத (ίʔυϕʔεࠃͱҟͳΔ) ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
11 UK Office (Bristol)
12 UK Office (Bristol) ϓϩμΫτ։ൃڌUK(Bristol)
13 UK Office (Bristol) ౦ژɺεϖΠϯɺΠϯυωγΞΛ த৺ʹελοϑੈքதʹࡏ
ɾຊҎ֎ͷࠃΛରʹ։ൃ͍ͯ͠ΔϨγϐαʔϏε ɾ݄ؒར༻ऀ3000ສਓҎ্ ɾ22ݴޠͰ68ΧࠃʹαʔϏεΛల։ ɾWebɺAndroidɺiOSͰఏڙத(ίʔυϕʔεࠃͱҟͳΔ) ɾੈքதͷΤϯδχΞ͕։ൃʹࢀՃ 14 ւ֎͚ͷϨγϐαʔϏε ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
2017ͷάϩʔόϧ αʔϏεͷ 15
ରԠݴޠ 16
2017ͷάϩʔόϧαʔϏεͷ 17 ରԠݴޠ͕૿Ճ 15ݴޠ 201612݄࣌
2017ͷάϩʔόϧαʔϏεͷ 18 ରԠݴޠ͕૿Ճ 15ݴޠ 22ݴޠ 20179݄࣌ 201612݄࣌
2017ͷάϩʔόϧαʔϏεͷ 19 ରԠݴޠ͕૿Ճ 15ݴޠ 22ݴޠ + 7 ݴޠ 201612݄࣌ 20179݄࣌
2017ͷάϩʔόϧαʔϏεͷ 20 ରԠݴޠ͕૿Ճ 15ݴޠ 22ݴޠ શݴޠରԠ ≠ ੈքରԠ 201612݄࣌ 20179݄࣌
21
22 Ҭੑͷҧ͍Λߟྀͨ͠αʔϏε։ൃ͕ඞཁ ref: https://techconf.cookpad.com/2017/rejasupotaro.html
ల։தͷࠃͷ 23
2017ͷάϩʔόϧαʔϏεͷ 24 ల։தͷࠃͷ͕૿Ճ 58ϲࠃ 201612݄࣌
2017ͷάϩʔόϧαʔϏεͷ 25 ల։தͷࠃͷ͕૿Ճ 58ϲࠃ 68ϲࠃ 201612݄࣌ 20179݄࣌
2017ͷάϩʔόϧαʔϏεͷ 26 ల։தͷࠃͷ͕૿Ճ 58ϲࠃ 68ϲࠃ + 10ϲࠃ 201612݄࣌ 20179݄࣌
UKΦϑΟεͰಇࣾ͘һ 27
2017ͷάϩʔόϧαʔϏεͷ 28 UKΦϑΟεͰಇࣾ͘һ 5ਓ 201612݄࣌
2017ͷάϩʔόϧαʔϏεͷ 29 UKΦϑΟεͰಇࣾ͘һ 5ਓ 50ਓ 201712݄࣌ 201612݄࣌
2017ͷάϩʔόϧαʔϏεͷ 30 UKΦϑΟεͰಇࣾ͘һ 5ਓ 50ਓ 10ഒ 201612݄࣌ 201712݄࣌
2017ͷάϩʔόϧαʔϏεͷ 31 Google Playʮϕετ Φϒ 2017ʯͰ7Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί
ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@
2017ͷάϩʔόϧαʔϏεͷ 32 Google Playʮϕετ Φϒ 2017ʯͰ7Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί
ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ ຊൃͷΞϓϦͰ ࠷ଟͷ7ҬͰಉ࣌ೖ
2017ͷάϩʔόϧαʔϏεͷ 33 Google Playʮϕετ Φϒ 2017ʯͰ7Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί
ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ αʔϏε৫͖ͯͨ͠
2017ͷάϩʔόϧαʔϏεͷ 34 Google Playʮϕετ Φϒ 2017ʯͰ7Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί
ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ ҰํͰࢁͷ՝͋ͬͨ
άϩʔόϧαʔϏεʹ͓͚Δ SREͷ՝ͱઓ 35
36 άϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝ͱઓ ՝ᶃɿಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ ՝ᶄɿΠϕϯτ͕ͱʹ͔͘ଟ͍ ՝ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍ ՝ᶆɿtoil ͕ٸ૿͢Δ άϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝
՝ᶃ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ 37
38 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕͍ ͙ݕࡧͰ͖ͳ͍
39 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕͍ ͙ݕࡧͰ͖ͳ͍ ༷ʑͳࠃʹαʔϏεΛల։͢Δ͜ͱͰ ࠃ͝ͱʹϢʔβମݧʹ͕ࠩग़࢝Ίͨ
40 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕͍ ͙ݕࡧͰ͖ͳ͍ ͦͷࠃʹॅΜͰ͍ͳ͍ͷͰ ಉ͡ڥͰςετͰ͖ͳ͍
41 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕͍ ͙ݕࡧͰ͖ͳ͍ ͕࣌ࠩ͋ΔࠃͷਓͱϝοηʔδΛ ަΘͤΔͷଟͯ͘1ճ
42 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕͍ ͙ݕࡧͰ͖ͳ͍ ݪҼ͕Θ͔Βͳ͍ͱ վળͰ͖ͳ͍
ઓᶃ ੈքதͷϢʔβମݧΛଌఆ͠վળ͢Δ 43
ɾఆ؍ଌʹCatchpoint SystemsͷSynthetic MonitoringΛར༻ - 30Χࠃ͔ΒఆظతʹϝτϦΫεΛऩू - Time To First Byte
- DNS lookup time - TLS established time - ϨεϙϯελΠϜ ͳͲ 44 ఆ؍ଌʹΑΔ֤ࠃͷϢʔβମݧͷଌఆ ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ
45 ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
46 ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
47 ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
48 ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ ݪҼΛௐࠪͰ͖Δঢ়ଶʹͳͬͨ
49 ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ ΠϯυωγΞͷTTFB ΠϯυωγΞͷTLSଓ࣌ؒ ถࠃͷTTFB ถࠃͷTLSଓ࣌ؒ 750ms 330ms 225ms
150ms
50 ΠϯυωγΞถࠃͱൺͯ TLSଓTime To First Byteʹ ͕͔͔࣌ؒΔ ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
51 ถࠃ͔Βཧతʹԕ͍ࠃͷϢʔβମݧ ͕ѱ͍ʹ͋Δ͜ͱ͕໌ ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
άϩʔόϧαʔϏεʹ͓͚ΔSREͷઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍ 52 SFGIUUQTUFDIDPOGDPPLQBEDPNTPSBIIUNM
άϩʔόϧαʔϏεʹ͓͚ΔSREͷઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍ 53 ɾαʔόถࠃϦʔδϣϯͰཧ ɾcookpad.comυϝΠϯΛࠃαʔϏεͱڞ༗ ɾRoute53ͷLatency Based RoutingͰIPΛฦ٫ - ౦ژ͔ถࠃͷ͍ۙσʔληϯλ·ͣΞΫηε
ɾࠃͱάϩʔόϧαʔϏεΛৼΓ͚ - ϦόʔεϓϩΩγͷਖ਼نදݱͰఆ - cookpad.comຊͷαʔϏε - cookpad.com/ukͳͲάϩʔόϧαʔϏε
άϩʔόϧαʔϏεʹ͓͚ΔSREͷઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍ 54 location ~ ^/(en|vi|fil|ae|al|ao|ar|arabic|at|au|az|ba|bd|be|bg|bh|bo| br|by|ch|ci|cl|cn|co|cr|cu|cy|cz|de|dj|dk|do|dz|ec|ee|eg| es|fi|fr|gh|gr|gt|hk|hn|hr|hu|id|ie|il|in|iq|ir|it|jo| ke|km|kr|kw|kz|lb|lk|lt|lu|lv|ly|ma|mo|mr|mx|my|mz|ng|ni| nl|no|nz|om|pa|pe|ph|pk|pl|ps|pt|py|qa|ro|rs|ru|sa|
sd|se|sg|si|sk|so|sv|sy|th|tn|tr|tt|tw|ua|uk|us|uy|uz|ve| vn|ye|za)(/|$) { proxy_pass http://cookpad_use1; } location / { proxy_pass https://cookpad_apne; }
55 ถࠃ͔Βԕ͍ࠃͷϢʔβ͕շదʹ͑ΔΑ͏ʹ͢Δ ɾΞϝϦΧʹαʔό͕ଘࡏ͍ͯͯ͠ωοτϫʔΫϨΠςϯγ͕ߴ͍ ɾӡ༻ίετ্͕͕Δٕज़·ͩೖΕͨ͘ͳ͍ - αʔόͷϚϧνϦʔδϣϯԽͳͲ ɾΠϯυωγΞҎ֎ͷࠃগͳ͍खͰޮతʹվળ͍ͨ͠ - ଞʹΒͳ͚ΕͳΒͳ͍͜ͱࢁ͋Δ ઓᶃɿ
ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ
56 ถࠃ͔Βԕ͍ࠃͷϢʔβ͕շదʹ͑ΔΑ͏ʹ͢Δ ɾΞϝϦΧʹαʔό͕ଘࡏ͍ͯͯ͠ωοτϫʔΫϨΠςϯγ͕ߴ͍ ɾӡ༻ίετ্͕͕Δٕज़·ͩೖΕͨ͘ͳ͍ - αʔόͷϚϧνϦʔδϣϯԽͱ͔ ɾΠϯυωγΞҎ֎ͷࠃগͳ͍खͰޮతʹվળ͍ͨ͠ - ଞʹΒͳ͚ΕͳΒͳ͍͜ͱࢁ͋Δ ઓᶃɿ
ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ CDNʹΑΔTCPͱTLSͷऴͰ ωοτϫʔΫϨΠςϯγΛվળ͢Δ
57 FastlyʹΑΔϨΠςϯγͷվળ ɾWeb / API શͯͷϦΫΤετΛFastlyܦ༝ʹ - Ωϟογϡͱ͍ͯͬͯ͠ΔΘ͚Ͱͳ͍ ɾFastlyͰTCPͱTLSͷऴΛ࣮ݱ -
Ϣʔβͱσʔληϯλͷڑ͕ۙ͘ͳΔ - TCP/TLSͷϋϯυγΣΠΫʹ͔͔Δ͕࣌ؒॖ͞ΕΔ ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
58 FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ Fastlyಋೖલ(1.25 sec) ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
59 FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ Fastlyಋೖલ(1.25 sec) Fastlyಋೖޙ(0.45 sec) ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
60 FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ Fastlyಋೖલ(1.25 sec) Fastlyಋೖޙ(0.45 sec) ΠϯυωγΞϢʔβͷ ϨεϙϯελΠϜ1/3ʹॖ ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
61 ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ Ξϧθϯνϯ ΠϯυωγΞ αδΞϥϏΞ UAE ΠΪϦε Fastlyಋೖ Fastlyಋೖ
Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ
62 ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ Ξϧθϯνϯ ΠϯυωγΞ αδΞϥϏΞ UAE ΠΪϦε Fastlyಋೖ Fastlyಋೖ
Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ ཧతʹถࠃ͔Βԕ͍ࠃͷ Ϣʔβମݧ͕վળ
՝ᶄ Πϕϯτ͕ͱʹ͔͘ଟ͍ 63
ɾ͜͜Ͱݴ͏ΠϕϯτॕࡇͳͲຖඞͣ։࠵͞ΕΔͷ ɾΠϕϯτͷࡍʹϢʔβ͕େྔʹΞΫηεͯ͘͠Δ ɾେ͖ͳΠϕϯτࠃफڭͳͲͷ୯ҐͰ࠵͞ΕΔ 64 ΠϕϯτͷόϦΤʔγϣϯ͕๛ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
ɾ͜͜Ͱݴ͏ΠϕϯτॕࡇͳͲຖඞͣ։࠵͞ΕΔͷ ɾΠϕϯτͷࡍʹϢʔβ͕େྔʹΞΫηεͯ͘͠Δ ɾେ͖ͳΠϕϯτࠃफڭͳͲͷ୯ҐͰ࠵͞ΕΔ - Ξϧθϯνϯ 65 ΠϕϯτͷόϦΤʔγϣϯ͕๛ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
66 ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
67 ֵ໋ه೦ɺಠཱه೦ʹ ύεςϦʔτεΛ࡞ͬͯ৯Δ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
ɾ͜͜Ͱݴ͏ΠϕϯτॕࡇͳͲຖඞͣ։࠵͞ΕΔͷ ɾΠϕϯτͷࡍʹϢʔβ͕େྔʹΞΫηεͯ͘͠Δ ɾେ͖ͳΠϕϯτࠃफڭͳͲͷ୯ҐͰ࠵͞ΕΔ - Ξϧθϯνϯ - ΠεϥϜݍ 68 ΠϕϯτͷόϦΤʔγϣϯ͕๛ ՝ᶄɿ
Πϕϯτ͕ͱʹ͔͘ଟ͍
69 ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
70 ΠυΡϧΞυϋʢ٘ਜ਼ࡇʣ ͰΛ৯Δ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
71 ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
72 ϥϚμϯظؒ ͔Β໌͚ͷؒͰ ՈͰ৯ࣄΛָ͠Ή ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
ɾຊͷ߹ɺόϨϯλΠϯͰϦΫΤετ͕2ഒʹͳΔ 73 ΠϕϯτʹΑΔϢʔβϦΫΤετͷมԽ ٸʹϦΫΤετ͕Ͷ্͕Δ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
74 ΠϕϯτʹΑΔϢʔβϦΫΤετͷมԽ ٸʹϦΫΤετ͕Ͷ্͕Δ ɾΠυΡϧΞυϋ͕࢝·ΔͱͷؒϦΫΤετ͕2ഒʹͳΔ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
ɾϥϚμϯ͕࢝·Δͱ1ϲ݄ͷؒϦΫΤετ͕2ʙ3ഒʹͳΔ 75 ΠϕϯτʹΑΔϢʔβϦΫΤετͷมԽ ٸʹϦΫΤετ͕Ͷ্͕Δ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
ɾւ֎ͷϨγϐαʔϏε͕CookpadͷؒʹͳΔ͜ͱ͋Δ 76 1Χࠃ૿͑ͯɺಥવϢʔβ͕100ສਓ૿͑Δέʔε͕༗Δ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
ɾւ֎ͷϨγϐαʔϏε͕CookpadͷؒʹͳΔ͜ͱ͋Δ 77 1Χࠃ૿͑ͯɺಥવϢʔβ͕100ສਓ૿͑Δέʔε͕༗Δ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍ ࠃͷ͕૿͑ͯ Πϕϯτ૿͑Δ
78 ຖ݄όϨϯλΠϯ —
79 શͯͷΠϕϯτʹ SRE͕ରࡦ͢Δͷ͍͠
ઓᶄ γεςϜͱ৫ͷεέʔϥϏϦςΟΛ ߟྀͨ͠ΈΛೖΕΔ 80
81 εέʔϥϏϦςΟɾՄ༻ੑΛߟྀٕͨ͠ज़ͷ࠾༻ ɾϓϩμΫγϣϯͷσʔλϕʔεશAmazon AuroraΛར༻ ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
82 DockerΞϓϦ։ൃڥͷఏڙ ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
ɾຊͰഓΘΕͨECS + hakoʹΑΔσϓϩΠγεςϜΛಋೖ - ΦʔτεέʔϧΛඪ४ࡌ (ϨΨγΞϓϦࣗͰ࣮͢Δඞཁ͕͋ͬͨ) - ։ൃεέʔϧ͢Δ - ։ൃऀґཔͳ͠ʹڥมൿಗใΛมߋՄೳ
- ECSΫϥελʹίϯςφΛσϓϩΠ͢ΔͷͰΠϯελϯεͷηοτΞοϓෆཁ - hako-consoleͳͲͷࣾͷΤίγεςϜͷԸܙΛڗड 83 ref https://speakerdeck.com/eagletmt/web-application-development-in-cookpad-2017 DockerΞϓϦ։ൃڥͷఏڙ ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
84 hako-console 1PXFSFECZ!FBHMFUNU ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
85 hako-console 1PXFSFECZ!FBHMFUNU ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
86 hako-console 1PXFSFECZ!FBHMFUNU ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ άϩʔόϧαʔϏεͰ ຊͰͷٕज़ܦݧΛ ׆͔͍ͯ͠Δ
87 ֤छμογϡϘʔυͷ࡞ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕σʔλετΞͷεϩʔΫΤϦΛੳͰ͖ΔΑ͏ʹ͢Δ ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
88 ֤छμογϡϘʔυͷ࡞ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕ओཁͳϝτϦΫεΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
89 ֤छμογϡϘʔυͷ࡞ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕HTTPͷεςʔλείʔυมԽΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
90 ֤छμογϡϘʔυͷ࡞ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕HTTPͷεςʔλείʔυมԽΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ ։ൃऀ͕ࣗͰΛൃݟͯ͠ ηϧϑϚωδϝϯτͰ͖Δ ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
՝ᶅ σϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍ 91
92 ຊͰՄೳͳΦϖϨʔγϣϯւ֎ͰͰ͖ͳ͍͜ͱ͋Δ ɾ࣌ؒίωΫγϣϯΛҡ࣋Ͱ͖ͳ͍ڥ - ωοτϫʔΫڥ͕ຊͷΑ͏ʹ͍ͬͯΔͱݶΒͳ͍ - ւ֎ͰఀిɾߑਫͳͲͷҟৗࣄଶ͕ৗతʹى͜Δ - खݩ͔ΒCapistranoͰσϓϩΠͰ͖ΔਓɾͰ͖ͳ͍ਓ͕͍Δ ՝ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
93 ຊͰՄೳͳΦϖϨʔγϣϯւ֎ͰͰ͖ͳ͍͜ͱ͋Δ ɾ࣌ؒίωΫγϣϯΛҡ࣋Ͱ͖ͳ͍ڥ - ωοτϫʔΫڥ͕ຊͷΑ͏ʹ͍ͬͯΔͱݶΒͳ͍ - ఀిɾߑਫͳͲͷҟৗࣄଶී௨ʹى͖Δ - खݩ͔ΒCapistranoͰσϓϩΠͰ͖ΔਓɾͰ͖ͳ͍ਓ͕͍Δ ͦͦσϓϩΠͰ͖ͳ͍
՝ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
94 σϓϩΠΛଞͷਓʹґཔ͢Δੈքઢ͕ੜ͢Δ ɾखݩ͔ΒσϓϩΠ͕Ͱ͖ͳ͍։ൃऀ͔ΒσϓϩΠґཔ͕དྷΔ ɾσϓϩΠͷཧ͓ޓ͍ʹγϯυΠ - ։ൃऀ1ʹproductionʹԿճσϓϩΠ͢Δ - ґཔ͢Δํɺґཔ͞ΕΔํγϯυΠ - σϓϩΠʹର͢Δ৺ཧతͳίετ্͕͕ͬͯ͠·͏
- ৽͍͠ػೳΛϦϦʔε͍ͨ͠ͷʹཧσϓϩΠऀͪ͋Γ͏Δ ՝ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
ઓᶅ ੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ 95
96 σϓϩΠαʔόΛถࠃϦʔδϣϯʹ༻ҙ͢Δ ɾssh ͰσϓϩΠαʔόʹΞΫηεͯ͠Β͍CapistranoΛ࣮ߦ ɾཧσϓϩΠͷѱເ͔Βղ์͞ΕΔ ઓᶅɿੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ
97 σϓϩΠαʔόͰͷϚχϡΞϧΦϖϨʔγϣϯͷ ɾΦϖϨʔγϣϯ͕౷Ұ͞Εͳ͍ - ։ൃऀ͕࠷৽ͷσϓϩΠεΫϦϓτΛ͏ͱݶΒͳ͍ - ఆ֎ͷ͜ͱΛαʔόͰߦ͏ਓ͕͍Δ ɾαʔόʹΞΫηε͢Δͷ͕໘ ɾσϓϩΠͷϩάΛ͍ʹ͍͘ ઓᶅɿੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ
98 σϓϩΠڥͷ৽ chat (slack) deploy server (cap & hako) bot
(ruboty) job scheduler (Rundeck) deploy targets ઓᶅɿੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ
99 σϓϩΠαʔόͷϩάΠϯ͕ෆཁʹͳΔ ઓᶅɿੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ
100 ఆ͍ͯ͠ΔΦϖϨʔγϣϯʹ౷ҰͰ͖Δ ઓᶅɿੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ
101 σϓϩΠͷϩάΛ͏ͷ͕༰қʹͳΔ σϓϩΠ࣌ࠁ σϓϩΠͨ͠ਓ ઓᶅɿੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ
102 σϓϩΠͷϩάΛ͏ͷ͕༰қʹͳΔ ઓᶅɿੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ
՝ᶆ toil ͕ٸ૿͢Δ 103
104 SFGIUUQTMBOEJOHHPPHMFDPNTSFCPPLIUNM ՝ᶆɿtoil͕ٸ૿͢Δ
ɾखಈͰରԠ͍ͯ͠Δ ɾࣗಈԽͷ༨͕͋Δ ɾ܁Γฦ͠ൃੜ͢Δ ɾൃੜ͔ͯ͠Β͡Όͳ͍ͱରԠͰ͖ͳ͍ ɾαʔϏεͷվળʹͭͳ͕Βͳ͍ ɾαʔϏεϢʔβͷʹൺྫͯ͠૿Ճ͢Δ 105 toilʹͯ·Δۀ ՝ᶆɿtoil͕ٸ૿͢Δ SFGIUUQTMBOEJOHHPPHMFDPNTSFCPPLDIBQUFSTFMJNJOBUJOHUPJMIUNM
106 ։ൃऀ͔ΒSREͷґཔ͕ٸ૿
֤ظؒͷSREͷґཔͷਪҠ ՝ᶆɿtoil͕ٸ૿͢Δ
107 ۀׂ߹ͷมԽ /PW %FD +BO
'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH ՝ᶆɿtoil͕ٸ૿͢Δ
108 ۀׂ߹ͷมԽ /PW %FD +BO
'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH SRE͕Γ͍ͨ͜ͱ ՝ᶆɿtoil͕ٸ૿͢Δ
109 ۀׂ߹ͷมԽ /PW %FD +BO
'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH toilͷ૿Ճʹൺྫͯ͠ ׂ͕͚࣌ؒͳ͘ͳΔ ՝ᶆɿtoil͕ٸ૿͢Δ
110 SREͷۀׂ߹ͷมԽ /PW %FD +BO
'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH Γ͍ͨ͜ͱ͕Ͱ͖ͳ͍ ՝ᶆɿtoil͕ٸ૿͢Δ
111 20171~5݄ʹ૿Ճͨ͠toilτοϓ3 ΧςΰϦ ґཔͷ "DDPVOU.BOBHFNFOU "843FMBUFE0QFSBUJPOT )VHF%BUBCBTF5BCMF.JHSBUJPOT
՝ᶆɿtoil͕ٸ૿͢Δ
112 20171~5݄ʹ૿Ճͨ͠toilτοϓ3 DBUFHPSZ PGJTTVFT "DDPVOU.BOBHFNFOU "843FMBUFE0QFSBUJPOT )VHF%BUBCBTF5BCMF.JHSBUJPOT
ΞΧϯτཧճΓ ͷґཔͷᓲ໓͕ඞཁ ՝ᶆɿtoil͕ٸ૿͢Δ
113 ΞΧϯτཧ͕ਏ͍ཧ༝ ɾࣾһ͕ٸ૿͍ͯ͠Δ͠ɺࠓޙ૿͑Δ͜ͱ͕૾Ͱ͖Δ ɾ։ൃऀ͕ར༻͢Δπʔϧ͕ଟ༷Խ͖͍ͯͯ͠Δ - Single Sign OnʹରԠ͍ͯ͠ͳ͍πʔϧ͋Δ - ͰɺΞΫηε੍ޚ͍ͨ͠
ɾ͕࣌ࠩ͋ΔͷͰίϛϡχέʔγϣϯͷλΠϛϯάΛҳ͍͢͠ - Կ͕͍͔ͨ͠Ѳ͢Δͷʹ͔͔Δ͜ͱ ՝ᶆɿtoil͕ٸ૿͢Δ
114 άϩʔόϧͰར༻࣮ͷ͋Δπʔϧ܈ ɾੳɿ Kibanaɺ GrafanaɺNewRelicɺCatchpointɺDatadog ɾίʔυཧɿ Github Enterpriseɺgithub.com ɾόάτϥοΧɿSentryɺBugsnagɺAirbrake ɾCIɿJenkinsɺCircle
CIɺSemaphore CI ɾPagerɿPagerDuty ɾͦͷଞɿAWSɺKuroko2ɺin-house πʔϧ ͳͲ ՝ᶆɿtoil͕ٸ૿͢Δ
115 ࣾπʔϧͷΞΫηεʹVPNΛ͑ΔͱݶΒͳ͍ ɾੈքதͷΦϑΟεͷ։ൃऀ͕VPNΛ͑Δͷ͔ʁ - શһ͕IPΞυϨεͰ੍ޚͰ͖Δڥʹ͍ͳ͍ ɾ౦ژͷࣾNWΛܦ༝ͨ͠ଓଥ͔ʁ - ࣾπʔϧʹੈքதͷελοϑ͕ΞΫηε͢Δ - ΞϑϦΧͷελοϑVPNΛ҆ఆͯ͠ுΕΔͩΖ͏͔
՝ᶆɿtoil͕ٸ૿͢Δ
116 ࣾπʔϧʹBasicೝূΛ͍ͨ͘ͳ͍ ɾSREࣾπʔϧ͕૿͑ΔʹBasicೝূͷઃఆΛ͢Δ - Ϣʔβ͝ͱʹύεϫʔυΛ༻ҙ͢Δͷ໘ - ελοϑͰϢʔβͱύεϫʔυΛڞ༗͢Δͷආ͚͍ͨ ɾϢʔβࣾπʔϧΛ͏࣌සൟʹύεϫʔυೖྗ͕ඞཁ ՝ᶆɿtoil͕ٸ૿͢Δ
ઓᶆ ΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ 117
ઓᶆɿΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ 118 nginx + omniauthʹΑΔࣾπʔϧͷΞΫηε੍ޚ ɾࣾһೖࣾ͢ΔͱඞͣG SuiteͷΞΧϯτ͕༩͞ΕΔ ɾࣾπʔϧͷΞΫηε੍ޚ nginx +
nginx_omniauth_adapter - ࣾ༻ͷG SuiteΞΧϯτͰ͋Δ͜ͱΛݕূ - VPNෆཁ - πʔϧຖʹΞΧϯτཧBasicೝূͷಋೖ͕ཁΒͳ͘ͳΔ
119 nginx + omniauthʹΑΔࣾπʔϧͷΞΫηε੍ޚ ઓᶆɿΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ
120 SRE͔͠Ͱ͖ͳ͍ΞΧϯτཧͷݖݶҠৡ ɾLDAPɺActive DirectoryͷΞΧϯτཧͷݖݶΛҠৡ - Github EnterpriseαʔόͷύεϫʔυΛࣗͰઃఆͰ͖Δ ɾ։ൃऀ͕͏ssh keyΛ֤छαʔόͷࣗಈσϓϩΠ -
Gatewayͷssh keyैདྷitamaeͳͲͰσϓϩΠ͍ͯͨ͠ - ਓ͕ଟ͍ͱରԠ͢Δͷਏ͍ - ࣗͰઃఆͰ͖ͯɺࣗಈతʹσϓϩΠ͞ΕΔ ઓᶆɿΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ
121 1PXFSFECZ!TPSBI ઓᶆɿΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ
122 DBUFHPSZ PGJTTVFT "DDPVOU.BOBHFNFOU "843FMBUFE0QFSBUJPOT )VHF%BUBCBTF5BCMF.JHSBUJPOT 20171~5݄ʹ૿Ճͨ͠toilτοϓ3ͷݱࡏ
ઓᶆɿΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ
123 20171~5݄ʹ૿Ճͨ͠toilτοϓ3ͷݱࡏ DBUFHPSZ PGJTTVFT "DDPVOU.BOBHFNFOU "843FMBUFE0QFSBUJPOT )VHF%BUBCBTF5BCMF.JHSBUJPOT
݅·Ͱݮগ ݅·Ͱݮগ ࣗಈԽத ઓᶆɿΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ
124 ֤छࢪࡦΛܦͯSREͷґཔݮগ +BO 'FC .BS
"QS .BZ +VO +VM "VH 4FQ 0DU /PW %FD 2017ͷ֤݄ͷSREͷґཔͷਪҠ ઓᶆɿΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ
՝ᶆ “toil͕૿͑Δ”ͷ͏ҰͭͷऔΓΈ 125
126 SREͷϚϧνϦʔδϣϯରԠ ՝ᶆ “toil͕૿͑Δ”ͷ͏ҰͭͷऔΓΈ Japan UK US
·ͱΊ 127
·ͱΊ 128 ɾαʔϏε৫த ɾάϩʔόϧͳΒͰͷऔΓΈ͕͋Δ ɾSREͱͯ͠ઓͰ͖Δ՝͕ࢁ΄Ͳ͋Δ
Thank you !! 129
[email protected]