Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Challenges for Global Service from a Perspectiv...
Search
Takayuki WATANABE (渡辺 喬之)
February 10, 2018
Technology
3
3.2k
Challenges for Global Service from a Perspective of SRE
Cookpad TechConf 2018:
https://techconf.cookpad.com/2018/
Takayuki WATANABE (渡辺 喬之)
February 10, 2018
Tweet
Share
More Decks by Takayuki WATANABE (渡辺 喬之)
See All by Takayuki WATANABE (渡辺 喬之)
[Money Forward x Shippio] BaySide-Tech-Nite (May 19, 2023)
takanabe
0
110
[Developers Summit 2023] ソフトウェアテスト新時代の幕開け: 機械学習とデータサイエンスで実現するテスト運用の高度化
takanabe
26
11k
SRE NEXT 2022: Sensible Incident Management for Software Startups
takanabe
2
7.6k
SRE NEXT 2020 [C6] Designing fault-tolerant microservices with SRE and circuit breaker centric architecture
takanabe
1
7.6k
Challenges for Global Service from a Perspective of SRE 2nd season
takanabe
3
4.5k
Practical Approaches to Achieve Continuous Deployment with Kubernetes
takanabe
0
230
Building infrastructure on AWS with Ruby
takanabe
0
290
SRE vs ラマダン
takanabe
0
230
Other Decks in Technology
See All in Technology
コード✕AIーソフトウェア開発者のための生成AI実践入門~
yuhattor
4
860
Qdrant を用いた検索改善施策の紹介 / Search Engineering Tech Talk 2024 Summer
visional_engineering_and_design
1
210
The People First Approach to Engineering Success - DevNot 2024
zikriyeurkmez
0
170
Oracle Database 23ai 新機能#4 Rolling Maintenance
oracle4engineer
PRO
0
140
SageMaker学習のツボ / The Key Points of Learning SageMaker
cmhiranofumio
0
200
入社半年(合計1年)でGoogle Cloud 認定を全冠した秘訣🤫
risatube
1
240
From naive to advanced RAG: the complete guide
glaforge
0
190
プロダクト開発の貢献をアピールするための目標設計や認知活動 / Goal design and recognition activities to promote product development contributions.
oomatomo
5
930
LINE-ChatGPT 倫理問題を整理する全力肯定彼氏くん [LuC4]に訪れたサービス開始以来の最大の危機
o_ob
2
160
I tried the newly introduced certification "Applied Skills" on Microsoft Learn
mappie_kochi
0
250
【㈱アイモバイル】エンジニア向け会社説明資料
imobile
0
470
とある事業会社にとっての Kaggler の魅力
hakubishin3
5
980
Featured
See All Featured
The Straight Up "How To Draw Better" Workshop
denniskardys
232
130k
What the flash - Photography Introduction
edds
67
11k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
7
590
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
110
6.9k
Building a Scalable Design System with Sketch
lauravandoore
459
33k
The World Runs on Bad Software
bkeepers
PRO
65
11k
5 minutes of I Can Smell Your CMS
philhawksworth
202
19k
Building Flexible Design Systems
yeseniaperezcruz
327
38k
Infographics Made Easy
chrislema
239
18k
Fireside Chat
paigeccino
32
3k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
131
33k
Transcript
Challenges for Global Service from a Perspective of SRE Takayuki
Watanabe Infrastructure Department SRE Group Cookpad Inc. Feb 10, 2018
Who? 2
ࣗݾհ ɾΠϯϑϥετϥΫνϟʔ෦ SREάϧʔϓ - ΫοΫύουͷάϩʔόϧαʔϏεͷ։ൃʹैࣄ ɾSRE = Site Reliability Engineering
- System & Software EngineeringͰ αΠτͷ৴པੑʹؔΘΔશͯͷ՝ʹऔΓΉ 3 ลڤ೭ (Takayuki Watanabe) twitter: takanabe_w / github: takanabe
ࠓ͢͜ͱ 4 ɾΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ ɾ2017ͷάϩʔόϧαʔϏεͷ ɾάϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝ͱઓ
ΫοΫύουͷ άϩʔόϧαʔϏεͬͯԿʁ 5
ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ 6 ւ֎͚ͷϨγϐαʔϏε
ւ֎͚ͷϨγϐαʔϏε 7 ຊͷΫοΫύουͱ ҟͳΔαʔϏε ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
ւ֎͚ͷϨγϐαʔϏε 8 ݄ؒར༻ऀ3000ສਓҎ্ ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ ※ 2017 1ʙ12݄ฏۉ
ւ֎͚ͷϨγϐαʔϏε 9 22ݴޠ ɾ68ΧࠃͰల։ ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ ※ 2017 12݄࣌
ւ֎͚ͷϨγϐαʔϏε 10 WebɾAndroidɾiOSͰఏڙத (ίʔυϕʔεࠃͱҟͳΔ) ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
11 UK Office (Bristol)
12 UK Office (Bristol) ϓϩμΫτ։ൃڌUK(Bristol)
13 UK Office (Bristol) ౦ژɺεϖΠϯɺΠϯυωγΞΛ த৺ʹελοϑੈքதʹࡏ
ɾຊҎ֎ͷࠃΛରʹ։ൃ͍ͯ͠ΔϨγϐαʔϏε ɾ݄ؒར༻ऀ3000ສਓҎ্ ɾ22ݴޠͰ68ΧࠃʹαʔϏεΛల։ ɾWebɺAndroidɺiOSͰఏڙத(ίʔυϕʔεࠃͱҟͳΔ) ɾੈքதͷΤϯδχΞ͕։ൃʹࢀՃ 14 ւ֎͚ͷϨγϐαʔϏε ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
2017ͷάϩʔόϧ αʔϏεͷ 15
ରԠݴޠ 16
2017ͷάϩʔόϧαʔϏεͷ 17 ରԠݴޠ͕૿Ճ 15ݴޠ 201612݄࣌
2017ͷάϩʔόϧαʔϏεͷ 18 ରԠݴޠ͕૿Ճ 15ݴޠ 22ݴޠ 20179݄࣌ 201612݄࣌
2017ͷάϩʔόϧαʔϏεͷ 19 ରԠݴޠ͕૿Ճ 15ݴޠ 22ݴޠ + 7 ݴޠ 201612݄࣌ 20179݄࣌
2017ͷάϩʔόϧαʔϏεͷ 20 ରԠݴޠ͕૿Ճ 15ݴޠ 22ݴޠ શݴޠରԠ ≠ ੈքରԠ 201612݄࣌ 20179݄࣌
21
22 Ҭੑͷҧ͍Λߟྀͨ͠αʔϏε։ൃ͕ඞཁ ref: https://techconf.cookpad.com/2017/rejasupotaro.html
ల։தͷࠃͷ 23
2017ͷάϩʔόϧαʔϏεͷ 24 ల։தͷࠃͷ͕૿Ճ 58ϲࠃ 201612݄࣌
2017ͷάϩʔόϧαʔϏεͷ 25 ల։தͷࠃͷ͕૿Ճ 58ϲࠃ 68ϲࠃ 201612݄࣌ 20179݄࣌
2017ͷάϩʔόϧαʔϏεͷ 26 ల։தͷࠃͷ͕૿Ճ 58ϲࠃ 68ϲࠃ + 10ϲࠃ 201612݄࣌ 20179݄࣌
UKΦϑΟεͰಇࣾ͘һ 27
2017ͷάϩʔόϧαʔϏεͷ 28 UKΦϑΟεͰಇࣾ͘һ 5ਓ 201612݄࣌
2017ͷάϩʔόϧαʔϏεͷ 29 UKΦϑΟεͰಇࣾ͘һ 5ਓ 50ਓ 201712݄࣌ 201612݄࣌
2017ͷάϩʔόϧαʔϏεͷ 30 UKΦϑΟεͰಇࣾ͘һ 5ਓ 50ਓ 10ഒ 201612݄࣌ 201712݄࣌
2017ͷάϩʔόϧαʔϏεͷ 31 Google Playʮϕετ Φϒ 2017ʯͰ7Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί
ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@
2017ͷάϩʔόϧαʔϏεͷ 32 Google Playʮϕετ Φϒ 2017ʯͰ7Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί
ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ ຊൃͷΞϓϦͰ ࠷ଟͷ7ҬͰಉ࣌ೖ
2017ͷάϩʔόϧαʔϏεͷ 33 Google Playʮϕετ Φϒ 2017ʯͰ7Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί
ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ αʔϏε৫͖ͯͨ͠
2017ͷάϩʔόϧαʔϏεͷ 34 Google Playʮϕετ Φϒ 2017ʯͰ7Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί
ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ ҰํͰࢁͷ՝͋ͬͨ
άϩʔόϧαʔϏεʹ͓͚Δ SREͷ՝ͱઓ 35
36 άϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝ͱઓ ՝ᶃɿಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ ՝ᶄɿΠϕϯτ͕ͱʹ͔͘ଟ͍ ՝ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍ ՝ᶆɿtoil ͕ٸ૿͢Δ άϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝
՝ᶃ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ 37
38 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕͍ ͙ݕࡧͰ͖ͳ͍
39 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕͍ ͙ݕࡧͰ͖ͳ͍ ༷ʑͳࠃʹαʔϏεΛల։͢Δ͜ͱͰ ࠃ͝ͱʹϢʔβମݧʹ͕ࠩग़࢝Ίͨ
40 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕͍ ͙ݕࡧͰ͖ͳ͍ ͦͷࠃʹॅΜͰ͍ͳ͍ͷͰ ಉ͡ڥͰςετͰ͖ͳ͍
41 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕͍ ͙ݕࡧͰ͖ͳ͍ ͕࣌ࠩ͋ΔࠃͷਓͱϝοηʔδΛ ަΘͤΔͷଟͯ͘1ճ
42 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕͍ ͙ݕࡧͰ͖ͳ͍ ݪҼ͕Θ͔Βͳ͍ͱ վળͰ͖ͳ͍
ઓᶃ ੈքதͷϢʔβମݧΛଌఆ͠վળ͢Δ 43
ɾఆ؍ଌʹCatchpoint SystemsͷSynthetic MonitoringΛར༻ - 30Χࠃ͔ΒఆظతʹϝτϦΫεΛऩू - Time To First Byte
- DNS lookup time - TLS established time - ϨεϙϯελΠϜ ͳͲ 44 ఆ؍ଌʹΑΔ֤ࠃͷϢʔβମݧͷଌఆ ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ
45 ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
46 ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
47 ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
48 ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ ݪҼΛௐࠪͰ͖Δঢ়ଶʹͳͬͨ
49 ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ ΠϯυωγΞͷTTFB ΠϯυωγΞͷTLSଓ࣌ؒ ถࠃͷTTFB ถࠃͷTLSଓ࣌ؒ 750ms 330ms 225ms
150ms
50 ΠϯυωγΞถࠃͱൺͯ TLSଓTime To First Byteʹ ͕͔͔࣌ؒΔ ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
51 ถࠃ͔Βཧతʹԕ͍ࠃͷϢʔβମݧ ͕ѱ͍ʹ͋Δ͜ͱ͕໌ ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
άϩʔόϧαʔϏεʹ͓͚ΔSREͷઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍ 52 SFGIUUQTUFDIDPOGDPPLQBEDPNTPSBIIUNM
άϩʔόϧαʔϏεʹ͓͚ΔSREͷઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍ 53 ɾαʔόถࠃϦʔδϣϯͰཧ ɾcookpad.comυϝΠϯΛࠃαʔϏεͱڞ༗ ɾRoute53ͷLatency Based RoutingͰIPΛฦ٫ - ౦ژ͔ถࠃͷ͍ۙσʔληϯλ·ͣΞΫηε
ɾࠃͱάϩʔόϧαʔϏεΛৼΓ͚ - ϦόʔεϓϩΩγͷਖ਼نදݱͰఆ - cookpad.comຊͷαʔϏε - cookpad.com/ukͳͲάϩʔόϧαʔϏε
άϩʔόϧαʔϏεʹ͓͚ΔSREͷઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍ 54 location ~ ^/(en|vi|fil|ae|al|ao|ar|arabic|at|au|az|ba|bd|be|bg|bh|bo| br|by|ch|ci|cl|cn|co|cr|cu|cy|cz|de|dj|dk|do|dz|ec|ee|eg| es|fi|fr|gh|gr|gt|hk|hn|hr|hu|id|ie|il|in|iq|ir|it|jo| ke|km|kr|kw|kz|lb|lk|lt|lu|lv|ly|ma|mo|mr|mx|my|mz|ng|ni| nl|no|nz|om|pa|pe|ph|pk|pl|ps|pt|py|qa|ro|rs|ru|sa|
sd|se|sg|si|sk|so|sv|sy|th|tn|tr|tt|tw|ua|uk|us|uy|uz|ve| vn|ye|za)(/|$) { proxy_pass http://cookpad_use1; } location / { proxy_pass https://cookpad_apne; }
55 ถࠃ͔Βԕ͍ࠃͷϢʔβ͕շదʹ͑ΔΑ͏ʹ͢Δ ɾΞϝϦΧʹαʔό͕ଘࡏ͍ͯͯ͠ωοτϫʔΫϨΠςϯγ͕ߴ͍ ɾӡ༻ίετ্͕͕Δٕज़·ͩೖΕͨ͘ͳ͍ - αʔόͷϚϧνϦʔδϣϯԽͳͲ ɾΠϯυωγΞҎ֎ͷࠃগͳ͍खͰޮతʹվળ͍ͨ͠ - ଞʹΒͳ͚ΕͳΒͳ͍͜ͱࢁ͋Δ ઓᶃɿ
ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ
56 ถࠃ͔Βԕ͍ࠃͷϢʔβ͕շదʹ͑ΔΑ͏ʹ͢Δ ɾΞϝϦΧʹαʔό͕ଘࡏ͍ͯͯ͠ωοτϫʔΫϨΠςϯγ͕ߴ͍ ɾӡ༻ίετ্͕͕Δٕज़·ͩೖΕͨ͘ͳ͍ - αʔόͷϚϧνϦʔδϣϯԽͱ͔ ɾΠϯυωγΞҎ֎ͷࠃগͳ͍खͰޮతʹվળ͍ͨ͠ - ଞʹΒͳ͚ΕͳΒͳ͍͜ͱࢁ͋Δ ઓᶃɿ
ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ CDNʹΑΔTCPͱTLSͷऴͰ ωοτϫʔΫϨΠςϯγΛվળ͢Δ
57 FastlyʹΑΔϨΠςϯγͷվળ ɾWeb / API શͯͷϦΫΤετΛFastlyܦ༝ʹ - Ωϟογϡͱ͍ͯͬͯ͠ΔΘ͚Ͱͳ͍ ɾFastlyͰTCPͱTLSͷऴΛ࣮ݱ -
Ϣʔβͱσʔληϯλͷڑ͕ۙ͘ͳΔ - TCP/TLSͷϋϯυγΣΠΫʹ͔͔Δ͕࣌ؒॖ͞ΕΔ ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
58 FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ Fastlyಋೖલ(1.25 sec) ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
59 FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ Fastlyಋೖલ(1.25 sec) Fastlyಋೖޙ(0.45 sec) ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
60 FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ Fastlyಋೖલ(1.25 sec) Fastlyಋೖޙ(0.45 sec) ΠϯυωγΞϢʔβͷ ϨεϙϯελΠϜ1/3ʹॖ ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
61 ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ Ξϧθϯνϯ ΠϯυωγΞ αδΞϥϏΞ UAE ΠΪϦε Fastlyಋೖ Fastlyಋೖ
Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ
62 ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ Ξϧθϯνϯ ΠϯυωγΞ αδΞϥϏΞ UAE ΠΪϦε Fastlyಋೖ Fastlyಋೖ
Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ ཧతʹถࠃ͔Βԕ͍ࠃͷ Ϣʔβମݧ͕վળ
՝ᶄ Πϕϯτ͕ͱʹ͔͘ଟ͍ 63
ɾ͜͜Ͱݴ͏ΠϕϯτॕࡇͳͲຖඞͣ։࠵͞ΕΔͷ ɾΠϕϯτͷࡍʹϢʔβ͕େྔʹΞΫηεͯ͘͠Δ ɾେ͖ͳΠϕϯτࠃफڭͳͲͷ୯ҐͰ࠵͞ΕΔ 64 ΠϕϯτͷόϦΤʔγϣϯ͕๛ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
ɾ͜͜Ͱݴ͏ΠϕϯτॕࡇͳͲຖඞͣ։࠵͞ΕΔͷ ɾΠϕϯτͷࡍʹϢʔβ͕େྔʹΞΫηεͯ͘͠Δ ɾେ͖ͳΠϕϯτࠃफڭͳͲͷ୯ҐͰ࠵͞ΕΔ - Ξϧθϯνϯ 65 ΠϕϯτͷόϦΤʔγϣϯ͕๛ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
66 ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
67 ֵ໋ه೦ɺಠཱه೦ʹ ύεςϦʔτεΛ࡞ͬͯ৯Δ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
ɾ͜͜Ͱݴ͏ΠϕϯτॕࡇͳͲຖඞͣ։࠵͞ΕΔͷ ɾΠϕϯτͷࡍʹϢʔβ͕େྔʹΞΫηεͯ͘͠Δ ɾେ͖ͳΠϕϯτࠃफڭͳͲͷ୯ҐͰ࠵͞ΕΔ - Ξϧθϯνϯ - ΠεϥϜݍ 68 ΠϕϯτͷόϦΤʔγϣϯ͕๛ ՝ᶄɿ
Πϕϯτ͕ͱʹ͔͘ଟ͍
69 ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
70 ΠυΡϧΞυϋʢ٘ਜ਼ࡇʣ ͰΛ৯Δ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
71 ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
72 ϥϚμϯظؒ ͔Β໌͚ͷؒͰ ՈͰ৯ࣄΛָ͠Ή ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
ɾຊͷ߹ɺόϨϯλΠϯͰϦΫΤετ͕2ഒʹͳΔ 73 ΠϕϯτʹΑΔϢʔβϦΫΤετͷมԽ ٸʹϦΫΤετ͕Ͷ্͕Δ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
74 ΠϕϯτʹΑΔϢʔβϦΫΤετͷมԽ ٸʹϦΫΤετ͕Ͷ্͕Δ ɾΠυΡϧΞυϋ͕࢝·ΔͱͷؒϦΫΤετ͕2ഒʹͳΔ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
ɾϥϚμϯ͕࢝·Δͱ1ϲ݄ͷؒϦΫΤετ͕2ʙ3ഒʹͳΔ 75 ΠϕϯτʹΑΔϢʔβϦΫΤετͷมԽ ٸʹϦΫΤετ͕Ͷ্͕Δ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
ɾւ֎ͷϨγϐαʔϏε͕CookpadͷؒʹͳΔ͜ͱ͋Δ 76 1Χࠃ૿͑ͯɺಥવϢʔβ͕100ສਓ૿͑Δέʔε͕༗Δ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
ɾւ֎ͷϨγϐαʔϏε͕CookpadͷؒʹͳΔ͜ͱ͋Δ 77 1Χࠃ૿͑ͯɺಥવϢʔβ͕100ສਓ૿͑Δέʔε͕༗Δ ՝ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍ ࠃͷ͕૿͑ͯ Πϕϯτ૿͑Δ
78 ຖ݄όϨϯλΠϯ —
79 શͯͷΠϕϯτʹ SRE͕ରࡦ͢Δͷ͍͠
ઓᶄ γεςϜͱ৫ͷεέʔϥϏϦςΟΛ ߟྀͨ͠ΈΛೖΕΔ 80
81 εέʔϥϏϦςΟɾՄ༻ੑΛߟྀٕͨ͠ज़ͷ࠾༻ ɾϓϩμΫγϣϯͷσʔλϕʔεશAmazon AuroraΛར༻ ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
82 DockerΞϓϦ։ൃڥͷఏڙ ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
ɾຊͰഓΘΕͨECS + hakoʹΑΔσϓϩΠγεςϜΛಋೖ - ΦʔτεέʔϧΛඪ४ࡌ (ϨΨγΞϓϦࣗͰ࣮͢Δඞཁ͕͋ͬͨ) - ։ൃεέʔϧ͢Δ - ։ൃऀґཔͳ͠ʹڥมൿಗใΛมߋՄೳ
- ECSΫϥελʹίϯςφΛσϓϩΠ͢ΔͷͰΠϯελϯεͷηοτΞοϓෆཁ - hako-consoleͳͲͷࣾͷΤίγεςϜͷԸܙΛڗड 83 ref https://speakerdeck.com/eagletmt/web-application-development-in-cookpad-2017 DockerΞϓϦ։ൃڥͷఏڙ ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
84 hako-console 1PXFSFECZ!FBHMFUNU ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
85 hako-console 1PXFSFECZ!FBHMFUNU ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
86 hako-console 1PXFSFECZ!FBHMFUNU ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ άϩʔόϧαʔϏεͰ ຊͰͷٕज़ܦݧΛ ׆͔͍ͯ͠Δ
87 ֤छμογϡϘʔυͷ࡞ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕σʔλετΞͷεϩʔΫΤϦΛੳͰ͖ΔΑ͏ʹ͢Δ ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
88 ֤छμογϡϘʔυͷ࡞ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕ओཁͳϝτϦΫεΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
89 ֤छμογϡϘʔυͷ࡞ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕HTTPͷεςʔλείʔυมԽΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
90 ֤छμογϡϘʔυͷ࡞ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕HTTPͷεςʔλείʔυมԽΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ ։ൃऀ͕ࣗͰΛൃݟͯ͠ ηϧϑϚωδϝϯτͰ͖Δ ઓᶄɿγεςϜͱ৫ͷεέʔϥϏϦςΟΛߟྀͨ͠ΈΛೖΕΔ
՝ᶅ σϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍ 91
92 ຊͰՄೳͳΦϖϨʔγϣϯւ֎ͰͰ͖ͳ͍͜ͱ͋Δ ɾ࣌ؒίωΫγϣϯΛҡ࣋Ͱ͖ͳ͍ڥ - ωοτϫʔΫڥ͕ຊͷΑ͏ʹ͍ͬͯΔͱݶΒͳ͍ - ւ֎ͰఀిɾߑਫͳͲͷҟৗࣄଶ͕ৗతʹى͜Δ - खݩ͔ΒCapistranoͰσϓϩΠͰ͖ΔਓɾͰ͖ͳ͍ਓ͕͍Δ ՝ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
93 ຊͰՄೳͳΦϖϨʔγϣϯւ֎ͰͰ͖ͳ͍͜ͱ͋Δ ɾ࣌ؒίωΫγϣϯΛҡ࣋Ͱ͖ͳ͍ڥ - ωοτϫʔΫڥ͕ຊͷΑ͏ʹ͍ͬͯΔͱݶΒͳ͍ - ఀిɾߑਫͳͲͷҟৗࣄଶී௨ʹى͖Δ - खݩ͔ΒCapistranoͰσϓϩΠͰ͖ΔਓɾͰ͖ͳ͍ਓ͕͍Δ ͦͦσϓϩΠͰ͖ͳ͍
՝ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
94 σϓϩΠΛଞͷਓʹґཔ͢Δੈքઢ͕ੜ͢Δ ɾखݩ͔ΒσϓϩΠ͕Ͱ͖ͳ͍։ൃऀ͔ΒσϓϩΠґཔ͕དྷΔ ɾσϓϩΠͷཧ͓ޓ͍ʹγϯυΠ - ։ൃऀ1ʹproductionʹԿճσϓϩΠ͢Δ - ґཔ͢Δํɺґཔ͞ΕΔํγϯυΠ - σϓϩΠʹର͢Δ৺ཧతͳίετ্͕͕ͬͯ͠·͏
- ৽͍͠ػೳΛϦϦʔε͍ͨ͠ͷʹཧσϓϩΠऀͪ͋Γ͏Δ ՝ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
ઓᶅ ੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ 95
96 σϓϩΠαʔόΛถࠃϦʔδϣϯʹ༻ҙ͢Δ ɾssh ͰσϓϩΠαʔόʹΞΫηεͯ͠Β͍CapistranoΛ࣮ߦ ɾཧσϓϩΠͷѱເ͔Βղ์͞ΕΔ ઓᶅɿੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ
97 σϓϩΠαʔόͰͷϚχϡΞϧΦϖϨʔγϣϯͷ ɾΦϖϨʔγϣϯ͕౷Ұ͞Εͳ͍ - ։ൃऀ͕࠷৽ͷσϓϩΠεΫϦϓτΛ͏ͱݶΒͳ͍ - ఆ֎ͷ͜ͱΛαʔόͰߦ͏ਓ͕͍Δ ɾαʔόʹΞΫηε͢Δͷ͕໘ ɾσϓϩΠͷϩάΛ͍ʹ͍͘ ઓᶅɿੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ
98 σϓϩΠڥͷ৽ chat (slack) deploy server (cap & hako) bot
(ruboty) job scheduler (Rundeck) deploy targets ઓᶅɿੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ
99 σϓϩΠαʔόͷϩάΠϯ͕ෆཁʹͳΔ ઓᶅɿੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ
100 ఆ͍ͯ͠ΔΦϖϨʔγϣϯʹ౷ҰͰ͖Δ ઓᶅɿੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ
101 σϓϩΠͷϩάΛ͏ͷ͕༰қʹͳΔ σϓϩΠ࣌ࠁ σϓϩΠͨ͠ਓ ઓᶅɿੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ
102 σϓϩΠͷϩάΛ͏ͷ͕༰қʹͳΔ ઓᶅɿੈքͷͲ͔͜ΒͰσϓϩΠͰ͖ΔΑ͏ʹ͢Δ
՝ᶆ toil ͕ٸ૿͢Δ 103
104 SFGIUUQTMBOEJOHHPPHMFDPNTSFCPPLIUNM ՝ᶆɿtoil͕ٸ૿͢Δ
ɾखಈͰରԠ͍ͯ͠Δ ɾࣗಈԽͷ༨͕͋Δ ɾ܁Γฦ͠ൃੜ͢Δ ɾൃੜ͔ͯ͠Β͡Όͳ͍ͱରԠͰ͖ͳ͍ ɾαʔϏεͷվળʹͭͳ͕Βͳ͍ ɾαʔϏεϢʔβͷʹൺྫͯ͠૿Ճ͢Δ 105 toilʹͯ·Δۀ ՝ᶆɿtoil͕ٸ૿͢Δ SFGIUUQTMBOEJOHHPPHMFDPNTSFCPPLDIBQUFSTFMJNJOBUJOHUPJMIUNM
106 ։ൃऀ͔ΒSREͷґཔ͕ٸ૿
֤ظؒͷSREͷґཔͷਪҠ ՝ᶆɿtoil͕ٸ૿͢Δ
107 ۀׂ߹ͷมԽ /PW %FD +BO
'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH ՝ᶆɿtoil͕ٸ૿͢Δ
108 ۀׂ߹ͷมԽ /PW %FD +BO
'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH SRE͕Γ͍ͨ͜ͱ ՝ᶆɿtoil͕ٸ૿͢Δ
109 ۀׂ߹ͷมԽ /PW %FD +BO
'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH toilͷ૿Ճʹൺྫͯ͠ ׂ͕͚࣌ؒͳ͘ͳΔ ՝ᶆɿtoil͕ٸ૿͢Δ
110 SREͷۀׂ߹ͷมԽ /PW %FD +BO
'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH Γ͍ͨ͜ͱ͕Ͱ͖ͳ͍ ՝ᶆɿtoil͕ٸ૿͢Δ
111 20171~5݄ʹ૿Ճͨ͠toilτοϓ3 ΧςΰϦ ґཔͷ "DDPVOU.BOBHFNFOU "843FMBUFE0QFSBUJPOT )VHF%BUBCBTF5BCMF.JHSBUJPOT
՝ᶆɿtoil͕ٸ૿͢Δ
112 20171~5݄ʹ૿Ճͨ͠toilτοϓ3 DBUFHPSZ PGJTTVFT "DDPVOU.BOBHFNFOU "843FMBUFE0QFSBUJPOT )VHF%BUBCBTF5BCMF.JHSBUJPOT
ΞΧϯτཧճΓ ͷґཔͷᓲ໓͕ඞཁ ՝ᶆɿtoil͕ٸ૿͢Δ
113 ΞΧϯτཧ͕ਏ͍ཧ༝ ɾࣾһ͕ٸ૿͍ͯ͠Δ͠ɺࠓޙ૿͑Δ͜ͱ͕૾Ͱ͖Δ ɾ։ൃऀ͕ར༻͢Δπʔϧ͕ଟ༷Խ͖͍ͯͯ͠Δ - Single Sign OnʹରԠ͍ͯ͠ͳ͍πʔϧ͋Δ - ͰɺΞΫηε੍ޚ͍ͨ͠
ɾ͕࣌ࠩ͋ΔͷͰίϛϡχέʔγϣϯͷλΠϛϯάΛҳ͍͢͠ - Կ͕͍͔ͨ͠Ѳ͢Δͷʹ͔͔Δ͜ͱ ՝ᶆɿtoil͕ٸ૿͢Δ
114 άϩʔόϧͰར༻࣮ͷ͋Δπʔϧ܈ ɾੳɿ Kibanaɺ GrafanaɺNewRelicɺCatchpointɺDatadog ɾίʔυཧɿ Github Enterpriseɺgithub.com ɾόάτϥοΧɿSentryɺBugsnagɺAirbrake ɾCIɿJenkinsɺCircle
CIɺSemaphore CI ɾPagerɿPagerDuty ɾͦͷଞɿAWSɺKuroko2ɺin-house πʔϧ ͳͲ ՝ᶆɿtoil͕ٸ૿͢Δ
115 ࣾπʔϧͷΞΫηεʹVPNΛ͑ΔͱݶΒͳ͍ ɾੈքதͷΦϑΟεͷ։ൃऀ͕VPNΛ͑Δͷ͔ʁ - શһ͕IPΞυϨεͰ੍ޚͰ͖Δڥʹ͍ͳ͍ ɾ౦ژͷࣾNWΛܦ༝ͨ͠ଓଥ͔ʁ - ࣾπʔϧʹੈքதͷελοϑ͕ΞΫηε͢Δ - ΞϑϦΧͷελοϑVPNΛ҆ఆͯ͠ுΕΔͩΖ͏͔
՝ᶆɿtoil͕ٸ૿͢Δ
116 ࣾπʔϧʹBasicೝূΛ͍ͨ͘ͳ͍ ɾSREࣾπʔϧ͕૿͑ΔʹBasicೝূͷઃఆΛ͢Δ - Ϣʔβ͝ͱʹύεϫʔυΛ༻ҙ͢Δͷ໘ - ελοϑͰϢʔβͱύεϫʔυΛڞ༗͢Δͷආ͚͍ͨ ɾϢʔβࣾπʔϧΛ͏࣌සൟʹύεϫʔυೖྗ͕ඞཁ ՝ᶆɿtoil͕ٸ૿͢Δ
ઓᶆ ΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ 117
ઓᶆɿΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ 118 nginx + omniauthʹΑΔࣾπʔϧͷΞΫηε੍ޚ ɾࣾһೖࣾ͢ΔͱඞͣG SuiteͷΞΧϯτ͕༩͞ΕΔ ɾࣾπʔϧͷΞΫηε੍ޚ nginx +
nginx_omniauth_adapter - ࣾ༻ͷG SuiteΞΧϯτͰ͋Δ͜ͱΛݕূ - VPNෆཁ - πʔϧຖʹΞΧϯτཧBasicೝূͷಋೖ͕ཁΒͳ͘ͳΔ
119 nginx + omniauthʹΑΔࣾπʔϧͷΞΫηε੍ޚ ઓᶆɿΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ
120 SRE͔͠Ͱ͖ͳ͍ΞΧϯτཧͷݖݶҠৡ ɾLDAPɺActive DirectoryͷΞΧϯτཧͷݖݶΛҠৡ - Github EnterpriseαʔόͷύεϫʔυΛࣗͰઃఆͰ͖Δ ɾ։ൃऀ͕͏ssh keyΛ֤छαʔόͷࣗಈσϓϩΠ -
Gatewayͷssh keyैདྷitamaeͳͲͰσϓϩΠ͍ͯͨ͠ - ਓ͕ଟ͍ͱରԠ͢Δͷਏ͍ - ࣗͰઃఆͰ͖ͯɺࣗಈతʹσϓϩΠ͞ΕΔ ઓᶆɿΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ
121 1PXFSFECZ!TPSBI ઓᶆɿΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ
122 DBUFHPSZ PGJTTVFT "DDPVOU.BOBHFNFOU "843FMBUFE0QFSBUJPOT )VHF%BUBCBTF5BCMF.JHSBUJPOT 20171~5݄ʹ૿Ճͨ͠toilτοϓ3ͷݱࡏ
ઓᶆɿΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ
123 20171~5݄ʹ૿Ճͨ͠toilτοϓ3ͷݱࡏ DBUFHPSZ PGJTTVFT "DDPVOU.BOBHFNFOU "843FMBUFE0QFSBUJPOT )VHF%BUBCBTF5BCMF.JHSBUJPOT
݅·Ͱݮগ ݅·Ͱݮগ ࣗಈԽத ઓᶆɿΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ
124 ֤छࢪࡦΛܦͯSREͷґཔݮগ +BO 'FC .BS
"QS .BZ +VO +VM "VH 4FQ 0DU /PW %FD 2017ͷ֤݄ͷSREͷґཔͷਪҠ ઓᶆɿΞΧϯτཧʹؔ͢ΔґཔΛᓲ໓͢Δ
՝ᶆ “toil͕૿͑Δ”ͷ͏ҰͭͷऔΓΈ 125
126 SREͷϚϧνϦʔδϣϯରԠ ՝ᶆ “toil͕૿͑Δ”ͷ͏ҰͭͷऔΓΈ Japan UK US
·ͱΊ 127
·ͱΊ 128 ɾαʔϏε৫த ɾάϩʔόϧͳΒͰͷऔΓΈ͕͋Δ ɾSREͱͯ͠ઓͰ͖Δ՝͕ࢁ΄Ͳ͋Δ
Thank you !! 129
[email protected]