Challenges for Global Service from a Perspective of SRE

Cookpad TechConf 2018: https://techconf.cookpad.com/2018/

Transcript

1. Challenges for Global Service from a Perspective of SRE Takayuki Watanabe Infrastructure Department SRE Group Cookpad Inc. Feb 10, 2018
2. Who? 2
3. ࣗݾ঺հ ɾΠϯϑϥετϥΫνϟʔ෦ SREάϧʔϓ - ΫοΫύουͷάϩʔόϧαʔϏεͷ։ൃʹैࣄ ɾSRE = Site Reliability Engineering - System & Software EngineeringͰ αΠτͷ৴པੑʹؔΘΔશͯͷ՝୊ʹऔΓ૊Ή 3 ౉ลڤ೭ (Takayuki Watanabe) twitter: takanabe_w / github: takanabe
4. ࠓ೔࿩͢͜ͱ 4 ɾΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ ɾ2017೥ͷάϩʔόϧαʔϏεͷ੒௕ ɾάϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝୊ͱ௅ઓ
5. ΫοΫύουͷ άϩʔόϧαʔϏεͬͯԿʁ 5
6. ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ 6 ւ֎޲͚ͷϨγϐαʔϏε
7. ւ֎޲͚ͷϨγϐαʔϏε 7 ೔ຊͷΫοΫύουͱ ҟͳΔαʔϏε ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
8. ւ֎޲͚ͷϨγϐαʔϏε 8 ݄ؒར༻ऀ਺3000ສਓҎ্ ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ ※ 2017೥ 1ʙ12݄ฏۉ
9. ւ֎޲͚ͷϨγϐαʔϏε 9 22ݴޠ ɾ68ΧࠃͰల։ ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ ※ 2017೥ 12݄຤࣌఺
10. ւ֎޲͚ͷϨγϐαʔϏε 10 WebɾAndroidɾiOSͰఏڙத (ίʔυϕʔε͸ࠃ಺ͱҟͳΔ) ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
11. 11 UK Office (Bristol)
12. 12 UK Office (Bristol) ϓϩμΫτ։ൃڌ఺͸UK(Bristol)
13. 13 UK Office (Bristol) ౦ژɺεϖΠϯɺΠϯυωγΞΛ த৺ʹελοϑ͸ੈքதʹ఺ࡏ
14. ɾ೔ຊҎ֎ͷࠃΛର৅ʹ։ൃ͍ͯ͠ΔϨγϐαʔϏε ɾ݄ؒར༻ऀ਺3000ສਓҎ্ ɾ22ݴޠͰ68ΧࠃʹαʔϏεΛల։ ɾWebɺAndroidɺiOSͰఏڙத(ίʔυϕʔε͸ࠃ಺ͱҟͳΔ) ɾੈքதͷΤϯδχΞ͕։ൃʹࢀՃ 14 ւ֎޲͚ͷϨγϐαʔϏε ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
15. 2017೥ͷάϩʔόϧ αʔϏεͷ੒௕ 15
16. ରԠݴޠ਺ 16
17. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 17 ରԠݴޠ਺͕૿Ճ 15ݴޠ 2016೥12݄຤࣌఺
18. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 18 ରԠݴޠ਺͕૿Ճ 15ݴޠ 22ݴޠ 2017೥9݄຤࣌఺ 2016೥12݄຤࣌఺
19. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 19 ରԠݴޠ਺͕૿Ճ 15ݴޠ 22ݴޠ + 7 ݴޠ 2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺
20. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 20 ରԠݴޠ਺͕૿Ճ 15ݴޠ 22ݴޠ શݴޠରԠ ≠ ੈքରԠ 2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺
22. 22 ஍Ҭੑͷҧ͍Λߟྀͨ͠αʔϏε։ൃ͕ඞཁ
 ref: https://techconf.cookpad.com/2017/rejasupotaro.html
23. ల։தͷࠃͷ਺ 23
24. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 24 ల։தͷࠃͷ਺͕૿Ճ 58ϲࠃ 2016೥12݄຤࣌఺
25. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 25 ల։தͷࠃͷ਺͕૿Ճ 58ϲࠃ 68ϲࠃ 2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺
26. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 26 ల։தͷࠃͷ਺͕૿Ճ 58ϲࠃ 68ϲࠃ + 10ϲࠃ 2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺
27. UKΦϑΟεͰಇࣾ͘һ਺ 27
28. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 28 UKΦϑΟεͰಇࣾ͘һ਺ 5ਓ 2016೥12݄຤࣌఺
29. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 29 UKΦϑΟεͰಇࣾ͘һ਺ 5ਓ ໿ 50ਓ 2017೥12݄຤࣌఺ 2016೥12݄຤࣌఺
30. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 30 UKΦϑΟεͰಇࣾ͘һ਺ 5ਓ ໿ 50ਓ 10ഒ 2016೥12݄຤࣌఺ 2017೥12݄຤࣌఺
31. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 31 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFG
    IUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@
    
32. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 32 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFG
    IUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@
    ೔ຊൃͷΞϓϦͰ ࠷ଟͷ7஍ҬͰಉ࣌ೖ৆
33. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 33 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFG
    IUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@
    αʔϏε΋૊৫΋੒௕͖ͯͨ͠
34. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 34 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFG
    IUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@
    ҰํͰ୔ࢁͷ՝୊΋͋ͬͨ
35. άϩʔόϧαʔϏεʹ͓͚Δ SREͷ՝୊ͱ௅ઓ 35
36. 36 άϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝୊ͱ௅ઓ ՝୊ᶃɿಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ ՝୊ᶄɿΠϕϯτ͕ͱʹ͔͘ଟ͍ ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍ ՝୊ᶆɿtoil ͕ٸ૿͢Δ άϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝୊
37. ՝୊ᶃ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ 37
38. 38 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕஗͍ ௚͙ݕࡧͰ͖ͳ͍
39. 39 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕஗͍ ௚͙ݕࡧͰ͖ͳ͍ ༷ʑͳࠃʹαʔϏεΛల։͢Δ͜ͱͰ ࠃ͝ͱʹϢʔβମݧʹ͕ࠩग़࢝Ίͨ
40. 40 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕஗͍ ௚͙ݕࡧͰ͖ͳ͍ ͦͷࠃʹॅΜͰ͍ͳ͍ͷͰ ಉ͡؀ڥͰςετͰ͖ͳ͍
41. 41 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕஗͍ ௚͙ݕࡧͰ͖ͳ͍ ͕࣌ࠩ͋ΔࠃͷਓͱϝοηʔδΛ ަΘͤΔͷ͸ଟͯ͘1೔਺ճ
42. 42 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕஗͍ ௚͙ݕࡧͰ͖ͳ͍ ݪҼ͕Θ͔Βͳ͍ͱ վળ͸Ͱ͖ͳ͍
43. ௅ઓᶃ ੈքதͷϢʔβମݧΛଌఆ͠վળ͢Δ 43
44. ɾఆ఺؍ଌʹ͸Catchpoint SystemsͷSynthetic MonitoringΛར༻ - ໿30Χࠃ͔ΒఆظతʹϝτϦΫεΛऩू - Time To First Byte - DNS lookup time - TLS established time - ϨεϙϯελΠϜ ͳͲ 44 ఆ఺؍ଌʹΑΔ֤ࠃͷϢʔβମݧͷଌఆ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFG
    IUUQUFDIMJGFDPPLQBEDPNFOUSZ
45. 45 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
46. 46 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
47. 47 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
48. 48 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ ݪҼΛௐࠪͰ͖Δঢ়ଶʹͳͬͨ
49. 49 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ ΠϯυωγΞͷTTFB ΠϯυωγΞͷTLS઀ଓ࣌ؒ ถࠃͷTTFB ถࠃͷTLS઀ଓ࣌ؒ ໿750ms ໿330ms ໿225ms ໿150ms
50. 50 ΠϯυωγΞ͸ถࠃͱൺ΂ͯ TLS઀ଓ΍Time To First Byteʹ ͕͔͔࣌ؒΔ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
51. 51 ถࠃ͔Β஍ཧతʹԕ͍ࠃͷϢʔβମݧ ͕ѱ͍܏޲ʹ͋Δ͜ͱ͕൑໌ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
52. άϩʔόϧαʔϏεʹ͓͚ΔSREͷ௅ઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍ 52 SFG
    IUUQTUFDIDPOGDPPLQBEDPNTPSBIIUNM
53. άϩʔόϧαʔϏεʹ͓͚ΔSREͷ௅ઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍ 53 ɾαʔό͸ถࠃϦʔδϣϯͰ؅ཧ ɾcookpad.comυϝΠϯΛࠃ಺αʔϏεͱڞ༗ ɾRoute53ͷLatency Based RoutingͰIPΛฦ٫ - ౦ژ͔ถࠃͷ͍ۙσʔληϯλ·ͣΞΫηε ɾࠃ಺ͱάϩʔόϧαʔϏεΛৼΓ෼͚ - ϦόʔεϓϩΩγͷਖ਼نදݱͰ൑ఆ - cookpad.com͸೔ຊͷαʔϏε - cookpad.com/ukͳͲ͸άϩʔόϧαʔϏε
54. άϩʔόϧαʔϏεʹ͓͚ΔSREͷ௅ઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍ 54 location ~ ^/(en|vi|fil|ae|al|ao|ar|arabic|at|au|az|ba|bd|be|bg|bh|bo| br|by|ch|ci|cl|cn|co|cr|cu|cy|cz|de|dj|dk|do|dz|ec|ee|eg| es|fi|fr|gh|gr|gt|hk|hn|hr|hu|id|ie|il|in|iq|ir|it|jo| ke|km|kr|kw|kz|lb|lk|lt|lu|lv|ly|ma|mo|mr|mx|my|mz|ng|ni| nl|no|nz|om|pa|pe|ph|pk|pl|ps|pt|py|qa|ro|rs|ru|sa| sd|se|sg|si|sk|so|sv|sy|th|tn|tr|tt|tw|ua|uk|us|uy|uz|ve| vn|ye|za)(/|$) { proxy_pass http://cookpad_use1; } location / { proxy_pass https://cookpad_apne; }
55. 55 ถࠃ͔Βԕ͍ࠃͷϢʔβ͕շదʹ࢖͑ΔΑ͏ʹ͢Δ ɾΞϝϦΧʹαʔό͕ଘࡏ͍ͯͯ͠ωοτϫʔΫϨΠςϯγ͕ߴ͍ ɾӡ༻ίετ্͕͕Δٕज़͸·ͩೖΕͨ͘ͳ͍ - αʔόͷϚϧνϦʔδϣϯԽͳͲ ɾΠϯυωγΞҎ֎ͷࠃ΋গͳ͍ख਺Ͱޮ཰తʹվળ͍ͨ͠ - ଞʹ΋΍Βͳ͚Ε͹ͳΒͳ͍͜ͱ͸୔ࢁ͋Δ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFG
    IUUQUFDIMJGFDPPLQBEDPNFOUSZ
56. 56 ถࠃ͔Βԕ͍ࠃͷϢʔβ͕շదʹ࢖͑ΔΑ͏ʹ͢Δ ɾΞϝϦΧʹαʔό͕ଘࡏ͍ͯͯ͠ωοτϫʔΫϨΠςϯγ͕ߴ͍ ɾӡ༻ίετ্͕͕Δٕज़͸·ͩೖΕͨ͘ͳ͍ - αʔόͷϚϧνϦʔδϣϯԽͱ͔ ɾΠϯυωγΞҎ֎ͷࠃ΋গͳ͍ख਺Ͱޮ཰తʹվળ͍ͨ͠ - ଞʹ΋΍Βͳ͚Ε͹ͳΒͳ͍͜ͱ͸୔ࢁ͋Δ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFG
    IUUQUFDIMJGFDPPLQBEDPNFOUSZ CDNʹΑΔTCPͱTLSͷऴ୺Ͱ ωοτϫʔΫϨΠςϯγΛվળ͢Δ
57. 57 FastlyʹΑΔϨΠςϯγͷվળ ɾWeb / API શͯͷϦΫΤετΛFastlyܦ༝ʹ - Ωϟογϡͱͯ͠࢖͍ͬͯΔΘ͚Ͱ͸ͳ͍ ɾFastlyͰTCPͱTLSͷऴ୺Λ࣮ݱ - Ϣʔβͱσʔληϯλͷڑ཭͕ۙ͘ͳΔ - TCP/TLSͷϋϯυγΣΠΫʹ͔͔Δ͕࣌ؒ୹ॖ͞ΕΔ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
58. 58 FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ Fastlyಋೖલ(໿1.25 sec) ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
59. 59 FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ Fastlyಋೖલ(໿1.25 sec) Fastlyಋೖޙ(໿0.45 sec) ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
60. 60 FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ Fastlyಋೖલ(໿1.25 sec) Fastlyಋೖޙ(໿0.45 sec) ΠϯυωγΞϢʔβͷ ϨεϙϯελΠϜ͸໿1/3ʹ୹ॖ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
61. 61 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ Ξϧθϯνϯ ΠϯυωγΞ α΢δΞϥϏΞ UAE ΠΪϦε Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ
62. 62 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ Ξϧθϯνϯ ΠϯυωγΞ α΢δΞϥϏΞ UAE ΠΪϦε Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ ஍ཧతʹถࠃ͔Βԕ͍ࠃͷ Ϣʔβମݧ͕վળ
63. ՝୊ᶄ Πϕϯτ͕ͱʹ͔͘ଟ͍ 63
64. ɾ͜͜Ͱݴ͏Πϕϯτ͸ॕࡇ೔ͳͲຖ೥ඞͣ։࠵͞ΕΔ΋ͷ ɾΠϕϯτͷࡍʹ͸Ϣʔβ͕େྔʹΞΫηεͯ͘͠Δ ɾେ͖ͳΠϕϯτ͸ࠃ΍फڭͳͲͷ୯ҐͰ࠵͞ΕΔ 64 ΠϕϯτͷόϦΤʔγϣϯ͕๛෋ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
65. ɾ͜͜Ͱݴ͏Πϕϯτ͸ॕࡇ೔ͳͲຖ೥ඞͣ։࠵͞ΕΔ΋ͷ ɾΠϕϯτͷࡍʹ͸Ϣʔβ͕େྔʹΞΫηεͯ͘͠Δ ɾେ͖ͳΠϕϯτ͸ࠃ΍फڭͳͲͷ୯ҐͰ࠵͞ΕΔ - Ξϧθϯνϯ 65 ΠϕϯτͷόϦΤʔγϣϯ͕๛෋ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
66. 66 ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
67. 67 ֵ໋ه೦೔ɺಠཱه೦೔ʹ ύεςϦʔτεΛ࡞ͬͯ৯΂Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
68. ɾ͜͜Ͱݴ͏Πϕϯτ͸ॕࡇ೔ͳͲຖ೥ඞͣ։࠵͞ΕΔ΋ͷ ɾΠϕϯτͷࡍʹ͸Ϣʔβ͕େྔʹΞΫηεͯ͘͠Δ ɾେ͖ͳΠϕϯτ͸ࠃ΍फڭͳͲͷ୯ҐͰ࠵͞ΕΔ - Ξϧθϯνϯ - ΠεϥϜݍ 68 ΠϕϯτͷόϦΤʔγϣϯ͕๛෋ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
69. 69 ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
70. 70 ΠυΡϧΞυϋʢ٘ਜ਼ࡇʣ Ͱ೑Λ৯΂Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
71. 71 ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
72. 72 ϥϚμϯظؒ͸ ೔຅͔Β໷໌͚ͷؒͰ Ո଒Ͱ৯ࣄΛָ͠Ή ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
73. ɾ೔ຊͷ৔߹ɺόϨϯλΠϯͰϦΫΤετ਺͕2ഒʹͳΔ 73 ΠϕϯτʹΑΔϢʔβϦΫΤετ਺ͷมԽ ٸʹϦΫΤετ਺͕௓Ͷ্͕Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
74. 74 ΠϕϯτʹΑΔϢʔβϦΫΤετ਺ͷมԽ ٸʹϦΫΤετ਺͕௓Ͷ্͕Δ ɾΠυΡϧΞυϋ͕࢝·Δͱ਺೔ͷؒϦΫΤετ਺͕2ഒʹͳΔ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
75. ɾϥϚμϯ͕࢝·Δͱ1ϲ݄ͷؒϦΫΤετ਺͕2ʙ3ഒʹͳΔ 75 ΠϕϯτʹΑΔϢʔβϦΫΤετ਺ͷมԽ ٸʹϦΫΤετ਺͕௓Ͷ্͕Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
76. ɾւ֎ͷϨγϐαʔϏε͕Cookpadͷ஥ؒʹͳΔ͜ͱ΋͋Δ 76 1Χࠃ૿͑ͯɺಥવϢʔβ਺͕100ສਓ૿͑Δέʔε͕༗Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
77. ɾւ֎ͷϨγϐαʔϏε͕Cookpadͷ஥ؒʹͳΔ͜ͱ΋͋Δ 77 1Χࠃ૿͑ͯɺಥવϢʔβ਺͕100ສਓ૿͑Δέʔε͕༗Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍ ࠃͷ਺͕૿͑ͯ Πϕϯτ΋૿͑Δ
78. 78 ຖ݄όϨϯλΠϯ —
79. 79 શͯͷΠϕϯτʹ౎౓ SRE͕ରࡦ͢Δͷ͸೉͍͠
80. ௅ઓᶄ γεςϜͱ૊৫ͷεέʔϥϏϦςΟΛ ߟྀͨ͠࢓૊ΈΛೖΕΔ 80
81. 81 εέʔϥϏϦςΟɾՄ༻ੑΛߟྀٕͨ͠ज़ͷ࠾༻ ɾϓϩμΫγϣϯͷσʔλϕʔε͸શ୆Amazon AuroraΛར༻ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
82. 82 DockerΞϓϦ։ൃ؀ڥͷఏڙ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
83. ɾ೔ຊͰഓΘΕͨECS + hakoʹΑΔσϓϩΠγεςϜΛಋೖ - ΦʔτεέʔϧΛඪ४౥ࡌ (ϨΨγΞϓϦ͸ࣗ෼Ͱ࣮૷͢Δඞཁ͕͋ͬͨ) - ։ൃ΋εέʔϧ͢Δ - ։ൃऀ͸ґཔͳ͠ʹ؀ڥม਺΍ൿಗ৘ใΛมߋՄೳ - ECSΫϥελʹίϯςφΛσϓϩΠ͢ΔͷͰΠϯελϯεͷηοτΞοϓෆཁ - hako-consoleͳͲͷࣾ಺ͷΤίγεςϜͷԸܙΛڗड 83 ref https://speakerdeck.com/eagletmt/web-application-development-in-cookpad-2017 DockerΞϓϦ։ൃ؀ڥͷఏڙ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
84. 84 hako-console 1PXFSFE
    CZ
    !FBHMFUNU ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
85. 85 hako-console 1PXFSFE
    CZ
    !FBHMFUNU ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
86. 86 hako-console 1PXFSFE
    CZ
    !FBHMFUNU ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ άϩʔόϧαʔϏεͰ΋ ೔ຊͰͷٕज़΍ܦݧΛ ׆͔͍ͯ͠Δ
87. 87 ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕σʔλετΞͷεϩʔΫΤϦΛ෼ੳͰ͖ΔΑ͏ʹ͢Δ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
88. 88 ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕ओཁͳϝτϦΫεΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
89. 89 ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕HTTPͷεςʔλείʔυมԽΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
90. 90 ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕HTTPͷεςʔλείʔυมԽΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ ։ൃऀ͕ࣗ෼Ͱ໰୊Λൃݟͯ͠ ηϧϑϚωδϝϯτͰ͖Δ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
91. ՝୊ᶅ σϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍ 91
92. 92 ೔ຊͰՄೳͳΦϖϨʔγϣϯ͸ւ֎Ͱ͸Ͱ͖ͳ͍͜ͱ΋͋Δ ɾ௕࣌ؒίωΫγϣϯΛҡ࣋Ͱ͖ͳ͍؀ڥ - ωοτϫʔΫ؀ڥ͕೔ຊͷΑ͏ʹ੔͍ͬͯΔͱ͸ݶΒͳ͍ - ւ֎Ͱ͸ఀిɾߑਫͳͲͷҟৗࣄଶ͕೔ৗతʹى͜Δ - खݩ͔ΒCapistranoͰσϓϩΠͰ͖ΔਓɾͰ͖ͳ͍ਓ͕͍Δ ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
93. 93 ೔ຊͰՄೳͳΦϖϨʔγϣϯ͸ւ֎Ͱ͸Ͱ͖ͳ͍͜ͱ΋͋Δ ɾ௕࣌ؒίωΫγϣϯΛҡ࣋Ͱ͖ͳ͍؀ڥ - ωοτϫʔΫ؀ڥ͕೔ຊͷΑ͏ʹ੔͍ͬͯΔͱ͸ݶΒͳ͍ - ఀిɾߑਫͳͲͷҟৗࣄଶ͸ී௨ʹى͖Δ - खݩ͔ΒCapistranoͰσϓϩΠͰ͖ΔਓɾͰ͖ͳ͍ਓ͕͍Δ ͦ΋ͦ΋σϓϩΠͰ͖ͳ͍ ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
94. 94 σϓϩΠΛଞͷਓʹґཔ͢Δੈքઢ͕஀ੜ͢Δ ɾखݩ͔ΒσϓϩΠ͕Ͱ͖ͳ͍։ൃऀ͔ΒσϓϩΠґཔ͕དྷΔ ɾσϓϩΠͷ୅ཧ͸͓ޓ͍ʹγϯυΠ - ։ൃऀ͸1೔ʹproductionʹԿճ΋σϓϩΠ͢Δ - ґཔ͢Δํ΋ɺґཔ͞ΕΔํ΋γϯυΠ - σϓϩΠʹର͢Δ৺ཧతͳίετ্͕͕ͬͯ͠·͏ - ৽͍͠ػೳΛϦϦʔε͍ͨ͠ͷʹ୅ཧσϓϩΠऀ଴ͪ΋͋Γ͏Δ ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
95. ௅ઓᶅ ੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ 95
96. 96 σϓϩΠαʔόΛถࠃϦʔδϣϯʹ༻ҙ͢Δ ɾssh ͰσϓϩΠαʔόʹΞΫηεͯ͠΋Β͍CapistranoΛ࣮ߦ ɾ୅ཧσϓϩΠͷѱເ͔Β͸ղ์͞ΕΔ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
97. 97 σϓϩΠαʔόͰͷϚχϡΞϧΦϖϨʔγϣϯͷ໰୊ ɾΦϖϨʔγϣϯ͕౷Ұ͞Εͳ͍ - ։ൃऀ͕࠷৽ͷσϓϩΠεΫϦϓτΛ࢖͏ͱ͸ݶΒͳ͍ - ૝ఆ֎ͷ͜ͱΛαʔό಺Ͱߦ͏ਓ͕͍Δ ɾαʔόʹΞΫηε͢Δͷ͕໘౗ ɾσϓϩΠͷϩάΛ௥͍ʹ͍͘ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
98. 98 σϓϩΠ؀ڥͷ࡮৽ chat (slack)
 deploy server (cap & hako) bot (ruboty)
 job scheduler (Rundeck)
 deploy targets ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
99. 99 σϓϩΠαʔό΁ͷϩάΠϯ͕ෆཁʹͳΔ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
100. 100 ૝ఆ͍ͯ͠ΔΦϖϨʔγϣϯʹ౷ҰͰ͖Δ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
101. 101 σϓϩΠͷϩάΛ௥͏ͷ͕༰қʹͳΔ σϓϩΠ࣌ࠁ σϓϩΠͨ͠ਓ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
102. 102 σϓϩΠͷϩάΛ௥͏ͷ͕༰қʹͳΔ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
103. ՝୊ᶆ toil ͕ٸ૿͢Δ 103
104. 104 SFG
     IUUQTMBOEJOHHPPHMFDPNTSFCPPLIUNM ՝୊ᶆɿtoil͕ٸ૿͢Δ
105. ɾखಈͰରԠ͍ͯ͠Δ ɾࣗಈԽͷ༨஍͕͋Δ ɾ܁Γฦ͠ൃੜ͢Δ ɾൃੜ͔ͯ͠Β͡Όͳ͍ͱରԠͰ͖ͳ͍ ɾαʔϏεͷվળʹͭͳ͕Βͳ͍ ɾαʔϏε΍Ϣʔβͷ਺ʹൺྫͯ͠૿Ճ͢Δ 105 toilʹ౰ͯ͸·Δۀ຿ ՝୊ᶆɿtoil͕ٸ૿͢Δ SFG
     IUUQTMBOEJOHHPPHMFDPNTSFCPPLDIBQUFSTFMJNJOBUJOHUPJMIUNM
106. 106 ։ൃऀ͔ΒSRE΁ͷґཔ͕ٸ૿      
     
     
      
     
     
      
     
     
         ֤ظؒͷSRE΁ͷґཔ਺ͷਪҠ ՝୊ᶆɿtoil͕ٸ૿͢Δ
107. 107 ۀ຿ׂ߹ͷมԽ      
     /PW 
     %FD 
     +BO 
     'FC 5PJM 4ZTUFN
     &OHJOFFSJOH 4PGUXBSF
     &OHJOFFSJOH ՝୊ᶆɿtoil͕ٸ૿͢Δ
108. 108 ۀ຿ׂ߹ͷมԽ      
     /PW 
     %FD 
     +BO 
     'FC 5PJM 4ZTUFN
     &OHJOFFSJOH 4PGUXBSF
     &OHJOFFSJOH SRE͕΍Γ͍ͨ͜ͱ ՝୊ᶆɿtoil͕ٸ૿͢Δ
109. 109 ۀ຿ׂ߹ͷมԽ      
     /PW 
     %FD 
     +BO 
     'FC 5PJM 4ZTUFN
     &OHJOFFSJOH 4PGUXBSF
     &OHJOFFSJOH toilͷ૿Ճʹൺྫͯ͠ ׂ͕͚࣌ؒͳ͘ͳΔ ՝୊ᶆɿtoil͕ٸ૿͢Δ
110. 110 SREͷۀ຿ׂ߹ͷมԽ      
     /PW 
     %FD 
     +BO 
     'FC 5PJM 4ZTUFN
     &OHJOFFSJOH 4PGUXBSF
     &OHJOFFSJOH ΍Γ͍ͨ͜ͱ͕Ͱ͖ͳ͍ ՝୊ᶆɿtoil͕ٸ૿͢Δ
111. 111 2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3 ΧςΰϦ ґཔͷ਺ "DDPVOU
     .BOBHFNFOU  "84
     3FMBUFE
     0QFSBUJPOT  )VHF
     %BUBCBTF
     5BCMF
     .JHSBUJPOT  ՝୊ᶆɿtoil͕ٸ૿͢Δ
112. 112 2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3 DBUFHPSZ 
     PG
     JTTVFT "DDPVOU
     .BOBHFNFOU  "84
     3FMBUFE
     0QFSBUJPOT  )VHF
     %BUBCBTF
     5BCMF
     .JHSBUJPOT  ΞΧ΢ϯτ؅ཧճΓ ͷґཔͷᓲ໓͕ඞཁ ՝୊ᶆɿtoil͕ٸ૿͢Δ
113. 113 ΞΧ΢ϯτ؅ཧ͕ਏ͍ཧ༝ ɾࣾһ਺͕ٸ૿͍ͯ͠Δ͠ɺࠓޙ΋૿͑Δ͜ͱ͕૝૾Ͱ͖Δ ɾ։ൃऀ͕ར༻͢Δπʔϧ͕ଟ༷Խ͖͍ͯͯ͠Δ - Single Sign OnʹରԠ͍ͯ͠ͳ͍πʔϧ΋͋Δ - Ͱ΋ɺΞΫηε੍ޚ͸͍ͨ͠ ɾ͕࣌ࠩ͋ΔͷͰίϛϡχέʔγϣϯͷλΠϛϯάΛҳ͠΍͍͢ - Կ͕͍͔ͨ͠೺Ѳ͢Δͷʹ਺೔͔͔Δ͜ͱ΋ ՝୊ᶆɿtoil͕ٸ૿͢Δ
114. 114 άϩʔόϧͰར༻࣮੷ͷ͋Δπʔϧ܈ ɾ෼ੳɿ Kibanaɺ GrafanaɺNewRelicɺCatchpointɺDatadog ɾίʔυ؅ཧɿ Github Enterpriseɺgithub.com ɾόάτϥοΧɿSentryɺBugsnagɺAirbrake ɾCIɿJenkinsɺCircle CIɺSemaphore CI ɾPagerɿPagerDuty ɾͦͷଞɿAWSɺKuroko2ɺin-house πʔϧ ͳͲ ՝୊ᶆɿtoil͕ٸ૿͢Δ
115. 115 ࣾ಺πʔϧ΁ͷΞΫηεʹVPNΛ࢖͑Δͱ͸ݶΒͳ͍ ɾੈքதͷΦϑΟεͷ։ൃऀ͕VPNΛ࢖͑Δͷ͔ʁ - શһ͕IPΞυϨεͰ੍ޚͰ͖Δ؀ڥʹ͍ͳ͍ ɾ౦ژͷࣾ಺NWΛܦ༝ͨ͠઀ଓ͸ଥ౰͔ʁ - ࣾ಺πʔϧʹ͸ੈքதͷελοϑ͕ΞΫηε͢Δ - ΞϑϦΧͷελοϑ͸VPNΛ҆ఆͯ͠ுΕΔͩΖ͏͔ ՝୊ᶆɿtoil͕ٸ૿͢Δ
116. 116 ࣾ಺πʔϧʹBasicೝূΛ࢖͍ͨ͘ͳ͍ ɾSRE͸ࣾ಺πʔϧ͕૿͑Δ౓ʹBasicೝূͷઃఆΛ͢Δ - ౎౓Ϣʔβ͝ͱʹύεϫʔυΛ༻ҙ͢Δͷ͸໘౗ - ελοϑͰϢʔβͱύεϫʔυΛڞ༗͢Δͷ΋ආ͚͍ͨ ɾϢʔβ͸ࣾ಺πʔϧΛ࢖͏࣌සൟʹύεϫʔυೖྗ͕ඞཁ ՝୊ᶆɿtoil͕ٸ૿͢Δ
117. ௅ઓᶆ ΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ 117
118. ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ 118 nginx + omniauthʹΑΔࣾ಺πʔϧͷΞΫηε੍ޚ ɾࣾһ͸ೖࣾ͢ΔͱඞͣG SuiteͷΞΧ΢ϯτ͕෇༩͞ΕΔ ɾࣾ಺πʔϧͷΞΫηε੍ޚ͸ nginx + nginx_omniauth_adapter - ࣾ༻ͷG SuiteΞΧ΢ϯτͰ͋Δ͜ͱΛݕূ - VPNෆཁ - πʔϧຖʹΞΧ΢ϯτ؅ཧ΍Basicೝূͷಋೖ͕ཁΒͳ͘ͳΔ
119. 119 nginx + omniauthʹΑΔࣾ಺πʔϧͷΞΫηε੍ޚ ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
120. 120 SRE͔͠Ͱ͖ͳ͍ΞΧ΢ϯτ؅ཧͷݖݶҠৡ ɾLDAPɺActive DirectoryͷΞΧ΢ϯτ؅ཧͷݖݶΛҠৡ - Github Enterprise΍αʔόͷύεϫʔυΛࣗ෼ͰઃఆͰ͖Δ ɾ։ൃऀ͕࢖͏ssh keyΛ֤छαʔό΁ͷࣗಈσϓϩΠ - Gatewayͷssh key͸ैདྷitamaeͳͲͰσϓϩΠ͍ͯͨ͠ - ਓ਺͕ଟ͍ͱ౎౓ରԠ͢Δͷ͸ਏ͍ - ࣗ෼ͰઃఆͰ͖ͯɺࣗಈతʹσϓϩΠ͞ΕΔ ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
121. 121 1PXFSFE
     CZ
     !TPSBI ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
122. 122 DBUFHPSZ 
     PG
     JTTVFT "DDPVOU
     .BOBHFNFOU  "84
     3FMBUFE
     0QFSBUJPOT  )VHF
     %BUBCBTF
     5BCMF
     .JHSBUJPOT  2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3ͷݱࡏ ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
123. 123 2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3ͷݱࡏ DBUFHPSZ 
     PG
     JTTVFT "DDPVOU
     .BOBHFNFOU  "84
     3FMBUFE
     0QFSBUJPOT  )VHF
     %BUBCBTF
     5BCMF
     .JHSBUJPOT  ਺݅·Ͱݮগ ਺݅·Ͱݮগ ࣗಈԽத ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
124. 124 ֤छࢪࡦΛܦͯSRE΁ͷґཔ਺͸ݮগ܏޲      +BO 'FC .BS "QS .BZ +VO +VM "VH 4FQ 0DU /PW %FD 2017೥ͷ֤݄ͷSRE΁ͷґཔ਺ͷਪҠ ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
125. ՝୊ᶆ “toil͕૿͑Δ”΁ͷ΋͏ҰͭͷऔΓ૊Έ 125
126. 126 SREͷϚϧνϦʔδϣϯରԠ ՝୊ᶆ “toil͕૿͑Δ”΁ͷ΋͏ҰͭͷऔΓ૊Έ Japan UK US
127. ·ͱΊ 127
128. ·ͱΊ 128 ɾαʔϏε΋૊৫΋੒௕த ɾάϩʔόϧͳΒͰ͸ͷऔΓ૊Έ͕͋Δ ɾSREͱͯ͠௅ઓͰ͖Δ՝୊͕ࢁ΄Ͳ͋Δ
129. Thank you !! 129 takayuki-watanabe@cookpad.com