Challenges for Global Service from a Perspective of SRE

Challenges for Global Service from a Perspective of SRE

Cookpad TechConf 2018: https://techconf.cookpad.com/2018/

Transcript

  1. Challenges for Global Service from a Perspective of SRE Takayuki

    Watanabe Infrastructure Department SRE Group Cookpad Inc. Feb 10, 2018
  2. Who? 2

  3. ࣗݾ঺հ ɾΠϯϑϥετϥΫνϟʔ෦ SREάϧʔϓ - ΫοΫύουͷάϩʔόϧαʔϏεͷ։ൃʹैࣄ ɾSRE = Site Reliability Engineering

    - System & Software EngineeringͰ αΠτͷ৴པੑʹؔΘΔશͯͷ՝୊ʹऔΓ૊Ή 3 ౉ลڤ೭ (Takayuki Watanabe) twitter: takanabe_w / github: takanabe
  4. ࠓ೔࿩͢͜ͱ 4 ɾΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ ɾ2017೥ͷάϩʔόϧαʔϏεͷ੒௕ ɾάϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝୊ͱ௅ઓ

  5. ΫοΫύουͷ άϩʔόϧαʔϏεͬͯԿʁ 5

  6. ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ 6 ւ֎޲͚ͷϨγϐαʔϏε

  7. ւ֎޲͚ͷϨγϐαʔϏε 7 ೔ຊͷΫοΫύουͱ ҟͳΔαʔϏε ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ

  8. ւ֎޲͚ͷϨγϐαʔϏε 8 ݄ؒར༻ऀ਺3000ສਓҎ্ ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ ※ 2017೥ 1ʙ12݄ฏۉ

  9. ւ֎޲͚ͷϨγϐαʔϏε 9 22ݴޠ ɾ68ΧࠃͰల։ ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ ※ 2017೥ 12݄຤࣌఺

  10. ւ֎޲͚ͷϨγϐαʔϏε 10 WebɾAndroidɾiOSͰఏڙத (ίʔυϕʔε͸ࠃ಺ͱҟͳΔ) ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ

  11. 11 UK Office (Bristol)

  12. 12 UK Office (Bristol) ϓϩμΫτ։ൃڌ఺͸UK(Bristol)

  13. 13 UK Office (Bristol) ౦ژɺεϖΠϯɺΠϯυωγΞΛ த৺ʹελοϑ͸ੈքதʹ఺ࡏ

  14. ɾ೔ຊҎ֎ͷࠃΛର৅ʹ։ൃ͍ͯ͠ΔϨγϐαʔϏε ɾ݄ؒར༻ऀ਺3000ສਓҎ্ ɾ22ݴޠͰ68ΧࠃʹαʔϏεΛల։ ɾWebɺAndroidɺiOSͰఏڙத(ίʔυϕʔε͸ࠃ಺ͱҟͳΔ) ɾੈքதͷΤϯδχΞ͕։ൃʹࢀՃ 14 ւ֎޲͚ͷϨγϐαʔϏε ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ

  15. 2017೥ͷάϩʔόϧ αʔϏεͷ੒௕ 15

  16. ରԠݴޠ਺ 16

  17. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 17 ରԠݴޠ਺͕૿Ճ 15ݴޠ 2016೥12݄຤࣌఺

  18. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 18 ରԠݴޠ਺͕૿Ճ 15ݴޠ 22ݴޠ 2017೥9݄຤࣌఺ 2016೥12݄຤࣌఺

  19. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 19 ରԠݴޠ਺͕૿Ճ 15ݴޠ 22ݴޠ + 7 ݴޠ 2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺

  20. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 20 ରԠݴޠ਺͕૿Ճ 15ݴޠ 22ݴޠ શݴޠରԠ ≠ ੈքରԠ 2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺

  21. 21

  22. 22 ஍Ҭੑͷҧ͍Λߟྀͨ͠αʔϏε։ൃ͕ඞཁ
 ref: https://techconf.cookpad.com/2017/rejasupotaro.html

  23. ల։தͷࠃͷ਺ 23

  24. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 24 ల։தͷࠃͷ਺͕૿Ճ 58ϲࠃ 2016೥12݄຤࣌఺

  25. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 25 ల։தͷࠃͷ਺͕૿Ճ 58ϲࠃ 68ϲࠃ 2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺

  26. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 26 ల։தͷࠃͷ਺͕૿Ճ 58ϲࠃ 68ϲࠃ + 10ϲࠃ 2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺

  27. UKΦϑΟεͰಇࣾ͘һ਺ 27

  28. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 28 UKΦϑΟεͰಇࣾ͘һ਺ 5ਓ 2016೥12݄຤࣌఺

  29. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 29 UKΦϑΟεͰಇࣾ͘һ਺ 5ਓ ໿ 50ਓ 2017೥12݄຤࣌఺ 2016೥12݄຤࣌఺

  30. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 30 UKΦϑΟεͰಇࣾ͘һ਺ 5ਓ ໿ 50ਓ 10ഒ 2016೥12݄຤࣌఺ 2017೥12݄຤࣌఺

  31. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 31 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί

    ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@
  32. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 32 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί

    ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ ೔ຊൃͷΞϓϦͰ ࠷ଟͷ7஍ҬͰಉ࣌ೖ৆
  33. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 33 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί

    ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ αʔϏε΋૊৫΋੒௕͖ͯͨ͠
  34. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 34 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί

    ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ ҰํͰ୔ࢁͷ՝୊΋͋ͬͨ
  35. άϩʔόϧαʔϏεʹ͓͚Δ SREͷ՝୊ͱ௅ઓ 35

  36. 36 άϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝୊ͱ௅ઓ ՝୊ᶃɿಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ ՝୊ᶄɿΠϕϯτ͕ͱʹ͔͘ଟ͍ ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍ ՝୊ᶆɿtoil ͕ٸ૿͢Δ άϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝୊

  37. ՝୊ᶃ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ 37

  38. 38 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕஗͍ ௚͙ݕࡧͰ͖ͳ͍

  39. 39 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕஗͍ ௚͙ݕࡧͰ͖ͳ͍ ༷ʑͳࠃʹαʔϏεΛల։͢Δ͜ͱͰ ࠃ͝ͱʹϢʔβମݧʹ͕ࠩग़࢝Ίͨ

  40. 40 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕஗͍ ௚͙ݕࡧͰ͖ͳ͍ ͦͷࠃʹॅΜͰ͍ͳ͍ͷͰ ಉ͡؀ڥͰςετͰ͖ͳ͍

  41. 41 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕஗͍ ௚͙ݕࡧͰ͖ͳ͍ ͕࣌ࠩ͋ΔࠃͷਓͱϝοηʔδΛ ަΘͤΔͷ͸ଟͯ͘1೔਺ճ

  42. 42 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕஗͍ ௚͙ݕࡧͰ͖ͳ͍ ݪҼ͕Θ͔Βͳ͍ͱ վળ͸Ͱ͖ͳ͍

  43. ௅ઓᶃ ੈքதͷϢʔβମݧΛଌఆ͠վળ͢Δ 43

  44. ɾఆ఺؍ଌʹ͸Catchpoint SystemsͷSynthetic MonitoringΛར༻ - ໿30Χࠃ͔ΒఆظతʹϝτϦΫεΛऩू - Time To First Byte

    - DNS lookup time - TLS established time - ϨεϙϯελΠϜ ͳͲ 44 ఆ఺؍ଌʹΑΔ֤ࠃͷϢʔβମݧͷଌఆ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ
  45. 45 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

  46. 46 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

  47. 47 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

  48. 48 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ ݪҼΛௐࠪͰ͖Δঢ়ଶʹͳͬͨ

  49. 49 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ ΠϯυωγΞͷTTFB ΠϯυωγΞͷTLS઀ଓ࣌ؒ ถࠃͷTTFB ถࠃͷTLS઀ଓ࣌ؒ ໿750ms ໿330ms ໿225ms

    ໿150ms
  50. 50 ΠϯυωγΞ͸ถࠃͱൺ΂ͯ TLS઀ଓ΍Time To First Byteʹ ͕͔͔࣌ؒΔ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

  51. 51 ถࠃ͔Β஍ཧతʹԕ͍ࠃͷϢʔβମݧ ͕ѱ͍܏޲ʹ͋Δ͜ͱ͕൑໌ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

  52. άϩʔόϧαʔϏεʹ͓͚ΔSREͷ௅ઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍ 52 SFGIUUQTUFDIDPOGDPPLQBEDPNTPSBIIUNM

  53. άϩʔόϧαʔϏεʹ͓͚ΔSREͷ௅ઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍ 53 ɾαʔό͸ถࠃϦʔδϣϯͰ؅ཧ ɾcookpad.comυϝΠϯΛࠃ಺αʔϏεͱڞ༗ ɾRoute53ͷLatency Based RoutingͰIPΛฦ٫ - ౦ژ͔ถࠃͷ͍ۙσʔληϯλ·ͣΞΫηε

    ɾࠃ಺ͱάϩʔόϧαʔϏεΛৼΓ෼͚ - ϦόʔεϓϩΩγͷਖ਼نදݱͰ൑ఆ - cookpad.com͸೔ຊͷαʔϏε - cookpad.com/ukͳͲ͸άϩʔόϧαʔϏε
  54. άϩʔόϧαʔϏεʹ͓͚ΔSREͷ௅ઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍ 54 location ~ ^/(en|vi|fil|ae|al|ao|ar|arabic|at|au|az|ba|bd|be|bg|bh|bo| br|by|ch|ci|cl|cn|co|cr|cu|cy|cz|de|dj|dk|do|dz|ec|ee|eg| es|fi|fr|gh|gr|gt|hk|hn|hr|hu|id|ie|il|in|iq|ir|it|jo| ke|km|kr|kw|kz|lb|lk|lt|lu|lv|ly|ma|mo|mr|mx|my|mz|ng|ni| nl|no|nz|om|pa|pe|ph|pk|pl|ps|pt|py|qa|ro|rs|ru|sa|

    sd|se|sg|si|sk|so|sv|sy|th|tn|tr|tt|tw|ua|uk|us|uy|uz|ve| vn|ye|za)(/|$) { proxy_pass http://cookpad_use1; } location / { proxy_pass https://cookpad_apne; }
  55. 55 ถࠃ͔Βԕ͍ࠃͷϢʔβ͕շదʹ࢖͑ΔΑ͏ʹ͢Δ ɾΞϝϦΧʹαʔό͕ଘࡏ͍ͯͯ͠ωοτϫʔΫϨΠςϯγ͕ߴ͍ ɾӡ༻ίετ্͕͕Δٕज़͸·ͩೖΕͨ͘ͳ͍ - αʔόͷϚϧνϦʔδϣϯԽͳͲ ɾΠϯυωγΞҎ֎ͷࠃ΋গͳ͍ख਺Ͱޮ཰తʹվળ͍ͨ͠ - ଞʹ΋΍Βͳ͚Ε͹ͳΒͳ͍͜ͱ͸୔ࢁ͋Δ ௅ઓᶃɿ

    ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ
  56. 56 ถࠃ͔Βԕ͍ࠃͷϢʔβ͕շదʹ࢖͑ΔΑ͏ʹ͢Δ ɾΞϝϦΧʹαʔό͕ଘࡏ͍ͯͯ͠ωοτϫʔΫϨΠςϯγ͕ߴ͍ ɾӡ༻ίετ্͕͕Δٕज़͸·ͩೖΕͨ͘ͳ͍ - αʔόͷϚϧνϦʔδϣϯԽͱ͔ ɾΠϯυωγΞҎ֎ͷࠃ΋গͳ͍ख਺Ͱޮ཰తʹվળ͍ͨ͠ - ଞʹ΋΍Βͳ͚Ε͹ͳΒͳ͍͜ͱ͸୔ࢁ͋Δ ௅ઓᶃɿ

    ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ CDNʹΑΔTCPͱTLSͷऴ୺Ͱ ωοτϫʔΫϨΠςϯγΛվળ͢Δ
  57. 57 FastlyʹΑΔϨΠςϯγͷվળ ɾWeb / API શͯͷϦΫΤετΛFastlyܦ༝ʹ - Ωϟογϡͱͯ͠࢖͍ͬͯΔΘ͚Ͱ͸ͳ͍ ɾFastlyͰTCPͱTLSͷऴ୺Λ࣮ݱ -

    Ϣʔβͱσʔληϯλͷڑ཭͕ۙ͘ͳΔ - TCP/TLSͷϋϯυγΣΠΫʹ͔͔Δ͕࣌ؒ୹ॖ͞ΕΔ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
  58. 58 FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ Fastlyಋೖલ(໿1.25 sec) ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

  59. 59 FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ Fastlyಋೖલ(໿1.25 sec) Fastlyಋೖޙ(໿0.45 sec) ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

  60. 60 FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ Fastlyಋೖલ(໿1.25 sec) Fastlyಋೖޙ(໿0.45 sec) ΠϯυωγΞϢʔβͷ ϨεϙϯελΠϜ͸໿1/3ʹ୹ॖ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

  61. 61 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ Ξϧθϯνϯ ΠϯυωγΞ α΢δΞϥϏΞ UAE ΠΪϦε Fastlyಋೖ Fastlyಋೖ

    Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ
  62. 62 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ Ξϧθϯνϯ ΠϯυωγΞ α΢δΞϥϏΞ UAE ΠΪϦε Fastlyಋೖ Fastlyಋೖ

    Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ ஍ཧతʹถࠃ͔Βԕ͍ࠃͷ Ϣʔβମݧ͕վળ
  63. ՝୊ᶄ Πϕϯτ͕ͱʹ͔͘ଟ͍ 63

  64. ɾ͜͜Ͱݴ͏Πϕϯτ͸ॕࡇ೔ͳͲຖ೥ඞͣ։࠵͞ΕΔ΋ͷ ɾΠϕϯτͷࡍʹ͸Ϣʔβ͕େྔʹΞΫηεͯ͘͠Δ ɾେ͖ͳΠϕϯτ͸ࠃ΍फڭͳͲͷ୯ҐͰ࠵͞ΕΔ 64 ΠϕϯτͷόϦΤʔγϣϯ͕๛෋ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

  65. ɾ͜͜Ͱݴ͏Πϕϯτ͸ॕࡇ೔ͳͲຖ೥ඞͣ։࠵͞ΕΔ΋ͷ ɾΠϕϯτͷࡍʹ͸Ϣʔβ͕େྔʹΞΫηεͯ͘͠Δ ɾେ͖ͳΠϕϯτ͸ࠃ΍फڭͳͲͷ୯ҐͰ࠵͞ΕΔ - Ξϧθϯνϯ 65 ΠϕϯτͷόϦΤʔγϣϯ͕๛෋ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

  66. 66 ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

  67. 67 ֵ໋ه೦೔ɺಠཱه೦೔ʹ ύεςϦʔτεΛ࡞ͬͯ৯΂Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

  68. ɾ͜͜Ͱݴ͏Πϕϯτ͸ॕࡇ೔ͳͲຖ೥ඞͣ։࠵͞ΕΔ΋ͷ ɾΠϕϯτͷࡍʹ͸Ϣʔβ͕େྔʹΞΫηεͯ͘͠Δ ɾେ͖ͳΠϕϯτ͸ࠃ΍फڭͳͲͷ୯ҐͰ࠵͞ΕΔ - Ξϧθϯνϯ - ΠεϥϜݍ 68 ΠϕϯτͷόϦΤʔγϣϯ͕๛෋ ՝୊ᶄɿ

    Πϕϯτ͕ͱʹ͔͘ଟ͍
  69. 69 ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

  70. 70 ΠυΡϧΞυϋʢ٘ਜ਼ࡇʣ Ͱ೑Λ৯΂Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

  71. 71 ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

  72. 72 ϥϚμϯظؒ͸ ೔຅͔Β໷໌͚ͷؒͰ Ո଒Ͱ৯ࣄΛָ͠Ή ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

  73. ɾ೔ຊͷ৔߹ɺόϨϯλΠϯͰϦΫΤετ਺͕2ഒʹͳΔ 73 ΠϕϯτʹΑΔϢʔβϦΫΤετ਺ͷมԽ ٸʹϦΫΤετ਺͕௓Ͷ্͕Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

  74. 74 ΠϕϯτʹΑΔϢʔβϦΫΤετ਺ͷมԽ ٸʹϦΫΤετ਺͕௓Ͷ্͕Δ ɾΠυΡϧΞυϋ͕࢝·Δͱ਺೔ͷؒϦΫΤετ਺͕2ഒʹͳΔ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

  75. ɾϥϚμϯ͕࢝·Δͱ1ϲ݄ͷؒϦΫΤετ਺͕2ʙ3ഒʹͳΔ 75 ΠϕϯτʹΑΔϢʔβϦΫΤετ਺ͷมԽ ٸʹϦΫΤετ਺͕௓Ͷ্͕Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

  76. ɾւ֎ͷϨγϐαʔϏε͕Cookpadͷ஥ؒʹͳΔ͜ͱ΋͋Δ 76 1Χࠃ૿͑ͯɺಥવϢʔβ਺͕100ສਓ૿͑Δέʔε͕༗Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

  77. ɾւ֎ͷϨγϐαʔϏε͕Cookpadͷ஥ؒʹͳΔ͜ͱ΋͋Δ 77 1Χࠃ૿͑ͯɺಥવϢʔβ਺͕100ສਓ૿͑Δέʔε͕༗Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍ ࠃͷ਺͕૿͑ͯ Πϕϯτ΋૿͑Δ

  78. 78 ຖ݄όϨϯλΠϯ —

  79. 79 શͯͷΠϕϯτʹ౎౓ SRE͕ରࡦ͢Δͷ͸೉͍͠

  80. ௅ઓᶄ γεςϜͱ૊৫ͷεέʔϥϏϦςΟΛ ߟྀͨ͠࢓૊ΈΛೖΕΔ 80

  81. 81 εέʔϥϏϦςΟɾՄ༻ੑΛߟྀٕͨ͠ज़ͷ࠾༻ ɾϓϩμΫγϣϯͷσʔλϕʔε͸શ୆Amazon AuroraΛར༻ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

  82. 82 DockerΞϓϦ։ൃ؀ڥͷఏڙ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

  83. ɾ೔ຊͰഓΘΕͨECS + hakoʹΑΔσϓϩΠγεςϜΛಋೖ - ΦʔτεέʔϧΛඪ४౥ࡌ (ϨΨγΞϓϦ͸ࣗ෼Ͱ࣮૷͢Δඞཁ͕͋ͬͨ) - ։ൃ΋εέʔϧ͢Δ - ։ൃऀ͸ґཔͳ͠ʹ؀ڥม਺΍ൿಗ৘ใΛมߋՄೳ

    - ECSΫϥελʹίϯςφΛσϓϩΠ͢ΔͷͰΠϯελϯεͷηοτΞοϓෆཁ - hako-consoleͳͲͷࣾ಺ͷΤίγεςϜͷԸܙΛڗड 83 ref https://speakerdeck.com/eagletmt/web-application-development-in-cookpad-2017 DockerΞϓϦ։ൃ؀ڥͷఏڙ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
  84. 84 hako-console 1PXFSFECZ!FBHMFUNU ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

  85. 85 hako-console 1PXFSFECZ!FBHMFUNU ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

  86. 86 hako-console 1PXFSFECZ!FBHMFUNU ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ άϩʔόϧαʔϏεͰ΋ ೔ຊͰͷٕज़΍ܦݧΛ ׆͔͍ͯ͠Δ

  87. 87 ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕σʔλετΞͷεϩʔΫΤϦΛ෼ੳͰ͖ΔΑ͏ʹ͢Δ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

  88. 88 ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕ओཁͳϝτϦΫεΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

  89. 89 ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕HTTPͷεςʔλείʔυมԽΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

  90. 90 ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕HTTPͷεςʔλείʔυมԽΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ ։ൃऀ͕ࣗ෼Ͱ໰୊Λൃݟͯ͠ ηϧϑϚωδϝϯτͰ͖Δ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

  91. ՝୊ᶅ σϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍ 91

  92. 92 ೔ຊͰՄೳͳΦϖϨʔγϣϯ͸ւ֎Ͱ͸Ͱ͖ͳ͍͜ͱ΋͋Δ ɾ௕࣌ؒίωΫγϣϯΛҡ࣋Ͱ͖ͳ͍؀ڥ - ωοτϫʔΫ؀ڥ͕೔ຊͷΑ͏ʹ੔͍ͬͯΔͱ͸ݶΒͳ͍ - ւ֎Ͱ͸ఀిɾߑਫͳͲͷҟৗࣄଶ͕೔ৗతʹى͜Δ - खݩ͔ΒCapistranoͰσϓϩΠͰ͖ΔਓɾͰ͖ͳ͍ਓ͕͍Δ ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍

  93. 93 ೔ຊͰՄೳͳΦϖϨʔγϣϯ͸ւ֎Ͱ͸Ͱ͖ͳ͍͜ͱ΋͋Δ ɾ௕࣌ؒίωΫγϣϯΛҡ࣋Ͱ͖ͳ͍؀ڥ - ωοτϫʔΫ؀ڥ͕೔ຊͷΑ͏ʹ੔͍ͬͯΔͱ͸ݶΒͳ͍ - ఀిɾߑਫͳͲͷҟৗࣄଶ͸ී௨ʹى͖Δ - खݩ͔ΒCapistranoͰσϓϩΠͰ͖ΔਓɾͰ͖ͳ͍ਓ͕͍Δ ͦ΋ͦ΋σϓϩΠͰ͖ͳ͍

    ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
  94. 94 σϓϩΠΛଞͷਓʹґཔ͢Δੈքઢ͕஀ੜ͢Δ ɾखݩ͔ΒσϓϩΠ͕Ͱ͖ͳ͍։ൃऀ͔ΒσϓϩΠґཔ͕དྷΔ ɾσϓϩΠͷ୅ཧ͸͓ޓ͍ʹγϯυΠ - ։ൃऀ͸1೔ʹproductionʹԿճ΋σϓϩΠ͢Δ - ґཔ͢Δํ΋ɺґཔ͞ΕΔํ΋γϯυΠ - σϓϩΠʹର͢Δ৺ཧతͳίετ্͕͕ͬͯ͠·͏

    - ৽͍͠ػೳΛϦϦʔε͍ͨ͠ͷʹ୅ཧσϓϩΠऀ଴ͪ΋͋Γ͏Δ ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
  95. ௅ઓᶅ ੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ 95

  96. 96 σϓϩΠαʔόΛถࠃϦʔδϣϯʹ༻ҙ͢Δ ɾssh ͰσϓϩΠαʔόʹΞΫηεͯ͠΋Β͍CapistranoΛ࣮ߦ ɾ୅ཧσϓϩΠͷѱເ͔Β͸ղ์͞ΕΔ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ

  97. 97 σϓϩΠαʔόͰͷϚχϡΞϧΦϖϨʔγϣϯͷ໰୊ ɾΦϖϨʔγϣϯ͕౷Ұ͞Εͳ͍ - ։ൃऀ͕࠷৽ͷσϓϩΠεΫϦϓτΛ࢖͏ͱ͸ݶΒͳ͍ - ૝ఆ֎ͷ͜ͱΛαʔό಺Ͱߦ͏ਓ͕͍Δ ɾαʔόʹΞΫηε͢Δͷ͕໘౗ ɾσϓϩΠͷϩάΛ௥͍ʹ͍͘ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ

  98. 98 σϓϩΠ؀ڥͷ࡮৽ chat (slack)
 deploy server (cap & hako) bot

    (ruboty)
 job scheduler (Rundeck)
 deploy targets ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
  99. 99 σϓϩΠαʔό΁ͷϩάΠϯ͕ෆཁʹͳΔ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ

  100. 100 ૝ఆ͍ͯ͠ΔΦϖϨʔγϣϯʹ౷ҰͰ͖Δ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ

  101. 101 σϓϩΠͷϩάΛ௥͏ͷ͕༰қʹͳΔ σϓϩΠ࣌ࠁ σϓϩΠͨ͠ਓ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ

  102. 102 σϓϩΠͷϩάΛ௥͏ͷ͕༰қʹͳΔ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ

  103. ՝୊ᶆ toil ͕ٸ૿͢Δ 103

  104. 104 SFGIUUQTMBOEJOHHPPHMFDPNTSFCPPLIUNM ՝୊ᶆɿtoil͕ٸ૿͢Δ

  105. ɾखಈͰରԠ͍ͯ͠Δ ɾࣗಈԽͷ༨஍͕͋Δ ɾ܁Γฦ͠ൃੜ͢Δ ɾൃੜ͔ͯ͠Β͡Όͳ͍ͱରԠͰ͖ͳ͍ ɾαʔϏεͷվળʹͭͳ͕Βͳ͍ ɾαʔϏε΍Ϣʔβͷ਺ʹൺྫͯ͠૿Ճ͢Δ 105 toilʹ౰ͯ͸·Δۀ຿ ՝୊ᶆɿtoil͕ٸ૿͢Δ SFGIUUQTMBOEJOHHPPHMFDPNTSFCPPLDIBQUFSTFMJNJOBUJOHUPJMIUNM

  106. 106 ։ൃऀ͔ΒSRE΁ͷґཔ͕ٸ૿        

       ֤ظؒͷSRE΁ͷґཔ਺ͷਪҠ ՝୊ᶆɿtoil͕ٸ૿͢Δ
  107. 107 ۀ຿ׂ߹ͷมԽ      /PW %FD +BO

    'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH ՝୊ᶆɿtoil͕ٸ૿͢Δ
  108. 108 ۀ຿ׂ߹ͷมԽ      /PW %FD +BO

    'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH SRE͕΍Γ͍ͨ͜ͱ ՝୊ᶆɿtoil͕ٸ૿͢Δ
  109. 109 ۀ຿ׂ߹ͷมԽ      /PW %FD +BO

    'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH toilͷ૿Ճʹൺྫͯ͠ ׂ͕͚࣌ؒͳ͘ͳΔ ՝୊ᶆɿtoil͕ٸ૿͢Δ
  110. 110 SREͷۀ຿ׂ߹ͷมԽ      /PW %FD +BO

    'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH ΍Γ͍ͨ͜ͱ͕Ͱ͖ͳ͍ ՝୊ᶆɿtoil͕ٸ૿͢Δ
  111. 111 2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3 ΧςΰϦ ґཔͷ਺ "DDPVOU.BOBHFNFOU  "843FMBUFE0QFSBUJPOT  )VHF%BUBCBTF5BCMF.JHSBUJPOT 

    ՝୊ᶆɿtoil͕ٸ૿͢Δ
  112. 112 2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3 DBUFHPSZ PGJTTVFT "DDPVOU.BOBHFNFOU  "843FMBUFE0QFSBUJPOT  )VHF%BUBCBTF5BCMF.JHSBUJPOT 

    ΞΧ΢ϯτ؅ཧճΓ ͷґཔͷᓲ໓͕ඞཁ ՝୊ᶆɿtoil͕ٸ૿͢Δ
  113. 113 ΞΧ΢ϯτ؅ཧ͕ਏ͍ཧ༝ ɾࣾһ਺͕ٸ૿͍ͯ͠Δ͠ɺࠓޙ΋૿͑Δ͜ͱ͕૝૾Ͱ͖Δ ɾ։ൃऀ͕ར༻͢Δπʔϧ͕ଟ༷Խ͖͍ͯͯ͠Δ - Single Sign OnʹରԠ͍ͯ͠ͳ͍πʔϧ΋͋Δ - Ͱ΋ɺΞΫηε੍ޚ͸͍ͨ͠

    ɾ͕࣌ࠩ͋ΔͷͰίϛϡχέʔγϣϯͷλΠϛϯάΛҳ͠΍͍͢ - Կ͕͍͔ͨ͠೺Ѳ͢Δͷʹ਺೔͔͔Δ͜ͱ΋ ՝୊ᶆɿtoil͕ٸ૿͢Δ
  114. 114 άϩʔόϧͰར༻࣮੷ͷ͋Δπʔϧ܈ ɾ෼ੳɿ Kibanaɺ GrafanaɺNewRelicɺCatchpointɺDatadog ɾίʔυ؅ཧɿ Github Enterpriseɺgithub.com ɾόάτϥοΧɿSentryɺBugsnagɺAirbrake ɾCIɿJenkinsɺCircle

    CIɺSemaphore CI ɾPagerɿPagerDuty ɾͦͷଞɿAWSɺKuroko2ɺin-house πʔϧ ͳͲ ՝୊ᶆɿtoil͕ٸ૿͢Δ
  115. 115 ࣾ಺πʔϧ΁ͷΞΫηεʹVPNΛ࢖͑Δͱ͸ݶΒͳ͍ ɾੈքதͷΦϑΟεͷ։ൃऀ͕VPNΛ࢖͑Δͷ͔ʁ - શһ͕IPΞυϨεͰ੍ޚͰ͖Δ؀ڥʹ͍ͳ͍ ɾ౦ژͷࣾ಺NWΛܦ༝ͨ͠઀ଓ͸ଥ౰͔ʁ - ࣾ಺πʔϧʹ͸ੈքதͷελοϑ͕ΞΫηε͢Δ - ΞϑϦΧͷελοϑ͸VPNΛ҆ఆͯ͠ுΕΔͩΖ͏͔

    ՝୊ᶆɿtoil͕ٸ૿͢Δ
  116. 116 ࣾ಺πʔϧʹBasicೝূΛ࢖͍ͨ͘ͳ͍ ɾSRE͸ࣾ಺πʔϧ͕૿͑Δ౓ʹBasicೝূͷઃఆΛ͢Δ - ౎౓Ϣʔβ͝ͱʹύεϫʔυΛ༻ҙ͢Δͷ͸໘౗ - ελοϑͰϢʔβͱύεϫʔυΛڞ༗͢Δͷ΋ආ͚͍ͨ ɾϢʔβ͸ࣾ಺πʔϧΛ࢖͏࣌සൟʹύεϫʔυೖྗ͕ඞཁ ՝୊ᶆɿtoil͕ٸ૿͢Δ

  117. ௅ઓᶆ ΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ 117

  118. ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ 118 nginx + omniauthʹΑΔࣾ಺πʔϧͷΞΫηε੍ޚ ɾࣾһ͸ೖࣾ͢ΔͱඞͣG SuiteͷΞΧ΢ϯτ͕෇༩͞ΕΔ ɾࣾ಺πʔϧͷΞΫηε੍ޚ͸ nginx +

    nginx_omniauth_adapter - ࣾ༻ͷG SuiteΞΧ΢ϯτͰ͋Δ͜ͱΛݕূ - VPNෆཁ - πʔϧຖʹΞΧ΢ϯτ؅ཧ΍Basicೝূͷಋೖ͕ཁΒͳ͘ͳΔ
  119. 119 nginx + omniauthʹΑΔࣾ಺πʔϧͷΞΫηε੍ޚ ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ

  120. 120 SRE͔͠Ͱ͖ͳ͍ΞΧ΢ϯτ؅ཧͷݖݶҠৡ ɾLDAPɺActive DirectoryͷΞΧ΢ϯτ؅ཧͷݖݶΛҠৡ - Github Enterprise΍αʔόͷύεϫʔυΛࣗ෼ͰઃఆͰ͖Δ ɾ։ൃऀ͕࢖͏ssh keyΛ֤छαʔό΁ͷࣗಈσϓϩΠ -

    Gatewayͷssh key͸ैདྷitamaeͳͲͰσϓϩΠ͍ͯͨ͠ - ਓ਺͕ଟ͍ͱ౎౓ରԠ͢Δͷ͸ਏ͍ - ࣗ෼ͰઃఆͰ͖ͯɺࣗಈతʹσϓϩΠ͞ΕΔ ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
  121. 121 1PXFSFECZ!TPSBI ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ

  122. 122 DBUFHPSZ PGJTTVFT "DDPVOU.BOBHFNFOU  "843FMBUFE0QFSBUJPOT  )VHF%BUBCBTF5BCMF.JHSBUJPOT  2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3ͷݱࡏ

    ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
  123. 123 2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3ͷݱࡏ DBUFHPSZ PGJTTVFT "DDPVOU.BOBHFNFOU  "843FMBUFE0QFSBUJPOT  )VHF%BUBCBTF5BCMF.JHSBUJPOT 

    ਺݅·Ͱݮগ ਺݅·Ͱݮগ ࣗಈԽத ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
  124. 124 ֤छࢪࡦΛܦͯSRE΁ͷґཔ਺͸ݮগ܏޲      +BO 'FC .BS

    "QS .BZ +VO +VM "VH 4FQ 0DU /PW %FD 2017೥ͷ֤݄ͷSRE΁ͷґཔ਺ͷਪҠ ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
  125. ՝୊ᶆ “toil͕૿͑Δ”΁ͷ΋͏ҰͭͷऔΓ૊Έ 125

  126. 126 SREͷϚϧνϦʔδϣϯରԠ ՝୊ᶆ “toil͕૿͑Δ”΁ͷ΋͏ҰͭͷऔΓ૊Έ Japan UK US

  127. ·ͱΊ 127

  128. ·ͱΊ 128 ɾαʔϏε΋૊৫΋੒௕த ɾάϩʔόϧͳΒͰ͸ͷऔΓ૊Έ͕͋Δ ɾSREͱͯ͠௅ઓͰ͖Δ՝୊͕ࢁ΄Ͳ͋Δ

  129. Thank you !! 129 takayuki-watanabe@cookpad.com