Challenges for Global Service from a Perspective of SRE

Cookpad TechConf 2018: https://techconf.cookpad.com/2018/

Transcript

1. Challenges for Global Service
   from a Perspective of SRE
   Takayuki Watanabe
   Infrastructure Department SRE Group
   Cookpad Inc.
   Feb 10, 2018
   

   View Slide

2. Who?
   2
   

   View Slide

3. ࣗݾ঺հ
   ɾΠϯϑϥετϥΫνϟʔ෦ SREάϧʔϓ
   - ΫοΫύουͷάϩʔόϧαʔϏεͷ։ൃʹैࣄ
   ɾSRE = Site Reliability Engineering
   - System & Software EngineeringͰ
   αΠτͷ৴པੑʹؔΘΔશͯͷ՝୊ʹऔΓ૊Ή
   3
   ౉ลڤ೭ (Takayuki Watanabe)
   twitter: takanabe_w / github: takanabe
   

   View Slide

4. ࠓ೔࿩͢͜ͱ
   4
   ɾΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
   ɾ2017೥ͷάϩʔόϧαʔϏεͷ੒௕
   ɾάϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝୊ͱ௅ઓ
   

   View Slide

5. ΫοΫύουͷ
   άϩʔόϧαʔϏεͬͯԿʁ
   5
   

   View Slide

6. ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
   6
   ւ֎޲͚ͷϨγϐαʔϏε
   

   View Slide

7. ւ֎޲͚ͷϨγϐαʔϏε
   7
   ೔ຊͷΫοΫύουͱ
   ҟͳΔαʔϏε
   ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
   

   View Slide

8. ւ֎޲͚ͷϨγϐαʔϏε
   8
   ݄ؒར༻ऀ਺3000ສਓҎ্
   ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
   ※ 2017೥ 1ʙ12݄ฏۉ
   

   View Slide

9. ւ֎޲͚ͷϨγϐαʔϏε
   9
   22ݴޠ ɾ68ΧࠃͰల։
   ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
   ※ 2017೥ 12݄຤࣌఺
   

   View Slide

10. ւ֎޲͚ͷϨγϐαʔϏε
    10
    WebɾAndroidɾiOSͰఏڙத
    (ίʔυϕʔε͸ࠃ಺ͱҟͳΔ)
    ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
    

    View Slide

11. 11
    UK Office (Bristol)
    

    View Slide

12. 12
    UK Office (Bristol)
    ϓϩμΫτ։ൃڌ఺͸UK(Bristol)
    

    View Slide

13. 13
    UK Office (Bristol)
    ౦ژɺεϖΠϯɺΠϯυωγΞΛ
    த৺ʹελοϑ͸ੈքதʹ఺ࡏ
    

    View Slide

14. ɾ೔ຊҎ֎ͷࠃΛର৅ʹ։ൃ͍ͯ͠ΔϨγϐαʔϏε
    ɾ݄ؒར༻ऀ਺3000ສਓҎ্
    ɾ22ݴޠͰ68ΧࠃʹαʔϏεΛల։
    ɾWebɺAndroidɺiOSͰఏڙத(ίʔυϕʔε͸ࠃ಺ͱҟͳΔ)
    ɾੈքதͷΤϯδχΞ͕։ൃʹࢀՃ
    14
    ւ֎޲͚ͷϨγϐαʔϏε
    ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ
    

    View Slide

15. 2017೥ͷάϩʔόϧ
    αʔϏεͷ੒௕
    15
    

    View Slide

16. ରԠݴޠ਺
    16
    

    View Slide

17. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕
    17
    ରԠݴޠ਺͕૿Ճ
    15ݴޠ
    2016೥12݄຤࣌఺
    

    View Slide

18. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕
    18
    ରԠݴޠ਺͕૿Ճ
    15ݴޠ 22ݴޠ
    2017೥9݄຤࣌఺
    2016೥12݄຤࣌఺
    

    View Slide

19. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕
    19
    ରԠݴޠ਺͕૿Ճ
    15ݴޠ 22ݴޠ
    + 7 ݴޠ
    2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺
    

    View Slide

20. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕
    20
    ରԠݴޠ਺͕૿Ճ
    15ݴޠ 22ݴޠ
    શݴޠରԠ ≠ ੈքରԠ
    2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺
    

    View Slide

21. View Slide

22. 22
    ஍Ҭੑͷҧ͍Λߟྀͨ͠αʔϏε։ൃ͕ඞཁ

    ref: https://techconf.cookpad.com/2017/rejasupotaro.html
    

    View Slide

23. ల։தͷࠃͷ਺
    23
    

    View Slide

24. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕
    24
    ల։தͷࠃͷ਺͕૿Ճ
    58ϲࠃ
    2016೥12݄຤࣌఺
    

    View Slide

25. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕
    25
    ల։தͷࠃͷ਺͕૿Ճ
    58ϲࠃ
    68ϲࠃ
    2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺
    

    View Slide

26. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕
    26
    ల։தͷࠃͷ਺͕૿Ճ
    58ϲࠃ
    68ϲࠃ
    + 10ϲࠃ
    2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺
    

    View Slide

27. UKΦϑΟεͰಇࣾ͘һ਺
    27
    

    View Slide

28. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕
    28
    UKΦϑΟεͰಇࣾ͘һ਺
    5ਓ
    2016೥12݄຤࣌఺
    

    View Slide

29. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕
    29
    UKΦϑΟεͰಇࣾ͘һ਺
    5ਓ ໿
    50ਓ
    2017೥12݄຤࣌఺
    2016೥12݄຤࣌఺
    

    View Slide

30. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕
    30
    UKΦϑΟεͰಇࣾ͘һ਺
    5ਓ ໿
    50ਓ
    10ഒ
    2016೥12݄຤࣌఺ 2017೥12݄຤࣌఺
    

    View Slide

31. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕
    31
    Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़
    ɾεϖΠϯ
    ɾΠλϦΞ
    ɾϑϥϯε
    ɾϝΩγί
    ɾϒϥδϧ
    ɾೆถ
    ɾΠϯυωγΞ
    SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@
    

    View Slide

32. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕
    32
    Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़
    ɾεϖΠϯ
    ɾΠλϦΞ
    ɾϑϥϯε
    ɾϝΩγί
    ɾϒϥδϧ
    ɾೆถ
    ɾΠϯυωγΞ
    SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@
    ೔ຊൃͷΞϓϦͰ
    ࠷ଟͷ7஍ҬͰಉ࣌ೖ৆
    

    View Slide

33. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕
    33
    Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़
    ɾεϖΠϯ
    ɾΠλϦΞ
    ɾϑϥϯε
    ɾϝΩγί
    ɾϒϥδϧ
    ɾೆถ
    ɾΠϯυωγΞ
    SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@
    αʔϏε΋૊৫΋੒௕͖ͯͨ͠
    

    View Slide

34. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕
    34
    Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़
    ɾεϖΠϯ
    ɾΠλϦΞ
    ɾϑϥϯε
    ɾϝΩγί
    ɾϒϥδϧ
    ɾೆถ
    ɾΠϯυωγΞ
    SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@
    ҰํͰ୔ࢁͷ՝୊΋͋ͬͨ
    

    View Slide

35. άϩʔόϧαʔϏεʹ͓͚Δ
    SREͷ՝୊ͱ௅ઓ
    35
    

    View Slide

36. 36
    άϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝୊ͱ௅ઓ
    ՝୊ᶃɿಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍
    ՝୊ᶄɿΠϕϯτ͕ͱʹ͔͘ଟ͍
    ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
    ՝୊ᶆɿtoil ͕ٸ૿͢Δ
    άϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝୊
    

    View Slide

37. ՝୊ᶃ
    ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍
    37
    

    View Slide

38. 38
    ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ
    ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍
    දࣔ͞ΕΔ·Ͱ͕஗͍
    ௚͙ݕࡧͰ͖ͳ͍
    

    View Slide

39. 39
    ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ
    ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍
    දࣔ͞ΕΔ·Ͱ͕஗͍
    ௚͙ݕࡧͰ͖ͳ͍
    ༷ʑͳࠃʹαʔϏεΛల։͢Δ͜ͱͰ
    ࠃ͝ͱʹϢʔβମݧʹ͕ࠩग़࢝Ίͨ
    

    View Slide

40. 40
    ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ
    ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍
    දࣔ͞ΕΔ·Ͱ͕஗͍
    ௚͙ݕࡧͰ͖ͳ͍
    ͦͷࠃʹॅΜͰ͍ͳ͍ͷͰ
    ಉ͡؀ڥͰςετͰ͖ͳ͍
    

    View Slide

41. 41
    ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ
    ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍
    දࣔ͞ΕΔ·Ͱ͕஗͍
    ௚͙ݕࡧͰ͖ͳ͍
    ͕࣌ࠩ͋ΔࠃͷਓͱϝοηʔδΛ
    ަΘͤΔͷ͸ଟͯ͘1೔਺ճ
    

    View Slide

42. 42
    ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ
    ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍
    දࣔ͞ΕΔ·Ͱ͕஗͍
    ௚͙ݕࡧͰ͖ͳ͍
    ݪҼ͕Θ͔Βͳ͍ͱ
    վળ͸Ͱ͖ͳ͍
    

    View Slide

43. ௅ઓᶃ
    ੈքதͷϢʔβମݧΛଌఆ͠վળ͢Δ
    43
    

    View Slide

44. ɾఆ఺؍ଌʹ͸Catchpoint SystemsͷSynthetic MonitoringΛར༻
    - ໿30Χࠃ͔ΒఆظతʹϝτϦΫεΛऩू
    - Time To First Byte
    - DNS lookup time
    - TLS established time
    - ϨεϙϯελΠϜ ͳͲ
    44
    ఆ఺؍ଌʹΑΔ֤ࠃͷϢʔβମݧͷଌఆ
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ
    

    View Slide

45. 45
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    

    View Slide

46. 46
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    

    View Slide

47. 47
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    

    View Slide

48. 48
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    ݪҼΛௐࠪͰ͖Δঢ়ଶʹͳͬͨ
    

    View Slide

49. 49
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    ΠϯυωγΞͷTTFB
    ΠϯυωγΞͷTLS઀ଓ࣌ؒ
    ถࠃͷTTFB
    ถࠃͷTLS઀ଓ࣌ؒ
    ໿750ms
    ໿330ms
    ໿225ms
    ໿150ms
    

    View Slide

50. 50
    ΠϯυωγΞ͸ถࠃͱൺ΂ͯ
    TLS઀ଓ΍Time To First Byteʹ
    ͕͔͔࣌ؒΔ
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    

    View Slide

51. 51
    ถࠃ͔Β஍ཧతʹԕ͍ࠃͷϢʔβମݧ
    ͕ѱ͍܏޲ʹ͋Δ͜ͱ͕൑໌
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    

    View Slide

52. άϩʔόϧαʔϏεʹ͓͚ΔSREͷ௅ઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍
    52
    SFGIUUQTUFDIDPOGDPPLQBEDPNTPSBIIUNM
    

    View Slide

53. άϩʔόϧαʔϏεʹ͓͚ΔSREͷ௅ઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍
    53
    ɾαʔό͸ถࠃϦʔδϣϯͰ؅ཧ
    ɾcookpad.comυϝΠϯΛࠃ಺αʔϏεͱڞ༗
    ɾRoute53ͷLatency Based RoutingͰIPΛฦ٫
    - ౦ژ͔ถࠃͷ͍ۙσʔληϯλ·ͣΞΫηε
    ɾࠃ಺ͱάϩʔόϧαʔϏεΛৼΓ෼͚
    - ϦόʔεϓϩΩγͷਖ਼نදݱͰ൑ఆ
    - cookpad.com͸೔ຊͷαʔϏε
    - cookpad.com/ukͳͲ͸άϩʔόϧαʔϏε
    

    View Slide

54. άϩʔόϧαʔϏεʹ͓͚ΔSREͷ௅ઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍
    54
    location ~ ^/(en|vi|fil|ae|al|ao|ar|arabic|at|au|az|ba|bd|be|bg|bh|bo|
    br|by|ch|ci|cl|cn|co|cr|cu|cy|cz|de|dj|dk|do|dz|ec|ee|eg|
    es|fi|fr|gh|gr|gt|hk|hn|hr|hu|id|ie|il|in|iq|ir|it|jo|
    ke|km|kr|kw|kz|lb|lk|lt|lu|lv|ly|ma|mo|mr|mx|my|mz|ng|ni|
    nl|no|nz|om|pa|pe|ph|pk|pl|ps|pt|py|qa|ro|rs|ru|sa|
    sd|se|sg|si|sk|so|sv|sy|th|tn|tr|tt|tw|ua|uk|us|uy|uz|ve|
    vn|ye|za)(/|$)
    {
    proxy_pass http://cookpad_use1;
    }
    location / {
    proxy_pass https://cookpad_apne;
    }
    

    View Slide

55. 55
    ถࠃ͔Βԕ͍ࠃͷϢʔβ͕շదʹ࢖͑ΔΑ͏ʹ͢Δ
    ɾΞϝϦΧʹαʔό͕ଘࡏ͍ͯͯ͠ωοτϫʔΫϨΠςϯγ͕ߴ͍
    ɾӡ༻ίετ্͕͕Δٕज़͸·ͩೖΕͨ͘ͳ͍
    - αʔόͷϚϧνϦʔδϣϯԽͳͲ
    ɾΠϯυωγΞҎ֎ͷࠃ΋গͳ͍ख਺Ͱޮ཰తʹվળ͍ͨ͠
    - ଞʹ΋΍Βͳ͚Ε͹ͳΒͳ͍͜ͱ͸୔ࢁ͋Δ
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ
    

    View Slide

56. 56
    ถࠃ͔Βԕ͍ࠃͷϢʔβ͕շదʹ࢖͑ΔΑ͏ʹ͢Δ
    ɾΞϝϦΧʹαʔό͕ଘࡏ͍ͯͯ͠ωοτϫʔΫϨΠςϯγ͕ߴ͍
    ɾӡ༻ίετ্͕͕Δٕज़͸·ͩೖΕͨ͘ͳ͍
    - αʔόͷϚϧνϦʔδϣϯԽͱ͔
    ɾΠϯυωγΞҎ֎ͷࠃ΋গͳ͍ख਺Ͱޮ཰తʹվળ͍ͨ͠
    - ଞʹ΋΍Βͳ͚Ε͹ͳΒͳ͍͜ͱ͸୔ࢁ͋Δ
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ
    CDNʹΑΔTCPͱTLSͷऴ୺Ͱ
    ωοτϫʔΫϨΠςϯγΛվળ͢Δ
    

    View Slide

57. 57
    FastlyʹΑΔϨΠςϯγͷվળ
    ɾWeb / API શͯͷϦΫΤετΛFastlyܦ༝ʹ
    - Ωϟογϡͱͯ͠࢖͍ͬͯΔΘ͚Ͱ͸ͳ͍
    ɾFastlyͰTCPͱTLSͷऴ୺Λ࣮ݱ
    - Ϣʔβͱσʔληϯλͷڑ཭͕ۙ͘ͳΔ
    - TCP/TLSͷϋϯυγΣΠΫʹ͔͔Δ͕࣌ؒ୹ॖ͞ΕΔ
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    

    View Slide

58. 58
    FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ
    Fastlyಋೖલ(໿1.25 sec)
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    

    View Slide

59. 59
    FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ
    Fastlyಋೖલ(໿1.25 sec)
    Fastlyಋೖޙ(໿0.45 sec)
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    

    View Slide

60. 60
    FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ
    Fastlyಋೖલ(໿1.25 sec)
    Fastlyಋೖޙ(໿0.45 sec)
    ΠϯυωγΞϢʔβͷ
    ϨεϙϯελΠϜ͸໿1/3ʹ୹ॖ
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    

    View Slide

61. 61
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    Ξϧθϯνϯ ΠϯυωγΞ α΢δΞϥϏΞ
    UAE ΠΪϦε
    Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ
    Fastlyಋೖ Fastlyಋೖ
    

    View Slide

62. 62
    ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
    Ξϧθϯνϯ ΠϯυωγΞ α΢δΞϥϏΞ
    UAE ΠΪϦε
    Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ
    Fastlyಋೖ Fastlyಋೖ
    ஍ཧతʹถࠃ͔Βԕ͍ࠃͷ
    Ϣʔβମݧ͕վળ
    

    View Slide

63. ՝୊ᶄ
    Πϕϯτ͕ͱʹ͔͘ଟ͍
    63
    

    View Slide

64. ɾ͜͜Ͱݴ͏Πϕϯτ͸ॕࡇ೔ͳͲຖ೥ඞͣ։࠵͞ΕΔ΋ͷ
    ɾΠϕϯτͷࡍʹ͸Ϣʔβ͕େྔʹΞΫηεͯ͘͠Δ
    ɾେ͖ͳΠϕϯτ͸ࠃ΍फڭͳͲͷ୯ҐͰ࠵͞ΕΔ
    64
    ΠϕϯτͷόϦΤʔγϣϯ͕๛෋
    ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
    

    View Slide

65. ɾ͜͜Ͱݴ͏Πϕϯτ͸ॕࡇ೔ͳͲຖ೥ඞͣ։࠵͞ΕΔ΋ͷ
    ɾΠϕϯτͷࡍʹ͸Ϣʔβ͕େྔʹΞΫηεͯ͘͠Δ
    ɾେ͖ͳΠϕϯτ͸ࠃ΍फڭͳͲͷ୯ҐͰ࠵͞ΕΔ
    - Ξϧθϯνϯ
    65
    ΠϕϯτͷόϦΤʔγϣϯ͕๛෋
    ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
    

    View Slide

66. 66
    ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
    

    View Slide

67. 67
    ֵ໋ه೦೔ɺಠཱه೦೔ʹ
    ύεςϦʔτεΛ࡞ͬͯ৯΂Δ
    ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
    

    View Slide

68. ɾ͜͜Ͱݴ͏Πϕϯτ͸ॕࡇ೔ͳͲຖ೥ඞͣ։࠵͞ΕΔ΋ͷ
    ɾΠϕϯτͷࡍʹ͸Ϣʔβ͕େྔʹΞΫηεͯ͘͠Δ
    ɾେ͖ͳΠϕϯτ͸ࠃ΍फڭͳͲͷ୯ҐͰ࠵͞ΕΔ
    - Ξϧθϯνϯ
    - ΠεϥϜݍ
    68
    ΠϕϯτͷόϦΤʔγϣϯ͕๛෋
    ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
    

    View Slide

69. 69
    ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
    

    View Slide

70. 70
    ΠυΡϧΞυϋʢ٘ਜ਼ࡇʣ
    Ͱ೑Λ৯΂Δ
    ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
    

    View Slide

71. 71
    ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
    

    View Slide

72. 72
    ϥϚμϯظؒ͸
    ೔຅͔Β໷໌͚ͷؒͰ
    Ո଒Ͱ৯ࣄΛָ͠Ή
    ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
    

    View Slide

73. ɾ೔ຊͷ৔߹ɺόϨϯλΠϯͰϦΫΤετ਺͕2ഒʹͳΔ
    73
    ΠϕϯτʹΑΔϢʔβϦΫΤετ਺ͷมԽ
    ٸʹϦΫΤετ਺͕௓Ͷ্͕Δ
    ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
    

    View Slide

74. 74
    ΠϕϯτʹΑΔϢʔβϦΫΤετ਺ͷมԽ
    ٸʹϦΫΤετ਺͕௓Ͷ্͕Δ
    ɾΠυΡϧΞυϋ͕࢝·Δͱ਺೔ͷؒϦΫΤετ਺͕2ഒʹͳΔ
    ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
    

    View Slide

75. ɾϥϚμϯ͕࢝·Δͱ1ϲ݄ͷؒϦΫΤετ਺͕2ʙ3ഒʹͳΔ
    75
    ΠϕϯτʹΑΔϢʔβϦΫΤετ਺ͷมԽ
    ٸʹϦΫΤετ਺͕௓Ͷ্͕Δ
    ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
    

    View Slide

76. ɾւ֎ͷϨγϐαʔϏε͕Cookpadͷ஥ؒʹͳΔ͜ͱ΋͋Δ
    76
    1Χࠃ૿͑ͯɺಥવϢʔβ਺͕100ສਓ૿͑Δέʔε͕༗Δ
    ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
    

    View Slide

77. ɾւ֎ͷϨγϐαʔϏε͕Cookpadͷ஥ؒʹͳΔ͜ͱ΋͋Δ
    77
    1Χࠃ૿͑ͯɺಥવϢʔβ਺͕100ສਓ૿͑Δέʔε͕༗Δ
    ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍
    ࠃͷ਺͕૿͑ͯ
    Πϕϯτ΋૿͑Δ
    

    View Slide

78. 78
    ຖ݄όϨϯλΠϯ —
    

    View Slide

79. 79
    શͯͷΠϕϯτʹ౎౓
    SRE͕ରࡦ͢Δͷ͸೉͍͠
    

    View Slide

80. ௅ઓᶄ
    γεςϜͱ૊৫ͷεέʔϥϏϦςΟΛ
    ߟྀͨ͠࢓૊ΈΛೖΕΔ
    80
    

    View Slide

81. 81
    εέʔϥϏϦςΟɾՄ༻ੑΛߟྀٕͨ͠ज़ͷ࠾༻
    ɾϓϩμΫγϣϯͷσʔλϕʔε͸શ୆Amazon AuroraΛར༻
    ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
    

    View Slide

82. 82
    DockerΞϓϦ։ൃ؀ڥͷఏڙ
    ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
    

    View Slide

83. ɾ೔ຊͰഓΘΕͨECS + hakoʹΑΔσϓϩΠγεςϜΛಋೖ
    - ΦʔτεέʔϧΛඪ४౥ࡌ (ϨΨγΞϓϦ͸ࣗ෼Ͱ࣮૷͢Δඞཁ͕͋ͬͨ)
    - ։ൃ΋εέʔϧ͢Δ
    - ։ൃऀ͸ґཔͳ͠ʹ؀ڥม਺΍ൿಗ৘ใΛมߋՄೳ
    - ECSΫϥελʹίϯςφΛσϓϩΠ͢ΔͷͰΠϯελϯεͷηοτΞοϓෆཁ
    - hako-consoleͳͲͷࣾ಺ͷΤίγεςϜͷԸܙΛڗड
    83
    ref https://speakerdeck.com/eagletmt/web-application-development-in-cookpad-2017
    DockerΞϓϦ։ൃ؀ڥͷఏڙ
    ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
    

    View Slide

84. 84
    hako-console
    1PXFSFECZ!FBHMFUNU
    ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
    

    View Slide

85. 85
    hako-console
    1PXFSFECZ!FBHMFUNU
    ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
    

    View Slide

86. 86
    hako-console
    1PXFSFECZ!FBHMFUNU
    ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
    άϩʔόϧαʔϏεͰ΋
    ೔ຊͰͷٕज़΍ܦݧΛ
    ׆͔͍ͯ͠Δ
    

    View Slide

87. 87
    ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗
    ɾ։ൃऀ͕σʔλετΞͷεϩʔΫΤϦΛ෼ੳͰ͖ΔΑ͏ʹ͢Δ
    ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
    

    View Slide

88. 88
    ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗
    ɾ։ൃऀ͕ओཁͳϝτϦΫεΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ
    ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
    

    View Slide

89. 89
    ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗
    ɾ։ൃऀ͕HTTPͷεςʔλείʔυมԽΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ
    ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
    

    View Slide

90. 90
    ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗
    ɾ։ൃऀ͕HTTPͷεςʔλείʔυมԽΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ
    ։ൃऀ͕ࣗ෼Ͱ໰୊Λൃݟͯ͠
    ηϧϑϚωδϝϯτͰ͖Δ
    ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
    

    View Slide

91. ՝୊ᶅ
    σϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
    91
    

    View Slide

92. 92
    ೔ຊͰՄೳͳΦϖϨʔγϣϯ͸ւ֎Ͱ͸Ͱ͖ͳ͍͜ͱ΋͋Δ
    ɾ௕࣌ؒίωΫγϣϯΛҡ࣋Ͱ͖ͳ͍؀ڥ
    - ωοτϫʔΫ؀ڥ͕೔ຊͷΑ͏ʹ੔͍ͬͯΔͱ͸ݶΒͳ͍
    - ւ֎Ͱ͸ఀిɾߑਫͳͲͷҟৗࣄଶ͕೔ৗతʹى͜Δ
    - खݩ͔ΒCapistranoͰσϓϩΠͰ͖ΔਓɾͰ͖ͳ͍ਓ͕͍Δ
    ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
    

    View Slide

93. 93
    ೔ຊͰՄೳͳΦϖϨʔγϣϯ͸ւ֎Ͱ͸Ͱ͖ͳ͍͜ͱ΋͋Δ
    ɾ௕࣌ؒίωΫγϣϯΛҡ࣋Ͱ͖ͳ͍؀ڥ
    - ωοτϫʔΫ؀ڥ͕೔ຊͷΑ͏ʹ੔͍ͬͯΔͱ͸ݶΒͳ͍
    - ఀిɾߑਫͳͲͷҟৗࣄଶ͸ී௨ʹى͖Δ
    - खݩ͔ΒCapistranoͰσϓϩΠͰ͖ΔਓɾͰ͖ͳ͍ਓ͕͍Δ
    ͦ΋ͦ΋σϓϩΠͰ͖ͳ͍
    ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
    

    View Slide

94. 94
    σϓϩΠΛଞͷਓʹґཔ͢Δੈքઢ͕஀ੜ͢Δ
    ɾखݩ͔ΒσϓϩΠ͕Ͱ͖ͳ͍։ൃऀ͔ΒσϓϩΠґཔ͕དྷΔ
    ɾσϓϩΠͷ୅ཧ͸͓ޓ͍ʹγϯυΠ
    - ։ൃऀ͸1೔ʹproductionʹԿճ΋σϓϩΠ͢Δ
    - ґཔ͢Δํ΋ɺґཔ͞ΕΔํ΋γϯυΠ
    - σϓϩΠʹର͢Δ৺ཧతͳίετ্͕͕ͬͯ͠·͏
    - ৽͍͠ػೳΛϦϦʔε͍ͨ͠ͷʹ୅ཧσϓϩΠऀ଴ͪ΋͋Γ͏Δ
    ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍
    

    View Slide

95. ௅ઓᶅ
    ੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
    95
    

    View Slide

96. 96
    σϓϩΠαʔόΛถࠃϦʔδϣϯʹ༻ҙ͢Δ
    ɾssh ͰσϓϩΠαʔόʹΞΫηεͯ͠΋Β͍CapistranoΛ࣮ߦ
    ɾ୅ཧσϓϩΠͷѱເ͔Β͸ղ์͞ΕΔ
    ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
    

    View Slide

97. 97
    σϓϩΠαʔόͰͷϚχϡΞϧΦϖϨʔγϣϯͷ໰୊
    ɾΦϖϨʔγϣϯ͕౷Ұ͞Εͳ͍
    - ։ൃऀ͕࠷৽ͷσϓϩΠεΫϦϓτΛ࢖͏ͱ͸ݶΒͳ͍
    - ૝ఆ֎ͷ͜ͱΛαʔό಺Ͱߦ͏ਓ͕͍Δ
    ɾαʔόʹΞΫηε͢Δͷ͕໘౗
    ɾσϓϩΠͷϩάΛ௥͍ʹ͍͘
    ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
    

    View Slide

98. 98
    σϓϩΠ؀ڥͷ࡮৽
    chat
    (slack)

    deploy server
    (cap & hako)
    bot
    (ruboty)

    job scheduler
    (Rundeck)
 deploy targets
    ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
    

    View Slide

99. 99
    σϓϩΠαʔό΁ͷϩάΠϯ͕ෆཁʹͳΔ
    ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
    

    View Slide

100. 100
     ૝ఆ͍ͯ͠ΔΦϖϨʔγϣϯʹ౷ҰͰ͖Δ
     ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
     

     View Slide

101. 101
     σϓϩΠͷϩάΛ௥͏ͷ͕༰қʹͳΔ
     σϓϩΠ࣌ࠁ σϓϩΠͨ͠ਓ
     ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
     

     View Slide

102. 102
     σϓϩΠͷϩάΛ௥͏ͷ͕༰қʹͳΔ
     ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
     

     View Slide

103. ՝୊ᶆ
     toil ͕ٸ૿͢Δ
     103
     

     View Slide

104. 104
     SFGIUUQTMBOEJOHHPPHMFDPNTSFCPPLIUNM
     ՝୊ᶆɿtoil͕ٸ૿͢Δ
     

     View Slide

105. ɾखಈͰରԠ͍ͯ͠Δ
     ɾࣗಈԽͷ༨஍͕͋Δ
     ɾ܁Γฦ͠ൃੜ͢Δ
     ɾൃੜ͔ͯ͠Β͡Όͳ͍ͱରԠͰ͖ͳ͍
     ɾαʔϏεͷվળʹͭͳ͕Βͳ͍
     ɾαʔϏε΍Ϣʔβͷ਺ʹൺྫͯ͠૿Ճ͢Δ
     105
     toilʹ౰ͯ͸·Δۀ຿
     ՝୊ᶆɿtoil͕ٸ૿͢Δ
     SFGIUUQTMBOEJOHHPPHMFDPNTSFCPPLDIBQUFSTFMJNJOBUJOHUPJMIUNM
     

     View Slide

106. 106
     ։ൃऀ͔ΒSRE΁ͷґཔ͕ٸ૿
     
     
     
     
     
     
     
     
     
     ֤ظؒͷSRE΁ͷґཔ਺ͷਪҠ
     ՝୊ᶆɿtoil͕ٸ૿͢Δ
     

     View Slide

107. 107
     ۀ຿ׂ߹ͷมԽ
     
     
     
     
     
     /PW %FD +BO 'FC
     5PJM
     4ZTUFN&OHJOFFSJOH
     4PGUXBSF&OHJOFFSJOH
     ՝୊ᶆɿtoil͕ٸ૿͢Δ
     

     View Slide

108. 108
     ۀ຿ׂ߹ͷมԽ
     
     
     
     
     
     /PW %FD +BO 'FC
     5PJM
     4ZTUFN&OHJOFFSJOH
     4PGUXBSF&OHJOFFSJOH
     SRE͕΍Γ͍ͨ͜ͱ
     ՝୊ᶆɿtoil͕ٸ૿͢Δ
     

     View Slide

109. 109
     ۀ຿ׂ߹ͷมԽ
     
     
     
     
     
     /PW %FD +BO 'FC
     5PJM
     4ZTUFN&OHJOFFSJOH
     4PGUXBSF&OHJOFFSJOH
     toilͷ૿Ճʹൺྫͯ͠
     ׂ͕͚࣌ؒͳ͘ͳΔ
     ՝୊ᶆɿtoil͕ٸ૿͢Δ
     

     View Slide

110. 110
     SREͷۀ຿ׂ߹ͷมԽ
     
     
     
     
     
     /PW %FD +BO 'FC
     5PJM
     4ZTUFN&OHJOFFSJOH
     4PGUXBSF&OHJOFFSJOH
     ΍Γ͍ͨ͜ͱ͕Ͱ͖ͳ͍
     ՝୊ᶆɿtoil͕ٸ૿͢Δ
     

     View Slide

111. 111
     2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3
     ΧςΰϦ ґཔͷ਺
     "DDPVOU.BOBHFNFOU
     "843FMBUFE0QFSBUJPOT
     )VHF%BUBCBTF5BCMF.JHSBUJPOT
     ՝୊ᶆɿtoil͕ٸ૿͢Δ
     

     View Slide

112. 112
     2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3
     DBUFHPSZ PGJTTVFT
     "DDPVOU.BOBHFNFOU
     "843FMBUFE0QFSBUJPOT
     )VHF%BUBCBTF5BCMF.JHSBUJPOT
     ΞΧ΢ϯτ؅ཧճΓ
     ͷґཔͷᓲ໓͕ඞཁ
     ՝୊ᶆɿtoil͕ٸ૿͢Δ
     

     View Slide

113. 113
     ΞΧ΢ϯτ؅ཧ͕ਏ͍ཧ༝
     ɾࣾһ਺͕ٸ૿͍ͯ͠Δ͠ɺࠓޙ΋૿͑Δ͜ͱ͕૝૾Ͱ͖Δ
     ɾ։ൃऀ͕ར༻͢Δπʔϧ͕ଟ༷Խ͖͍ͯͯ͠Δ
     - Single Sign OnʹରԠ͍ͯ͠ͳ͍πʔϧ΋͋Δ
     - Ͱ΋ɺΞΫηε੍ޚ͸͍ͨ͠
     ɾ͕࣌ࠩ͋ΔͷͰίϛϡχέʔγϣϯͷλΠϛϯάΛҳ͠΍͍͢
     - Կ͕͍͔ͨ͠೺Ѳ͢Δͷʹ਺೔͔͔Δ͜ͱ΋
     ՝୊ᶆɿtoil͕ٸ૿͢Δ
     

     View Slide

114. 114
     άϩʔόϧͰར༻࣮੷ͷ͋Δπʔϧ܈
     ɾ෼ੳɿ Kibanaɺ GrafanaɺNewRelicɺCatchpointɺDatadog
     ɾίʔυ؅ཧɿ Github Enterpriseɺgithub.com
     ɾόάτϥοΧɿSentryɺBugsnagɺAirbrake
     ɾCIɿJenkinsɺCircle CIɺSemaphore CI
     ɾPagerɿPagerDuty
     ɾͦͷଞɿAWSɺKuroko2ɺin-house πʔϧ ͳͲ
     ՝୊ᶆɿtoil͕ٸ૿͢Δ
     

     View Slide

115. 115
     ࣾ಺πʔϧ΁ͷΞΫηεʹVPNΛ࢖͑Δͱ͸ݶΒͳ͍
     ɾੈքதͷΦϑΟεͷ։ൃऀ͕VPNΛ࢖͑Δͷ͔ʁ
     - શһ͕IPΞυϨεͰ੍ޚͰ͖Δ؀ڥʹ͍ͳ͍
     ɾ౦ژͷࣾ಺NWΛܦ༝ͨ͠઀ଓ͸ଥ౰͔ʁ
     - ࣾ಺πʔϧʹ͸ੈքதͷελοϑ͕ΞΫηε͢Δ
     - ΞϑϦΧͷελοϑ͸VPNΛ҆ఆͯ͠ுΕΔͩΖ͏͔
     ՝୊ᶆɿtoil͕ٸ૿͢Δ
     

     View Slide

116. 116
     ࣾ಺πʔϧʹBasicೝূΛ࢖͍ͨ͘ͳ͍
     ɾSRE͸ࣾ಺πʔϧ͕૿͑Δ౓ʹBasicೝূͷઃఆΛ͢Δ
     - ౎౓Ϣʔβ͝ͱʹύεϫʔυΛ༻ҙ͢Δͷ͸໘౗
     - ελοϑͰϢʔβͱύεϫʔυΛڞ༗͢Δͷ΋ආ͚͍ͨ
     ɾϢʔβ͸ࣾ಺πʔϧΛ࢖͏࣌සൟʹύεϫʔυೖྗ͕ඞཁ
     ՝୊ᶆɿtoil͕ٸ૿͢Δ
     

     View Slide

117. ௅ઓᶆ
     ΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
     117
     

     View Slide

118. ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
     118
     nginx + omniauthʹΑΔࣾ಺πʔϧͷΞΫηε੍ޚ
     ɾࣾһ͸ೖࣾ͢ΔͱඞͣG SuiteͷΞΧ΢ϯτ͕෇༩͞ΕΔ
     ɾࣾ಺πʔϧͷΞΫηε੍ޚ͸ nginx + nginx_omniauth_adapter
     - ࣾ༻ͷG SuiteΞΧ΢ϯτͰ͋Δ͜ͱΛݕূ
     - VPNෆཁ
     - πʔϧຖʹΞΧ΢ϯτ؅ཧ΍Basicೝূͷಋೖ͕ཁΒͳ͘ͳΔ
     

     View Slide

119. 119
     nginx + omniauthʹΑΔࣾ಺πʔϧͷΞΫηε੍ޚ
     ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
     

     View Slide

120. 120
     SRE͔͠Ͱ͖ͳ͍ΞΧ΢ϯτ؅ཧͷݖݶҠৡ
     ɾLDAPɺActive DirectoryͷΞΧ΢ϯτ؅ཧͷݖݶΛҠৡ
     - Github Enterprise΍αʔόͷύεϫʔυΛࣗ෼ͰઃఆͰ͖Δ
     ɾ։ൃऀ͕࢖͏ssh keyΛ֤छαʔό΁ͷࣗಈσϓϩΠ
     - Gatewayͷssh key͸ैདྷitamaeͳͲͰσϓϩΠ͍ͯͨ͠
     - ਓ਺͕ଟ͍ͱ౎౓ରԠ͢Δͷ͸ਏ͍
     - ࣗ෼ͰઃఆͰ͖ͯɺࣗಈతʹσϓϩΠ͞ΕΔ
     ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
     

     View Slide

121. 121
     1PXFSFECZ!TPSBI
     ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
     

     View Slide

122. 122
     DBUFHPSZ PGJTTVFT
     "DDPVOU.BOBHFNFOU
     "843FMBUFE0QFSBUJPOT
     )VHF%BUBCBTF5BCMF.JHSBUJPOT
     2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3ͷݱࡏ
     ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
     

     View Slide

123. 123
     2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3ͷݱࡏ
     DBUFHPSZ PGJTTVFT
     "DDPVOU.BOBHFNFOU
     "843FMBUFE0QFSBUJPOT
     )VHF%BUBCBTF5BCMF.JHSBUJPOT
     ਺݅·Ͱݮগ
     ਺݅·Ͱݮগ
     ࣗಈԽத
     ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
     

     View Slide

124. 124
     ֤छࢪࡦΛܦͯSRE΁ͷґཔ਺͸ݮগ܏޲
     
     
     
     
     
     +BO 'FC .BS "QS .BZ +VO +VM "VH 4FQ 0DU /PW %FD
     2017೥ͷ֤݄ͷSRE΁ͷґཔ਺ͷਪҠ
     ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
     

     View Slide

125. ՝୊ᶆ “toil͕૿͑Δ”΁ͷ΋͏ҰͭͷऔΓ૊Έ
     125
     

     View Slide

126. 126
     SREͷϚϧνϦʔδϣϯରԠ
     ՝୊ᶆ “toil͕૿͑Δ”΁ͷ΋͏ҰͭͷऔΓ૊Έ
     Japan
     UK
     US
     

     View Slide

127. ·ͱΊ
     127
     

     View Slide

128. ·ͱΊ
     128
     ɾαʔϏε΋૊৫΋੒௕த
     ɾάϩʔόϧͳΒͰ͸ͷऔΓ૊Έ͕͋Δ
     ɾSREͱͯ͠௅ઓͰ͖Δ՝୊͕ࢁ΄Ͳ͋Δ
     

     View Slide

129. Thank you !!
     129
     takayuki-watanabe@cookpad.com
     

     View Slide