Challenges for Global Service from a Perspective of SRE

Slide 1

Slide 1 text

Challenges for Global Service from a Perspective of SRE Takayuki Watanabe Infrastructure Department SRE Group Cookpad Inc. Feb 10, 2018

Slide 2

Slide 2 text

Who? 2

Slide 3

Slide 3 text

ࣗݾ঺հ ɾΠϯϑϥετϥΫνϟʔ෦ SREάϧʔϓ - ΫοΫύουͷάϩʔόϧαʔϏεͷ։ൃʹैࣄ ɾSRE = Site Reliability Engineering - System & Software EngineeringͰ αΠτͷ৴པੑʹؔΘΔશͯͷ՝୊ʹऔΓ૊Ή 3 ౉ลڤ೭ (Takayuki Watanabe) twitter: takanabe_w / github: takanabe

Slide 4

Slide 4 text

ࠓ೔࿩͢͜ͱ 4 ɾΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ ɾ2017೥ͷάϩʔόϧαʔϏεͷ੒௕ ɾάϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝୊ͱ௅ઓ

Slide 5

Slide 5 text

ΫοΫύουͷ άϩʔόϧαʔϏεͬͯԿʁ 5

Slide 6

Slide 6 text

ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ 6 ւ֎޲͚ͷϨγϐαʔϏε

Slide 7

Slide 7 text

ւ֎޲͚ͷϨγϐαʔϏε 7 ೔ຊͷΫοΫύουͱ ҟͳΔαʔϏε ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ

Slide 8

Slide 8 text

ւ֎޲͚ͷϨγϐαʔϏε 8 ݄ؒར༻ऀ਺3000ສਓҎ্ ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ ※ 2017೥ 1ʙ12݄ฏۉ

Slide 9

Slide 9 text

ւ֎޲͚ͷϨγϐαʔϏε 9 22ݴޠ ɾ68ΧࠃͰల։ ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ ※ 2017೥ 12݄຤࣌఺

Slide 10

Slide 10 text

ւ֎޲͚ͷϨγϐαʔϏε 10 WebɾAndroidɾiOSͰఏڙத (ίʔυϕʔε͸ࠃ಺ͱҟͳΔ) ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ

Slide 11

Slide 11 text

11 UK Oﬃce (Bristol)

Slide 12

Slide 12 text

12 UK Oﬃce (Bristol) ϓϩμΫτ։ൃڌ఺͸UK(Bristol)

Slide 13

Slide 13 text

13 UK Oﬃce (Bristol) ౦ژɺεϖΠϯɺΠϯυωγΞΛ த৺ʹελοϑ͸ੈքதʹ఺ࡏ

Slide 14

Slide 14 text

ɾ೔ຊҎ֎ͷࠃΛର৅ʹ։ൃ͍ͯ͠ΔϨγϐαʔϏε ɾ݄ؒར༻ऀ਺3000ສਓҎ্ ɾ22ݴޠͰ68ΧࠃʹαʔϏεΛల։ ɾWebɺAndroidɺiOSͰఏڙத(ίʔυϕʔε͸ࠃ಺ͱҟͳΔ) ɾੈքதͷΤϯδχΞ͕։ൃʹࢀՃ 14 ւ֎޲͚ͷϨγϐαʔϏε ΫοΫύουͷάϩʔόϧαʔϏεͬͯԿʁ

Slide 15

Slide 15 text

2017೥ͷάϩʔόϧ αʔϏεͷ੒௕ 15

Slide 16

Slide 16 text

ରԠݴޠ਺ 16

Slide 17

Slide 17 text

2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 17 ରԠݴޠ਺͕૿Ճ 15ݴޠ 2016೥12݄຤࣌఺

Slide 18

Slide 18 text

2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 18 ରԠݴޠ਺͕૿Ճ 15ݴޠ 22ݴޠ 2017೥9݄຤࣌఺ 2016೥12݄຤࣌఺

Slide 19

Slide 19 text

2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 19 ରԠݴޠ਺͕૿Ճ 15ݴޠ 22ݴޠ + 7 ݴޠ 2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺

Slide 20

Slide 20 text

2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 20 ରԠݴޠ਺͕૿Ճ 15ݴޠ 22ݴޠ શݴޠରԠ ≠ ੈքରԠ 2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺

Slide 21

Slide 21 text

Slide 22

Slide 22 text

22 ஍Ҭੑͷҧ͍Λߟྀͨ͠αʔϏε։ൃ͕ඞཁ  ref: https://techconf.cookpad.com/2017/rejasupotaro.html

Slide 23

Slide 23 text

ల։தͷࠃͷ਺ 23

Slide 24

Slide 24 text

2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 24 ల։தͷࠃͷ਺͕૿Ճ 58ϲࠃ 2016೥12݄຤࣌఺

Slide 25

Slide 25 text

2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 25 ల։தͷࠃͷ਺͕૿Ճ 58ϲࠃ 68ϲࠃ 2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺

Slide 26

Slide 26 text

2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 26 ల։தͷࠃͷ਺͕૿Ճ 58ϲࠃ 68ϲࠃ + 10ϲࠃ 2016೥12݄຤࣌఺ 2017೥9݄຤࣌఺

Slide 27

Slide 27 text

UKΦϑΟεͰಇࣾ͘һ਺ 27

Slide 28

Slide 28 text

2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 28 UKΦϑΟεͰಇࣾ͘һ਺ 5ਓ 2016೥12݄຤࣌఺

Slide 29

Slide 29 text

2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 29 UKΦϑΟεͰಇࣾ͘һ਺ 5ਓ ໿ 50ਓ 2017೥12݄຤࣌఺ 2016೥12݄຤࣌఺

Slide 30

Slide 30 text

2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 30 UKΦϑΟεͰಇࣾ͘һ਺ 5ਓ ໿ 50ਓ 10ഒ 2016೥12݄຤࣌఺ 2017೥12݄຤࣌఺

Slide 31

Slide 31 text

2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 31 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@

Slide 32

Slide 32 text

2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 32 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ ೔ຊൃͷΞϓϦͰ ࠷ଟͷ7஍ҬͰಉ࣌ೖ৆

Slide 33

Slide 33 text

2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 33 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ αʔϏε΋૊৫΋੒௕͖ͯͨ͠

Slide 34

Slide 34 text

2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 34 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ ҰํͰ୔ࢁͷ՝୊΋͋ͬͨ

Slide 35

Slide 35 text

άϩʔόϧαʔϏεʹ͓͚Δ SREͷ՝୊ͱ௅ઓ 35

Slide 36

Slide 36 text

36 άϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝୊ͱ௅ઓ ՝୊ᶃɿಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ ՝୊ᶄɿΠϕϯτ͕ͱʹ͔͘ଟ͍ ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍ ՝୊ᶆɿtoil ͕ٸ૿͢Δ άϩʔόϧαʔϏεʹ͓͚ΔSREͷ՝୊

Slide 37

Slide 37 text

՝୊ᶃ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ 37

Slide 38

Slide 38 text

38 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕஗͍ ௚͙ݕࡧͰ͖ͳ͍

Slide 39

Slide 39 text

39 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕஗͍ ௚͙ݕࡧͰ͖ͳ͍ ༷ʑͳࠃʹαʔϏεΛల։͢Δ͜ͱͰ ࠃ͝ͱʹϢʔβମݧʹ͕ࠩग़࢝Ίͨ

Slide 40

Slide 40 text

40 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕஗͍ ௚͙ݕࡧͰ͖ͳ͍ ͦͷࠃʹॅΜͰ͍ͳ͍ͷͰ ಉ͡؀ڥͰςετͰ͖ͳ͍

Slide 41

Slide 41 text

41 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕஗͍ ௚͙ݕࡧͰ͖ͳ͍ ͕࣌ࠩ͋ΔࠃͷਓͱϝοηʔδΛ ަΘͤΔͷ͸ଟͯ͘1೔਺ճ

Slide 42

Slide 42 text

42 ͋ΔࠃͷίϛϡχςΟϚωʔδϟ͔ΒϢʔβମݧ͕ѱ͍ͱݴΘΕΔ ՝୊ᶃɿ ಛఆͷࠃʹ͓͍ͯϢʔβମݧ͕ѱ͍ දࣔ͞ΕΔ·Ͱ͕஗͍ ௚͙ݕࡧͰ͖ͳ͍ ݪҼ͕Θ͔Βͳ͍ͱ վળ͸Ͱ͖ͳ͍

Slide 43

Slide 43 text

௅ઓᶃ ੈքதͷϢʔβମݧΛଌఆ͠վળ͢Δ 43

Slide 44

Slide 44 text

ɾఆ఺؍ଌʹ͸Catchpoint SystemsͷSynthetic MonitoringΛར༻ - ໿30Χࠃ͔ΒఆظతʹϝτϦΫεΛऩू - Time To First Byte - DNS lookup time - TLS established time - ϨεϙϯελΠϜ ͳͲ 44 ఆ఺؍ଌʹΑΔ֤ࠃͷϢʔβମݧͷଌఆ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ

Slide 45

Slide 45 text

45 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

Slide 46

Slide 46 text

46 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

Slide 47

Slide 47 text

47 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

Slide 48

Slide 48 text

48 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ ݪҼΛௐࠪͰ͖Δঢ়ଶʹͳͬͨ

Slide 49

Slide 49 text

49 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ ΠϯυωγΞͷTTFB ΠϯυωγΞͷTLS઀ଓ࣌ؒ ถࠃͷTTFB ถࠃͷTLS઀ଓ࣌ؒ ໿750ms ໿330ms ໿225ms ໿150ms

Slide 50

Slide 50 text

50 ΠϯυωγΞ͸ถࠃͱൺ΂ͯ TLS઀ଓ΍Time To First Byteʹ ͕͔͔࣌ؒΔ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

Slide 51

Slide 51 text

51 ถࠃ͔Β஍ཧతʹԕ͍ࠃͷϢʔβମݧ ͕ѱ͍܏޲ʹ͋Δ͜ͱ͕൑໌ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

Slide 52

Slide 52 text

άϩʔόϧαʔϏεʹ͓͚ΔSREͷ௅ઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍ 52 SFGIUUQTUFDIDPOGDPPLQBEDPNTPSBIIUNM

Slide 53

Slide 53 text

άϩʔόϧαʔϏεʹ͓͚ΔSREͷ௅ઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍ 53 ɾαʔό͸ถࠃϦʔδϣϯͰ؅ཧ ɾcookpad.comυϝΠϯΛࠃ಺αʔϏεͱڞ༗ ɾRoute53ͷLatency Based RoutingͰIPΛฦ٫ - ౦ژ͔ถࠃͷ͍ۙσʔληϯλ·ͣΞΫηε ɾࠃ಺ͱάϩʔόϧαʔϏεΛৼΓ෼͚ - ϦόʔεϓϩΩγͷਖ਼نදݱͰ൑ఆ - cookpad.com͸೔ຊͷαʔϏε - cookpad.com/ukͳͲ͸άϩʔόϧαʔϏε

Slide 54

Slide 54 text

άϩʔόϧαʔϏεʹ͓͚ΔSREͷ௅ઓ: ถࠃ͔Βԕ͍ࠃͰͷϢʔβମݧ͕ѱ͍ 54 location ~ ^/(en|vi|fil|ae|al|ao|ar|arabic|at|au|az|ba|bd|be|bg|bh|bo| br|by|ch|ci|cl|cn|co|cr|cu|cy|cz|de|dj|dk|do|dz|ec|ee|eg| es|fi|fr|gh|gr|gt|hk|hn|hr|hu|id|ie|il|in|iq|ir|it|jo| ke|km|kr|kw|kz|lb|lk|lt|lu|lv|ly|ma|mo|mr|mx|my|mz|ng|ni| nl|no|nz|om|pa|pe|ph|pk|pl|ps|pt|py|qa|ro|rs|ru|sa| sd|se|sg|si|sk|so|sv|sy|th|tn|tr|tt|tw|ua|uk|us|uy|uz|ve| vn|ye|za)(/|$) { proxy_pass http://cookpad_use1; } location / { proxy_pass https://cookpad_apne; }

Slide 55

Slide 55 text

55 ถࠃ͔Βԕ͍ࠃͷϢʔβ͕շదʹ࢖͑ΔΑ͏ʹ͢Δ ɾΞϝϦΧʹαʔό͕ଘࡏ͍ͯͯ͠ωοτϫʔΫϨΠςϯγ͕ߴ͍ ɾӡ༻ίετ্͕͕Δٕज़͸·ͩೖΕͨ͘ͳ͍ - αʔόͷϚϧνϦʔδϣϯԽͳͲ ɾΠϯυωγΞҎ֎ͷࠃ΋গͳ͍ख਺Ͱޮ཰తʹվળ͍ͨ͠ - ଞʹ΋΍Βͳ͚Ε͹ͳΒͳ͍͜ͱ͸୔ࢁ͋Δ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ

Slide 56

Slide 56 text

56 ถࠃ͔Βԕ͍ࠃͷϢʔβ͕շదʹ࢖͑ΔΑ͏ʹ͢Δ ɾΞϝϦΧʹαʔό͕ଘࡏ͍ͯͯ͠ωοτϫʔΫϨΠςϯγ͕ߴ͍ ɾӡ༻ίετ্͕͕Δٕज़͸·ͩೖΕͨ͘ͳ͍ - αʔόͷϚϧνϦʔδϣϯԽͱ͔ ɾΠϯυωγΞҎ֎ͷࠃ΋গͳ͍ख਺Ͱޮ཰తʹվળ͍ͨ͠ - ଞʹ΋΍Βͳ͚Ε͹ͳΒͳ͍͜ͱ͸୔ࢁ͋Δ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ CDNʹΑΔTCPͱTLSͷऴ୺Ͱ ωοτϫʔΫϨΠςϯγΛվળ͢Δ

Slide 57

Slide 57 text

57 FastlyʹΑΔϨΠςϯγͷվળ ɾWeb / API શͯͷϦΫΤετΛFastlyܦ༝ʹ - Ωϟογϡͱͯ͠࢖͍ͬͯΔΘ͚Ͱ͸ͳ͍ ɾFastlyͰTCPͱTLSͷऴ୺Λ࣮ݱ - Ϣʔβͱσʔληϯλͷڑ཭͕ۙ͘ͳΔ - TCP/TLSͷϋϯυγΣΠΫʹ͔͔Δ͕࣌ؒ୹ॖ͞ΕΔ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

Slide 58

Slide 58 text

58 FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ Fastlyಋೖલ(໿1.25 sec) ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

Slide 59

Slide 59 text

59 FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ Fastlyಋೖલ(໿1.25 sec) Fastlyಋೖޙ(໿0.45 sec) ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

Slide 60

Slide 60 text

60 FastlyಋೖલޙͷΠϯυωγΞͷϨεϙϯελΠϜͷมԽ Fastlyಋೖલ(໿1.25 sec) Fastlyಋೖޙ(໿0.45 sec) ΠϯυωγΞϢʔβͷ ϨεϙϯελΠϜ͸໿1/3ʹ୹ॖ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ

Slide 61

Slide 61 text

61 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ Ξϧθϯνϯ ΠϯυωγΞ α΢δΞϥϏΞ UAE ΠΪϦε Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ

Slide 62

Slide 62 text

62 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ Ξϧθϯνϯ ΠϯυωγΞ α΢δΞϥϏΞ UAE ΠΪϦε Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ ஍ཧతʹถࠃ͔Βԕ͍ࠃͷ Ϣʔβମݧ͕վળ

Slide 63

Slide 63 text

՝୊ᶄ Πϕϯτ͕ͱʹ͔͘ଟ͍ 63

Slide 64

Slide 64 text

ɾ͜͜Ͱݴ͏Πϕϯτ͸ॕࡇ೔ͳͲຖ೥ඞͣ։࠵͞ΕΔ΋ͷ ɾΠϕϯτͷࡍʹ͸Ϣʔβ͕େྔʹΞΫηεͯ͘͠Δ ɾେ͖ͳΠϕϯτ͸ࠃ΍फڭͳͲͷ୯ҐͰ࠵͞ΕΔ 64 ΠϕϯτͷόϦΤʔγϣϯ͕๛෋ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

Slide 65

Slide 65 text

ɾ͜͜Ͱݴ͏Πϕϯτ͸ॕࡇ೔ͳͲຖ೥ඞͣ։࠵͞ΕΔ΋ͷ ɾΠϕϯτͷࡍʹ͸Ϣʔβ͕େྔʹΞΫηεͯ͘͠Δ ɾେ͖ͳΠϕϯτ͸ࠃ΍फڭͳͲͷ୯ҐͰ࠵͞ΕΔ - Ξϧθϯνϯ 65 ΠϕϯτͷόϦΤʔγϣϯ͕๛෋ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

Slide 66

Slide 66 text

66 ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

Slide 67

Slide 67 text

67 ֵ໋ه೦೔ɺಠཱه೦೔ʹ ύεςϦʔτεΛ࡞ͬͯ৯΂Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

Slide 68

Slide 68 text

ɾ͜͜Ͱݴ͏Πϕϯτ͸ॕࡇ೔ͳͲຖ೥ඞͣ։࠵͞ΕΔ΋ͷ ɾΠϕϯτͷࡍʹ͸Ϣʔβ͕େྔʹΞΫηεͯ͘͠Δ ɾେ͖ͳΠϕϯτ͸ࠃ΍फڭͳͲͷ୯ҐͰ࠵͞ΕΔ - Ξϧθϯνϯ - ΠεϥϜݍ 68 ΠϕϯτͷόϦΤʔγϣϯ͕๛෋ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

Slide 69

Slide 69 text

69 ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

Slide 70

Slide 70 text

70 ΠυΡϧΞυϋʢ٘ਜ਼ࡇʣ Ͱ೑Λ৯΂Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

Slide 71

Slide 71 text

71 ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

Slide 72

Slide 72 text

72 ϥϚμϯظؒ͸ ೔຅͔Β໷໌͚ͷؒͰ Ո଒Ͱ৯ࣄΛָ͠Ή ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

Slide 73

Slide 73 text

ɾ೔ຊͷ৔߹ɺόϨϯλΠϯͰϦΫΤετ਺͕2ഒʹͳΔ 73 ΠϕϯτʹΑΔϢʔβϦΫΤετ਺ͷมԽ ٸʹϦΫΤετ਺͕௓Ͷ্͕Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

Slide 74

Slide 74 text

74 ΠϕϯτʹΑΔϢʔβϦΫΤετ਺ͷมԽ ٸʹϦΫΤετ਺͕௓Ͷ্͕Δ ɾΠυΡϧΞυϋ͕࢝·Δͱ਺೔ͷؒϦΫΤετ਺͕2ഒʹͳΔ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

Slide 75

Slide 75 text

ɾϥϚμϯ͕࢝·Δͱ1ϲ݄ͷؒϦΫΤετ਺͕2ʙ3ഒʹͳΔ 75 ΠϕϯτʹΑΔϢʔβϦΫΤετ਺ͷมԽ ٸʹϦΫΤετ਺͕௓Ͷ্͕Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

Slide 76

Slide 76 text

ɾւ֎ͷϨγϐαʔϏε͕Cookpadͷ஥ؒʹͳΔ͜ͱ΋͋Δ 76 1Χࠃ૿͑ͯɺಥવϢʔβ਺͕100ສਓ૿͑Δέʔε͕༗Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍

Slide 77

Slide 77 text

ɾւ֎ͷϨγϐαʔϏε͕Cookpadͷ஥ؒʹͳΔ͜ͱ΋͋Δ 77 1Χࠃ૿͑ͯɺಥવϢʔβ਺͕100ສਓ૿͑Δέʔε͕༗Δ ՝୊ᶄɿ Πϕϯτ͕ͱʹ͔͘ଟ͍ ࠃͷ਺͕૿͑ͯ Πϕϯτ΋૿͑Δ

Slide 78

Slide 78 text

78 ຖ݄όϨϯλΠϯ —

Slide 79

Slide 79 text

79 શͯͷΠϕϯτʹ౎౓ SRE͕ରࡦ͢Δͷ͸೉͍͠

Slide 80

Slide 80 text

௅ઓᶄ γεςϜͱ૊৫ͷεέʔϥϏϦςΟΛ ߟྀͨ͠࢓૊ΈΛೖΕΔ 80

Slide 81

Slide 81 text

81 εέʔϥϏϦςΟɾՄ༻ੑΛߟྀٕͨ͠ज़ͷ࠾༻ ɾϓϩμΫγϣϯͷσʔλϕʔε͸શ୆Amazon AuroraΛར༻ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

Slide 82

Slide 82 text

82 DockerΞϓϦ։ൃ؀ڥͷఏڙ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

Slide 83

Slide 83 text

ɾ೔ຊͰഓΘΕͨECS + hakoʹΑΔσϓϩΠγεςϜΛಋೖ - ΦʔτεέʔϧΛඪ४౥ࡌ (ϨΨγΞϓϦ͸ࣗ෼Ͱ࣮૷͢Δඞཁ͕͋ͬͨ) - ։ൃ΋εέʔϧ͢Δ - ։ൃऀ͸ґཔͳ͠ʹ؀ڥม਺΍ൿಗ৘ใΛมߋՄೳ - ECSΫϥελʹίϯςφΛσϓϩΠ͢ΔͷͰΠϯελϯεͷηοτΞοϓෆཁ - hako-consoleͳͲͷࣾ಺ͷΤίγεςϜͷԸܙΛڗड 83 ref https://speakerdeck.com/eagletmt/web-application-development-in-cookpad-2017 DockerΞϓϦ։ൃ؀ڥͷఏڙ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

Slide 84

Slide 84 text

84 hako-console 1PXFSFECZ!FBHMFUNU ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

Slide 85

Slide 85 text

85 hako-console 1PXFSFECZ!FBHMFUNU ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

Slide 86

Slide 86 text

86 hako-console 1PXFSFECZ!FBHMFUNU ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ άϩʔόϧαʔϏεͰ΋ ೔ຊͰͷٕज़΍ܦݧΛ ׆͔͍ͯ͠Δ

Slide 87

Slide 87 text

87 ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕σʔλετΞͷεϩʔΫΤϦΛ෼ੳͰ͖ΔΑ͏ʹ͢Δ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

Slide 88

Slide 88 text

88 ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕ओཁͳϝτϦΫεΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

Slide 89

Slide 89 text

89 ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕HTTPͷεςʔλείʔυมԽΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

Slide 90

Slide 90 text

90 ֤छμογϡϘʔυͷ࡞੒ͱϘτϧωοΫͷڞ༗ ɾ։ൃऀ͕HTTPͷεςʔλείʔυมԽΛ֬ೝͰ͖ΔΑ͏ʹ͢Δ ։ൃऀ͕ࣗ෼Ͱ໰୊Λൃݟͯ͠ ηϧϑϚωδϝϯτͰ͖Δ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ

Slide 91

Slide 91 text

՝୊ᶅ σϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍ 91

Slide 92

Slide 92 text

92 ೔ຊͰՄೳͳΦϖϨʔγϣϯ͸ւ֎Ͱ͸Ͱ͖ͳ͍͜ͱ΋͋Δ ɾ௕࣌ؒίωΫγϣϯΛҡ࣋Ͱ͖ͳ͍؀ڥ - ωοτϫʔΫ؀ڥ͕೔ຊͷΑ͏ʹ੔͍ͬͯΔͱ͸ݶΒͳ͍ - ւ֎Ͱ͸ఀిɾߑਫͳͲͷҟৗࣄଶ͕೔ৗతʹى͜Δ - खݩ͔ΒCapistranoͰσϓϩΠͰ͖ΔਓɾͰ͖ͳ͍ਓ͕͍Δ ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍

Slide 93

Slide 93 text

93 ೔ຊͰՄೳͳΦϖϨʔγϣϯ͸ւ֎Ͱ͸Ͱ͖ͳ͍͜ͱ΋͋Δ ɾ௕࣌ؒίωΫγϣϯΛҡ࣋Ͱ͖ͳ͍؀ڥ - ωοτϫʔΫ؀ڥ͕೔ຊͷΑ͏ʹ੔͍ͬͯΔͱ͸ݶΒͳ͍ - ఀిɾߑਫͳͲͷҟৗࣄଶ͸ී௨ʹى͖Δ - खݩ͔ΒCapistranoͰσϓϩΠͰ͖ΔਓɾͰ͖ͳ͍ਓ͕͍Δ ͦ΋ͦ΋σϓϩΠͰ͖ͳ͍ ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍

Slide 94

Slide 94 text

94 σϓϩΠΛଞͷਓʹґཔ͢Δੈքઢ͕஀ੜ͢Δ ɾखݩ͔ΒσϓϩΠ͕Ͱ͖ͳ͍։ൃऀ͔ΒσϓϩΠґཔ͕དྷΔ ɾσϓϩΠͷ୅ཧ͸͓ޓ͍ʹγϯυΠ - ։ൃऀ͸1೔ʹproductionʹԿճ΋σϓϩΠ͢Δ - ґཔ͢Δํ΋ɺґཔ͞ΕΔํ΋γϯυΠ - σϓϩΠʹର͢Δ৺ཧతͳίετ্͕͕ͬͯ͠·͏ - ৽͍͠ػೳΛϦϦʔε͍ͨ͠ͷʹ୅ཧσϓϩΠऀ଴ͪ΋͋Γ͏Δ ՝୊ᶅɿσϓϩΠͷΦϖϨʔγϣϯίετ͕ߴ͍

Slide 95

Slide 95 text

௅ઓᶅ ੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ 95

Slide 96

Slide 96 text

96 σϓϩΠαʔόΛถࠃϦʔδϣϯʹ༻ҙ͢Δ ɾssh ͰσϓϩΠαʔόʹΞΫηεͯ͠΋Β͍CapistranoΛ࣮ߦ ɾ୅ཧσϓϩΠͷѱເ͔Β͸ղ์͞ΕΔ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ

Slide 97

Slide 97 text

97 σϓϩΠαʔόͰͷϚχϡΞϧΦϖϨʔγϣϯͷ໰୊ ɾΦϖϨʔγϣϯ͕౷Ұ͞Εͳ͍ - ։ൃऀ͕࠷৽ͷσϓϩΠεΫϦϓτΛ࢖͏ͱ͸ݶΒͳ͍ - ૝ఆ֎ͷ͜ͱΛαʔό಺Ͱߦ͏ਓ͕͍Δ ɾαʔόʹΞΫηε͢Δͷ͕໘౗ ɾσϓϩΠͷϩάΛ௥͍ʹ͍͘ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ

Slide 98

Slide 98 text

98 σϓϩΠ؀ڥͷ࡮৽ chat (slack)  deploy server (cap & hako) bot (ruboty)  job scheduler (Rundeck)  deploy targets ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ

Slide 99

Slide 99 text

99 σϓϩΠαʔό΁ͷϩάΠϯ͕ෆཁʹͳΔ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ

Slide 100

Slide 100 text

100 ૝ఆ͍ͯ͠ΔΦϖϨʔγϣϯʹ౷ҰͰ͖Δ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ

Slide 101

Slide 101 text

101 σϓϩΠͷϩάΛ௥͏ͷ͕༰қʹͳΔ σϓϩΠ࣌ࠁ σϓϩΠͨ͠ਓ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ

Slide 102

Slide 102 text

102 σϓϩΠͷϩάΛ௥͏ͷ͕༰қʹͳΔ ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ

Slide 103

Slide 103 text

՝୊ᶆ toil ͕ٸ૿͢Δ 103

Slide 104

Slide 104 text

104 SFGIUUQTMBOEJOHHPPHMFDPNTSFCPPLIUNM ՝୊ᶆɿtoil͕ٸ૿͢Δ

Slide 105

Slide 105 text

ɾखಈͰରԠ͍ͯ͠Δ ɾࣗಈԽͷ༨஍͕͋Δ ɾ܁Γฦ͠ൃੜ͢Δ ɾൃੜ͔ͯ͠Β͡Όͳ͍ͱରԠͰ͖ͳ͍ ɾαʔϏεͷվળʹͭͳ͕Βͳ͍ ɾαʔϏε΍Ϣʔβͷ਺ʹൺྫͯ͠૿Ճ͢Δ 105 toilʹ౰ͯ͸·Δۀ຿ ՝୊ᶆɿtoil͕ٸ૿͢Δ SFGIUUQTMBOEJOHHPPHMFDPNTSFCPPLDIBQUFSTFMJNJOBUJOHUPJMIUNM

Slide 106

Slide 106 text

106 ։ൃऀ͔ΒSRE΁ͷґཔ͕ٸ૿ ֤ظؒͷSRE΁ͷґཔ਺ͷਪҠ ՝୊ᶆɿtoil͕ٸ૿͢Δ

Slide 107

Slide 107 text

107 ۀ຿ׂ߹ͷมԽ /PW %FD +BO 'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH ՝୊ᶆɿtoil͕ٸ૿͢Δ

Slide 108

Slide 108 text

108 ۀ຿ׂ߹ͷมԽ /PW %FD +BO 'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH SRE͕΍Γ͍ͨ͜ͱ ՝୊ᶆɿtoil͕ٸ૿͢Δ

Slide 109

Slide 109 text

109 ۀ຿ׂ߹ͷมԽ /PW %FD +BO 'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH toilͷ૿Ճʹൺྫͯ͠ ׂ͕͚࣌ؒͳ͘ͳΔ ՝୊ᶆɿtoil͕ٸ૿͢Δ

Slide 110

Slide 110 text

110 SREͷۀ຿ׂ߹ͷมԽ /PW %FD +BO 'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH ΍Γ͍ͨ͜ͱ͕Ͱ͖ͳ͍ ՝୊ᶆɿtoil͕ٸ૿͢Δ

Slide 111

Slide 111 text

111 2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3 ΧςΰϦ ґཔͷ਺ "DDPVOU.BOBHFNFOU "843FMBUFE0QFSBUJPOT )VHF%BUBCBTF5BCMF.JHSBUJPOT ՝୊ᶆɿtoil͕ٸ૿͢Δ

Slide 112

Slide 112 text

112 2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3 DBUFHPSZ PGJTTVFT "DDPVOU.BOBHFNFOU "843FMBUFE0QFSBUJPOT )VHF%BUBCBTF5BCMF.JHSBUJPOT ΞΧ΢ϯτ؅ཧճΓ ͷґཔͷᓲ໓͕ඞཁ ՝୊ᶆɿtoil͕ٸ૿͢Δ

Slide 113

Slide 113 text

113 ΞΧ΢ϯτ؅ཧ͕ਏ͍ཧ༝ ɾࣾһ਺͕ٸ૿͍ͯ͠Δ͠ɺࠓޙ΋૿͑Δ͜ͱ͕૝૾Ͱ͖Δ ɾ։ൃऀ͕ར༻͢Δπʔϧ͕ଟ༷Խ͖͍ͯͯ͠Δ - Single Sign OnʹରԠ͍ͯ͠ͳ͍πʔϧ΋͋Δ - Ͱ΋ɺΞΫηε੍ޚ͸͍ͨ͠ ɾ͕࣌ࠩ͋ΔͷͰίϛϡχέʔγϣϯͷλΠϛϯάΛҳ͠΍͍͢ - Կ͕͍͔ͨ͠೺Ѳ͢Δͷʹ਺೔͔͔Δ͜ͱ΋ ՝୊ᶆɿtoil͕ٸ૿͢Δ

Slide 114

Slide 114 text

114 άϩʔόϧͰར༻࣮੷ͷ͋Δπʔϧ܈ ɾ෼ੳɿ Kibanaɺ GrafanaɺNewRelicɺCatchpointɺDatadog ɾίʔυ؅ཧɿ Github Enterpriseɺgithub.com ɾόάτϥοΧɿSentryɺBugsnagɺAirbrake ɾCIɿJenkinsɺCircle CIɺSemaphore CI ɾPagerɿPagerDuty ɾͦͷଞɿAWSɺKuroko2ɺin-house πʔϧ ͳͲ ՝୊ᶆɿtoil͕ٸ૿͢Δ

Slide 115

Slide 115 text

115 ࣾ಺πʔϧ΁ͷΞΫηεʹVPNΛ࢖͑Δͱ͸ݶΒͳ͍ ɾੈքதͷΦϑΟεͷ։ൃऀ͕VPNΛ࢖͑Δͷ͔ʁ - શһ͕IPΞυϨεͰ੍ޚͰ͖Δ؀ڥʹ͍ͳ͍ ɾ౦ژͷࣾ಺NWΛܦ༝ͨ͠઀ଓ͸ଥ౰͔ʁ - ࣾ಺πʔϧʹ͸ੈքதͷελοϑ͕ΞΫηε͢Δ - ΞϑϦΧͷελοϑ͸VPNΛ҆ఆͯ͠ுΕΔͩΖ͏͔ ՝୊ᶆɿtoil͕ٸ૿͢Δ

Slide 116

Slide 116 text

116 ࣾ಺πʔϧʹBasicೝূΛ࢖͍ͨ͘ͳ͍ ɾSRE͸ࣾ಺πʔϧ͕૿͑Δ౓ʹBasicೝূͷઃఆΛ͢Δ - ౎౓Ϣʔβ͝ͱʹύεϫʔυΛ༻ҙ͢Δͷ͸໘౗ - ελοϑͰϢʔβͱύεϫʔυΛڞ༗͢Δͷ΋ආ͚͍ͨ ɾϢʔβ͸ࣾ಺πʔϧΛ࢖͏࣌සൟʹύεϫʔυೖྗ͕ඞཁ ՝୊ᶆɿtoil͕ٸ૿͢Δ

Slide 117

Slide 117 text

௅ઓᶆ ΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ 117

Slide 118

Slide 118 text

௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ 118 nginx + omniauthʹΑΔࣾ಺πʔϧͷΞΫηε੍ޚ ɾࣾһ͸ೖࣾ͢ΔͱඞͣG SuiteͷΞΧ΢ϯτ͕෇༩͞ΕΔ ɾࣾ಺πʔϧͷΞΫηε੍ޚ͸ nginx + nginx_omniauth_adapter - ࣾ༻ͷG SuiteΞΧ΢ϯτͰ͋Δ͜ͱΛݕূ - VPNෆཁ - πʔϧຖʹΞΧ΢ϯτ؅ཧ΍Basicೝূͷಋೖ͕ཁΒͳ͘ͳΔ

Slide 119

Slide 119 text

119 nginx + omniauthʹΑΔࣾ಺πʔϧͷΞΫηε੍ޚ ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ

Slide 120

Slide 120 text

120 SRE͔͠Ͱ͖ͳ͍ΞΧ΢ϯτ؅ཧͷݖݶҠৡ ɾLDAPɺActive DirectoryͷΞΧ΢ϯτ؅ཧͷݖݶΛҠৡ - Github Enterprise΍αʔόͷύεϫʔυΛࣗ෼ͰઃఆͰ͖Δ ɾ։ൃऀ͕࢖͏ssh keyΛ֤छαʔό΁ͷࣗಈσϓϩΠ - Gatewayͷssh key͸ैདྷitamaeͳͲͰσϓϩΠ͍ͯͨ͠ - ਓ਺͕ଟ͍ͱ౎౓ରԠ͢Δͷ͸ਏ͍ - ࣗ෼ͰઃఆͰ͖ͯɺࣗಈతʹσϓϩΠ͞ΕΔ ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ

Slide 121

Slide 121 text

121 1PXFSFECZ!TPSBI ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ

Slide 122

Slide 122 text

122 DBUFHPSZ PGJTTVFT "DDPVOU.BOBHFNFOU "843FMBUFE0QFSBUJPOT )VHF%BUBCBTF5BCMF.JHSBUJPOT 2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3ͷݱࡏ ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ

Slide 123

Slide 123 text

123 2017೥1~5݄ʹ૿Ճͨ͠toilτοϓ3ͷݱࡏ DBUFHPSZ PGJTTVFT "DDPVOU.BOBHFNFOU "843FMBUFE0QFSBUJPOT )VHF%BUBCBTF5BCMF.JHSBUJPOT ਺݅·Ͱݮগ ਺݅·Ͱݮগ ࣗಈԽத ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ

Slide 124

Slide 124 text

124 ֤छࢪࡦΛܦͯSRE΁ͷґཔ਺͸ݮগ܏޲ +BO 'FC .BS "QS .BZ +VO +VM "VH 4FQ 0DU /PW %FD 2017೥ͷ֤݄ͷSRE΁ͷґཔ਺ͷਪҠ ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ

Slide 125

Slide 125 text

՝୊ᶆ “toil͕૿͑Δ”΁ͷ΋͏ҰͭͷऔΓ૊Έ 125

Slide 126

Slide 126 text

126 SREͷϚϧνϦʔδϣϯରԠ ՝୊ᶆ “toil͕૿͑Δ”΁ͷ΋͏ҰͭͷऔΓ૊Έ Japan UK US

Slide 127

Slide 127 text

·ͱΊ 127

Slide 128

Slide 128 text

·ͱΊ 128 ɾαʔϏε΋૊৫΋੒௕த ɾάϩʔόϧͳΒͰ͸ͷऔΓ૊Έ͕͋Δ ɾSREͱͯ͠௅ઓͰ͖Δ՝୊͕ࢁ΄Ͳ͋Δ

Slide 129

Slide 129 text

Thank you !! 129 [email protected]