Challenges for Global Service from a Perspective of SRE

Challenges for Global Service from a Perspective of SRE

Cookpad TechConf 2018: https://techconf.cookpad.com/2018/

Transcript

  1. 1.

    Challenges for Global Service from a Perspective of SRE Takayuki

    Watanabe Infrastructure Department SRE Group Cookpad Inc. Feb 10, 2018
  2. 2.
  3. 3.

    ࣗݾ঺հ ɾΠϯϑϥετϥΫνϟʔ෦ SREάϧʔϓ - ΫοΫύουͷάϩʔόϧαʔϏεͷ։ൃʹैࣄ ɾSRE = Site Reliability Engineering

    - System & Software EngineeringͰ αΠτͷ৴པੑʹؔΘΔશͯͷ՝୊ʹऔΓ૊Ή 3 ౉ลڤ೭ (Takayuki Watanabe) twitter: takanabe_w / github: takanabe
  4. 21.

    21

  5. 31.

    2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 31 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί

    ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@
  6. 32.

    2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 32 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί

    ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ ೔ຊൃͷΞϓϦͰ ࠷ଟͷ7஍ҬͰಉ࣌ೖ৆
  7. 33.

    2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 33 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί

    ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ αʔϏε΋૊৫΋੒௕͖ͯͨ͠
  8. 34.

    2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 34 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί

    ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ ҰํͰ୔ࢁͷ՝୊΋͋ͬͨ
  9. 44.

    ɾఆ఺؍ଌʹ͸Catchpoint SystemsͷSynthetic MonitoringΛར༻ - ໿30Χࠃ͔ΒఆظతʹϝτϦΫεΛऩू - Time To First Byte

    - DNS lookup time - TLS established time - ϨεϙϯελΠϜ ͳͲ 44 ఆ఺؍ଌʹΑΔ֤ࠃͷϢʔβମݧͷଌఆ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ
  10. 57.

    57 FastlyʹΑΔϨΠςϯγͷվળ ɾWeb / API શͯͷϦΫΤετΛFastlyܦ༝ʹ - Ωϟογϡͱͯ͠࢖͍ͬͯΔΘ͚Ͱ͸ͳ͍ ɾFastlyͰTCPͱTLSͷऴ୺Λ࣮ݱ -

    Ϣʔβͱσʔληϯλͷڑ཭͕ۙ͘ͳΔ - TCP/TLSͷϋϯυγΣΠΫʹ͔͔Δ͕࣌ؒ୹ॖ͞ΕΔ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
  11. 62.

    62 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ Ξϧθϯνϯ ΠϯυωγΞ α΢δΞϥϏΞ UAE ΠΪϦε Fastlyಋೖ Fastlyಋೖ

    Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ ஍ཧతʹถࠃ͔Βԕ͍ࠃͷ Ϣʔβମݧ͕վળ
  12. 83.

    ɾ೔ຊͰഓΘΕͨECS + hakoʹΑΔσϓϩΠγεςϜΛಋೖ - ΦʔτεέʔϧΛඪ४౥ࡌ (ϨΨγΞϓϦ͸ࣗ෼Ͱ࣮૷͢Δඞཁ͕͋ͬͨ) - ։ൃ΋εέʔϧ͢Δ - ։ൃऀ͸ґཔͳ͠ʹ؀ڥม਺΍ൿಗ৘ใΛมߋՄೳ

    - ECSΫϥελʹίϯςφΛσϓϩΠ͢ΔͷͰΠϯελϯεͷηοτΞοϓෆཁ - hako-consoleͳͲͷࣾ಺ͷΤίγεςϜͷԸܙΛڗड 83 ref https://speakerdeck.com/eagletmt/web-application-development-in-cookpad-2017 DockerΞϓϦ։ൃ؀ڥͷఏڙ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
  13. 98.

    98 σϓϩΠ؀ڥͷ࡮৽ chat (slack)
 deploy server (cap & hako) bot

    (ruboty)
 job scheduler (Rundeck)
 deploy targets ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
  14. 106.

    106 ։ൃऀ͔ΒSRE΁ͷґཔ͕ٸ૿        

       ֤ظؒͷSRE΁ͷґཔ਺ͷਪҠ ՝୊ᶆɿtoil͕ٸ૿͢Δ
  15. 107.

    107 ۀ຿ׂ߹ͷมԽ      /PW %FD +BO

    'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH ՝୊ᶆɿtoil͕ٸ૿͢Δ
  16. 108.

    108 ۀ຿ׂ߹ͷมԽ      /PW %FD +BO

    'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH SRE͕΍Γ͍ͨ͜ͱ ՝୊ᶆɿtoil͕ٸ૿͢Δ
  17. 109.

    109 ۀ຿ׂ߹ͷมԽ      /PW %FD +BO

    'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH toilͷ૿Ճʹൺྫͯ͠ ׂ͕͚࣌ؒͳ͘ͳΔ ՝୊ᶆɿtoil͕ٸ૿͢Δ
  18. 110.

    110 SREͷۀ຿ׂ߹ͷมԽ      /PW %FD +BO

    'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH ΍Γ͍ͨ͜ͱ͕Ͱ͖ͳ͍ ՝୊ᶆɿtoil͕ٸ૿͢Δ
  19. 118.

    ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ 118 nginx + omniauthʹΑΔࣾ಺πʔϧͷΞΫηε੍ޚ ɾࣾһ͸ೖࣾ͢ΔͱඞͣG SuiteͷΞΧ΢ϯτ͕෇༩͞ΕΔ ɾࣾ಺πʔϧͷΞΫηε੍ޚ͸ nginx +

    nginx_omniauth_adapter - ࣾ༻ͷG SuiteΞΧ΢ϯτͰ͋Δ͜ͱΛݕূ - VPNෆཁ - πʔϧຖʹΞΧ΢ϯτ؅ཧ΍Basicೝূͷಋೖ͕ཁΒͳ͘ͳΔ
  20. 120.

    120 SRE͔͠Ͱ͖ͳ͍ΞΧ΢ϯτ؅ཧͷݖݶҠৡ ɾLDAPɺActive DirectoryͷΞΧ΢ϯτ؅ཧͷݖݶΛҠৡ - Github Enterprise΍αʔόͷύεϫʔυΛࣗ෼ͰઃఆͰ͖Δ ɾ։ൃऀ͕࢖͏ssh keyΛ֤छαʔό΁ͷࣗಈσϓϩΠ -

    Gatewayͷssh key͸ैདྷitamaeͳͲͰσϓϩΠ͍ͯͨ͠ - ਓ਺͕ଟ͍ͱ౎౓ରԠ͢Δͷ͸ਏ͍ - ࣗ෼ͰઃఆͰ͖ͯɺࣗಈతʹσϓϩΠ͞ΕΔ ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
  21. 123.
  22. 124.

    124 ֤छࢪࡦΛܦͯSRE΁ͷґཔ਺͸ݮগ܏޲      +BO 'FC .BS

    "QS .BZ +VO +VM "VH 4FQ 0DU /PW %FD 2017೥ͷ֤݄ͷSRE΁ͷґཔ਺ͷਪҠ ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
  23. 127.