Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Challenges for Global Service from a Perspective of SRE

Challenges for Global Service from a Perspective of SRE

Cookpad TechConf 2018: https://techconf.cookpad.com/2018/

More Decks by Takayuki WATANABE (渡辺 喬之)

Other Decks in Technology

Transcript

  1. Challenges for Global Service from a Perspective of SRE Takayuki

    Watanabe Infrastructure Department SRE Group Cookpad Inc. Feb 10, 2018
  2. ࣗݾ঺հ ɾΠϯϑϥετϥΫνϟʔ෦ SREάϧʔϓ - ΫοΫύουͷάϩʔόϧαʔϏεͷ։ൃʹैࣄ ɾSRE = Site Reliability Engineering

    - System & Software EngineeringͰ αΠτͷ৴པੑʹؔΘΔશͯͷ՝୊ʹऔΓ૊Ή 3 ౉ลڤ೭ (Takayuki Watanabe) twitter: takanabe_w / github: takanabe
  3. 21

  4. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 31 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί

    ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@
  5. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 32 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί

    ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ ೔ຊൃͷΞϓϦͰ ࠷ଟͷ7஍ҬͰಉ࣌ೖ৆
  6. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 33 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί

    ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ αʔϏε΋૊৫΋੒௕͖ͯͨ͠
  7. 2017೥ͷάϩʔόϧαʔϏεͷ੒௕ 34 Google Playʮϕετ Φϒ 2017ʯͰ7஍Ҭ͕બग़ ɾεϖΠϯ ɾΠλϦΞ ɾϑϥϯε ɾϝΩγί

    ɾϒϥδϧ ɾೆถ ɾΠϯυωγΞ SFGIUUQTJOGPDPPLQBEDPNQSOFXTQSFTT@@ ҰํͰ୔ࢁͷ՝୊΋͋ͬͨ
  8. ɾఆ఺؍ଌʹ͸Catchpoint SystemsͷSynthetic MonitoringΛར༻ - ໿30Χࠃ͔ΒఆظతʹϝτϦΫεΛऩू - Time To First Byte

    - DNS lookup time - TLS established time - ϨεϙϯελΠϜ ͳͲ 44 ఆ఺؍ଌʹΑΔ֤ࠃͷϢʔβମݧͷଌఆ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ SFGIUUQUFDIMJGFDPPLQBEDPNFOUSZ
  9. 57 FastlyʹΑΔϨΠςϯγͷվળ ɾWeb / API શͯͷϦΫΤετΛFastlyܦ༝ʹ - Ωϟογϡͱͯ͠࢖͍ͬͯΔΘ͚Ͱ͸ͳ͍ ɾFastlyͰTCPͱTLSͷऴ୺Λ࣮ݱ -

    Ϣʔβͱσʔληϯλͷڑ཭͕ۙ͘ͳΔ - TCP/TLSͷϋϯυγΣΠΫʹ͔͔Δ͕࣌ؒ୹ॖ͞ΕΔ ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ
  10. 62 ௅ઓᶃɿ ੈքதͷϢʔβମݧΛܭଌ͠վળ͢Δ Ξϧθϯνϯ ΠϯυωγΞ α΢δΞϥϏΞ UAE ΠΪϦε Fastlyಋೖ Fastlyಋೖ

    Fastlyಋೖ Fastlyಋೖ Fastlyಋೖ ஍ཧతʹถࠃ͔Βԕ͍ࠃͷ Ϣʔβମݧ͕վળ
  11. ɾ೔ຊͰഓΘΕͨECS + hakoʹΑΔσϓϩΠγεςϜΛಋೖ - ΦʔτεέʔϧΛඪ४౥ࡌ (ϨΨγΞϓϦ͸ࣗ෼Ͱ࣮૷͢Δඞཁ͕͋ͬͨ) - ։ൃ΋εέʔϧ͢Δ - ։ൃऀ͸ґཔͳ͠ʹ؀ڥม਺΍ൿಗ৘ใΛมߋՄೳ

    - ECSΫϥελʹίϯςφΛσϓϩΠ͢ΔͷͰΠϯελϯεͷηοτΞοϓෆཁ - hako-consoleͳͲͷࣾ಺ͷΤίγεςϜͷԸܙΛڗड 83 ref https://speakerdeck.com/eagletmt/web-application-development-in-cookpad-2017 DockerΞϓϦ։ൃ؀ڥͷఏڙ ௅ઓᶄɿγεςϜͱ૊৫ͷεέʔϥϏϦςΟΛߟྀͨ͠࢓૊ΈΛೖΕΔ
  12. 98 σϓϩΠ؀ڥͷ࡮৽ chat (slack)
 deploy server (cap & hako) bot

    (ruboty)
 job scheduler (Rundeck)
 deploy targets ௅ઓᶅɿੈքͷͲ͔͜ΒͰ΋σϓϩΠͰ͖ΔΑ͏ʹ͢Δ
  13. 106 ։ൃऀ͔ΒSRE΁ͷґཔ͕ٸ૿        

       ֤ظؒͷSRE΁ͷґཔ਺ͷਪҠ ՝୊ᶆɿtoil͕ٸ૿͢Δ
  14. 107 ۀ຿ׂ߹ͷมԽ      /PW %FD +BO

    'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH ՝୊ᶆɿtoil͕ٸ૿͢Δ
  15. 108 ۀ຿ׂ߹ͷมԽ      /PW %FD +BO

    'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH SRE͕΍Γ͍ͨ͜ͱ ՝୊ᶆɿtoil͕ٸ૿͢Δ
  16. 109 ۀ຿ׂ߹ͷมԽ      /PW %FD +BO

    'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH toilͷ૿Ճʹൺྫͯ͠ ׂ͕͚࣌ؒͳ͘ͳΔ ՝୊ᶆɿtoil͕ٸ૿͢Δ
  17. 110 SREͷۀ຿ׂ߹ͷมԽ      /PW %FD +BO

    'FC 5PJM 4ZTUFN&OHJOFFSJOH 4PGUXBSF&OHJOFFSJOH ΍Γ͍ͨ͜ͱ͕Ͱ͖ͳ͍ ՝୊ᶆɿtoil͕ٸ૿͢Δ
  18. ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ 118 nginx + omniauthʹΑΔࣾ಺πʔϧͷΞΫηε੍ޚ ɾࣾһ͸ೖࣾ͢ΔͱඞͣG SuiteͷΞΧ΢ϯτ͕෇༩͞ΕΔ ɾࣾ಺πʔϧͷΞΫηε੍ޚ͸ nginx +

    nginx_omniauth_adapter - ࣾ༻ͷG SuiteΞΧ΢ϯτͰ͋Δ͜ͱΛݕূ - VPNෆཁ - πʔϧຖʹΞΧ΢ϯτ؅ཧ΍Basicೝূͷಋೖ͕ཁΒͳ͘ͳΔ
  19. 120 SRE͔͠Ͱ͖ͳ͍ΞΧ΢ϯτ؅ཧͷݖݶҠৡ ɾLDAPɺActive DirectoryͷΞΧ΢ϯτ؅ཧͷݖݶΛҠৡ - Github Enterprise΍αʔόͷύεϫʔυΛࣗ෼ͰઃఆͰ͖Δ ɾ։ൃऀ͕࢖͏ssh keyΛ֤छαʔό΁ͷࣗಈσϓϩΠ -

    Gatewayͷssh key͸ैདྷitamaeͳͲͰσϓϩΠ͍ͯͨ͠ - ਓ਺͕ଟ͍ͱ౎౓ରԠ͢Δͷ͸ਏ͍ - ࣗ෼ͰઃఆͰ͖ͯɺࣗಈతʹσϓϩΠ͞ΕΔ ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ
  20. 124 ֤छࢪࡦΛܦͯSRE΁ͷґཔ਺͸ݮগ܏޲      +BO 'FC .BS

    "QS .BZ +VO +VM "VH 4FQ 0DU /PW %FD 2017೥ͷ֤݄ͷSRE΁ͷґཔ਺ͷਪҠ ௅ઓᶆɿΞΧ΢ϯτ؅ཧʹؔ͢ΔґཔΛᓲ໓͢Δ