クックパッドの機械学習を支える基盤のつくりかた / Machine Learning ops at Cookpad

クックパッドの機械学習を支える基盤のつくりかた / Machine Learning ops at Cookpad

7dc8611c26c3ca62c551109c65d04270?s=128

Yuichiro Someya

June 01, 2017
Tweet

Transcript

  1. 1.

    ؙحؙػحسך 堣 唒㷕 统׾  佄ִ׷㛇 湍ךא ֻ׶ַ ׋ 2017/06/01

    AWS Summit Tokyo ΫοΫύουגࣜձࣾ છ୩ ༔Ұ࿠ ੕ ๺ే
  2. 2.
  3. 3.
  4. 5.
  5. 7.

     ΫοΫύουͱ"84  "NB[PO&$Πϯελϯε d ୆ఔ౓ "NB[PO4ΦϒδΣΫτ ԯҎ্ ‣ ೥ʹ%$͔Β׬શҠߦ

    ‣ BQOPSUIFBTU͓ΑͼVTFBTUΛओʹར༻  άϩʔόϧαʔϏε͸ओʹ64ͰՔಇ  SFRTҎ্ ϐʔΫ࣌ಉ࣌ϦΫΤετ਺
  6. 8.

     ར༻͍ͯ͠ΔαʔϏε ‣ ࠓճͷ͓࿩ʹؔ܎͢ΔαʔϏε  "NB[PO&$ "NB[PO71$ "84-BNCEB
 Ҏ߱&$ 71$

    -BNCEBͱදه   "NB[PO424 "NB[PO4/4 Ҏ߱424 4/4ͱදه   "NB[PO4 3FETIJGU Ҏ߱4 3FETIJGUͱදه   "84$POpH "NB[PO$MPVE8BUDI "84$MPVE5SBJM "84*".
 Ҏ߱$POpH $MPVE8BUDI $MPVE5SBJM *".ͱදه 
  7. 10.

     ΫοΫύουͷݚڀ։ൃ෦ʹ͍ͭͯ ‣ ೥݄ʹൃ଍  ݚڀ։ൃνʔϜ ೥݄d ɺݚڀ։ൃࣨ ೥݄d Λܦͯ


    ݱࡏ͸ਖ਼ࣾһ໊ ΞϧόΠτʗΠϯλʔϯ໊ͷ૊৫ʹ ‣ ػցֶशΛத৺ʹɺ৯จԽܥͷݚڀʹ͓͚Δ࣮੷΋  ΫοΫύουߐށ͝൧  ୈճਓ޻஌ೳֶձશࠃେձ ‣ ྉཧ͖Ζ͘ͷެ։ ೥݄ 
  8. 12.

     ྉཧ͖Ζ͘ͷΞʔΩςΫνϟ ‣ ը૾ͷΞοϓϩʔυɺ൑ఆॲཧʹ͸͕͔͔࣌ؒΔ d਺ඦNT   "1*αʔόʔͱ౷߹ͯ͠ಉظॲཧ͢Δͷ͸ඇݱ࣮త  ඇಉظͳྉཧʗඇྉཧͷ൑ఆॲཧ͕ඞཁ

    ‣ ඞཁͳܭࢉػ؀ڥ͕ΞϓϦέʔγϣϯαʔόʔͱେ͖͘ҟͳΔ  3VCZ˱1ZUIPO 3BJMT˱$IBJOFS $16˱(16  ൑ఆॲཧͱΞϓϦέʔγϣϯΛૄ݁߹ʹ͢Δͷ͕ϕλʔ 
  9. 15.

     ྉཧ͖Ζ͘ͷΞʔΩςΫνϟ ‣ "1*αʔόʔͰ41SF4JHOFE63-Λੜ੒  ੜ੒͞Εͨ63-Λར༻ͯ͠ΫϥΠΞϯτ J04 "OEPJSE ͔Β൑ఆ༻ ʹॖখ͞ΕͨࣸਅΛ4ʹ௚઀Ξοϓϩʔυ

     ൑ఆ༻ը૾ͷ
 Ξοϓϩʔυ 1SF4JHOFE63-
 ͷൃߦ https://s3.amazonaws.com/thumbnails/123.jpg? expires=1495436210&Signature=XXXYYYZZZ
  10. 22.

     ྉཧ͖Ζ͘ͷΞʔΩςΫνϟ  "QQSPYJNBUF"HF0G0MEFTU.FTTBHF &$4$MVTUFS εέʔϧΞ΢τ
 (desired_count++) ‣ 424ͷ$MPVE8BUDIϝτϦΫεʹج͍ͮͨ"MBSNͰXPSLFSΛΦʔτεέʔϧ 

    ΞϓϦέʔγϣϯͷੑೳཁٻΛຬͨ͢Α͏ʹ
 ྫ͑͹ɺ৽͘͠Ξοϓϩʔυ͞Εͨࣸਅ͸࣌ؒҎ಺ʹ൑ఆ͞Εͯཉ͍͠   "QQSPYJNBUF"HF0G0MEFTU.FTTBHF͕ར༻Ͱ͖Δ
  11. 23.

     &$4$MVTUFS ྉཧ͖Ζ͘ͷΞʔΩςΫνϟ  "QQSPYJNBUF"HF0G0MEFTU.FTTBHF εέʔϧΞ΢τ
 (desired_count++) ‣ 424ͷ$MPVE8BUDIϝτϦΫεʹج͍ͮͨ"MBSNͰXPSLFSΛΦʔτεέʔϧ 

    ΞϓϦέʔγϣϯͷੑೳཁٻΛຬͨ͢Α͏ʹ
 ྫ͑͹ɺ৽͘͠Ξοϓϩʔυ͞Εͨࣸਅ͸࣌ؒҎ಺ʹ൑ఆ͞Εͯཉ͍͠   "QQSPYJNBUF"HF0G0MEFTU.FTTBHF͕ར༻Ͱ͖Δ
  12. 24.

     &$4্Ͱͷࣸਅ൑ఆ  ‣ FBHMFUNUIBLP  Λར༻  ZBNMܗࣜͷϑΝΠϧͰ൑ఆॲཧXPSLFSΛఆٛɺ$-*ίϚϯυͰ্ཱͪ͛ 

    Φʔτεέʔϧͷઃఆ΋IBLPͰߦ͑Δ scheduler: cluster: hako-production-g2 desired_count: 1 autoscaling: min_capacity: 4 max_capacity: 10 policies: - alarms: [cookpadnet-worker-busy] cooldown: 300 adjustment_type: ChangeInCapacity scaling_adjustment: 1 app: image: cookpadnet-worker cpu: 128 memory: 1024 env: AWS_REGION: ap-northeast-1 COOKPADNET_ENV: production ... &$4$MVTUFS "QQSPYJNBUF"HF0G0MEFTU.FTTBHF IUUQTHJUIVCDPNFBHMFUNUIBLP
  13. 26.

     &$4$MVTUFS H 5BTL $POUBJOFS &$4্Ͱͷࣸਅ൑ఆ ‣ $POUBJOFS͔Β(16Λૢ࡞͢Δҝʹඞཁͳઃఆ  /7*%*"%SJWFSͷΠϯετʔϧ


    $MVTUFSʹΠϯετʔϧ͠ɺ5BTL͔Β͸
 WPMVNFͰݟΔ  ద੾ͳ-JOVY$BQBCJMJUZͷ෇༩
 QSJWJMFHFEΦϓγϣϯΛར༻  (16 /7*%*"%SJWFS
 LFSOFMNPEVMFT /7*%*"%SJWFS
 VTFSMJCSBSJFT "QQ 8PSLFS
  14. 27.

     &$4$MVTUFS H &$4্Ͱͷࣸਅ൑ఆ ‣ QSJWJMFHFEͳ5BTL ͷSPPUϢʔβʔ ͸ɺ&$4Ϋϥελ ʹରͯ͠SPPUϢʔβʔͱಉ༷ͳݖݶΛ΋ͭ 

    ΫϥελʹӨڹΛ༩͑ΔΑ͏ͳૢ࡞͕Մೳ
 NMPDL LJMM VUJNF ʜ   &$4ͰAEFWJDFAΦϓγϣϯ͕࢖͑Δͱʜ ‣ 5BTLͰඇSPPUͳϢʔβʔΛར༻͢Δ
 %PDLFS'JMF಺ͰA64&3XPSLFSA౳   $MVTUFS্ͷ/7*%*"%SJWFS VTFSMJCSBSJFT ͕
 ݟ͑ͳ͘ͳΔ  5BTL $POUBJOFS (16 /7*%*"%SJWFS
 LFSOFMNPEVMFT /7*%*"%SJWFS
 VTFSMJCSBSJFT "QQ 8PSLFS
  15. 28.

     &$4$MVTUFS H 5BTL $POUBJOFS &$4্Ͱͷࣸਅ൑ఆ ‣ $MVTUFSͱಉ͡όʔδϣϯͷVTFSMJCSBSJFTΛ
 $POUBJOFSʹΠϯετʔϧ 

    $POUBJOFS͕$MVTUFSʹόʔδϣϯґଘ͢Δ
 ผͷ$MVTUFSͰ͸ಈ͔ͳ͍͔΋͠Εͳ͍  (16 /7*%*"%SJWFS
 LFSOFMNPEVMFT /7*%*"%SJWFS
 VTFSMJCSBSJFT "QQ 8PSLFS
  16. 31.

     ੕๺ే ΄͠΄͘ͱ !LBOJ@C ‣ ΫοΫύουגࣜձࣾ
 ΠϯϑϥετϥΫνϟʔ෦෦௕  ೥৽ଔೖࣾ ‣

    43& ηΩϡϦςΟΤϯδχΞ  ઃܭߏஙӡ༻ *%4 8"' ੬ऑੑ਍அ *4.4 FUD ‣ "84ೝఆ4" %FW0QTΤϯδχΞ 1SPGFTTJPOBM 
  17. 34.

     ݚڀ։ൃ෦ઃཱ౰࣌ͷ"84؅ཧମ੍ ‣ Πϯϑϥ෦ͷӡ༻୲౰ऀͷΈ͕શݖݶΛ࣋ͭ  ৽͍͠Ϧιʔεͷ४උ͸جຊతʹ
 Πϯϑϥ෦͕ߦ͏  ίʔυԽ͞Ε͍ͯΔՕॴ΋l࣮ߦz͸
 Πϯϑϥ෦͕୲౰

    ‣ "84ʹؔ܎͢Δશͯͷ͜ͱΛΠϯϑϥ෦ʹू໿  ઃܭɺӡ༻ɺ໰͍߹ΘͤɺFUD  ֤ࣄۀ෦ Πϯϑϥ෦ ٕज़ج൫ܥ෦ॺ ݚڀ ։ൃ෦
  18. 36.

     தԝ؅ཧͷݶք ‣ ن໛֦େʹ൐͍Πϯϑϥ෦ࣗ਎͕ϘτϧωοΫʹͳΔ͜ͱ͕૿͑ͨ  ৽Ϧιʔεͷ࡞੒ ઃܭͳͲͷ૬ஊ ໰͍߹ΘͤͳͲ ‣ զʑ͕ಘҙͱ͢ΔΑ͏ͳӡ༻ΛٻΊΒΕͳ͍΋ͷ͕ग़͖ͯͨ

    ݚڀ։ൃ  ‣ ʮͦ΋ͦ΋"84ͷར఺Λ׆͔͍ͤͯͳ͍ͷͰ͸ʯͱ͍͏ٙ໰  ηϧϑϚωʔδͰ͖Δ͜ͱ͕ྑ͞ͷҰͭͰ͋Δ͸ͣ ‣ ຊདྷ΍Δ΂͖͜ͱʹ࣌ؒΛ࢖͑ΔΑ͏ʹ͍ͨ͠  l"84ͷݕূ΍؅ཧzͰ͸ͳ͘lϢʔβʹαʔϏεΛಧ͚Δ͜ͱz͕࢓ࣄ 
  19. 41.
  20. 44.
  21. 49.

     "84૬ஊ ϊ΢ϋ΢ ‣ "84ʹؔ͢Δϊ΢ϋ΢͸جຊతʹΠϯϑϥ෦͕͍࣋ͬͯͨ  ૬ஊ΋΄΅Πϯϑϥ෦͕ड͚Δ ‣ αϙʔτϓϥϯ͸σϕϩούʔ ೥ͷҠߦҎདྷ

      έʔεͷԠ౴ʹ΋͕͔͔࣌ؒΔͨΊجຊతʹࣗ෼ͨͪͰௐࠪ  "84΁ͷ໰͍߹Θͤ୅ߦͷΑ͏ʹͳ͍ͬͯͨ͜ͱ΋͋ͬͨ ‣ ຊདྷࣗ෼͕ͨͪղ͔ͳͯ͘΋ྑ͍໰୊ʹ͕͔͔͍࣌ؒͬͯͨ 
  22. 52.

     ͦͷଞ ‣ 5".ΞαΠϯʹΑΓίϯςΩετڞ༗͕͠΍͘͢  έʔεͷऔΓ·ͱΊ΍ྉۚϨϙʔτͷ࡞੒΋͓ئ͍͍ͯ͠Δ ‣ -JNJU*ODSFBTFͷॲཧ͕ര଎ʹ  ίϯγΣϧδϡ͕ΞαΠϯ͞ΕͯରԠͯ͘͠ΕΔ

    ࠷୹ ෼  ‣ "84؅ཧӡ༻Ͱ஍ຯʹ͔͔͍ͬͯͨίετ͕͔ͳΓԼ͕ͬͨ  "84΁ͷґଘ౓͕ߴ͍૊৫ʹͱͬͯ͸ಛʹ༗༻ͩͱࢥ͏ 
  23. 58.

     (16Πϯελϯε H Q ͷར༻ ‣ ϓϩϏδϣχϯάπʔϧ *UBNBF౳ Ͱͷࣗಈ؀ڥߏஙΛݕ౼ 

    /7*%*"%SJWFS  $6%"  DV%//  ͳͲ͸Πϯετʔϧʹ ͕͔͔࣌ؒΔ ‣ 1BDLFS  Λར༻ͯ͠όʔδϣϯຖʹϓϩϏδϣχϯάࡁΈ".*Λ
 ࡞੒ʗ؅ཧ  Πϯελϯε্ཱ͕͕͙ͪͬͯ͢(16؀ڥ͕ར༻Ͱ͖Δ  VTFBTUͰ͸%FFQ-FBSOJOH".*΋ࢼ༻ʗݕ౼͍ͯ͠Δ  IUUQXXXOWJEJBDPN%PXOMPBEJOEFYBTQY MBOHFOVTIUUQXXXOWJEJBDPKQPCKFDUDVEBQBSBMMFMDPNQVUJOHQMBUGPSNKQIUNM IUUQTEFWFMPQFSOWJEJBDPNDVEOOIUUQTXXXQBDLFSJP
  24. 63.
  25. 65.

      ඇಉظͰૄ݁߹ͳ
 ࣸਅ൑ఆϑϩʔ 424 4 ྉཧ͖Ζ͘ͷެ։ &$4(16Ϋϥελʔ )BLPʹΑΔσϓϩΠͱ
 Φʔτεέʔϧ

    &$4 (16ܭࢉػ؀ڥ 1BDLFSΛར༻ͨ͠
 (16".*ͷ࡞੒ͱӡ༻ ".* Πϯελϯεͷ
 མͱ͠๨Ε؂ࢹ Πϯελϯε؅ཧ$IBU#PU -BNCEB $MPVE8BUDI
  26. 66.

    PR