Upgrade to Pro — share decks privately, control downloads, hide ads and more …

WantedlyでFeature Storeを導入する際に考えたこと

WantedlyでFeature Storeを導入する際に考えたこと

第36回 MLOps 勉強会
https://mlops.connpass.com/event/300857/

Higuchi kokoro

November 29, 2023
Tweet

More Decks by Higuchi kokoro

Other Decks in Programming

Transcript

  1. ©2023 Wantedly, Inc. 'FBUVSF4UPSFͱ͸ͳʹ͔ 8BOUFEMZʹಋೖͯ͠Ͳ͏͔ͩͬͨ ຊ೔ͷ͓࿩͢͠Δ͜ͱ Part1 w 'FBUVSF4UPSFͱ͸ͳʹ͔ w

    'FBUVSF4UPSFͷඪ४తͳ࣮૷ߏ੒ w 'FBUVSF4UPSFϓϩμΫτͷ঺հ 8BOUFEMZͰ'FBUVSF4UPSFΛಋೖͨ͠എܠ Part2 w 8BOUFEMZͷਪનγεςϜͷཁ݅ w ͜Ε·Ͱͷ՝୊ w 'FBUVSF4UPSFͷࣗ࡞ʹࢸͬͨഎܠ ࣗ࡞ͨ͠'FBUVSF4UPSFͷߏ੒ Part3 w શମߏ੒ w ֤ػೳͷ࣮૷ߏ੒ Part4 w ࡞ͬͯྑ͔ͬͨ͜ͱ w ࣗ࡞ͯ͠ྑ͔ͬͨ͜ͱ w ΋ͬͱΑ͍ͨ͘͠ͱ͜Ζ ࡞ͬͨॴײ 'FBUVSF4UPSF֓ཁ
  2. ©2023 Wantedly, Inc. Ṥޱ৺ )JHVDIJ,PLPSP  ॴଐ8BOUFEMZ JOD 9![FSFCPN@ σʔλαΠΤϯςΟετͱͯ͠ਪનγεςϜͷਫ਼౓վળɾධՁɾ

    .-0QT·ΘΓͷ։ൃΛ͍ͯ͠·͢ ਖ਼͘͠πʔϧ΍ϓϩάϥϛϯάݴޠΛ࢖͍ɺ 
 ϩόετʹϓϩμΫτ։ൃ͢Δ͜ͱʹؔ৺͕͋Γ·͢ ࣗݾ঺հ
  3. ©2023 Wantedly, Inc. Create A World Where Work Drives Passion.

    ڀۃͷదࡐదॴʹΑΓ 
 γΰτͰίίϩΦυϧͻͱΛ;΍͢ ձࣾ঺հ ࢲͨͪͷϛογϣϯ ©2023 Wantedly, Inc.
  4. ©2023 Wantedly, Inc. Կ͕Ͱ͖Δ  Ұ؏ͨ͠ํ๏ͰσʔλΛॲཧ͠ɺ ෳ਺ͷϞσϧؒͰڞ༗Մೳͳ 
 ಛ௃ྔΛੜ੒ɾ؅ཧͰ͖Δ 'FBUVSF4UPSFͱ͸

    ͲΜͳγεςϜ  ಛ௃ྔσʔλࣗମ΍ɺ 
 ͦͷϝλσʔλΛޮ཰తʹอଘɺ ΞΫηεɺ࠶ར༻͢ΔͨΊʹઃܭ ͞ΕͨγεςϜ ࢖͏ͱͲ͏ͳΔ  ػցֶशϞσϧ։ൃͷεϐʔυ ΞοϓɺσʔλͷҰ؏ੑͷ֬อɺ ͓ΑͼϞσϧੑೳͷվળʹد༩ ػցֶशͷͨΊͷಛ௃ྔσʔλͷதԝ؅ཧɾڞ༗γεςϜ
  5. ©2023 Wantedly, Inc. 'FBUVSF4UPSFͷඪ४తͳ࣮૷ߏ੒ ओཁͳͭͷίϯϙʔωϯτ 'FBUVSF3FHJTUSZ ಛ௃ྔ΍ϝλ৘ใͳͲΛҰཡͰ͖Δ ར༻ऀ޲͚ͷΠϯλʔϑΣʔεͷఏڙ 0QFSBUJPOBM.POJUPSJOH $PODFQU%SJGUͳͲϞσϧͷܦ࣌ྼԽ

    Λݕ஌͢Δಛ௃ྔͷϞχλϦϯάػߏ ref: https://neptune.ai/blog/feature-stores-components-of-a-data-science-factory-guide 5SBOTGPSN ੜσʔλΛಛ௃ྔʹม׵͢Δػߏ ཁ݅ʹΑΓม׵ํ๏͕ҟͳΔ 4USFBNPS#BUDIͳͲ 4UPSBHF ಛ௃ྔΛอ؅͢Δػߏ#BUDIͳΒ %8) ϦΞϧλΠϜͳΒ3FEJTͳͲ 4FSWJOH ಛ௃ྔΛఏڙ͢ΔػߏϦʔΫ͕ 
 ى͖ͳ͍Α͏ʹ࣌ܥྻΛ 
 ࢀর͢Δ΋ͷ΋͋Δ QPJOUJOUJNF
  6. ©2023 Wantedly, Inc.  044 w ೥ʹ(PKFLͱ(PPHMF$MPVE͕ϦϦʔε  .-ͱ%BUB*OGSBͷ෼཭ w

    σʔλΠϯϑϥΛந৅Խ೚ҙͷΠϯϑϥ 
 ($1 "84 4OPX fl BLF౳ Ͱར༻Մೳ  σʔλϦʔΫΛճආՄೳ w QPJOUJOUJNFKPJO ޙड़ Λαϙʔτ  ֶशͱαʔϏϯάͷҰ؏ੑ୲อ w ֶशɾਪ࿦࣌ʹಉ͡ΠϯλʔϑΣʔεͰར༻Մೳ 'FBUVSF4UPSFϓϩμΫτ঺հd'FBTUd 3FGIUUQTEPDTGFBTUEFWWNBTUFSXIBUJTGFBTU
  7. ©2023 Wantedly, Inc.  ϑϧϚωʔδυαʔϏε w #2ͷσʔλΞΫηεɾΨόφϯεΛ׆༻Ͱ͖Δ  ௿ϨΠςϯγͷαʔϏϯά w

    ಛ௃ྔͷϦΫΤετͷ͕NTͰ׬ྃ  .-ɾੜ੒"*ͷ྆ํʹରԠ w ಛ௃ྔͱಉ͡Α͏ʹϕΫτϧຒΊࠐΈ΋֨ೲՄೳ  σʔλϦʔΫΛճආՄೳ w QPJOUJOUJNFKPJO ޙड़ Λαϙʔτ 'FBUVSF4UPSFϓϩμΫτ঺հd7FSUFY"*d 3FGIUUQTDMPVEHPPHMFDPNWFSUFYBJEPDTGFBUVSFTUPSF IMKBWBJGT IUUQTDMPVEHPPHMFDPNCMPHKBQSPEVDUTBJNBDIJOFMFBSOJOHOFXWFSUFYBJGFBUVSFTUPSFCJHRVFSZQPXFSFEHFOBJSFBEZ
  8. ©2023 Wantedly, Inc. ิ଍ಛ௃ྔͷάϧʔϐϯά w ݕࡧੑ΍Ϧιʔε؅ཧͷͨΊɺಛ௃ྔ͸͍͔ͭ͘ͷ؍఺ Ͱάϧʔϐϯά͠͞ΕΔ͜ͱ͕ଟ͍ 7FSUFY"*ʹ͓͚Δάϧʔϐϯάྫ w &OUJUZ'FBUVSFΛඥ෇͚Δ୯Ґ

    w FY Ϣʔβʔ ΞΠςϜ w 'FBUVSF(SPVQσʔλιʔεͷϩέʔγϣϯʹ 
 جͮ͘άϧʔϓ w 'FBUVSF7JFXҙຯͷ୯Ґʹجͮ͘άϧʔϓ 3FGIUUQTDMPVEHPPHMFDPNWFSUFYBJEPDTGFBUVSFTUPSFMBUFTUPWFSWJFX
  9. ©2023 Wantedly, Inc.  ఏڙ͢ΔϢʔβʔମݧ w ϢʔβʹืूΛɺϦΫϧʔλʔʹީิऀΛਪન͢Δ w ީิͱͳΔΞΠςϜʹείΞ͚ͮ͠ɺϥϯΩϯάͱͯ͠ఏڙ͢Δ 

    νʔϜߏ੒ w .-1E.ਓ %4ਓ ਪનج൫ΤϯδχΞਓ  ݱঢ়ͷγεςϜߏ੒ w 1ZUIPOͷKPCΛఆظ࣮ߦ͠ɺਪ࿦݁ՌΛ#JH2VFSZʹ֨ೲ ओʹ%4  w #2ͷ݁ՌΛ3FEJTܦ༝ͰɺϢʔβʹ഑৴ ओʹਪનج൫ΤϯδχΞ   ࣮૷࣌ͷཁٻ w 'FBUVSF4UPSF௥Ճਫ਼౓վળΛؚΊϲ݄ ਓͰϦϦʔε w ಛ௃ྔ࡞੒Λߦ͏%4ࣗ਎͕ϝϯςͰ͖ΔγεςϜ͕๬·͍͠ 'FBUVSF4UPSFͷಋೖର৅ͱͳͬͨਪનγεςϜͷཁ݅ɾલఏ
  10. ©2023 Wantedly, Inc. ՝୊ͦͷಛ௃ྔʹ༧ଌ௚લͷσʔλΛར༻Ͱ͖ͳ͍ w ैདྷͷ༧ଌλεΫ͸5SBJO7BMJE5FTUΛ࣌ܥྻ෼ׂ͠ɺ ༧ଌσʔλऩूظؒΛઃ͚͍ͯͨ w ༧ଌσʔλऩूظ͕ؒ͋ΔͨΊɺ௚લͷߦಈ·ͰΛ 


    ೖྗʹؚΊΔ͜ͱ͕ग़དྷͳ͍ w ਫ਼౓վળͷͨΊʹɺϢʔβͷΠϕϯτൃੜλΠϛϯάͷ ௚લ·ͰΛೖྗσʔλͷऩूظؒͱ͍ͨ͠ w ಛ௃ྔΛఆظతʹอଘ͠ɺϢʔβຖʹར༻͢Δ 
 ೖྗظؒΛม͑Δඞཁ͕ग़͖ͯͨ
  11. ©2023 Wantedly, Inc. 'FBUVSF4UPSFͷػೳͱ8BOUFEMZͷ՝୊ͱඞཁੑ ػೳ ݱঢ়ͷ՝୊ ඞཁੑ Feature Registry Ұݩ؅ཧ͍ͯ͠ͳ͍ͨΊɺ

    
 Ͳ͜ͰͲͷಛ௃ྔ͕࢖ΘΕ͍ͯΔͷ͔୳ͮ͠Β͍&࠶ར༻͠ʹ͍͘ ߴ Operational Monitoring ܽଛͳͲͷσάϨʹؾ͚ͮͳ͍ ͨͩ͠ΦϑϥΠϯςετͰ͋Δఔ౓ΧόʔͰ͖Δ ௿ Storage ಛ௃ྔ͕Ϋϥ΢υετϨʔδʹ͔͠࢒͓ͬͯΒͣɺ 
 σόοά΍࠶ར༻ɺϢʔβຖʹऔಘ࣌ظΛม͑Δ͜ͱ͕೉͍͠ ߴ Transform PythonϨΠϠʔͰBatchॲཧ͍ͯ͠ΔͷͰ஗͍ த Serving Servingػߏ͸طʹ͋ΔͷͰඞཁͳ͍ ແ Ұൠతͳ'FBUVSF4UPSFͷػೳ͕શͯඞཁͰ͋ΔΘ͚Ͱ͸ແ͍ͱΘ͔ͬͨ💡
  12. ©2023 Wantedly, Inc. ಺੡WT֎෦πʔϧ ಺੡ VertexAI Feast or ͦͷଞOSS Pros

    ΧελϚΠζ͠΍͍͢ 
 ඞཁͳ෦෼͚࣮ͩ૷Ͱ͖Δ εέʔϥϒϧ 
 ϦΞϧλΠϜͳಛ௃ྔͷαʔϏϯά ແྉɻίʔυΛશͯಡΊΔ Cons ࣮૷ίετ͕͔͔ΔɻυΩϡϝϯτ౳͸શͯ 
 ࣗ෼Ͱ༻ҙ͢Δඞཁ͕͋Δ 2023/10ϦϦʔεͰ 
 ৘ใ͕গͳ͍ 
 ྉ͕͔͔ۚΔ 
 Transformػߏ͸݁ہ࣮૷͕ඞཁɻ ґଘϥΠϒϥϦͷߋ৽͕ 
 ΍΍஗͍ 
 ࠃ಺ͷར༻ࣄྫ͕͋·Γͳ͍
  13. ©2023 Wantedly, Inc. ࣗ࡞͕ಋೖ΋อक΋ 
 ࠷΋ίετ͕খͦ͞͏ ࣗ࡞͢Δ͜ͱʹͨࠜ͠ڌ ࣗ࡞ˠطଘ੡඼΁ͷ Ҡߦ͸͠΍ͦ͢͏ w

    طଘ੡඼͸େن໛αʔϏεʹ΋దԠͰ͖ΔΑ͏ʹػೳ͕ଟ͍ w FY ಛ௃ྔͷάϧʔϐϯά΍ϦΞϧλΠϜαʔϏϯά w ϝϯςφϯε͞Ε͍ͯΔϥΠϒϥϦ͚ͩΛબΜͰಋೖͰ͖Δ w $IBU(15ʹΑΓɺ࣮૷ίετ͕Լ͕͍ͬͯΔ w 7FSUFY"*͸ಛ௃ྔ͕#2ʹΞοϓϩʔυ͞Ε͍ͯΔඞཁ͋Γ w ͲͷΈͪɺˢΛߦ͏ػߏ͸ඞཁ w ΑΓଞࣾࣄྫ͕ग़͖͔ͯͯΒஈ֊తʹҠߦ΋Մೳ
  14. ©2023 Wantedly, Inc.  ࣗ࡞ͯ͠ྑ͔ͬͨ͜ͱ w ઃܭΛܦͯɺࣗࣾͰٻΊ͍ͯΔ΋ͷͷཁ͕݅໌֬ʹͳͬͨ w 1ZUIPOͷΈͰ࣮૷͞Ε͍ͯΔͷͰɺ%4ࣗ਎͕ཉ͍͠ػೳΛؾܰʹ13Ͱ௥ՃͰ͖ΔΑ͏ʹͳͬͨ w

    ಛ௃ྔੜ੒ʹؔ͢Δ։ൃੜ࢈ੑͷ޲্ͷऔΓ૊Έ͕ҰΧॴʹूத͠ɺεέʔϧ͢ΔΑ͏ʹͳͬͨ w େن໛ͳνʔϜߏ੒Λ૝ఆ͍ͯ͠Δπʔϧ͕ଟ͍ͳ͔ɺඞཁे෼ͳػೳΛબ୒ͯ͠ಋೖ͢Δ͜ͱ͕ग़དྷͨ w $IBU(15ͷ୆಄ʹΑΓɺίʔυΛॻ͘ίετ͕֨ஈʹԼ͕͍ͬͯΔͨΊɺૉૣ࣮͘૷Ͱ͖ͨ w 1BOEBT⁶#JH2VFSZ⁶ZBNMؒͷܕΛࣄલʹόϦσʔγϣϯͰ͖ΔͨΊɺ&SSPS΍ܽଛ͕ݮͬͨ  ࣗ࡞ͯ͠೉͍͠ͱײͨ͜͡ͱ w ಛ௃ྔΛϦΞϧλΠϜͰαʔϏϯά͢Δػߏͷࣗ࡞͸͔ͳΓ೉ͦ͠͏͓ͦΒ͘7FSUFY"*ʹ৐ͬͨํ͕Α͍ w ੜ੒ͨ͠ಛ௃ྔಉ࢜ΛKPJO͢ΔΫΤϦΛಈతʹ࡞Δػߏ͸ෳࡶԽ͠΍͍͢ɻKJOKBUFNQMBUFͷ׆༻ͳͲ͕ඞཁͦ͏ ࣗ࡞ͨ͠ॴײ
  15. ©2023 Wantedly, Inc. ·ͱΊ 'FBUVSF4UPSFʹؚ·ΕΔػೳ΍ؔ࿈ϓϩμΫτʹ͍ͭͯ঺հͨ͠ w ಛ௃ྔσʔλࣗମ΍ɺϝλσʔλΛޮ཰తʹอଘɺΞΫηεɺ࠶ར༻͢ΔͨΊʹઃܭ͞ΕͨγεςϜ w Ұ؏ͨ͠ํ๏ͰσʔλΛॲཧ͠ɺෳ਺ͷϞσϧؒͰڞ༗Մೳͳಛ௃ྔΛੜ੒Ͱ͖Δ 8BOUFEMZͰ͸ൺֱݕ౼ͨ݁͠Ռ'FBUVSF4UPSFΛ಺੡Ͱ։ൃͨ͠

    w ؔ࿈ϓϩμΫτͷશͯͷػೳ͕ඞཁͰແ͍͜ͱ͕Θ͔ͬͨͨΊ w ࣗ࡞͢Δ͜ͱͰඞཁे෼ͳػೳ͚ͩΛ࣮૷Ͱ͖ͨ 'FBUVSF4UPSFΛಋೖ͢Δ͜ͱͰߴ଎ͳ։ൃͱਫ਼౓ͷվળͰ͖ͨ w σόοά΍ϞχλϦϯάɺϨϏϡʔͷ͠΍͢͞ͳͲɺӡ༻্ͷϖΠϯ͕େ෯ʹվળ͞Εͨ w Ϣʔβຖʹ࠷৽ͷσʔλΛඥ෇͚Δ͜ͱ͕ग़དྷΔͨΊɺਫ਼౓͕վળ͞Εͨ