Tweet
Tweet

Slide 1

Slide 1 text

©2021 Wantedly, Inc. %7$Λ׆༻ͨ͠ ػցֶशύΠϓϥΠϯ։ൃͷߴ଎Խ ୈ8ճ MLOps ษڧձ Tokyo (Online) Takayuki Kasai (@unblee) Wantedly, Inc. 2021.6.16

Slide 2

Slide 2 text

©2021 Wantedly, Inc. ࿩͢͜ͱ 8BOUFEMZ7JTJUͷ.BUDIJOHνʔϜͷ ػցֶशύΠϓϥΠϯ։ൃʹ͓͍ͯൃੜ͍ͯͨ͠໰୊ͱ ͦΕΛղܾ͢ΔͨΊʹ࠾ͬͨΞϓϩʔνʹ͍ͭͯ঺հ͠·͢

Slide 3

Slide 3 text

©2021 Wantedly, Inc. ։ൃ؀ڥͷ໰୊఺ طଘπʔϧͷػೳͱ໰୊఺ ΍ͬͨ͜ͱ ݁ՌͲ͏ͳ͔ͬͨ ݱঢ়ͷ՝୊఺ ࿩͢͜ͱ

Slide 4

Slide 4 text

©2021 Wantedly, Inc. ։ൃ؀ڥͷ໰୊఺

Slide 5

Slide 5 text

©2021 Wantedly, Inc. ػցֶशύΠϓϥΠϯ ։ൃ؀ڥͷ໰୊఺ w ύΠϓϥΠϯͷྲྀΕࣗମ͸γϯϓϧ w ։ൃ͸ओʹLT্ͷ1PEͰߦ͍ͬͯΔ • ຊ൪ͱ։ൃ؀ڥͷ࠶ݱੑͷ୲อͱϦιʔεʢओʹϝϞϦʣΛεέʔϧͤ͞ΔͨΊ w ୯ମͰ͸ϫʔΫϑϩʔΤϯδϯ͸࢖͍ͬͯͳ͍ • ϚΠΫϩαʔϏεؒΛ·͙ͨґଘؔ܎Λ͍࣋ͬͯΔ৔߹͸ Argo Workflows Λར༻͍ͯ͠Δ આ໌ͷ୯७ԽͷͨΊʹৄࡉ͸লུ #JH2VFSZ QSFQSPDFTTPS EBUB@MPBEFS USBJO #JH2VFSZ தؒग़ྗ QSFEJDU தؒग़ྗ தؒग़ྗ

Slide 6

Slide 6 text

©2021 Wantedly, Inc. ໰୊఺ ։ൃ؀ڥͷ໰୊఺ #JH2VFSZ QSFQSPDFTTPS EBUB@MPBEFS USBJO #JH2VFSZ தؒग़ྗ QSFEJDU தؒग़ྗ தؒग़ྗ w શମͷ࣮ߦ͕࣌ؒͱͯ΋௕͍ w ։ൃதʹ్தͷεςοϓΛมߋ͢Δͱ࠷ॳ͔Β΍Γ௚͢ඞཁ ͕͋Δ • Ճ͑ͯɺPod ্Ͱ௕࣮࣌ؒߦͯ͠์͓ͬͯ͘ͱʢk8s ͷʣGC Ͱσʔλ͕ফ͑Δέʔε͕ଘࡏͨ͠

Slide 7

Slide 7 text

©2021 Wantedly, Inc. Ξϓϩʔν ։ൃ؀ڥͷ໰୊఺ #JH2VFSZ QSFQSPDFTTPS EBUB@MPBEFS USBJO #JH2VFSZ தؒग़ྗ QSFEJDU தؒग़ྗ தؒग़ྗ w ֤εςοϓͷߴ଎Խࣦഊͨ͠ w தؒग़ྗͷΩϟογϡԽ

Slide 8

Slide 8 text

©2021 Wantedly, Inc. தؒੜ੒෺ΛΩϟογϡͱͯ͠׆༻͢Δ͜ͱʹΑͬͯ ։ൃதʹ͓͚ΔσʔλύΠϓϥΠϯͷ ్த͔Βͷ࠶࣮ߦʹ͔͔Δ࣌ؒͱ֤εςοϓͷ࣮ߦස౓Λ࡟ݮ͢Δ ΰʔϧ ։ൃ؀ڥͷ໰୊఺

Slide 9

Slide 9 text

©2021 Wantedly, Inc. ୡ੒͢ΔͨΊʹ΍Δ͜ͱɾཁ݅ ։ൃ؀ڥͷ໰୊఺ ඞཁͳதؒੜ੒෺͕ੜ੒ࡁΈͷεςοϓͷ࣮ߦΛεΩοϓ͢Δ • ظ଴ͨ͠ग़ྗ͕͢ͰʹߦΘΕ͍ͯͨΒͦͷεςοϓΛεΩοϓग़དྷΔ • தؒੜ੒෺Λར༻͢Δࡍ͸ࣗಈతʹద੾ͳ΋ͷΛબ୒ͯ͠ར༻ग़དྷΔΑ͏ʹ͢Δ • ຊ൪؀ڥ౳ΩϟογϡϛεΛආ͚͍ͨ؀ڥͰΩϟογϡΛແޮԽग़དྷΔ Ұ౓ੜ੒ͨ͠தؒੜ੒෺ΛʢPod ͷ࡟আ౳ʹΑͬͯʣฆࣦ͠ͳ͍Α͏ʹ͢Δ • ֤εςοϓऴྃ࣌ʹதؒੜ੒෺Λ GCS ͔Կ͔͠Βͷ֎෦ετϨʔδʹΞοϓϩʔυग़དྷΔ • ϩʔΧϧετϨʔδʹೖྗͱͯ͠ඞཁͳதؒੜ੒෺͕ଘࡏ͠ͳ͍ͱ͖ GCS ౳֎෦ετϨʔδʹଘࡏ ͢Δద੾ͳதؒੜ੒෺Λ୳ࡧͦ͠ΕΛೖྗͱͯ͠ར༻ग़དྷΔ

Slide 10

Slide 10 text

©2021 Wantedly, Inc. طଘπʔϧͷػೳͱ໰୊఺

Slide 11

Slide 11 text

©2021 Wantedly, Inc. طଘπʔϧ طଘπʔϧͷػೳͱ໰୊఺ ࣮ݧ؅ཧతͳଆ໘͸ҎԼͷࢿྉ͕ࢀߟʹͳΓ·͢ ୈ4ճ MLOps ษڧձ https://mlops.connpass.com/event/202359/ Data Version Control ʹΑΔ࣮ݧ؅ཧͷ࣮຿Ͱͷద༻ࣄྫ https://speakerdeck.com/sansandsoc/an-experiment-management-example-by-data-version-control DSOC R&Dݚڀһ ߴڮ ׮࣏ w %7$IUUQTEWDPSH w ػցֶशϓϩδΣΫτΛόʔδϣϯ؅ཧ͢ΔͨΊͷ$-* πʔϧ w ύΠϓϥΠϯͷ࣮ߦ؅ཧػೳؚ͕·ΕΔ

Slide 12

Slide 12 text

©2021 Wantedly, Inc. %7$ͷػೳ طଘπʔϧͷػೳͱ໰୊఺ w ύΠϓϥΠϯͷ࣮ߦίϚϯυ w AEWDSFQSPA w ࣮ߦ͢ΔͨΊʹ͸ҎԼͷϑΝΠϧ͕ඞཁ w EWDZBNM w EWDMPDLʢͪ͜Β͸ίϚϯυ࣮ߦޙʹࣗಈੜ੒͞ΕΔʣ

Slide 13

Slide 13 text

©2021 Wantedly, Inc. ࣮ߦͷྲྀΕ EWDZBNM EWDMPDL AEWDSFQSPA ಡΈࠐΈ ࣮ߦɾੜ੒ طଘπʔϧͷػೳͱ໰୊఺

Slide 14

Slide 14 text

©2021 Wantedly, Inc. %7$ͷػೳ طଘπʔϧͷػೳͱ໰୊఺ EWDZBNM ύΠϓϥΠϯͰ࣮ߦ͢ΔίϚϯυͱͦͷग़ྗϑΝΠϧɺ֤ εςʔδؒͷґଘؔ܎Λهड़͢Δ εςʔδؒͷґଘؔ܎Λεςʔδ໊Ͱ͸ͳ͘ɺϑΝΠϧ୯ ҐͰදݱ͍ͯ͠Δͷ͕ಛ௃త

Slide 15

Slide 15 text

©2021 Wantedly, Inc. %7$ͷػೳ طଘπʔϧͷػೳͱ໰୊఺ EWDMPDL ֤εςʔδͷग़ྗϑΝΠϧͱґଘ͍ͯ͠ΔϑΝΠ ϧͷϋογϡ஋ɾϑΝΠϧαΠζΛܭࢉͯ͠ه࿥ ͢Δʢdvc ίϚϯυʹΑͬͯࣗಈੜ੒͞ΕΔʣ ϋογϡ஋ͱϑΝΠϧαΠζΛݩʹΩϟογϡ͕ ୳ࡧ͞ΕΔʢϩʔΧϧ or ϦϞʔτʣ

Slide 16

Slide 16 text

©2021 Wantedly, Inc. ࣮ߦ͢Δ ࣮ߦ͢Δ ࣮ߦΛεΩοϓ͢Δ ࣮ߦͷྲྀΕ طଘπʔϧͷػೳͱ໰୊఺ ճ໨ 4UBHF" 4UBHF; 4UBHF# தؒग़ྗ" தؒग़ྗ# EWDMPDL ϑΝΠϧύεɾϋογϡ஋ɾϑΝΠϧαΠζ͕ه࿥͞ΕΔ /ճ໨ 4UBHF" 4UBHF; 4UBHF#` தؒग़ྗ" தؒग़ྗ#` EWDMPDL ه࿥͞Εͨϋογϡ஋ɾϑΝΠϧαΠζͷҰகͱ தؒग़ྗ"ϑΝΠϧͷଘࡏΛ֬ೝ ه࿥ͷதʹҰக͢Δ΋ͷ͕ͳ͍ ϑΝΠϧ͕ଘࡏ͠ͳ͍

Slide 17

Slide 17 text

©2021 Wantedly, Inc. தؒग़ྗͷύε؅ཧ͕େม طଘπʔϧͷػೳͱ໰୊఺ ໰୊ • ґଘؔ܎؅ཧͷͨΊʹϑΝΠϧύε͕େྔʹฒͿͨΊอकੑɾՄಡੑ͕͔ͳΓ௿͍ ཁҼ • DVC ʹ͓͚Δεςʔδؒͷґଘؔ܎ͷදݱํ๏͕ґଘઌͷλεΫ໊Ͱ͸ͳ͘ɺ֤εςʔδʹ͓͚ Δೖग़ྗϑΝΠϧ୯Ґʹͳ͍ͬͯΔ • εςʔδʹ͓͚Δґଘઌɾग़ྗϑΝΠϧΛ dvc.yaml ʹྻڍ͢Δඞཁ͕͋ΔͷͰɺґଘؔ܎Λ දݱ͔ͨͬͨ͠Βґଘ͞ΕΔεςʔδͷग़ྗϑΝΠϧϦετΛґଘ͢ΔεςʔδͷґଘϑΝΠ ϧϦετʹॏෳͯ͠ॻ͘ඞཁ͕͋Δ • શͯͷεςʔδͰڞ௨ͯ͠ґଘ͢ΔϑΝΠϧʢe.g. poetry.lockʣ͕͋Δ৔߹ɺશͯͷεςʔ δʹಉ͡΋ͷΛॻ͘ඞཁ͕͋Δ • dvc.yaml ͱίʔυதͰϑΝΠϧύεͷ੔߹ੑΛอͭඞཁੑ͕͋Δ • ࡉ੍͔͍ޚ͸΍Γ΍͍͢ͱ͍͏ϝϦοτ͸͋Δ͕զʑʹͱͬͯ͸ա৒

Slide 18

Slide 18 text

©2021 Wantedly, Inc. தؒग़ྗͷύε؅ཧ͕େม طଘπʔϧͷػೳͱ໰୊఺ զʑʹͱͬͯա৒ͳϙΠϯτ • ґଘؔ܎ͷදݱํ๏͕λεΫ୯ҐͰ͸ͳ͘ɺϑΝΠϧ୯Ґʹͳ͍ͬͯΔ • ґଘઌͱग़ྗઌͰॏෳͯ͠ϑΝΠϧύεΛॻ͘ඞཁ͕͋Δ վળͰ͖ͦ͏ͳϙΠϯτ • ґଘؔ܎Λεςʔδ୯Ґʹ͍ͨ͠ • ϑΝΠϧύεͷॏෳ؅ཧΛͨ͘͠ͳ͍ • ॻ͘ͷ͸1Օॴ͚ͩʹ͍ͨ͠ • ڞ௨ͯ͠ґଘ͢ΔϑΝΠϧ΋ॻ͘ͷ͸1Օॴ͚ͩʹ͍ͨ͠ • dvc.yaml ͷݟ௨͠ͱΞΫηεੑΛվળ͍ͨ͠ • dvc.yaml ͷՄಡੑΛ্͍͛ͨ • Python ίʔυ͔Β؆୯ʹΞΫηε͍ͨ͠

Slide 19

Slide 19 text

©2021 Wantedly, Inc. ΍ͬͨ͜ͱ

Slide 20

Slide 20 text

©2021 Wantedly, Inc. EWDͷ8SBQQFSπʔϧΛ࡞ͬͨ ίʔυδΣωϨʔλ • զʑͷ։ൃ؀ڥͰศརʹ࢖͏ͨΊͷઐ༻ Config ΛಡΈࠐΉ • dvc.yaml, stageouts.pyʢ͜ͷޙઆ໌ʣ Λੜ੒͢Δ ύΠϓϥΠϯͷ࣮ߦ • ύΠϓϥΠϯͷ࣮ߦࣗମ͸ dvc repro Λͦͷ··ར༻ • GCS ͷೝূ • Ωϟογϡͷ Pull/Push ͷࣗಈԽ

Slide 21

Slide 21 text

©2021 Wantedly, Inc. σΟϨΫτϦߏ੒

Slide 22

Slide 22 text

©2021 Wantedly, Inc. ࣮ߦͷྲྀΕ ઐ༻$PO fi H EWDZBNM TUBHFPVUTQZ XSBQQFS AEWDSFQSPA ಡΈࠐΈ ੜ੒ ಡΈࠐΈ ࣮ߦɾੜ੒ EWDMPDL

Slide 23

Slide 23 text

©2021 Wantedly, Inc. ίʔυδΣωϨʔλ ઐ༻$PO fi H dvc ͰύΠϓϥΠϯΛ࣮ߦ͢ΔͨΊʹඞཁͳ dvc.yaml ͱதؒग़ྗͷύε؅ཧΛߦ͏ stageouts.py ͷੜ੒Λߦ͏ͨΊʹඞཁ ʢdvc.yaml ͔Βͷมߋ಺༰͸͋͘·Ͱզʑͷ։ ൃ؀ڥʹ߹ΘͤΔͨΊͷ΋ͷʣ dvc.yaml ͷ໰୊఺Λվળ͢Δ • λεΫؒͷґଘؔ܎ͷએݴ • ڞ௨ͷґଘؔ܎ͷએݴ • ύε؅ཧͷάϧʔϐϯά

Slide 24

Slide 24 text

©2021 Wantedly, Inc. ίʔυδΣωϨʔλ TUBHFPVUTQZ தؒग़ྗઌͷ੔߹ੑΛ୲อ͠΍͘͢͢Δʢख ಈͰॏෳ؅ཧ͠ͳ͍ʣͨΊʹઐ༻ Config Ͱ ઃఆ֤ͨ͠εςʔδͷதؒग़ྗઌΛ Python ίʔυͱͯࣗ͠ಈੜ੒ͨ͠΋ͷ ֤εςʔδ͝ͱʹग़ྗઌϑΝΠϧύε͕࡞੒ ͞Ε͍ͯΔͷͰɺimport ͯ͠࢖͏

Slide 25

Slide 25 text

©2021 Wantedly, Inc. ݁ՌͲ͏ͳ͔ͬͨ

Slide 26

Slide 26 text

©2021 Wantedly, Inc. ݁ՌͲ͏ͳ͔ͬͨ ౰ॳͷΰʔϧࣗମ͸ୡ੒ 🎉 • ʮதؒੜ੒෺ΛΩϟογϡͱͯ͠׆༻͢Δ͜ͱʹΑͬͯɺ։ൃதʹ͓ ͚ΔσʔλύΠϓϥΠϯͷ్த͔Βͷ࠶࣮ߦʹ͔͔Δ࣌ؒͱ֤εςο ϓͷ࣮ߦස౓Λ࡟ݮ͢Δʯ • શͯஔ͖׵͑Δͱ͜Ζ·Ͱ͸͍͍ͬͯͳ͍ ෭࣍ޮՌ • ֤εςʔδʹ͓͚Δೖग़ྗ͕੔ཧ͞Εͨ

Slide 27

Slide 27 text

©2021 Wantedly, Inc. ݱঢ়ͷ՝୊఺

Slide 28

Slide 28 text

©2021 Wantedly, Inc. ݱঢ়ͷ՝୊఺ • ΩϟογϡΛͪΌΜͱ༗ޮ׆༻͢ΔͨΊʹ͸͔ͳΓࡉ͔͘εςʔδΛ੾ Δඞཁੑ͕͋Δ • εςʔδ͕੒ޭͨ͠ͱ͖ʹ͔͠Ωϟογϡ͕༗ޮʹͳΒͳ͍ʢdvc.lock ʹه࿥͞Εͳ͍ʣͷͰεςʔδ ಺Ͱࣦഊͨ͠৔߹͸ͦͷεςʔδΛؙ͝ͱ΍Γ௚͢ඞཁ͕͋Δ • ಈతʹ࡞੒͞ΕΔϑΝΠϧʹରԠͰ͖ͳ͍ • εςʔδ׬ྃ࣌ʹ dvc.yaml ʹॻ͔ΕͨϑΝΠϧύε͕ଘࡏ͠ͳ͍ͱΤϥʔʹͳΔ • ྫ͑͹։ൃ؀ڥ͔ຊ൪؀ڥ͔ʹΑͬͯੜ੒͞ΕΔ͔Ͳ͏͔ܾఆ͞ΕΔϑΝΠϧ͕ґଘؔ܎ʹؚ·Ε͍ͯΔ ৔߹ɺ։ൃ؀ڥͰ͸੒ޭ͢Δ͕ຊ൪؀ڥͰ͸ࣦഊ͢Δͱ͍ͬͨΑ͏ͳ͜ͱ͕ى͜Γ͑Δ • ಋೖίετ͕ߴ͍ • ίʔυΛ python ίϚϯυͰ࣮ߦͰ͖ΔϑΝΠϧ୯Ґʹ෼ׂ͢Δ࡞ۀ͕ඞཁʹͳΔ

Slide 29

Slide 29 text

©2021 Wantedly, Inc. ·ͱΊ WHY ύΠϓϥΠϯͷ࣮ߦ͕࣌ؒ௕ͯ͘։ൃੜ࢈ੑ͕མ͍ͪͯͨ WHAT தؒग़ྗΛΩϟογϡͱ্ͯ͠ख͘ѻͬͯɺ։ൃதͷ࣮ߦ࣌ؒͱස౓Λ࡟ ݮͨ͠ HOW DVC ͷ׆༻ͱ Wrapper πʔϧͷ࡞੒