推薦のためのdata pipelineのモニタリングとロギング

Transcript

1. ©2019 Wantedly, Inc. ਪનͷͨΊͷdata pipelineͷ ϞχλϦϯάͱϩΪϯά Apr 16, 2019 -

   Hazumi Ichijo Data Pipeline Casual Talk #2

2. ©2018 Wantedly, Inc. ࣗݾ঺հ Ұᑍ
   ୺੅
   !IB[VNJSS!SFSPTU
   %BUB
   &OHJOFFS
   JO
   8BOUFEMZ
   (P
   3BJMT
   3FBDUʜ
   ๺ւಓग़਎

3. ©2019 Wantedly, Inc.  ͳͥϞχλϦϯάɾϩΪϯά͢Δ͔
    ԿΛϞχλϦϯάɾϩΪϯά͢΂͖͔
    ͲΕ͘Β͍कΔ΂͖͔
   

   ԿΛ͢΂͖͔

4. ©2019 Wantedly, Inc. +PC
    +PC
    +PC
   
   +PC
    +PC
    σʔλ
   
   σʔλ
   

   σʔλ
    αʔϏε

5. ©2019 Wantedly, Inc. ໰୊͕ൃੜͨ͠ͱ͖Ͳ͏͠·͢ʁ

6. ©2019 Wantedly, Inc. +PC
    +PC
    +PC
   
   +PC
    +PC
    +PC
    +PC
   

   +PC
    σʔλ
   
   σʔλ
    σʔλ
   

7. ©2019 Wantedly, Inc. +PC
    +PC
    +PC
   
   +PC
    +PC
    +PC
    +PC
   

   +PC
    σʔλ
   
   σʔλ
    σʔλ
   

8. ©2019 Wantedly, Inc. +PC
    +PC
    +PC
   
   +PC
    +PC
    +PC
    +PC
   

   +PC
    σʔλ
   
   σʔλ
    σʔλ
    ʁ ʁ ʁ

9. ©2019 Wantedly, Inc.  ͦ΋ͦ΋ͳ͍
    ζϨΔʢྫ
   ݹ͍σʔλΛݩʹͨ͠ਪન͕ग़Δʣ ग़ͯ͘Δ঱ঢ় ਪનͷͨΊͷσʔλ͕

10. ©2019 Wantedly, Inc.  ͦ΋ͦ΋ͳ͍
     ζϨΔʢྫ
    ݹ͍σʔλΛݩʹͨ͠ਪન͕ग़Δʣ ग़ͯ͘Δ঱ঢ় ਪનͷͨΊͷσʔλ͕ 
    ؾ͖ͮʹ͍͘

11. ©2019 Wantedly, Inc.  ໰୊͕ൃੜͨ͠ࡍʹؾ͖ͮʹ͍͘
     ໰୊͕ൃੜͨ࣌͠ͷௐࠪํ๏͕໘౗ ͳͥϞχλϦϯάɾϩΪϯάΛ͢Δ͔ ਪનͷͨΊͷσʔλ͸

12. ©2019 Wantedly, Inc.  ໰୊͕ൃੜͨ͠ࡍʹؾ͖ͮʹ͍͘
     ໰୊͕ൃੜͨ࣌͠ͷௐࠪํ๏͕໘౗ ͳͥϞχλϦϯάɾϩΪϯάΛ͢Δ͔ ਪનͷͨΊͷσʔλ͸ 
    ϞχλϦϯάɾϩΪϯάΛͯ͠໰୊ൃੜ࣌ʹඋ͑Δ

13. ©2019 Wantedly, Inc. ԿΛϞχλϦϯάɾϩΪϯά͢Δʁ

14. ©2019 Wantedly, Inc.  ຖճܾΊΒΕͨ৔ॴʹσʔλ͕ग़ྗ͞ΕΔ΂͖
     ग़ྗ͞ΕΔσʔλ͕ਖ਼͍͠ඞཁ͕͋Δ
    w ґଘ͢Δσʔλ͕࠷৽ͷঢ়ଶ
    w

    ܾΊΒΕͨ࣌ࠁʹ࣮ߦ͞Εͯ΄͍͠
    w FUD
    ԿΛϞχλϦϯάɾϩΪϯά͢΂͖͔ ਪનͷͨΊͷσʔλ͸

15. ©2019 Wantedly, Inc.  ຖճܾΊΒΕͨ৔ॴʹσʔλ͕ग़ྗ͞ΕΔ΂͖
     ग़ྗ͞ΕΔσʔλ͕ਖ਼͍͠ඞཁ͕͋Δ
    w ґଘ͢Δσʔλ͕࠷৽ͷঢ়ଶ
    w

    ܾΊΒΕͨ࣌ࠁʹ࣮ߦ͞Εͯ΄͍͠
    w FUD
    ԿΛϞχλϦϯάɾϩΪϯά͢΂͖͔ ਪનͷͨΊͷσʔλ͸ 
    ϞχλϦϯάɾϩΪϯάΛͯ͠कΕΔΑ͏ʹ͢Δ

16. ©2019 Wantedly, Inc.  ΞϓϦέʔγϣϯ
    w δϣϒ࣮ߦ࣌ͷ࣮ߦঢ়گͷ஋ʢྫ
    ֶश཰ʣ
    w ΦϯϥΠϯධՁͷ஋
    w

    ॻ͖ग़͠ઌʹσʔλ͕ॻ͖ࠐ·Ε͔ͨ
    w Τϥʔ௨஌
    w Կʹ͕͔͔࣌ؒΔ͔
     Πϯϑϥ
    w δϣϒ͕੒ޭ͔ͨ͠ɾࣦഊ͔ͨ͠ɾ࣮ߦ͞Ε͔ͨ
    w ઃఆ͍ͯ͠Δ࣌ؒʹ࣮ߦ͞Ε͔ͨ ԿΛϞχλϦϯάɾϩΪϯά͢΂͖͔ ϞχλϦϯά

17. ©2019 Wantedly, Inc. w ॲཧͷ಺༰͕Θ͔Δ΋ͷʢྫ
    42-ʣ
    w δϣϒͷ։࢝ɾऴྃ
    w ॻ͖ग़ͨ͠σʔλ
    w

    FUD ԿΛϞχλϦϯάɾϩΪϯά͢΂͖͔ ϩΪϯά

18. ©2019 Wantedly, Inc. ͲΕ͘Β͍कΒΕΔ΂͖΋ͷʁ

19. ©2019 Wantedly, Inc.  αʔϏεશମ͕࢖͑ͳ͘ͳΔ
     ,1*ʹ99Өڹ͕ͰΔ
     Ұ෦ͷػೳ͕࢖͑ͳ͘ͳΔ ͲΕ͘Β͍कΔ΂͖͔

    ͋Δਪનʹ໰୊͕͋Δͱɾɾɾ

20. ©2019 Wantedly, Inc.  αʔϏεશମ͕࢖͑ͳ͘ͳΔ
     ,1*ʹ99Өڹ͕ͰΔ
     Ұ෦ͷػೳ͕࢖͑ͳ͘ͳΔ ͲΕ͘Β͍कΔ΂͖͔

    ͋Δਪનʹ໰୊͕͋Δͱɾɾɾ 
    ͦ΋ͦ΋ͦͷσʔλͷॏཁ౓͕Θ͔Δඞཁ͕͋Δ

21. ©2019 Wantedly, Inc. ԿΛ͢΂͖͔

22. ©2019 Wantedly, Inc.  ॏཁ౓ΛͰ͖Δ͚ͩ໌֬ʹ͓ͯ͘͠
     ϞχλϦϯάɾϩΪϯάͷ߲໨ચ͍ग़͠
     ΠϯϑϥνʔϜͱڠྗͯ͠ɺ༧Ί
    w

    ॏཁ౓Λ఻͑Δ
    w ϞχλϦϯάɾϩΪϯάͷλεΫͷׂΓৼΓΛߦ͏
    w ໰୊͕ൃੜͨ࣌͠ɺͲ͏͍ͬͨϑϩʔͰ֬ೝ͢Δ͔ɺͲ͏ରॲ ͢Δ͔Λ఻͑Δ
    
    ඞཁͰ͋Ε͹࣮ࡍʹϦϋʔαϧΛߦ͏ ԿΛ͢΂͖͔