推薦のためのdata pipelineのモニタリングとロギング

推薦のためのdata pipelineのモニタリングとロギング

9ec2ceee3f782e64fd67fa7d82b6c1b4?s=128

Hazumi Ichijo

April 16, 2019
Tweet

Transcript

  1. ©2019 Wantedly, Inc. ਪનͷͨΊͷdata pipelineͷ ϞχλϦϯάͱϩΪϯά Apr 16, 2019 -

    Hazumi Ichijo Data Pipeline Casual Talk #2
  2. ©2018 Wantedly, Inc. ࣗݾ঺հ Ұᑍ୺੅!IB[VNJSS!SFSPTU %BUB&OHJOFFSJO8BOUFEMZ (P 3BJMT 3FBDUʜ ๺ւಓग़਎

  3. ©2019 Wantedly, Inc.  ͳͥϞχλϦϯάɾϩΪϯά͢Δ͔  ԿΛϞχλϦϯάɾϩΪϯά͢΂͖͔  ͲΕ͘Β͍कΔ΂͖͔ 

    ԿΛ͢΂͖͔
  4. ©2019 Wantedly, Inc. +PC +PC +PC +PC +PC σʔλ σʔλ

    σʔλ αʔϏε
  5. ©2019 Wantedly, Inc. ໰୊͕ൃੜͨ͠ͱ͖Ͳ͏͠·͢ʁ

  6. ©2019 Wantedly, Inc. +PC +PC +PC +PC +PC +PC +PC

    +PC σʔλ σʔλ σʔλ
  7. ©2019 Wantedly, Inc. +PC +PC +PC +PC +PC +PC +PC

    +PC σʔλ σʔλ σʔλ
  8. ©2019 Wantedly, Inc. +PC +PC +PC +PC +PC +PC +PC

    +PC σʔλ σʔλ σʔλ ʁ ʁ ʁ
  9. ©2019 Wantedly, Inc.  ͦ΋ͦ΋ͳ͍  ζϨΔʢྫݹ͍σʔλΛݩʹͨ͠ਪન͕ग़Δʣ ग़ͯ͘Δ঱ঢ় ਪનͷͨΊͷσʔλ͕

  10. ©2019 Wantedly, Inc.  ͦ΋ͦ΋ͳ͍  ζϨΔʢྫݹ͍σʔλΛݩʹͨ͠ਪન͕ग़Δʣ ग़ͯ͘Δ঱ঢ় ਪનͷͨΊͷσʔλ͕ ؾ͖ͮʹ͍͘

  11. ©2019 Wantedly, Inc.  ໰୊͕ൃੜͨ͠ࡍʹؾ͖ͮʹ͍͘  ໰୊͕ൃੜͨ࣌͠ͷௐࠪํ๏͕໘౗ ͳͥϞχλϦϯάɾϩΪϯάΛ͢Δ͔ ਪનͷͨΊͷσʔλ͸

  12. ©2019 Wantedly, Inc.  ໰୊͕ൃੜͨ͠ࡍʹؾ͖ͮʹ͍͘  ໰୊͕ൃੜͨ࣌͠ͷௐࠪํ๏͕໘౗ ͳͥϞχλϦϯάɾϩΪϯάΛ͢Δ͔ ਪનͷͨΊͷσʔλ͸ ϞχλϦϯάɾϩΪϯάΛͯ͠໰୊ൃੜ࣌ʹඋ͑Δ

  13. ©2019 Wantedly, Inc. ԿΛϞχλϦϯάɾϩΪϯά͢Δʁ

  14. ©2019 Wantedly, Inc.  ຖճܾΊΒΕͨ৔ॴʹσʔλ͕ग़ྗ͞ΕΔ΂͖  ग़ྗ͞ΕΔσʔλ͕ਖ਼͍͠ඞཁ͕͋Δ w ґଘ͢Δσʔλ͕࠷৽ͷঢ়ଶ w

    ܾΊΒΕͨ࣌ࠁʹ࣮ߦ͞Εͯ΄͍͠ w FUD ԿΛϞχλϦϯάɾϩΪϯά͢΂͖͔ ਪનͷͨΊͷσʔλ͸
  15. ©2019 Wantedly, Inc.  ຖճܾΊΒΕͨ৔ॴʹσʔλ͕ग़ྗ͞ΕΔ΂͖  ग़ྗ͞ΕΔσʔλ͕ਖ਼͍͠ඞཁ͕͋Δ w ґଘ͢Δσʔλ͕࠷৽ͷঢ়ଶ w

    ܾΊΒΕͨ࣌ࠁʹ࣮ߦ͞Εͯ΄͍͠ w FUD ԿΛϞχλϦϯάɾϩΪϯά͢΂͖͔ ਪનͷͨΊͷσʔλ͸ ϞχλϦϯάɾϩΪϯάΛͯ͠कΕΔΑ͏ʹ͢Δ
  16. ©2019 Wantedly, Inc.  ΞϓϦέʔγϣϯ w δϣϒ࣮ߦ࣌ͷ࣮ߦঢ়گͷ஋ʢྫֶश཰ʣ w ΦϯϥΠϯධՁͷ஋ w

    ॻ͖ग़͠ઌʹσʔλ͕ॻ͖ࠐ·Ε͔ͨ w Τϥʔ௨஌ w Կʹ͕͔͔࣌ؒΔ͔  Πϯϑϥ w δϣϒ͕੒ޭ͔ͨ͠ɾࣦഊ͔ͨ͠ɾ࣮ߦ͞Ε͔ͨ w ઃఆ͍ͯ͠Δ࣌ؒʹ࣮ߦ͞Ε͔ͨ ԿΛϞχλϦϯάɾϩΪϯά͢΂͖͔ ϞχλϦϯά
  17. ©2019 Wantedly, Inc. w ॲཧͷ಺༰͕Θ͔Δ΋ͷʢྫ42-ʣ w δϣϒͷ։࢝ɾऴྃ w ॻ͖ग़ͨ͠σʔλ w

    FUD ԿΛϞχλϦϯάɾϩΪϯά͢΂͖͔ ϩΪϯά
  18. ©2019 Wantedly, Inc. ͲΕ͘Β͍कΒΕΔ΂͖΋ͷʁ

  19. ©2019 Wantedly, Inc.  αʔϏεશମ͕࢖͑ͳ͘ͳΔ  ,1*ʹ99Өڹ͕ͰΔ  Ұ෦ͷػೳ͕࢖͑ͳ͘ͳΔ ͲΕ͘Β͍कΔ΂͖͔

    ͋Δਪનʹ໰୊͕͋Δͱɾɾɾ
  20. ©2019 Wantedly, Inc.  αʔϏεશମ͕࢖͑ͳ͘ͳΔ  ,1*ʹ99Өڹ͕ͰΔ  Ұ෦ͷػೳ͕࢖͑ͳ͘ͳΔ ͲΕ͘Β͍कΔ΂͖͔

    ͋Δਪનʹ໰୊͕͋Δͱɾɾɾ ͦ΋ͦ΋ͦͷσʔλͷॏཁ౓͕Θ͔Δඞཁ͕͋Δ
  21. ©2019 Wantedly, Inc. ԿΛ͢΂͖͔

  22. ©2019 Wantedly, Inc.  ॏཁ౓ΛͰ͖Δ͚ͩ໌֬ʹ͓ͯ͘͠  ϞχλϦϯάɾϩΪϯάͷ߲໨ચ͍ग़͠  ΠϯϑϥνʔϜͱڠྗͯ͠ɺ༧Ί w

    ॏཁ౓Λ఻͑Δ w ϞχλϦϯάɾϩΪϯάͷλεΫͷׂΓৼΓΛߦ͏ w ໰୊͕ൃੜͨ࣌͠ɺͲ͏͍ͬͨϑϩʔͰ֬ೝ͢Δ͔ɺͲ͏ରॲ ͢Δ͔Λ఻͑ΔඞཁͰ͋Ε͹࣮ࡍʹϦϋʔαϧΛߦ͏ ԿΛ͢΂͖͔