Gunosyのデータ分析基盤、ログ基盤の全容

Transcript

1. Gunosyͷσʔλ෼ੳج൫ɺϩάج൫ͷશ༰
   σʔλ෼ੳج൫Night #1
   2017.2
   ৿ຊ ३࢘
   

   View Slide

2. ©Gunosy Inc. 2
   גࣜձࣾGunosy – ʮ৘ใΛੈքதͷਓʹ࠷దʹಧ͚Δʯ
   Gunosy͸ ৘ใΩϡϨʔγϣϯαʔϏεʮά
   ϊγʔʯͱ
   2016೥6݄1೔ʹKDDIגࣜձࣾͱڞಉͰϦϦʔεͨ͠

   ແྉχϡʔε഑৴ΞϓϦʮχϡʔεύεʯΛఏڙ͢Δ
   ձࣾͰ͢ɻʮ৘ใΛੈքதͷਓʹ࠷దʹಧ͚ΔʯΛ

   Ϗδϣϯʹ׆ಈ͍ͯ͠·͢ɻ
   ωοτ্ʹଘࡏ͢Δ͞·͟·ͳ৘ใΛɺ

   ಠࣗͷΞϧΰϦζϜͰऩूɺධՁ෇͚Λߦ͍

   Ϣʔβʔʹಧ͚·͢ɻ
   ৘ใΩϡϨʔγϣϯαʔϏε

   ʮά
   ϊγʔʯ
   200ഔମҎ্ͷχϡʔειʔεΛϕʔεʹɺ

   ৽ͨʹ։ൃͨ͠৘ใղੳɾ഑৴ٕज़Λ༻͍ͯࣗಈతʹ

   બఆͨ͠χϡʔε΍৘ใΛϢʔβʔʹಧ͚·͢ɻ
   ແྉχϡʔε഑৴ΞϓϦ

   ʮχϡʔεύεʯ
   

   View Slide

3. ©Gunosy Inc. 3
   ࣗݾ঺հ
   ● ৿ຊ ३࢘
   ● גࣜձࣾGunosy σʔλ෼ੳ෦
   ● هࣄ഑৴ΞϧΰϦζϜͷվળ
   ● ϩάج൫ͷ੔උ
   

   View Slide

4. ©Gunosy Inc. 4
   ࠓ೔͓࿩͢Δ͜ͱ
   ● Gunosyσʔλ෼ੳ෦ͷओͳۀ຿
   ● σʔλ෼ੳج൫ɺϩάج൫ͷίϯηϓτ
   ● σʔλ෼ੳج൫ɺϩάج൫ͷશମ૾
   ● ֬ఆ஋ϩάج൫
   ● ଎ใ஋ϩάج൫
   ● ͏·͍͍ͬͯ͘Δ͜ͱɾݱঢ়ͷ՝୊
   

   View Slide

5. ©Gunosy Inc. 5
   σʔλ෼ੳ෦ͷओͳۀ຿
   ● ΞϓϦΛ࢖༻͢ΔϢʔβʔͷຬ଍౓޲্ͷͨΊͷ෼ੳ
   ● ଺ࡏ࣌ؒɺܧଓ཰౳޲্ͷͨΊͷσʔλ෼ੳ
   ● هࣄ഑৴ΞϧΰϦζϜͷվળ
   ● ΞϧΰϦζϜݕূɺϓϩμΫγϣϯίʔυͷ࣮૷
   ● KPIूܭ
   ● ଞ෦ॺ͔ΒͷूܭґཔɺABςετ౳ͷ਺஋ఆ఺؍ଌ
   ● KPIʹҟৗ͕͋Δ৔߹ͷݪҼڀ໌
   ● σʔλ෼ੳج൫ɺϩάج൫ͷ੔උ
   

   View Slide

6. ©Gunosy Inc. 6
   ߈ΊͷλεΫɺकΓͷλεΫ
   ● ΞϓϦΛ࢖༻͢ΔϢʔβʔͷຬ଍౓޲্ͷͨΊͷ෼ੳ
   ● ଺ࡏ࣌ؒɺܧଓ཰౳޲্ͷͨΊͷσʔλ෼ੳ
   ● هࣄ഑৴ΞϧΰϦζϜͷվળ
   ● ΞϧΰϦζϜݕূɺϓϩμΫγϣϯίʔυͷ࣮૷
   ● KPIूܭ
   ● ଞ෦ॺ͔ΒͷूܭґཔɺABςετ౳ͷ਺஋ఆ఺؍ଌ
   ● KPIʹҟৗ͕͋Δ৔߹ͷݪҼڀ໌
   ● σʔλ෼ੳج൫ɺϩάج൫ͷ੔උ
   

   View Slide

7. ©Gunosy Inc. 7
   σʔλ෼ੳج൫ͷϙΠϯτ
   ● कΓͷλεΫ͸ࣗಈԽͯ͠ɺ߈ΊͷλεΫʹ஫ྗ͍ͨ͠
   ● ͍͔ʹଟ͘ͷ߈ΊλεΫʹνϟϨϯδ͕Ͱ͖Δ͔
   ● ੜ͖࢒ͬͨ΋ͷ͕Ձ஋ΛੜΈग़͢
   ● ͦͷͨΊͷج൫ΛͲͷΑ͏ʹͭ͘Δ͔
   ● Θ͔Γ΍͍͢μογϡϘʔυͷߏங
   ● KPIʹҟৗ͕͋ͬͨ৔߹ͷݪҼڀ໌লྗԽ
   ● େن໛σʔλͰ΋εέʔϧͰ͖Δϩάج൫
   ● σʔλ෼ੳ͠΍͍͢؀ڥ
   

   View Slide

8. ©Gunosy Inc. 8
   Gunosyͷσʔλ෼ੳج൫ɺϩάج൫
   ● ϩάج൫શମ૾Λ঺հ
   ● ཁॴͰσʔλ෼ੳج൫͕ొ৔
   ● Gunosyͷϩάج൫͸େ͖͘2ͭ
   ● ֬ఆ஋ϩάج൫
   ● KPI
   ● هࣄ഑৴ΞϧΰϦζϜ
   ● σʔλ෼ੳ
   ● ଎ใ஋ϩάج൫
   ● ଎ใ༻KPIʢHourly Active Userʣ
   ● هࣄ഑৴ΞϧΰϦζϜ
   

   View Slide

9. ©Gunosy Inc. 9
   ֬ఆ஋ϩάج൫
   ● શମ૾
   Redshift
   ϩάαʔόʔ S3
   SQS
   ίϯόʔλʔ
   Fluentd
   BigQuery
   KPIόον
   αʔόʔ
   μογϡϘʔυ
   

   View Slide

10. ©Gunosy Inc. 10
    KPIμογϡϘʔυ
    ● Redash
    ● ༷ʑͳσʔλɾιʔεʹ౷ҰతʹΞΫηεͰ͖ΔWeb
    αʔϏε
    ● σʔλՄࢹԽπʔϧ
    ● SQLͰ׬݁Ͱ͖ɺը໘ͷ࣮૷ෆཁ
    ● DjangoΦϦδφϧμογϡϘʔυ
    ● ϑϧεΫϥον࣮૷ͳͷͰɺࣗ༝౓͸ߴ͍
    ● ͍ʹ͑͠ΑΓར༻
    ● SQLͰ׬݁͠ͳ͍ࢦඪΛݟΔͱ͖ʹ࣮૷
    

    View Slide

11. ©Gunosy Inc. 11
    μογϡϘʔυͷ޻෉
    ● 2ͭͷμογϡϘʔυͷॅΈ෼͚͕Ͱ͖͍ͯΔ
    ● RedashͰඇΤϯδχΞ΋ूܭͰ͖Δ
    ● SQLษڧձ
    ● SlackͰSQL෦԰
    ● ҟৗ࣌ʹݪҼڀ໌ΛμογϡϘʔυͷΈͰ׬݁Ͱ͖Δ
    ● λΠϜϦʔʹҟৗʹؾ͚ͮΔ
    ● SQLൃߦ͠ͳ͍ɺੜϩάݟͳ͍
    

    View Slide

12. ©Gunosy Inc. 12
    μογϡϘʔυͷ՝୊
    ● ҟৗ͔Ͳ͏͔ͷධՁࢦඪ͕ෆे෼
    ● ҟৗ࣌͸μογϡϘʔυ͑͞ݟͨ͘ͳ͍
    ● Ͳͷࢦඪ͕ͳͥ໰୊͔Λݕग़͠ɺSlack΁Ξϥʔτ͢Δ
    ࢓૊Έ͕΄͍͠
    

    View Slide

13. ©Gunosy Inc. 13
    RedshiftͱBigQueryͷ࢖͍෼͚
    ● ҎલΑΓRedshiftΛ࢖༻͖ͯͨ͠
    ● BigQuery͸2016೥ՆΑΓຊ֨ར༻։࢝
    ● RedshiftɺBigQuery྆ํʹ΄΅͓ͳ͡σʔλ͕ೖ͍ͬͯΔ
    ● Ұ෦ͷ͔ͳΓେ͖ͳσʔλ͸BigQueryͷΈ
    ● Redshiftʹґଘͨ͠KPIΫΤϦʢKPIूܭόονʣ͕ଟ਺
    ● BigQuery΁ͷҠߦ͸ఘΊͨ
    ● ৽͍͠KPI΍࣌ؒͷ͔͔ΔΫΤϦ͸BigQuery΁
    

    View Slide

14. ©Gunosy Inc. 14
    Redshiftͷྑ͍ͱ͜Ζ
    Good in Redshift:
    ● ʢओ؍తʹʣγϯϓϧͳΫΤϦ
    ● Postgresqlϕʔε
    ● created_at::date
    else:
    ● ઃܭɺӡ༻ਏ͍
    ● distkeyͷઃܭ೉͍͠
    ● node௥Ճ࣌͸ϝϯςφϯεϞʔυ
    

    View Slide

15. ©Gunosy Inc. 15
    BigQueryͷྑ͍ͱ͜Ζ
    Good in BigQuery:
    ● ετϨʔδεέʔϥϒϧ
    else:
    ● Standard SQL΋ͬͱαϙʔτͯ͠΄͍͠
    ● percentile͸Α
    ● SELECTΫΤϦͰςʔϒϧεΩϟϯ͢ΔύʔςΟγϣϯ
    ࢦఆͨ͘͠ͳ͍
    

    View Slide

16. ©Gunosy Inc. 16
    ࣗՈ੡ϩάίϯόʔλʔ
    ● ੜϩάͷ੔ܗΛ୲౰
    ● JSONϩάΛύΠϓ۠੾Γͷϩάʹ
    ● ΞϓϦόʔδϣϯɺσόΠε(iOS, Android)ʹΑͬͯ
    ϑΥʔϚοτ͕ҟͳΔϩάͷ੔ܗ
    ● ϩά੔ܗ͸୭͕୲౰͢Δ͔
    ● ϩάαʔόʔʢ಄Ͱ΍Δ͔ʣ
    ● ϩάίϯόʔλʔʢඌͰ΍Δ͔ʣ
    ● ϩά੔ܗ͠ͳ͍
    ● ੔ܗΛʮඌʯͰ࣮ࢪͨ͠΄͏͕ӡ༻͸ָ
    ● ੜϩάɺ੔ܗϩά྆ऀΛอଘ͢Δӡ༻ָ͕
    

    View Slide

17. ©Gunosy Inc. 17
    ϩάઃܭΛϛεͬͨΒ
    ● ׬ᘳͳϩάઃܭ͸೉͍͠
    ● σόΠεɺόʔδϣϯɺ৽ػೳɺ৽UIͰࢥΘ͵ࠩҟ͕Ͱ
    ͯ͠·͏
    ● Ͳ͔͜ͰέΞ͢Δඞཁ͕ੜ͡Δ
    ● ϩάAPIαʔόʔʁ
    ● ϩάίϯόʔλʔʁ
    ● SQLΫΤϦʁ
    

    View Slide

18. ©Gunosy Inc. 18
    σʔλ෼ੳج൫
    ● πʔϧηοτ͸Jupyter+pandas+DB
    ● ϩάΛ࢖༻ͨ͠KPIʹؔ͢Δσʔλ෼ੳ
    ● هࣄσʔλ౳Λ࢖༻ͨ͠ΞϧΰϦζϜվળͷͨΊͷ෼ੳ
    Redshift
    BigQuery
    Pandas
    

    View Slide

19. ©Gunosy Inc. 19
    σʔλ෼ੳج൫ͷ՝୊
    ● ΞϧΰϦζϜվળͷͨΊͷ෼ੳج൫͸ෳ਺σʔλɾιʔε
    ʹ·͕͍ͨͬͯΔ
    ● هࣄσʔλ
    ● هࣄΫϥελϦϯάσʔλ
    ● είΞϦϯάσʔλ
    ● ౷ҰతʹΞΫηεͰ͖Δͱ෼ੳͷޮ཰͕޲্͢Δ
    ● ϩάΛ࢖༻ͨ͠෼ੳ͸BigQueryҰຊͰ׬݁͢Δ
    

    View Slide

20. ©Gunosy Inc. 20
    ABςετج൫ɺΦϑϥΠϯςετج൫
    ● ΰδϥʢABςετج൫ʣ
    ● ༝དྷɿABςετΛࢧ഑͢Δଘࡏ
    ● ABςετͷׂ౰ɺ࣮ࢪঢ়گΛWebπʔϧͰ؅ཧ
    ● σϩϦΞϯʢΦϑϥΠϯςετج൫ʣ
    ● աڈϩάΛ࢖༻ͯ͠ΞϧΰϦζϜมߋͷྑ͠ѱ͠Λ൑அ
    ͢Δ
    ● ABςετʢΦϯϥΠϯςετʣ͸࣌ؒΛཁ͢Δ
    ● ධՁࢦඪͷબͼํ͕؊
    ● ݱࡏ͸ίϚϯυϥΠϯπʔϧͷΈ
    

    View Slide

21. ©Gunosy Inc. 21
    ଎ใ஋ϩάج൫
    ● هࣄ഑৴ΞϧΰϦζϜͰ࢖༻
    ● KPI଎ใ஋ʢHourly Active User౳ʣͰ࢖༻
    ● σʔλ෼ੳʹ͸࢖༻ͯ͠ͳ͍
    ● Spark Streaming͔ΒKinesis Analytics΁Ҡߦத
    

    View Slide

22. ©Gunosy Inc. 22
    Spark StreamingΛ࢖༻ͨ͠଎ใ஋ج൫
    ● Spark StreamingͰ୹࣌ؒͷϩάूܭΛ࣮ࢪ
    ● ଎ใ஋Λ࢖༻͢ΔγεςϜ͕ѻ͍΍͍͢Α͏ʹRDSʹอଘ
    ● ՝୊
    ● αʔόʔίετʢEMRίετʣ͕ͦΕͳΓ
    ● ӡ༻ίετʢΫϥελ؅ཧίετʣ͕ͦΕͳΓ
    

    View Slide

23. ©Gunosy Inc. 23
    Kinesis AnalyticsΛ࢖༻ͨ͠଎ใ஋ج൫
    ● ετϦʔϛϯάσʔλΛSQLΫΤϦͰूܭͰ͖ΔαʔϏε
    ● ूܭ݁Ռ͸FirehoseͰElasticsearchʹอଘ
    ● ΤϯδχΞ͕࣮૷͢Δͷ͸ूܭΫΤϦͷΈ
    ● ௕ظؒͷӡ༻࣮੷͸͜Ε͔Β
    ● ίετ࡟ݮ
    

    View Slide

24. ©Gunosy Inc. 24
    ·ͱΊ
    ● Gunosyσʔλ෼ੳج൫ɺϩάج൫ͷશ༰ʹ͍ͭͯ঺հ
    ● ֬ఆ஋ϩάج൫
    ● ଎ใ஋ϩάج൫
    ● ߈ΊͷλεΫ
    ● ෼ੳ͠΍͍͢؀ڥͮ͘Γ
    ● ෳ਺ͷσʔλιʔεʹ౷ҰతʹΞΫηε
    ● େྔͷσʔλΛ༰қʹѻ͑Δ؀ڥ
    ● कΓͷλεΫ͸ՄೳͳݶΓࣗಈԽ
    

    View Slide