Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Gunosyのデータ分析基盤、ログ基盤の全容

moyomot
February 28, 2017
8.9k

 Gunosyのデータ分析基盤、ログ基盤の全容

moyomot

February 28, 2017
Tweet

Transcript

  1. Gunosyͷσʔλ෼ੳج൫ɺϩάج൫ͷશ༰
    σʔλ෼ੳج൫Night #1
    2017.2
    ৿ຊ ३࢘

    View Slide

  2. ©Gunosy Inc. 2
    גࣜձࣾGunosy – ʮ৘ใΛੈքதͷਓʹ࠷దʹಧ͚Δʯ
    Gunosy͸ ৘ใΩϡϨʔγϣϯαʔϏεʮά
    ϊγʔʯͱ
    2016೥6݄1೔ʹKDDIגࣜձࣾͱڞಉͰϦϦʔεͨ͠

    ແྉχϡʔε഑৴ΞϓϦʮχϡʔεύεʯΛఏڙ͢Δ
    ձࣾͰ͢ɻʮ৘ใΛੈքதͷਓʹ࠷దʹಧ͚ΔʯΛ

    Ϗδϣϯʹ׆ಈ͍ͯ͠·͢ɻ
    ωοτ্ʹଘࡏ͢Δ͞·͟·ͳ৘ใΛɺ

    ಠࣗͷΞϧΰϦζϜͰऩूɺධՁ෇͚Λߦ͍

    Ϣʔβʔʹಧ͚·͢ɻ
    ৘ใΩϡϨʔγϣϯαʔϏε

    ʮά
    ϊγʔʯ
    200ഔମҎ্ͷχϡʔειʔεΛϕʔεʹɺ

    ৽ͨʹ։ൃͨ͠৘ใղੳɾ഑৴ٕज़Λ༻͍ͯࣗಈతʹ

    બఆͨ͠χϡʔε΍৘ใΛϢʔβʔʹಧ͚·͢ɻ
    ແྉχϡʔε഑৴ΞϓϦ

    ʮχϡʔεύεʯ

    View Slide

  3. ©Gunosy Inc. 3
    ࣗݾ঺հ
    ● ৿ຊ ३࢘
    ● גࣜձࣾGunosy σʔλ෼ੳ෦
    ● هࣄ഑৴ΞϧΰϦζϜͷվળ
    ● ϩάج൫ͷ੔උ

    View Slide

  4. ©Gunosy Inc. 4
    ࠓ೔͓࿩͢Δ͜ͱ
    ● Gunosyσʔλ෼ੳ෦ͷओͳۀ຿
    ● σʔλ෼ੳج൫ɺϩάج൫ͷίϯηϓτ
    ● σʔλ෼ੳج൫ɺϩάج൫ͷશମ૾
    ● ֬ఆ஋ϩάج൫
    ● ଎ใ஋ϩάج൫
    ● ͏·͍͍ͬͯ͘Δ͜ͱɾݱঢ়ͷ՝୊

    View Slide

  5. ©Gunosy Inc. 5
    σʔλ෼ੳ෦ͷओͳۀ຿
    ● ΞϓϦΛ࢖༻͢ΔϢʔβʔͷຬ଍౓޲্ͷͨΊͷ෼ੳ
    ● ଺ࡏ࣌ؒɺܧଓ཰౳޲্ͷͨΊͷσʔλ෼ੳ
    ● هࣄ഑৴ΞϧΰϦζϜͷվળ
    ● ΞϧΰϦζϜݕূɺϓϩμΫγϣϯίʔυͷ࣮૷
    ● KPIूܭ
    ● ଞ෦ॺ͔ΒͷूܭґཔɺABςετ౳ͷ਺஋ఆ఺؍ଌ
    ● KPIʹҟৗ͕͋Δ৔߹ͷݪҼڀ໌
    ● σʔλ෼ੳج൫ɺϩάج൫ͷ੔උ

    View Slide

  6. ©Gunosy Inc. 6
    ߈ΊͷλεΫɺकΓͷλεΫ
    ● ΞϓϦΛ࢖༻͢ΔϢʔβʔͷຬ଍౓޲্ͷͨΊͷ෼ੳ
    ● ଺ࡏ࣌ؒɺܧଓ཰౳޲্ͷͨΊͷσʔλ෼ੳ
    ● هࣄ഑৴ΞϧΰϦζϜͷվળ
    ● ΞϧΰϦζϜݕূɺϓϩμΫγϣϯίʔυͷ࣮૷
    ● KPIूܭ
    ● ଞ෦ॺ͔ΒͷूܭґཔɺABςετ౳ͷ਺஋ఆ఺؍ଌ
    ● KPIʹҟৗ͕͋Δ৔߹ͷݪҼڀ໌
    ● σʔλ෼ੳج൫ɺϩάج൫ͷ੔උ

    View Slide

  7. ©Gunosy Inc. 7
    σʔλ෼ੳج൫ͷϙΠϯτ
    ● कΓͷλεΫ͸ࣗಈԽͯ͠ɺ߈ΊͷλεΫʹ஫ྗ͍ͨ͠
    ● ͍͔ʹଟ͘ͷ߈ΊλεΫʹνϟϨϯδ͕Ͱ͖Δ͔
    ● ੜ͖࢒ͬͨ΋ͷ͕Ձ஋ΛੜΈग़͢
    ● ͦͷͨΊͷج൫ΛͲͷΑ͏ʹͭ͘Δ͔
    ● Θ͔Γ΍͍͢μογϡϘʔυͷߏங
    ● KPIʹҟৗ͕͋ͬͨ৔߹ͷݪҼڀ໌লྗԽ
    ● େن໛σʔλͰ΋εέʔϧͰ͖Δϩάج൫
    ● σʔλ෼ੳ͠΍͍͢؀ڥ

    View Slide

  8. ©Gunosy Inc. 8
    Gunosyͷσʔλ෼ੳج൫ɺϩάج൫
    ● ϩάج൫શମ૾Λ঺հ
    ● ཁॴͰσʔλ෼ੳج൫͕ొ৔
    ● Gunosyͷϩάج൫͸େ͖͘2ͭ
    ● ֬ఆ஋ϩάج൫
    ● KPI
    ● هࣄ഑৴ΞϧΰϦζϜ
    ● σʔλ෼ੳ
    ● ଎ใ஋ϩάج൫
    ● ଎ใ༻KPIʢHourly Active Userʣ
    ● هࣄ഑৴ΞϧΰϦζϜ

    View Slide

  9. ©Gunosy Inc. 9
    ֬ఆ஋ϩάج൫
    ● શମ૾
    Redshift
    ϩάαʔόʔ S3
    SQS
    ίϯόʔλʔ
    Fluentd
    BigQuery
    KPIόον
    αʔόʔ
    μογϡϘʔυ

    View Slide

  10. ©Gunosy Inc. 10
    KPIμογϡϘʔυ
    ● Redash
    ● ༷ʑͳσʔλɾιʔεʹ౷ҰతʹΞΫηεͰ͖ΔWeb
    αʔϏε
    ● σʔλՄࢹԽπʔϧ
    ● SQLͰ׬݁Ͱ͖ɺը໘ͷ࣮૷ෆཁ
    ● DjangoΦϦδφϧμογϡϘʔυ
    ● ϑϧεΫϥον࣮૷ͳͷͰɺࣗ༝౓͸ߴ͍
    ● ͍ʹ͑͠ΑΓར༻
    ● SQLͰ׬݁͠ͳ͍ࢦඪΛݟΔͱ͖ʹ࣮૷

    View Slide

  11. ©Gunosy Inc. 11
    μογϡϘʔυͷ޻෉
    ● 2ͭͷμογϡϘʔυͷॅΈ෼͚͕Ͱ͖͍ͯΔ
    ● RedashͰඇΤϯδχΞ΋ूܭͰ͖Δ
    ● SQLษڧձ
    ● SlackͰSQL෦԰
    ● ҟৗ࣌ʹݪҼڀ໌ΛμογϡϘʔυͷΈͰ׬݁Ͱ͖Δ
    ● λΠϜϦʔʹҟৗʹؾ͚ͮΔ
    ● SQLൃߦ͠ͳ͍ɺੜϩάݟͳ͍

    View Slide

  12. ©Gunosy Inc. 12
    μογϡϘʔυͷ՝୊
    ● ҟৗ͔Ͳ͏͔ͷධՁࢦඪ͕ෆे෼
    ● ҟৗ࣌͸μογϡϘʔυ͑͞ݟͨ͘ͳ͍
    ● Ͳͷࢦඪ͕ͳͥ໰୊͔Λݕग़͠ɺSlack΁Ξϥʔτ͢Δ
    ࢓૊Έ͕΄͍͠

    View Slide

  13. ©Gunosy Inc. 13
    RedshiftͱBigQueryͷ࢖͍෼͚
    ● ҎલΑΓRedshiftΛ࢖༻͖ͯͨ͠
    ● BigQuery͸2016೥ՆΑΓຊ֨ར༻։࢝
    ● RedshiftɺBigQuery྆ํʹ΄΅͓ͳ͡σʔλ͕ೖ͍ͬͯΔ
    ● Ұ෦ͷ͔ͳΓେ͖ͳσʔλ͸BigQueryͷΈ
    ● Redshiftʹґଘͨ͠KPIΫΤϦʢKPIूܭόονʣ͕ଟ਺
    ● BigQuery΁ͷҠߦ͸ఘΊͨ
    ● ৽͍͠KPI΍࣌ؒͷ͔͔ΔΫΤϦ͸BigQuery΁

    View Slide

  14. ©Gunosy Inc. 14
    Redshiftͷྑ͍ͱ͜Ζ
    Good in Redshift:
    ● ʢओ؍తʹʣγϯϓϧͳΫΤϦ
    ● Postgresqlϕʔε
    ● created_at::date
    else:
    ● ઃܭɺӡ༻ਏ͍
    ● distkeyͷઃܭ೉͍͠
    ● node௥Ճ࣌͸ϝϯςφϯεϞʔυ

    View Slide

  15. ©Gunosy Inc. 15
    BigQueryͷྑ͍ͱ͜Ζ
    Good in BigQuery:
    ● ετϨʔδεέʔϥϒϧ
    else:
    ● Standard SQL΋ͬͱαϙʔτͯ͠΄͍͠
    ● percentile͸Α
    ● SELECTΫΤϦͰςʔϒϧεΩϟϯ͢ΔύʔςΟγϣϯ
    ࢦఆͨ͘͠ͳ͍

    View Slide

  16. ©Gunosy Inc. 16
    ࣗՈ੡ϩάίϯόʔλʔ
    ● ੜϩάͷ੔ܗΛ୲౰
    ● JSONϩάΛύΠϓ۠੾Γͷϩάʹ
    ● ΞϓϦόʔδϣϯɺσόΠε(iOS, Android)ʹΑͬͯ
    ϑΥʔϚοτ͕ҟͳΔϩάͷ੔ܗ
    ● ϩά੔ܗ͸୭͕୲౰͢Δ͔
    ● ϩάαʔόʔʢ಄Ͱ΍Δ͔ʣ
    ● ϩάίϯόʔλʔʢඌͰ΍Δ͔ʣ
    ● ϩά੔ܗ͠ͳ͍
    ● ੔ܗΛʮඌʯͰ࣮ࢪͨ͠΄͏͕ӡ༻͸ָ
    ● ੜϩάɺ੔ܗϩά྆ऀΛอଘ͢Δӡ༻ָ͕

    View Slide

  17. ©Gunosy Inc. 17
    ϩάઃܭΛϛεͬͨΒ
    ● ׬ᘳͳϩάઃܭ͸೉͍͠
    ● σόΠεɺόʔδϣϯɺ৽ػೳɺ৽UIͰࢥΘ͵ࠩҟ͕Ͱ
    ͯ͠·͏
    ● Ͳ͔͜ͰέΞ͢Δඞཁ͕ੜ͡Δ
    ● ϩάAPIαʔόʔʁ
    ● ϩάίϯόʔλʔʁ
    ● SQLΫΤϦʁ

    View Slide

  18. ©Gunosy Inc. 18
    σʔλ෼ੳج൫
    ● πʔϧηοτ͸Jupyter+pandas+DB
    ● ϩάΛ࢖༻ͨ͠KPIʹؔ͢Δσʔλ෼ੳ
    ● هࣄσʔλ౳Λ࢖༻ͨ͠ΞϧΰϦζϜվળͷͨΊͷ෼ੳ
    Redshift
    BigQuery
    Pandas

    View Slide

  19. ©Gunosy Inc. 19
    σʔλ෼ੳج൫ͷ՝୊
    ● ΞϧΰϦζϜվળͷͨΊͷ෼ੳج൫͸ෳ਺σʔλɾιʔε
    ʹ·͕͍ͨͬͯΔ
    ● هࣄσʔλ
    ● هࣄΫϥελϦϯάσʔλ
    ● είΞϦϯάσʔλ
    ● ౷ҰతʹΞΫηεͰ͖Δͱ෼ੳͷޮ཰͕޲্͢Δ
    ● ϩάΛ࢖༻ͨ͠෼ੳ͸BigQueryҰຊͰ׬݁͢Δ

    View Slide

  20. ©Gunosy Inc. 20
    ABςετج൫ɺΦϑϥΠϯςετج൫
    ● ΰδϥʢABςετج൫ʣ
    ● ༝དྷɿABςετΛࢧ഑͢Δଘࡏ
    ● ABςετͷׂ౰ɺ࣮ࢪঢ়گΛWebπʔϧͰ؅ཧ
    ● σϩϦΞϯʢΦϑϥΠϯςετج൫ʣ
    ● աڈϩάΛ࢖༻ͯ͠ΞϧΰϦζϜมߋͷྑ͠ѱ͠Λ൑அ
    ͢Δ
    ● ABςετʢΦϯϥΠϯςετʣ͸࣌ؒΛཁ͢Δ
    ● ධՁࢦඪͷબͼํ͕؊
    ● ݱࡏ͸ίϚϯυϥΠϯπʔϧͷΈ

    View Slide

  21. ©Gunosy Inc. 21
    ଎ใ஋ϩάج൫
    ● هࣄ഑৴ΞϧΰϦζϜͰ࢖༻
    ● KPI଎ใ஋ʢHourly Active User౳ʣͰ࢖༻
    ● σʔλ෼ੳʹ͸࢖༻ͯ͠ͳ͍
    ● Spark Streaming͔ΒKinesis Analytics΁Ҡߦத

    View Slide

  22. ©Gunosy Inc. 22
    Spark StreamingΛ࢖༻ͨ͠଎ใ஋ج൫
    ● Spark StreamingͰ୹࣌ؒͷϩάूܭΛ࣮ࢪ
    ● ଎ใ஋Λ࢖༻͢ΔγεςϜ͕ѻ͍΍͍͢Α͏ʹRDSʹอଘ
    ● ՝୊
    ● αʔόʔίετʢEMRίετʣ͕ͦΕͳΓ
    ● ӡ༻ίετʢΫϥελ؅ཧίετʣ͕ͦΕͳΓ

    View Slide

  23. ©Gunosy Inc. 23
    Kinesis AnalyticsΛ࢖༻ͨ͠଎ใ஋ج൫
    ● ετϦʔϛϯάσʔλΛSQLΫΤϦͰूܭͰ͖ΔαʔϏε
    ● ूܭ݁Ռ͸FirehoseͰElasticsearchʹอଘ
    ● ΤϯδχΞ͕࣮૷͢Δͷ͸ूܭΫΤϦͷΈ
    ● ௕ظؒͷӡ༻࣮੷͸͜Ε͔Β
    ● ίετ࡟ݮ

    View Slide

  24. ©Gunosy Inc. 24
    ·ͱΊ
    ● Gunosyσʔλ෼ੳج൫ɺϩάج൫ͷશ༰ʹ͍ͭͯ঺հ
    ● ֬ఆ஋ϩάج൫
    ● ଎ใ஋ϩάج൫
    ● ߈ΊͷλεΫ
    ● ෼ੳ͠΍͍͢؀ڥͮ͘Γ
    ● ෳ਺ͷσʔλιʔεʹ౷ҰతʹΞΫηε
    ● େྔͷσʔλΛ༰қʹѻ͑Δ؀ڥ
    ● कΓͷλεΫ͸ՄೳͳݶΓࣗಈԽ

    View Slide