Upgrade to Pro — share decks privately, control downloads, hide ads and more …

アドテク企業の本番環境からTD使ってみた / Treasure Data Tech Talk 20160425

アドテク企業の本番環境からTD使ってみた / Treasure Data Tech Talk 20160425

機械学習の基礎から、本番環境へのTreasureDataを使った機械学習導入部分までカバーします。
nex8という株式会社ファンコミュニケーションズの開発・運用するDSPにおけるお話です。

Takayuki Sakai

April 26, 2016
Tweet

More Decks by Takayuki Sakai

Other Decks in Technology

Transcript

  1. ΞυςΫاۀͷ

    ຊ൪؀ڥ͔ΒTD࢖ͬͯΈͨ
    Scala x TreasureData
    ΦϯϥΠϯCTR༧ଌ

    View full-size slide

  2. ञҪ ਸࢸ
    - 2016/01- F@N Communicationsגࣜձࣾ
    - ๭CAࣾΞυςΫελδΦͰΠϯλʔϯͱ͔ͯͨ͠
    - ScalaΤϯδχΞ (ଞʹRuby, Python, JS, Go…)
    - ػցֶश΋΍ΔΑ
    - Slack & Raspberry PiͰΤΞίϯ͚ͭͨΓ

    View full-size slide

  3. ର৅ऀ
    - ػցֶश or CTR༧ଌʹڵຯ͕͋Δਓ
    - Scala͔ΒTreasureDataΛ࢖ͬͯΈ͍ͨਓ

    View full-size slide

  4. ΞυςΫۀքͷதͰ΋

    DSPͱ͍͏ͷΛ࡞ͬͯ·͢

    View full-size slide

  5. What’s DSP?

    View full-size slide

  6. What DSPs do
    SSP DSP
    ͜ͷαΠτʹϦΫΤετ

    དྷͯΔ͚Ͳ޿ࠂग़͞΁Μʁ

    View full-size slide

  7. What DSPs do
    SSP DSP
    ͦ͜΍ͬͨΒ

    0.1ԁͳΒങ͏Θ

    View full-size slide

  8. What DSPs do
    SSP DSP
    Αͬ͠Ό͋Μͨʹ

    ചͬͨΖ
    ଞͷձࣾͷํ͕

    ͍͍஋ஈ͚ͭͯ͘ΕͨΘ

    View full-size slide

  9. What DSPs do
    SSP DSP
    Αͬ͠Ό͋Μͨʹ

    ചͬͨΖ
    ଞͷձࣾͷํ͕

    ͍͍஋ஈ͚ͭͯ͘ΕͨΘ

    View full-size slide

  10. ͜ͷؒΘ͔ͣ50ms

    View full-size slide

  11. ࠓͷҰ࿈ͷ΍ΓͱΓΛ

    RTBͱ͍͏Α
    RTB: Real-Time Bidding
    ςετʹग़Δͧʂ

    View full-size slide

  12. RTBͷಛ௃
    େྔΞΫηε
    ɾඵؒ5ສͱ͔
    ૣ͍Ϩεϙϯε
    ɾ100msҎ಺ʹฦ͞ͳ͍ͱΦʔΫγϣϯʹࢀՃͰ͖ͳ͍

    View full-size slide

  13. ސ٬ʢ޿ࠂओʣʹͱͬͯ

    ΑΓՁ஋ͷ͋ΔDSPΛ࡞Γ͍ͨʂ

    View full-size slide

  14. ΫϦοΫ཰(CTR)ͷ

    ༧ଌ͕େࣄ
    CTR: Click Through Rate

    View full-size slide

  15. DSP
    Site A
    Site B
    ޿ࠂग़͞΁Μʁ
    ޿ࠂग़͞΁Μʁ

    View full-size slide

  16. DSP
    Site A

    (CTR=0.1%)
    Site B

    (CTR=1%)
    0.5ԁͳΒങ͏Ͱ
    5ԁͳΒങ͏Ͱ

    View full-size slide

  17. CTR͕Θ͔Δͱ

    దਖ਼ͳ஋ஈͰೖࡳͰ͖Δ
    ΫϦοΫ཰

    View full-size slide

  18. RTBͷ࣌ʹΘ͔͍ͬͯΔ৘ใ
    - ϢʔβID
    - αΠτID
    - ޿ࠂID
    - etc…ʢͨ͘͞Μʣ

    View full-size slide

  19. - ϢʔβID
    - αΠτID
    - ޿ࠂID
    - etc…ʢͨ͘͞Μʣ
    ͜ΕΒͷ৘ใ͔Β

    CTRΛ༧ଌͯ͠ΈΑ͏ʂ

    View full-size slide

  20. ͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ

    View full-size slide

  21. ͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ
    Ͱ΋ͦͷϢʔβͷCTR͸1%ͩͥ

    View full-size slide

  22. ͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ
    Ͱ΋ͦͷϢʔβͷCTR͸1%ͩͥ
    ͡Ό͋ؒΛऔͬͯ0.5%ͬͯ͜ͱʹ͢Δʁ

    View full-size slide

  23. ͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ
    Ͱ΋ͦͷϢʔβͷCTR͸1%ͩͥ
    ͡Ό͋ؒΛऔͬͯ0.5%ͬͯ͜ͱʹ͢Δʁ
    Ϣʔβ͝ͱͷ৘ใͷํ͕ਖ਼֬ͩΖ

    0.8%͘Β͍͡ΌͶ

    View full-size slide

  24. ͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ
    Ͱ΋ͦͷϢʔβͷCTR͸1%ͩͥ
    ͡Ό͋ؒΛऔͬͯ0.5%ͬͯ͜ͱʹ͢Δʁ
    Ϣʔβ͝ͱͷ৘ใͷํ͕ਖ਼֬ͩΖ

    0.8%͘Β͍͡ΌͶ
    ͋ɺ޿ࠂ͝ͱͷCTR΋ߟ͑ͳ͍ͱ…

    View full-size slide

  25. ͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ
    Ͱ΋ͦͷϢʔβͷCTR͸1%ͩͥ
    ͡Ό͋ؒΛऔͬͯ0.5%ͬͯ͜ͱʹ͢Δʁ
    Ϣʔβ͝ͱͷ৘ใͷํ͕ਖ਼֬ͩΖ

    0.8%͘Β͍͡ΌͶ
    ͋ɺ޿ࠂͷCTR΋ߟ͑ͳ͍ͱ…
    ߟ͑ग़͢ͱେม

    View full-size slide

  26. - ϢʔβID
    - αΠτID
    - ޿ࠂID
    - etc…ʢͨ͘͞Μʣ
    ͪͳΈʹɺ͜ͷΑ͏ͳ

    ༧ଌͷࡐྉʹͳΔ৘ใΛ

    ಛ௃ྔͱ͍͏Α

    View full-size slide

  27. Machine Learning
    ػցֶश

    View full-size slide

  28. Machine LearningͳΒ…

    View full-size slide

  29. Machine LearningͳΒ…
    - ෳ਺ͷಛ௃ྔʹରͯ͠ (ϢʔβID, αΠτID…)

    View full-size slide

  30. Machine LearningͳΒ…
    - ෳ਺ͷಛ௃ྔʹରͯ͠ (ϢʔβID, αΠτID…)
    - ਺ֶతࠜڌʹج͍ͮͯ

    View full-size slide

  31. Machine LearningͳΒ…
    - ෳ਺ͷಛ௃ྔʹରͯ͠ (ϢʔβID, αΠτID…)
    - ਺ֶతࠜڌʹج͍ͮͯ
    - ࣗಈͰ

    View full-size slide

  32. Machine LearningͳΒ…
    - ෳ਺ͷಛ௃ྔʹରͯ͠ (ϢʔβID, αΠτID…)
    - ਺ֶతࠜڌʹج͍ͮͯ
    - ࣗಈͰ
    CTR͕༧ଌͰ͖Δʂ

    View full-size slide

  33. ػցֶशͬͯͲ͏΍Δͷʁ

    View full-size slide

  34. ࠓճͷख๏ɻৄ͍͠ਓ޲͚
    - ڭࢣ͋Γֶश
    - ڭࢣσʔλ͸ϩά͔Β࡞੒
    - ࠓճ͸ϩδεςΟοΫճؼͷઆ໌Ͱ͢
    ஌Βͳ͍ਓ͸ಡΈඈ͹ͯ͠OK

    View full-size slide

  35. ػցֶशͷجຊ
    1. ֶशσʔλͷ࡞੒
    2. ༧ଌϞσϧͷ࡞੒
    3. ༧ଌ

    View full-size slide

  36. 1. ֶशσʔλͷ࡞੒

    View full-size slide

  37. Ұൠతͳֶशσʔλ
    1 1 1 …… 0
    ಛ௃ྔ1 ಛ௃ྔ2 ಛ௃ྔ3 …… ਖ਼ղϥϕϧ
    2 3 2 …… 0
    2 2 3 …… 1
    ……

    View full-size slide

  38. CTR༧ଌͷ৔߹
    1 1 1 …… 0
    αΠτ Ϣʔβ ޿ࠂ …… ΫϦοΫ

    ͞Ε͔ͨ
    2 3 2 …… 0
    2 2 3 …… 1
    ……
    1ߦ͕

    1ΠϯϓϨογϣϯ

    View full-size slide

  39. CSVͰද͢ͱ…
    # αΠτ, Ϣʔβ, ޿ࠂ, …, ਖ਼ղϥϕϧ
    site_1, user_1, campaign_1, …, 0
    site_2, user_3, campaign_2, …, 0
    site_2, user_2, campaign_3, …, 1

    View full-size slide

  40. 2. ༧ଌϞσϧͷ࡞੒

    View full-size slide

  41. ֶशσʔλ
    …… 0
    ……
    ……
    ……
    Ξ

    ϧ

    ΰ


    Ϧ

    ζ

    Ϝ
    ༧ଌϞσϧ
    0
    1
    ࠓճ͸

    ϩδεςΟοΫճؼ
    …… 0

    View full-size slide

  42. αΠτ1
    αΠτ2
    Ϣʔβ1
    Ϣʔβ2
    ޿ࠂ1
    ޿ࠂ2
    ಛ௃ྔ ॏΈ
    ༧ଌϞσϧͷத਎

    View full-size slide

  43. ಛ௃ྔ ॏΈ
    0.1
    -0.2
    1.0
    -0.6
    -0.3
    -0.05
    αΠτ1
    αΠτ2
    Ϣʔβ1
    Ϣʔβ2
    ޿ࠂ1
    ޿ࠂ2

    View full-size slide

  44. CSVͰද͢ͱ…
    # ಛ௃ྔ, ॏΈ
    site_1, 0.1
    site_2, -0.2
    user_1, 1.0
    user_2, -0.6
    campaign_1,-0.3
    campaign_2,-0.05

    View full-size slide

  45. CTRΛ஌Γ͍ͨσʔλ
    αΠτ1 Ϣʔβ2 ޿ࠂ1 ……
    ֶशσʔλͱ΄΅ಉ͡

    ਖ਼ղϥϕϧ͚ͩͳ͍

    View full-size slide

  46. ޿ࠂ1 ……
    ಛ௃ྔ ॏΈ
    0.1
    -0.2
    1.0
    -0.6
    -0.3
    -0.05
    ༧ଌϞσϧ ͜ͷಛ௃ྔͷॏΈ͸…ʁ
    αΠτ1
    αΠτ2
    Ϣʔβ1
    Ϣʔβ2
    ޿ࠂ1
    ޿ࠂ2
    αΠτ1 Ϣʔβ2

    View full-size slide

  47. ……
    ಛ௃ྔ ॏΈ
    0.1
    -0.2
    1.0
    -0.6
    -0.3
    -0.05
    ༧ଌϞσϧ
    ޿ࠂ1
    αΠτ1
    αΠτ2
    Ϣʔβ1
    Ϣʔβ2
    ޿ࠂ1
    ޿ࠂ2
    αΠτ1 Ϣʔβ2

    View full-size slide

  48. ……
    ಛ௃ྔ ॏΈ
    0.1
    -0.2
    1.0
    -0.6
    -0.3
    -0.05
    ༧ଌϞσϧ
    ଍͠߹Θͤͯ
    -0.8
    ޿ࠂ1
    αΠτ1
    αΠτ2
    Ϣʔβ1
    Ϣʔβ2
    ޿ࠂ1
    ޿ࠂ2
    αΠτ1 Ϣʔβ2

    View full-size slide

  49. ……
    ಛ௃ྔ ॏΈ
    0.1
    -0.2
    1.0
    -0.6
    -0.3
    -0.05
    ༧ଌϞσϧ
    ຐ๏ͷؔ਺Λ͔͚Δͱ…
    sigmoid(-0.8)
    ޿ࠂ1
    αΠτ1 Ϣʔβ2
    αΠτ1
    αΠτ2
    Ϣʔβ1
    Ϣʔβ2
    ޿ࠂ1
    ޿ࠂ2

    View full-size slide

  50. ……
    ಛ௃ྔ ॏΈ
    0.1
    -0.2
    1.0
    -0.6
    -0.3
    -0.05
    ༧ଌϞσϧ
    CTRग़͖ͯͨʂ
    sigmoid(-0.8)
    0.31
    ※஋͸ద౰Ͱ͢
    ޿ࠂ1
    αΠτ1 Ϣʔβ2
    αΠτ1
    αΠτ2
    Ϣʔβ1
    Ϣʔβ2
    ޿ࠂ1
    ޿ࠂ2

    View full-size slide

  51. ֶशσʔλ
    …… 0
    ……
    ……
    ……
    0
    1
    1. ֶशσʔλͷ࡞੒
    ϩά
    …… 0

    View full-size slide

  52. ֶशσʔλ
    …… 0
    ……
    ……
    ……
    Ξ

    ϧ

    ΰ


    Ϧ

    ζ

    Ϝ
    0
    1
    …… 0
    2. ༧ଌϞσϧͷ࡞੒
    ༧ଌϞσϧ
    0.1
    -0.2
    1.0
    -0.6
    -0.3
    -0.05
    ಛ௃ྔ ॏΈ

    View full-size slide

  53. 3. ༧ଌ
    ……
    ༧ଌϞσϧ
    0.1
    -0.2
    1.0
    -0.6
    -0.3
    -0.05
    ಛ௃ྔ ॏΈ
    ༧ଌ͍ͨ͠

    σʔλ
    0.31
    ༧ଌCTR

    View full-size slide

  54. զʑͷγεςϜߏ੒

    View full-size slide

  55. RTBαʔό
    ϩά
    ϩάςʔϒϧ
    fluentd
    SQLͷੈք
    ֶशσʔλ
    0.1
    0.3
    0.2
    ༧ଌϞσϧ
    Treasure Data
    redis
    ίϐʔ
    ϝϞϦΩϟογϡ
    ϦΫΤετ
    Ϩεϙϯε
    CTRΛ༧ଌ
    0.31
    ༧ଌϞσϧʹ

    ΞΫηε
    όοναʔό td-client-java

    View full-size slide

  56. RTBαʔό
    ϩά
    ϩάςʔϒϧ
    fluentd
    SQLͷੈք
    ֶशσʔλ
    0.1
    0.3
    0.2
    ༧ଌϞσϧ
    Treasure Data
    redis
    ίϐʔ
    ϝϞϦΩϟογϡ
    ϦΫΤετ
    Ϩεϙϯε
    CTRΛ༧ଌ
    0.31
    ༧ଌϞσϧʹ

    ΞΫηε
    όοναʔό td-client-java
    1. ֶशσʔλͷ࡞੒

    View full-size slide

  57. RTBαʔό
    ϩά
    ϩάςʔϒϧ
    fluentd
    SQLͷੈք
    ֶशσʔλ
    0.1
    0.3
    0.2
    ༧ଌϞσϧ
    Treasure Data
    redis
    ίϐʔ
    ϝϞϦΩϟογϡ
    ϦΫΤετ
    Ϩεϙϯε
    CTRΛ༧ଌ
    0.31
    ༧ଌϞσϧʹ

    ΞΫηε
    όοναʔό td-client-java
    2. ༧ଌϞσϧͷ࡞੒

    View full-size slide

  58. RTBαʔό
    ϩά
    ϩάςʔϒϧ
    fluentd
    SQLͷੈք
    ֶशσʔλ
    0.1
    0.3
    0.2
    ༧ଌϞσϧ
    Treasure Data
    redis
    ίϐʔ
    ϝϞϦΩϟογϡ
    ϦΫΤετ
    Ϩεϙϯε
    CTRΛ༧ଌ
    0.31
    ༧ଌϞσϧʹ

    ΞΫηε
    όοναʔό td-client-java
    3. ༧ଌ

    View full-size slide

  59. ͓ؾ͖ͮͩΖ͏͔…

    View full-size slide

  60. RTBαʔό
    ϩά
    ϩάςʔϒϧ
    fluentd
    SQLͷੈք
    ֶशσʔλ
    0.1
    0.3
    0.2
    ༧ଌϞσϧ
    Treasure Data
    redis
    ίϐʔ
    ϝϞϦΩϟογϡ
    ϦΫΤετ
    Ϩεϙϯε
    CTRΛ༧ଌ
    0.31
    ༧ଌϞσϧʹ

    ΞΫηε
    όοναʔό td-client-java
    ࠷ॳͷ2εςοϓ͕

    SQLͰ׬݁ͯ͠Δʂ

    View full-size slide

  61. \ ŪƄźō… /

    View full-size slide

  62. ࠷ॳͷ2εςοϓΛSQLͰ࣮ݱ͢Δํ๏ʹ

    ؔͯ͠͸ɺHivemall։ൃऀͷ༉Ҫ͞Μ͕

    ॻ͍ͨQIitaͷૉ੖Β͍͠هࣄ͕

    ͋Γ·͢ͷͰɺͦͪΒΛࢀর͍ͯͩ͘͠͞ɻ
    Hive/HivemallΛར༻ͨ͠޿ࠂΫϦοΫεϧʔ཰(CTR)ͷਪఆ

    http://qiita.com/myui/items/f726ca3dcc48410abe45

    View full-size slide

  63. ΍ͬͱϗϯτʹຊ୊

    View full-size slide

  64. Scala͔ΒTDΛ࢖͏

    View full-size slide

  65. RTBαʔό
    ϩά
    ϩάςʔϒϧ
    fluentd
    SQLͷੈք
    ֶशσʔλ
    0.1
    0.3
    0.2
    ༧ଌϞσϧ
    Treasure Data
    redis
    ίϐʔ
    ϝϞϦΩϟογϡ
    ϦΫΤετ
    Ϩεϙϯε
    CTRΛ༧ଌ
    0.31
    ༧ଌϞσϧʹ

    ΞΫηε
    όοναʔό td-client-java
    ࠷ॳͷਤ

    View full-size slide

  66. RTBαʔό
    ϩά
    ϩάςʔϒϧ
    fluentd
    SQLͷੈք
    ֶशσʔλ
    0.1
    0.3
    0.2
    ༧ଌϞσϧ
    Treasure Data
    redis
    ίϐʔ
    ϝϞϦΩϟογϡ
    ϦΫΤετ
    Ϩεϙϯε
    CTRΛ༧ଌ
    0.31
    ༧ଌϞσϧʹ

    ΞΫηε
    όοναʔό td-client-java
    ͜ͷ෦෼

    View full-size slide

  67. td-client-java
    - Java੡ΫϥΠΞϯτϥΠϒϥϦ

    - Treasure Dataެࣜ
    - جຊతʹTDͷAPIΛhttpͰୟ͍ͯΔ͚ͩ

    View full-size slide

  68. ΫΤϦΛ౤͛ͯ

    ݁ՌΛऔಘͯ͠ΈΔ

    View full-size slide

  69. // hogeςʔϒϧͷத਎Λऔಘ
    val sql = ‘SELECT * FROM hoge’
    val client = TDClient.newClient()

    val jobRequest = TDJobRequest.newPrestoQuery(dbName, sql)

    val jobId = client.submit(jobRequest)

    val backOff = new ExponentialBackOff

    while (!client.jobStatus(jobId).getStatus.isFinished) {

    Thread.sleep(backOff.nextWaitTimeMillis)

    }
    val input = client.jobResult(jobId, TDResultFormat.MESSAGE_PACK_GZ, new
    Function[InputStream, InputStream] {

    def apply(input: InputStream) = input
    }
    val unpacker = MessagePack.newDefaultUnpacker(new
    GZIPInputStream(input))

    View full-size slide

  70. ௕͍…ʢ´ɾωɾʆʣ

    View full-size slide

  71. // hogeςʔϒϧͷத਎Λऔಘ
    val sql = ‘SELECT * FROM hoge’
    val client = TDClient.newClient()

    val jobRequest = TDJobRequest.newPrestoQuery(dbName, sql)

    val jobId = client.submit(jobRequest)

    val backOff = new ExponentialBackOff

    while (!client.jobStatus(jobId).getStatus.isFinished) {

    Thread.sleep(backOff.nextWaitTimeMillis)

    }
    val input = client.jobResult(jobId, TDResultFormat.MESSAGE_PACK_GZ, new
    Function[InputStream, InputStream] {

    def apply(input: InputStream) = input
    }
    val unpacker = MessagePack.newDefaultUnpacker(new
    GZIPInputStream(input))
    1. ΫΤϦΛ࣮ߦ

    View full-size slide

  72. // hogeςʔϒϧͷத਎Λऔಘ
    val sql = ‘SELECT * FROM hoge’
    val client = TDClient.newClient()

    val jobRequest = TDJobRequest.newPrestoQuery(dbName, sql)

    val jobId = client.submit(jobRequest)

    val backOff = new ExponentialBackOff

    while (!client.jobStatus(jobId).getStatus.isFinished) {

    Thread.sleep(backOff.nextWaitTimeMillis)

    }
    val input = client.jobResult(jobId, TDResultFormat.MESSAGE_PACK_GZ, new
    Function[InputStream, InputStream] {

    def apply(input: InputStream) = input
    }
    val unpacker = MessagePack.newDefaultUnpacker(new
    GZIPInputStream(input))
    2. ΫΤϦऴྃ·Ͱ଴ͭ

    View full-size slide

  73. // hogeςʔϒϧͷத਎Λऔಘ
    val sql = ‘SELECT * FROM hoge’
    val client = TDClient.newClient()

    val jobRequest = TDJobRequest.newPrestoQuery(dbName, sql)

    val jobId = client.submit(jobRequest)

    val backOff = new ExponentialBackOff

    while (!client.jobStatus(jobId).getStatus.isFinished) {

    Thread.sleep(backOff.nextWaitTimeMillis)

    }
    val input = client.jobResult(jobId, TDResultFormat.MESSAGE_PACK_GZ, new
    Function[InputStream, InputStream] {

    def apply(input: InputStream) = input
    }
    val unpacker = MessagePack.newDefaultUnpacker(new
    GZIPInputStream(input))
    3. ݁ՌΛऔಘ

    View full-size slide

  74. - ਖ਼௚࢖͍ʹ͍͘
    - ScalaͬΆ͘ͳ͍
    - ͦ΋ͦ΋TDͷςʔϒϧΛ௚઀ϓϩάϥϜ

    ͔ΒಡΉ͜ͱࣗମ͋·Γ૝ఆ͞Εͯͳ͍

    View full-size slide

  75. - ਖ਼௚࢖͍ʹ͍͘
    - ScalaͬΆ͘ͳ͍
    - ͦ΋ͦ΋TDͷςʔϒϧΛ௚઀ϓϩάϥϜ

    ͔ΒಡΉ͜ͱࣗମ͋·Γ૝ఆ͞Εͯͳ͍
    ͡Ό͋Ͳ͏͢Δ

    View full-size slide

  76. Result ExportΛ

    ࢖͍·͠ΐ͏

    View full-size slide

  77. Result Export
    - ΫΤϦ͕ऴΘͬͨλΠϛϯάͰ

    ݁ՌΛࢦఆͨ͠৔ॴʹసૹ͢Δ
    - సૹઌ
    - S3
    - RDB
    - Mongo
    - etc…

    View full-size slide

  78. RTBαʔό
    ϩά
    ϩάςʔϒϧ
    fluentd
    SQLͷੈք
    ֶशσʔλ
    0.1
    0.3
    0.2
    ༧ଌϞσϧ
    Treasure Data
    redis
    ίϐʔ
    ϝϞϦΩϟογϡ
    ϦΫΤετ
    Ϩεϙϯε
    CTRΛ༧ଌ
    0.31
    ༧ଌϞσϧʹ

    ΞΫηε
    όοναʔό td-client-java
    ༧ଌϞσϧͷ࡞੒࣌ʹ

    S3ʹͰ΋Export͓͚ͯ͠͹…
    S3

    View full-size slide

  79. RTBαʔό
    ϩά
    ϩάςʔϒϧ
    fluentd
    SQLͷੈք
    ֶशσʔλ
    0.1
    0.3
    0.2
    ༧ଌϞσϧ
    Treasure Data
    redis
    ίϐʔ
    ϝϞϦΩϟογϡ
    ϦΫΤετ
    Ϩεϙϯε
    CTRΛ༧ଌ
    0.31
    ༧ଌϞσϧʹ

    ΞΫηε
    όοναʔό td-client-java
    ؆୯ʂ
    S3

    View full-size slide

  80. ͦͷଞͷϢʔεέʔε
    Scala x TreasureData

    View full-size slide

  81. ScalaͰੜ੒ͨ͠σʔλΛ

    TDʹΞοϓϩʔυ

    View full-size slide

  82. Bulk Import
    - TDʹσʔλΛΞοϓϩʔυ͢ΔίϚϯυ
    - ίϚϯυϥΠϯͳͲ͔Β࢖͑Δ
    - JavaϥΠϒϥϦʹ΋ରԠؔ਺͕͋Δ

    View full-size slide

  83. ͋Εɺಈ͔ͳ͍…

    View full-size slide

  84. ͋Εɺಈ͔ͳ͍…
    ໰͍߹ΘͤΔ

    View full-size slide

  85. ͱ͍͏Θ͚ͰEmbulk

    ࢖͍·͠ΐ͏

    View full-size slide

  86. - Ϗοάσʔλ༻σʔλϩʔμ
    - fluentdͷϏοάσʔλ൛Έ͍ͨͳײ͡
    - TD͕։ൃ͍ͯ͠Δ
    - Φʔϓϯιʔε
    - Ϋδϥ γϟν͕͔Θ͍͍

    View full-size slide

  87. Αʔ͠Scala͔Β

    Embulk࢖͏ͧʔ…

    View full-size slide

  88. ͦ͜·Ͱ͸Ͱ͖·ͤΜͰͨ͠

    ʢ࣌ؒ੾Εʣ

    View full-size slide

  89. 1. TDͱHivemallͰCTR༧ଌϞσϧ࡞੒·Ͱ

    SQLͰ׬݁͢ΔΑʂ
    2. Scala͔ΒTDͷςʔϒϧಡΉͷେม
    => Result ExportΛ͏·͘࢖͓͏
    3. Scala͔ΒTDʹσʔλ্͛Δͷ͸EmbulkͰ
    => ୭͔΍Γํڭ͍͑ͯͩ͘͞

    View full-size slide

  90. \ ͋Γ͕ͱ͏͍͟͝·ͨ͠ /

    View full-size slide