アドテク企業の本番環境からTD使ってみた / Treasure Data Tech Talk 20160425

Transcript

1. ΞυςΫاۀͷ
 ຊ൪؀ڥ͔ΒTD࢖ͬͯΈͨ Scala x TreasureData ΦϯϥΠϯCTR༧ଌ

2. ञҪ ਸࢸ - 2016/01- F@N Communicationsגࣜձࣾ - ๭CAࣾΞυςΫελδΦͰΠϯλʔϯͱ͔ͯͨ͠ - ScalaΤϯδχΞ

   (ଞʹRuby, Python, JS, Go…) - ػցֶश΋΍ΔΑ - Slack & Raspberry PiͰΤΞίϯ͚ͭͨΓ

3. ର৅ऀ - ػցֶश or CTR༧ଌʹڵຯ͕͋Δਓ - Scala͔ΒTreasureDataΛ࢖ͬͯΈ͍ͨਓ

4. ΞυςΫۀքͷதͰ΋
 DSPͱ͍͏ͷΛ࡞ͬͯ·͢

5. What’s DSP?

6. What DSPs do SSP DSP ͜ͷαΠτʹϦΫΤετ
 དྷͯΔ͚Ͳ޿ࠂग़͞΁Μʁ

7. What DSPs do SSP DSP ͦ͜΍ͬͨΒ
 0.1ԁͳΒങ͏Θ

8. What DSPs do SSP DSP Αͬ͠Ό͋Μͨʹ
 ചͬͨΖ ଞͷձࣾͷํ͕
 ͍͍஋ஈ͚ͭͯ͘ΕͨΘ

9. What DSPs do SSP DSP Αͬ͠Ό͋Μͨʹ
 ചͬͨΖ ଞͷձࣾͷํ͕
 ͍͍஋ஈ͚ͭͯ͘ΕͨΘ

10. ͜ͷؒΘ͔ͣ50ms

11. ࠓͷҰ࿈ͷ΍ΓͱΓΛ
 RTBͱ͍͏Α RTB: Real-Time Bidding ςετʹग़Δͧʂ

12. RTBͷಛ௃ େྔΞΫηε ɾඵؒ5ສͱ͔ ૣ͍Ϩεϙϯε ɾ100msҎ಺ʹฦ͞ͳ͍ͱΦʔΫγϣϯʹࢀՃͰ͖ͳ͍

13. ຊ୊

14. ސ٬ʢ޿ࠂओʣʹͱͬͯ
 ΑΓՁ஋ͷ͋ΔDSPΛ࡞Γ͍ͨʂ

15. ΫϦοΫ཰(CTR)ͷ
 ༧ଌ͕େࣄ CTR: Click Through Rate

16. DSP Site A Site B ޿ࠂग़͞΁Μʁ ޿ࠂग़͞΁Μʁ

17. DSP Site A
 (CTR=0.1%) Site B
 (CTR=1%) 0.5ԁͳΒങ͏Ͱ 5ԁͳΒങ͏Ͱ

18. CTR͕Θ͔Δͱ
 దਖ਼ͳ஋ஈͰೖࡳͰ͖Δ ΫϦοΫ཰

19. RTBͷ࣌ʹΘ͔͍ͬͯΔ৘ใ - ϢʔβID - αΠτID - ޿ࠂID - etc…ʢͨ͘͞Μʣ

20. - ϢʔβID - αΠτID - ޿ࠂID - etc…ʢͨ͘͞Μʣ ͜ΕΒͷ৘ใ͔Β
 CTRΛ༧ଌͯ͠ΈΑ͏ʂ

21. ͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ

22. ͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ Ͱ΋ͦͷϢʔβͷCTR͸1%ͩͥ

23. ͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ Ͱ΋ͦͷϢʔβͷCTR͸1%ͩͥ ͡Ό͋ؒΛऔͬͯ0.5%ͬͯ͜ͱʹ͢Δʁ

24. ͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ Ͱ΋ͦͷϢʔβͷCTR͸1%ͩͥ ͡Ό͋ؒΛऔͬͯ0.5%ͬͯ͜ͱʹ͢Δʁ Ϣʔβ͝ͱͷ৘ใͷํ͕ਖ਼֬ͩΖ
 0.8%͘Β͍͡ΌͶ

25. ͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ Ͱ΋ͦͷϢʔβͷCTR͸1%ͩͥ ͡Ό͋ؒΛऔͬͯ0.5%ͬͯ͜ͱʹ͢Δʁ Ϣʔβ͝ͱͷ৘ใͷํ͕ਖ਼֬ͩΖ
 0.8%͘Β͍͡ΌͶ ͋ɺ޿ࠂ͝ͱͷCTR΋ߟ͑ͳ͍ͱ…

26. ͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ Ͱ΋ͦͷϢʔβͷCTR͸1%ͩͥ ͡Ό͋ؒΛऔͬͯ0.5%ͬͯ͜ͱʹ͢Δʁ Ϣʔβ͝ͱͷ৘ใͷํ͕ਖ਼֬ͩΖ
 0.8%͘Β͍͡ΌͶ ͋ɺ޿ࠂͷCTR΋ߟ͑ͳ͍ͱ… ߟ͑ग़͢ͱେม

27. - ϢʔβID - αΠτID - ޿ࠂID - etc…ʢͨ͘͞Μʣ ͪͳΈʹɺ͜ͷΑ͏ͳ
 ༧ଌͷࡐྉʹͳΔ৘ใΛ


    ಛ௃ྔͱ͍͏Α

28. Machine Learning ػցֶश

29. Machine LearningͳΒ…

30. Machine LearningͳΒ… - ෳ਺ͷಛ௃ྔʹରͯ͠ (ϢʔβID, αΠτID…)

31. Machine LearningͳΒ… - ෳ਺ͷಛ௃ྔʹରͯ͠ (ϢʔβID, αΠτID…) - ਺ֶతࠜڌʹج͍ͮͯ

32. Machine LearningͳΒ… - ෳ਺ͷಛ௃ྔʹରͯ͠ (ϢʔβID, αΠτID…) - ਺ֶతࠜڌʹج͍ͮͯ - ࣗಈͰ

33. Machine LearningͳΒ… - ෳ਺ͷಛ௃ྔʹରͯ͠ (ϢʔβID, αΠτID…) - ਺ֶతࠜڌʹج͍ͮͯ - ࣗಈͰ

    CTR͕༧ଌͰ͖Δʂ

34. ػցֶशͬͯͲ͏΍Δͷʁ

35. ࠓճͷख๏ɻৄ͍͠ਓ޲͚ - ڭࢣ͋Γֶश - ڭࢣσʔλ͸ϩά͔Β࡞੒ - ࠓճ͸ϩδεςΟοΫճؼͷઆ໌Ͱ͢ ஌Βͳ͍ਓ͸ಡΈඈ͹ͯ͠OK

36. ػցֶशͷجຊ 1. ֶशσʔλͷ࡞੒ 2. ༧ଌϞσϧͷ࡞੒ 3. ༧ଌ

37. 1. ֶशσʔλͷ࡞੒

38. Ұൠతͳֶशσʔλ 1 1 1 …… 0 ಛ௃ྔ1 ಛ௃ྔ2 ಛ௃ྔ3 ……

    ਖ਼ղϥϕϧ 2 3 2 …… 0 2 2 3 …… 1 ……

39. CTR༧ଌͷ৔߹ 1 1 1 …… 0 αΠτ Ϣʔβ ޿ࠂ ……

    ΫϦοΫ
 ͞Ε͔ͨ 2 3 2 …… 0 2 2 3 …… 1 …… 1ߦ͕
 1ΠϯϓϨογϣϯ

40. CSVͰද͢ͱ… # αΠτ, Ϣʔβ, ޿ࠂ, …, ਖ਼ղϥϕϧ site_1, user_1, campaign_1,

    …, 0 site_2, user_3, campaign_2, …, 0 site_2, user_2, campaign_3, …, 1 …

41. 2. ༧ଌϞσϧͷ࡞੒

42. ֶशσʔλ …… 0 …… …… …… Ξ
 ϧ
 ΰ
 


    Ϧ
 ζ
 Ϝ ༧ଌϞσϧ 0 1 ࠓճ͸
 ϩδεςΟοΫճؼ …… 0

43. αΠτ1 αΠτ2 Ϣʔβ1 Ϣʔβ2 ޿ࠂ1 ޿ࠂ2 ಛ௃ྔ ॏΈ ༧ଌϞσϧͷத਎

44. ಛ௃ྔ ॏΈ 0.1 -0.2 1.0 -0.6 -0.3 -0.05 αΠτ1 αΠτ2

    Ϣʔβ1 Ϣʔβ2 ޿ࠂ1 ޿ࠂ2

45. CSVͰද͢ͱ… # ಛ௃ྔ, ॏΈ site_1, 0.1 site_2, -0.2 user_1, 1.0

    user_2, -0.6 campaign_1,-0.3 campaign_2,-0.05 …

46. 3. ༧ଌ

47. CTRΛ஌Γ͍ͨσʔλ αΠτ1 Ϣʔβ2 ޿ࠂ1 …… ֶशσʔλͱ΄΅ಉ͡
 ਖ਼ղϥϕϧ͚ͩͳ͍

48. ޿ࠂ1 …… ಛ௃ྔ ॏΈ 0.1 -0.2 1.0 -0.6 -0.3 -0.05

    ༧ଌϞσϧ ͜ͷಛ௃ྔͷॏΈ͸…ʁ αΠτ1 αΠτ2 Ϣʔβ1 Ϣʔβ2 ޿ࠂ1 ޿ࠂ2 αΠτ1 Ϣʔβ2

49. …… ಛ௃ྔ ॏΈ 0.1 -0.2 1.0 -0.6 -0.3 -0.05 ༧ଌϞσϧ

    ޿ࠂ1 αΠτ1 αΠτ2 Ϣʔβ1 Ϣʔβ2 ޿ࠂ1 ޿ࠂ2 αΠτ1 Ϣʔβ2

50. …… ಛ௃ྔ ॏΈ 0.1 -0.2 1.0 -0.6 -0.3 -0.05 ༧ଌϞσϧ

    ଍͠߹Θͤͯ -0.8 ޿ࠂ1 αΠτ1 αΠτ2 Ϣʔβ1 Ϣʔβ2 ޿ࠂ1 ޿ࠂ2 αΠτ1 Ϣʔβ2

51. …… ಛ௃ྔ ॏΈ 0.1 -0.2 1.0 -0.6 -0.3 -0.05 ༧ଌϞσϧ

    ຐ๏ͷؔ਺Λ͔͚Δͱ… sigmoid(-0.8) ޿ࠂ1 αΠτ1 Ϣʔβ2 αΠτ1 αΠτ2 Ϣʔβ1 Ϣʔβ2 ޿ࠂ1 ޿ࠂ2

52. …… ಛ௃ྔ ॏΈ 0.1 -0.2 1.0 -0.6 -0.3 -0.05 ༧ଌϞσϧ

    CTRग़͖ͯͨʂ sigmoid(-0.8) 0.31 ※஋͸ద౰Ͱ͢ ޿ࠂ1 αΠτ1 Ϣʔβ2 αΠτ1 αΠτ2 Ϣʔβ1 Ϣʔβ2 ޿ࠂ1 ޿ࠂ2

53. ͓͞Β͍

54. ֶशσʔλ …… 0 …… …… …… 0 1 1. ֶशσʔλͷ࡞੒

    ϩά …… 0

55. ֶशσʔλ …… 0 …… …… …… Ξ
 ϧ
 ΰ
 


    Ϧ
 ζ
 Ϝ 0 1 …… 0 2. ༧ଌϞσϧͷ࡞੒ ༧ଌϞσϧ 0.1 -0.2 1.0 -0.6 -0.3 -0.05 ಛ௃ྔ ॏΈ

56. 3. ༧ଌ …… ༧ଌϞσϧ 0.1 -0.2 1.0 -0.6 -0.3 -0.05

    ಛ௃ྔ ॏΈ ༧ଌ͍ͨ͠
 σʔλ 0.31 ༧ଌCTR

57. զʑͷγεςϜߏ੒

58. RTBαʔό ϩά ϩάςʔϒϧ fluentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ

    Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ
 ΞΫηε όοναʔό td-client-java

59. RTBαʔό ϩά ϩάςʔϒϧ fluentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ

    Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ
 ΞΫηε όοναʔό td-client-java 1. ֶशσʔλͷ࡞੒

60. RTBαʔό ϩά ϩάςʔϒϧ fluentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ

    Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ
 ΞΫηε όοναʔό td-client-java 2. ༧ଌϞσϧͷ࡞੒

61. RTBαʔό ϩά ϩάςʔϒϧ fluentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ

    Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ
 ΞΫηε όοναʔό td-client-java 3. ༧ଌ

62. ͓ؾ͖ͮͩΖ͏͔…

63. RTBαʔό ϩά ϩάςʔϒϧ fluentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ

    Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ
 ΞΫηε όοναʔό td-client-java ࠷ॳͷ2εςοϓ͕
 SQLͰ׬݁ͯ͠Δʂ

64. \ ŪƄźō… /

65. ࠷ॳͷ2εςοϓΛSQLͰ࣮ݱ͢Δํ๏ʹ
 ؔͯ͠͸ɺHivemall։ൃऀͷ༉Ҫ͞Μ͕
 ॻ͍ͨQIitaͷૉ੖Β͍͠هࣄ͕
 ͋Γ·͢ͷͰɺͦͪΒΛࢀর͍ͯͩ͘͠͞ɻ Hive/HivemallΛར༻ͨ͠޿ࠂΫϦοΫεϧʔ཰(CTR)ͷਪఆ
 http://qiita.com/myui/items/f726ca3dcc48410abe45

66. ΍ͬͱϗϯτʹຊ୊

67. Scala͔ΒTDΛ࢖͏

68. RTBαʔό ϩά ϩάςʔϒϧ fluentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ

    Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ
 ΞΫηε όοναʔό td-client-java ࠷ॳͷਤ

69. RTBαʔό ϩά ϩάςʔϒϧ fluentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ

    Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ
 ΞΫηε όοναʔό td-client-java ͜ͷ෦෼

70. td-client-java - Java੡ΫϥΠΞϯτϥΠϒϥϦ
 - Treasure Dataެࣜ - جຊతʹTDͷAPIΛhttpͰୟ͍ͯΔ͚ͩ

71. ΫΤϦΛ౤͛ͯ
 ݁ՌΛऔಘͯ͠ΈΔ

72. // hogeςʔϒϧͷத਎Λऔಘ val sql = ‘SELECT * FROM hoge’ val

    client = TDClient.newClient()
 val jobRequest = TDJobRequest.newPrestoQuery(dbName, sql)
 val jobId = client.submit(jobRequest)
 val backOff = new ExponentialBackOff
 while (!client.jobStatus(jobId).getStatus.isFinished) {
 Thread.sleep(backOff.nextWaitTimeMillis)
 } val input = client.jobResult(jobId, TDResultFormat.MESSAGE_PACK_GZ, new Function[InputStream, InputStream] {
 def apply(input: InputStream) = input } val unpacker = MessagePack.newDefaultUnpacker(new GZIPInputStream(input))

73. ௕͍…ʢ´ɾωɾʆʣ

74. // hogeςʔϒϧͷத਎Λऔಘ val sql = ‘SELECT * FROM hoge’ val

    client = TDClient.newClient()
 val jobRequest = TDJobRequest.newPrestoQuery(dbName, sql)
 val jobId = client.submit(jobRequest)
 val backOff = new ExponentialBackOff
 while (!client.jobStatus(jobId).getStatus.isFinished) {
 Thread.sleep(backOff.nextWaitTimeMillis)
 } val input = client.jobResult(jobId, TDResultFormat.MESSAGE_PACK_GZ, new Function[InputStream, InputStream] {
 def apply(input: InputStream) = input } val unpacker = MessagePack.newDefaultUnpacker(new GZIPInputStream(input)) 1. ΫΤϦΛ࣮ߦ

75. // hogeςʔϒϧͷத਎Λऔಘ val sql = ‘SELECT * FROM hoge’ val

    client = TDClient.newClient()
 val jobRequest = TDJobRequest.newPrestoQuery(dbName, sql)
 val jobId = client.submit(jobRequest)
 val backOff = new ExponentialBackOff
 while (!client.jobStatus(jobId).getStatus.isFinished) {
 Thread.sleep(backOff.nextWaitTimeMillis)
 } val input = client.jobResult(jobId, TDResultFormat.MESSAGE_PACK_GZ, new Function[InputStream, InputStream] {
 def apply(input: InputStream) = input } val unpacker = MessagePack.newDefaultUnpacker(new GZIPInputStream(input)) 2. ΫΤϦऴྃ·Ͱ଴ͭ

76. // hogeςʔϒϧͷத਎Λऔಘ val sql = ‘SELECT * FROM hoge’ val

    client = TDClient.newClient()
 val jobRequest = TDJobRequest.newPrestoQuery(dbName, sql)
 val jobId = client.submit(jobRequest)
 val backOff = new ExponentialBackOff
 while (!client.jobStatus(jobId).getStatus.isFinished) {
 Thread.sleep(backOff.nextWaitTimeMillis)
 } val input = client.jobResult(jobId, TDResultFormat.MESSAGE_PACK_GZ, new Function[InputStream, InputStream] {
 def apply(input: InputStream) = input } val unpacker = MessagePack.newDefaultUnpacker(new GZIPInputStream(input)) 3. ݁ՌΛऔಘ

77. - ਖ਼௚࢖͍ʹ͍͘ - ScalaͬΆ͘ͳ͍ - ͦ΋ͦ΋TDͷςʔϒϧΛ௚઀ϓϩάϥϜ
 ͔ΒಡΉ͜ͱࣗମ͋·Γ૝ఆ͞Εͯͳ͍

78. - ਖ਼௚࢖͍ʹ͍͘ - ScalaͬΆ͘ͳ͍ - ͦ΋ͦ΋TDͷςʔϒϧΛ௚઀ϓϩάϥϜ
 ͔ΒಡΉ͜ͱࣗମ͋·Γ૝ఆ͞Εͯͳ͍ ͡Ό͋Ͳ͏͢Δ

79. Result ExportΛ
 ࢖͍·͠ΐ͏

80. Result Export - ΫΤϦ͕ऴΘͬͨλΠϛϯάͰ
 ݁ՌΛࢦఆͨ͠৔ॴʹసૹ͢Δ - సૹઌ - S3 -

    RDB - Mongo - etc…

81. RTBαʔό ϩά ϩάςʔϒϧ fluentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ

    Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ
 ΞΫηε όοναʔό td-client-java ༧ଌϞσϧͷ࡞੒࣌ʹ
 S3ʹͰ΋Export͓͚ͯ͠͹… S3

82. RTBαʔό ϩά ϩάςʔϒϧ fluentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ

    Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ
 ΞΫηε όοναʔό td-client-java ؆୯ʂ S3

83. ͦͷଞͷϢʔεέʔε Scala x TreasureData

84. ScalaͰੜ੒ͨ͠σʔλΛ
 TDʹΞοϓϩʔυ

85. Bulk Import - TDʹσʔλΛΞοϓϩʔυ͢ΔίϚϯυ - ίϚϯυϥΠϯͳͲ͔Β࢖͑Δ - JavaϥΠϒϥϦʹ΋ରԠؔ਺͕͋Δ

86. ͋Εɺಈ͔ͳ͍…

87. ͋Εɺಈ͔ͳ͍… ໰͍߹ΘͤΔ

88. None

89. ʂʁ

90. None

91. ͱ͍͏Θ͚ͰEmbulk
 ࢖͍·͠ΐ͏

92. - Ϗοάσʔλ༻σʔλϩʔμ - fluentdͷϏοάσʔλ൛Έ͍ͨͳײ͡ - TD͕։ൃ͍ͯ͠Δ - Φʔϓϯιʔε - Ϋδϥ

    γϟν͕͔Θ͍͍

93. Αʔ͠Scala͔Β
 Embulk࢖͏ͧʔ…

94. ͦ͜·Ͱ͸Ͱ͖·ͤΜͰͨ͠
 ʢ࣌ؒ੾Εʣ

95. ·ͱΊ

96. 1. TDͱHivemallͰCTR༧ଌϞσϧ࡞੒·Ͱ
 SQLͰ׬݁͢ΔΑʂ 2. Scala͔ΒTDͷςʔϒϧಡΉͷେม => Result ExportΛ͏·͘࢖͓͏ 3. Scala͔ΒTDʹσʔλ্͛Δͷ͸EmbulkͰ

    => ୭͔΍Γํڭ͍͑ͯͩ͘͞

97. \ ͋Γ͕ͱ͏͍͟͝·ͨ͠ /

98. None