アドテク企業の本番環境からTD使ってみた / Treasure Data Tech Talk 20160425

by Takayuki Sakai

Slide 1

Slide 1 text

ΞυςΫاۀͷ  ຊ൪؀ڥ͔ΒTD࢖ͬͯΈͨ Scala x TreasureData ΦϯϥΠϯCTR༧ଌ

Slide 2

Slide 2 text

ञҪ ਸࢸ - 2016/01- F@N Communicationsגࣜձࣾ - ๭CAࣾΞυςΫελδΦͰΠϯλʔϯͱ͔ͯͨ͠ - ScalaΤϯδχΞ (ଞʹRuby, Python, JS, Go…) - ػցֶश΋΍ΔΑ - Slack & Raspberry PiͰΤΞίϯ͚ͭͨΓ

Slide 3

Slide 3 text

ର৅ऀ - ػցֶश or CTR༧ଌʹڵຯ͕͋Δਓ - Scala͔ΒTreasureDataΛ࢖ͬͯΈ͍ͨਓ

Slide 4

Slide 4 text

ΞυςΫۀքͷதͰ΋  DSPͱ͍͏ͷΛ࡞ͬͯ·͢

Slide 5

Slide 5 text

What’s DSP?

Slide 6

Slide 6 text

What DSPs do SSP DSP ͜ͷαΠτʹϦΫΤετ  དྷͯΔ͚Ͳ޿ࠂग़͞΁Μʁ

Slide 7

Slide 7 text

What DSPs do SSP DSP ͦ͜΍ͬͨΒ  0.1ԁͳΒങ͏Θ

Slide 8

Slide 8 text

What DSPs do SSP DSP Αͬ͠Ό͋Μͨʹ  ചͬͨΖ ଞͷձࣾͷํ͕  ͍͍஋ஈ͚ͭͯ͘ΕͨΘ

Slide 9

Slide 9 text

What DSPs do SSP DSP Αͬ͠Ό͋Μͨʹ  ചͬͨΖ ଞͷձࣾͷํ͕  ͍͍஋ஈ͚ͭͯ͘ΕͨΘ

Slide 10

Slide 10 text

͜ͷؒΘ͔ͣ50ms

Slide 11

Slide 11 text

ࠓͷҰ࿈ͷ΍ΓͱΓΛ  RTBͱ͍͏Α RTB: Real-Time Bidding ςετʹग़Δͧʂ

Slide 12

Slide 12 text

RTBͷಛ௃ େྔΞΫηε ɾඵؒ5ສͱ͔ ૣ͍Ϩεϙϯε ɾ100msҎ಺ʹฦ͞ͳ͍ͱΦʔΫγϣϯʹࢀՃͰ͖ͳ͍

Slide 13

Slide 13 text

ຊ୊

Slide 14

Slide 14 text

ސ٬ʢ޿ࠂओʣʹͱͬͯ  ΑΓՁ஋ͷ͋ΔDSPΛ࡞Γ͍ͨʂ

Slide 15

Slide 15 text

ΫϦοΫ཰(CTR)ͷ  ༧ଌ͕େࣄ CTR: Click Through Rate

Slide 16

Slide 16 text

DSP Site A Site B ޿ࠂग़͞΁Μʁ ޿ࠂग़͞΁Μʁ

Slide 17

Slide 17 text

DSP Site A  (CTR=0.1%) Site B  (CTR=1%) 0.5ԁͳΒങ͏Ͱ 5ԁͳΒങ͏Ͱ

Slide 18

Slide 18 text

CTR͕Θ͔Δͱ  దਖ਼ͳ஋ஈͰೖࡳͰ͖Δ ΫϦοΫ཰

Slide 19

Slide 19 text

RTBͷ࣌ʹΘ͔͍ͬͯΔ৘ใ - ϢʔβID - αΠτID - ޿ࠂID - etc…ʢͨ͘͞Μʣ

Slide 20

Slide 20 text

- ϢʔβID - αΠτID - ޿ࠂID - etc…ʢͨ͘͞Μʣ ͜ΕΒͷ৘ใ͔Β  CTRΛ༧ଌͯ͠ΈΑ͏ʂ

Slide 21

Slide 21 text

͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ

Slide 22

Slide 22 text

͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ Ͱ΋ͦͷϢʔβͷCTR͸1%ͩͥ

Slide 23

Slide 23 text

͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ Ͱ΋ͦͷϢʔβͷCTR͸1%ͩͥ ͡Ό͋ؒΛऔͬͯ0.5%ͬͯ͜ͱʹ͢Δʁ

Slide 24

Slide 24 text

͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ Ͱ΋ͦͷϢʔβͷCTR͸1%ͩͥ ͡Ό͋ؒΛऔͬͯ0.5%ͬͯ͜ͱʹ͢Δʁ Ϣʔβ͝ͱͷ৘ใͷํ͕ਖ਼֬ͩΖ  0.8%͘Β͍͡ΌͶ

Slide 25

Slide 25 text

͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ Ͱ΋ͦͷϢʔβͷCTR͸1%ͩͥ ͡Ό͋ؒΛऔͬͯ0.5%ͬͯ͜ͱʹ͢Δʁ Ϣʔβ͝ͱͷ৘ใͷํ͕ਖ਼֬ͩΖ  0.8%͘Β͍͡ΌͶ ͋ɺ޿ࠂ͝ͱͷCTR΋ߟ͑ͳ͍ͱ…

Slide 26

Slide 26 text

͜ͷαΠτͰͷࠓ·ͰͷCTR͸0.1%ͩΑ Ͱ΋ͦͷϢʔβͷCTR͸1%ͩͥ ͡Ό͋ؒΛऔͬͯ0.5%ͬͯ͜ͱʹ͢Δʁ Ϣʔβ͝ͱͷ৘ใͷํ͕ਖ਼֬ͩΖ  0.8%͘Β͍͡ΌͶ ͋ɺ޿ࠂͷCTR΋ߟ͑ͳ͍ͱ… ߟ͑ग़͢ͱେม

Slide 27

Slide 27 text

- ϢʔβID - αΠτID - ޿ࠂID - etc…ʢͨ͘͞Μʣ ͪͳΈʹɺ͜ͷΑ͏ͳ  ༧ଌͷࡐྉʹͳΔ৘ใΛ  ಛ௃ྔͱ͍͏Α

Slide 28

Slide 28 text

Machine Learning ػցֶश

Slide 29

Slide 29 text

Machine LearningͳΒ…

Slide 30

Slide 30 text

Machine LearningͳΒ… - ෳ਺ͷಛ௃ྔʹରͯ͠ (ϢʔβID, αΠτID…)

Slide 31

Slide 31 text

Machine LearningͳΒ… - ෳ਺ͷಛ௃ྔʹରͯ͠ (ϢʔβID, αΠτID…) - ਺ֶతࠜڌʹج͍ͮͯ

Slide 32

Slide 32 text

Machine LearningͳΒ… - ෳ਺ͷಛ௃ྔʹରͯ͠ (ϢʔβID, αΠτID…) - ਺ֶతࠜڌʹج͍ͮͯ - ࣗಈͰ

Slide 33

Slide 33 text

Machine LearningͳΒ… - ෳ਺ͷಛ௃ྔʹରͯ͠ (ϢʔβID, αΠτID…) - ਺ֶతࠜڌʹج͍ͮͯ - ࣗಈͰ CTR͕༧ଌͰ͖Δʂ

Slide 34

Slide 34 text

ػցֶशͬͯͲ͏΍Δͷʁ

Slide 35

Slide 35 text

ࠓճͷख๏ɻৄ͍͠ਓ޲͚ - ڭࢣ͋Γֶश - ڭࢣσʔλ͸ϩά͔Β࡞੒ - ࠓճ͸ϩδεςΟοΫճؼͷઆ໌Ͱ͢ ஌Βͳ͍ਓ͸ಡΈඈ͹ͯ͠OK

Slide 36

Slide 36 text

ػցֶशͷجຊ 1. ֶशσʔλͷ࡞੒ 2. ༧ଌϞσϧͷ࡞੒ 3. ༧ଌ

Slide 37

Slide 37 text

1. ֶशσʔλͷ࡞੒

Slide 38

Slide 38 text

Ұൠతͳֶशσʔλ 1 1 1 …… 0 ಛ௃ྔ1 ಛ௃ྔ2 ಛ௃ྔ3 …… ਖ਼ղϥϕϧ 2 3 2 …… 0 2 2 3 …… 1 ……

Slide 39

Slide 39 text

CTR༧ଌͷ৔߹ 1 1 1 …… 0 αΠτ Ϣʔβ ޿ࠂ …… ΫϦοΫ  ͞Ε͔ͨ 2 3 2 …… 0 2 2 3 …… 1 …… 1ߦ͕  1ΠϯϓϨογϣϯ

Slide 40

Slide 40 text

CSVͰද͢ͱ… # αΠτ, Ϣʔβ, ޿ࠂ, …, ਖ਼ղϥϕϧ site_1, user_1, campaign_1, …, 0 site_2, user_3, campaign_2, …, 0 site_2, user_2, campaign_3, …, 1 …

Slide 41

Slide 41 text

2. ༧ଌϞσϧͷ࡞੒

Slide 42

Slide 42 text

ֶशσʔλ …… 0 …… …… …… Ξ  ϧ  ΰ    Ϧ  ζ  Ϝ ༧ଌϞσϧ 0 1 ࠓճ͸  ϩδεςΟοΫճؼ …… 0

Slide 43

Slide 43 text

αΠτ1 αΠτ2 Ϣʔβ1 Ϣʔβ2 ޿ࠂ1 ޿ࠂ2 ಛ௃ྔ ॏΈ ༧ଌϞσϧͷத਎

Slide 44

Slide 44 text

ಛ௃ྔ ॏΈ 0.1 -0.2 1.0 -0.6 -0.3 -0.05 αΠτ1 αΠτ2 Ϣʔβ1 Ϣʔβ2 ޿ࠂ1 ޿ࠂ2

Slide 45

Slide 45 text

CSVͰද͢ͱ… # ಛ௃ྔ, ॏΈ site_1, 0.1 site_2, -0.2 user_1, 1.0 user_2, -0.6 campaign_1,-0.3 campaign_2,-0.05 …

Slide 46

Slide 46 text

3. ༧ଌ

Slide 47

Slide 47 text

CTRΛ஌Γ͍ͨσʔλ αΠτ1 Ϣʔβ2 ޿ࠂ1 …… ֶशσʔλͱ΄΅ಉ͡  ਖ਼ղϥϕϧ͚ͩͳ͍

Slide 48

Slide 48 text

޿ࠂ1 …… ಛ௃ྔ ॏΈ 0.1 -0.2 1.0 -0.6 -0.3 -0.05 ༧ଌϞσϧ ͜ͷಛ௃ྔͷॏΈ͸…ʁ αΠτ1 αΠτ2 Ϣʔβ1 Ϣʔβ2 ޿ࠂ1 ޿ࠂ2 αΠτ1 Ϣʔβ2

Slide 49

Slide 49 text

…… ಛ௃ྔ ॏΈ 0.1 -0.2 1.0 -0.6 -0.3 -0.05 ༧ଌϞσϧ ޿ࠂ1 αΠτ1 αΠτ2 Ϣʔβ1 Ϣʔβ2 ޿ࠂ1 ޿ࠂ2 αΠτ1 Ϣʔβ2

Slide 50

Slide 50 text

…… ಛ௃ྔ ॏΈ 0.1 -0.2 1.0 -0.6 -0.3 -0.05 ༧ଌϞσϧ ଍͠߹Θͤͯ -0.8 ޿ࠂ1 αΠτ1 αΠτ2 Ϣʔβ1 Ϣʔβ2 ޿ࠂ1 ޿ࠂ2 αΠτ1 Ϣʔβ2

Slide 51

Slide 51 text

…… ಛ௃ྔ ॏΈ 0.1 -0.2 1.0 -0.6 -0.3 -0.05 ༧ଌϞσϧ ຐ๏ͷؔ਺Λ͔͚Δͱ… sigmoid(-0.8) ޿ࠂ1 αΠτ1 Ϣʔβ2 αΠτ1 αΠτ2 Ϣʔβ1 Ϣʔβ2 ޿ࠂ1 ޿ࠂ2

Slide 52

Slide 52 text

…… ಛ௃ྔ ॏΈ 0.1 -0.2 1.0 -0.6 -0.3 -0.05 ༧ଌϞσϧ CTRग़͖ͯͨʂ sigmoid(-0.8) 0.31 ※஋͸ద౰Ͱ͢ ޿ࠂ1 αΠτ1 Ϣʔβ2 αΠτ1 αΠτ2 Ϣʔβ1 Ϣʔβ2 ޿ࠂ1 ޿ࠂ2

Slide 53

Slide 53 text

͓͞Β͍

Slide 54

Slide 54 text

ֶशσʔλ …… 0 …… …… …… 0 1 1. ֶशσʔλͷ࡞੒ ϩά …… 0

Slide 55

Slide 55 text

ֶशσʔλ …… 0 …… …… …… Ξ  ϧ  ΰ    Ϧ  ζ  Ϝ 0 1 …… 0 2. ༧ଌϞσϧͷ࡞੒ ༧ଌϞσϧ 0.1 -0.2 1.0 -0.6 -0.3 -0.05 ಛ௃ྔ ॏΈ

Slide 56

Slide 56 text

3. ༧ଌ …… ༧ଌϞσϧ 0.1 -0.2 1.0 -0.6 -0.3 -0.05 ಛ௃ྔ ॏΈ ༧ଌ͍ͨ͠  σʔλ 0.31 ༧ଌCTR

Slide 57

Slide 57 text

զʑͷγεςϜߏ੒

Slide 58

Slide 58 text

RTBαʔό ϩά ϩάςʔϒϧ ﬂuentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ  ΞΫηε όοναʔό td-client-java

Slide 59

Slide 59 text

RTBαʔό ϩά ϩάςʔϒϧ ﬂuentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ  ΞΫηε όοναʔό td-client-java 1. ֶशσʔλͷ࡞੒

Slide 60

Slide 60 text

RTBαʔό ϩά ϩάςʔϒϧ ﬂuentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ  ΞΫηε όοναʔό td-client-java 2. ༧ଌϞσϧͷ࡞੒

Slide 61

Slide 61 text

RTBαʔό ϩά ϩάςʔϒϧ ﬂuentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ  ΞΫηε όοναʔό td-client-java 3. ༧ଌ

Slide 62

Slide 62 text

͓ؾ͖ͮͩΖ͏͔…

Slide 63

Slide 63 text

RTBαʔό ϩά ϩάςʔϒϧ ﬂuentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ  ΞΫηε όοναʔό td-client-java ࠷ॳͷ2εςοϓ͕  SQLͰ׬݁ͯ͠Δʂ

Slide 64

Slide 64 text

\ ŪƄźō… /

Slide 65

Slide 65 text

࠷ॳͷ2εςοϓΛSQLͰ࣮ݱ͢Δํ๏ʹ  ؔͯ͠͸ɺHivemall։ൃऀͷ༉Ҫ͞Μ͕  ॻ͍ͨQIitaͷૉ੖Β͍͠هࣄ͕  ͋Γ·͢ͷͰɺͦͪΒΛࢀর͍ͯͩ͘͠͞ɻ Hive/HivemallΛར༻ͨ͠޿ࠂΫϦοΫεϧʔ཰(CTR)ͷਪఆ  http://qiita.com/myui/items/f726ca3dcc48410abe45

Slide 66

Slide 66 text

΍ͬͱϗϯτʹຊ୊

Slide 67

Slide 67 text

Scala͔ΒTDΛ࢖͏

Slide 68

Slide 68 text

RTBαʔό ϩά ϩάςʔϒϧ ﬂuentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ  ΞΫηε όοναʔό td-client-java ࠷ॳͷਤ

Slide 69

Slide 69 text

RTBαʔό ϩά ϩάςʔϒϧ ﬂuentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ  ΞΫηε όοναʔό td-client-java ͜ͷ෦෼

Slide 70

Slide 70 text

td-client-java - Java੡ΫϥΠΞϯτϥΠϒϥϦ  - Treasure Dataެࣜ - جຊతʹTDͷAPIΛhttpͰୟ͍ͯΔ͚ͩ

Slide 71

Slide 71 text

ΫΤϦΛ౤͛ͯ  ݁ՌΛऔಘͯ͠ΈΔ

Slide 72

Slide 72 text

// hogeςʔϒϧͷத਎Λऔಘ val sql = ‘SELECT * FROM hoge’ val client = TDClient.newClient()  val jobRequest = TDJobRequest.newPrestoQuery(dbName, sql)  val jobId = client.submit(jobRequest)  val backOff = new ExponentialBackOff  while (!client.jobStatus(jobId).getStatus.isFinished) {  Thread.sleep(backOff.nextWaitTimeMillis)  } val input = client.jobResult(jobId, TDResultFormat.MESSAGE_PACK_GZ, new Function[InputStream, InputStream] {  def apply(input: InputStream) = input } val unpacker = MessagePack.newDefaultUnpacker(new GZIPInputStream(input))

Slide 73

Slide 73 text

௕͍…ʢ´ɾωɾʆʣ

Slide 74

Slide 74 text

Slide 75

Slide 75 text

Slide 76

Slide 76 text

Slide 77

Slide 77 text

- ਖ਼௚࢖͍ʹ͍͘ - ScalaͬΆ͘ͳ͍ - ͦ΋ͦ΋TDͷςʔϒϧΛ௚઀ϓϩάϥϜ  ͔ΒಡΉ͜ͱࣗମ͋·Γ૝ఆ͞Εͯͳ͍

Slide 78

Slide 78 text

- ਖ਼௚࢖͍ʹ͍͘ - ScalaͬΆ͘ͳ͍ - ͦ΋ͦ΋TDͷςʔϒϧΛ௚઀ϓϩάϥϜ  ͔ΒಡΉ͜ͱࣗମ͋·Γ૝ఆ͞Εͯͳ͍ ͡Ό͋Ͳ͏͢Δ

Slide 79

Slide 79 text

Result ExportΛ  ࢖͍·͠ΐ͏

Slide 80

Slide 80 text

Result Export - ΫΤϦ͕ऴΘͬͨλΠϛϯάͰ  ݁ՌΛࢦఆͨ͠৔ॴʹసૹ͢Δ - సૹઌ - S3 - RDB - Mongo - etc…

Slide 81

Slide 81 text

RTBαʔό ϩά ϩάςʔϒϧ ﬂuentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ  ΞΫηε όοναʔό td-client-java ༧ଌϞσϧͷ࡞੒࣌ʹ  S3ʹͰ΋Export͓͚ͯ͠͹… S3

Slide 82

Slide 82 text

RTBαʔό ϩά ϩάςʔϒϧ ﬂuentd SQLͷੈք ֶशσʔλ 0.1 0.3 0.2 ༧ଌϞσϧ Treasure Data redis ίϐʔ ϝϞϦΩϟογϡ ϦΫΤετ Ϩεϙϯε CTRΛ༧ଌ 0.31 ༧ଌϞσϧʹ  ΞΫηε όοναʔό td-client-java ؆୯ʂ S3

Slide 83

Slide 83 text

ͦͷଞͷϢʔεέʔε Scala x TreasureData

Slide 84

Slide 84 text

ScalaͰੜ੒ͨ͠σʔλΛ  TDʹΞοϓϩʔυ

Slide 85

Slide 85 text

Bulk Import - TDʹσʔλΛΞοϓϩʔυ͢ΔίϚϯυ - ίϚϯυϥΠϯͳͲ͔Β࢖͑Δ - JavaϥΠϒϥϦʹ΋ରԠؔ਺͕͋Δ

Slide 86

Slide 86 text

͋Εɺಈ͔ͳ͍…

Slide 87

Slide 87 text

͋Εɺಈ͔ͳ͍… ໰͍߹ΘͤΔ

Slide 88

Slide 88 text

No content

Slide 89

Slide 89 text

ʂʁ

Slide 90

Slide 90 text

No content

Slide 91

Slide 91 text

ͱ͍͏Θ͚ͰEmbulk  ࢖͍·͠ΐ͏

Slide 92

Slide 92 text

- Ϗοάσʔλ༻σʔλϩʔμ - ﬂuentdͷϏοάσʔλ൛Έ͍ͨͳײ͡ - TD͕։ൃ͍ͯ͠Δ - Φʔϓϯιʔε - Ϋδϥ γϟν͕͔Θ͍͍

Slide 93

Slide 93 text

Αʔ͠Scala͔Β  Embulk࢖͏ͧʔ…

Slide 94

Slide 94 text

ͦ͜·Ͱ͸Ͱ͖·ͤΜͰͨ͠  ʢ࣌ؒ੾Εʣ

Slide 95

Slide 95 text

·ͱΊ

Slide 96

Slide 96 text

1. TDͱHivemallͰCTR༧ଌϞσϧ࡞੒·Ͱ  SQLͰ׬݁͢ΔΑʂ 2. Scala͔ΒTDͷςʔϒϧಡΉͷେม => Result ExportΛ͏·͘࢖͓͏ 3. Scala͔ΒTDʹσʔλ্͛Δͷ͸EmbulkͰ => ୭͔΍Γํڭ͍͑ͯͩ͘͞