GBDT によるクリック率予測を高速化したい #オレシカナイト vol.4

Slide 1

Slide 1 text

GBDT ʹΑΔ ΫϦοΫ཰༧ଌΛߴ଎Խ͍ͨ͠ ΦϨγΧφΠτ vol.4 2017-11-22 KOMIYA Atsushi

Slide 2

Slide 2 text

޿ࠂ഑৴࠷దԽ͓͡͞Μۀ

Slide 3

Slide 3 text

@komiya_atsushi

Slide 4

Slide 4 text

ࠓ೔ͷ͓࿩

Slide 5

Slide 5 text

͍ΘΏΔ “50ms or die” ͱ ݴΘΕ͍ͯΔք۾ʹ͓͚Δ

Slide 6

Slide 6 text

޿ࠂΫϦοΫ཰ͷ ΦϯϥΠϯɾϦΞϧλΠϜ༧ଌ

Slide 7

Slide 7 text

※ ػցֶशدΓͷ ࠐΈೖͬͨ࿩͸͠·ͤΜ (ػցֶशΏΔ;Θ੎ͳͷͰͰ͖·ͤΜ)

Slide 8

Slide 8 text

ΫϦοΫ཰Λ༧ଌ͢Δํ๏ (Ұྫ) • ϩδεςΟοΫճؼ • (Field-aware) factorization machines • Gradient boosting decision trees (GBDT)

Slide 9

Slide 9 text

GBDT ʹΑΔ޿ࠂΫϦοΫ཰༧ଌ • Practical Lessons from Predicting Clicks on Ads at Facebook • https://research.fb.com/publications/practical- lessons-from-predicting-clicks-on-ads-at-facebook/ • Using boosted trees for click-through rate prediction for sponsored search (Yandex) • http://wan.poly.edu/KDD2012/forms/workshop/ ADKDD12/doc/a3.pdf

Slide 10

Slide 10 text

GBDT is Կʁ

Slide 11

Slide 11 text

ܾఆ໦ IUUQYHCPPTUSFBEUIFEPDTJPFOMBUFTUNPEFMIUNM

Slide 12

Slide 12 text

GBDT IUUQYHCPPTUSFBEUIFEPDTJPFOMBUFTUNPEFMIUNM

Slide 13

Slide 13 text

(ͱͬͯ΋ࡶʹදݱ͢Δͱ)  ܾఆ໦Λԣʹͨ͘͞Μ ฒ΂ͨ΋ͷ

Slide 14

Slide 14 text

୅දతͳ GBDT ͷ࣮૷ • XGBoost (DMLC ⭐9,661) • LightGBM (Microsoft ⭐4,184) • CatBoost (Yandex ⭐2,129)

Slide 15

Slide 15 text

GBDT ͰΫϦοΫ཰༧ଌ  ͏Ε͍͜͠ͱɾ൵͍͜͠ͱ

Slide 16

Slide 16 text

Կ͕͏Ε͍͠ͷʁ • ༧ଌਫ਼౓͸ͦΕͳΓʹ͍͍Α • n ࣍ͷಛ௃ྔͷަޓ࡞༻Λѻ͑ΔΑ • ʮੑผ x ޿ࠂΧςΰϦʯΈ͍ͨͳಛ௃ྔͷ ૊߹ΘͤΛ༻ҙ͓ͯ͘͠ඞཁ͕ͳ͍ • (n ͸πϦʔͷਂ͞)

Slide 17

Slide 17 text

޿ࠂΫϦοΫ཰༧ଌͷ࣮ࡍ Ϣʔβ" ộ ޿ࠂ ޿ࠂ ޿ࠂ ޿ࠂO ʴ Ϣʔβͷಛ௃ྔ ޿ࠂ͝ͱͷಛ௃ྔ ༧ଌϞσϧ ộ ޿ࠂ͝ͱʹ ༧ଌ

Slide 18

Slide 18 text

޿ࠂΫϦοΫ཰༧ଌͷ࣮ࡍ Ϣʔβ" ộ ޿ࠂ ޿ࠂ ޿ࠂ ޿ࠂO ʴ Ϣʔβͷಛ௃ྔ ޿ࠂ͝ͱͷಛ௃ྔ ༧ଌϞσϧ ộ ޿ࠂ͝ͱʹ ༧ଌ ผʑʹଘࡏ͢ΔϢʔβͷಛ௃ྔͱ ޿ࠂͷಛ௃ྔΛΦϯϥΠϯͰ݁߹ KPJO ͯ͠ಛ௃ϕΫτϧΛ࡞Δ

Slide 19

Slide 19 text

޿ࠂΫϦοΫ཰༧ଌͷ࣮ࡍ Ϣʔβ" ộ ޿ࠂ ޿ࠂ ޿ࠂ ޿ࠂO ʴ Ϣʔβͷಛ௃ྔ ޿ࠂ͝ͱͷಛ௃ྔ ༧ଌϞσϧ ộ ޿ࠂ͝ͱʹ ༧ଌ ݁߹ͨ͠ಛ௃ϕΫτϧΛ ༧ଌϞσϧʹ༩͑ͯ݁ՌΛಘΔ

Slide 20

Slide 20 text

Կ͕൵͍͠ͷʁ • ֶश͸ͦΕͳΓʹ͕͔͔࣌ؒΔΑ • ༧ଌ΋ͦΕͳΓʹ͕͔͔࣌ؒΔΑ • πϦʔ਺Λ૿΍͢΄Ͳ (աֶशͷϦεΫ͸͋ Δ͚Ͳ) ਫ਼౓͕ߴ͘ͳΔ • ҰํͰɺ༧ଌʹ͔͔Δ࣌ؒܭࢉྔ͸πϦʔ ਺ʹൺྫͯ͠૿Ճ͢Δ

Slide 21

Slide 21 text

༧ଌਫ਼౓ͱॲཧ଎౓͸ τϨʔυΦϑؔ܎ʹͳΓ͕ͪ

Slide 22

Slide 22 text

༧ଌਫ਼౓ΛߴΊΔͨΊʹ΋  ༧ଌͷॲཧ଎౓ΛߴΊ͍ͨ❗

Slide 23

Slide 23 text

GBDT ʹΑΔΫϦοΫ཰༧ଌͷ ߴ଎ԽΛਤΔ

Slide 24

Slide 24 text

⚠

Slide 25

Slide 25 text

͝஫ҙ • ࠓճ͝঺հ͢ΔɺGBDT ʹ͓͚Δ༧ଌॲཧͷ ߴ଎ԽϙΠϯτ͸ɺ޿ࠂࡏݿͷྔɺར༻͢Δ ϋʔυ΢ΣΞ΍ݴޠɺಛ௃ϕΫτϧͷେ͖͞ ͳͲΑͬͯੑೳͷྑ͠ѱ͕͠େ͖͘มΘΔՄ ೳੑ͕͋Γ·͢ • ※ඞͣߴ଎ԽͰ͖Δ͜ͱΛอূ͢Δ΋ͷͰ͸͋ Γ·ͤΜ

Slide 26

Slide 26 text

GBDT ༧ଌॲཧͷߴ଎ԽϙΠϯτ • GBDT ͷ֤πϦʔͷϝϞϦ্Ͱͷදݱํ๏Λ ޻෉͢Δ • ಛ௃ϕΫτϧͷදݱํ๏Λ޻෉͢Δ • ༧ଌͷॲཧखॱʹେ͖͘ӨڹΛ༩͑͏Δ  ͞ΒͳΔ޻෉Λ͢Δ

Slide 27

Slide 27 text

πϦʔͷϝϞϦ্Ͱͷ දݱํ๏Λ޻෉͢Δ

Slide 28

Slide 28 text

ϕʔεϥΠϯɿφΠʔϒͳ໦ߏ଄ Y<> Y<> Y<>

Slide 29

Slide 29 text

ϕʔεϥΠϯɿφΠʔϒͳ໦ߏ଄ Y<> Y<> Y<> ɾಛ௃ϕΫτϧ্ͷΠϯσΫε ɾ෼ׂ৚݅ͷ஋ ɾ಺෦ϊʔυPS༿ϊʔυʁ ɾࠨͷࢠϊʔυ΁ͷࢀর ɾӈͷࢠϊʔυ΁ͷࢀর

Slide 30

Slide 30 text

ϕʔεϥΠϯɿφΠʔϒͳ໦ߏ଄ • ࠨӈͷࢠϊʔυ΁ͷࢀরΛ࣋ͨͳ͚Ε͹  ͍͚ͳͯ͘৑௕Ͱ͋Δ • ΩϟογϡϝϞϦతʹܾͯ͠๬·͍͠දݱͰ ͸ͳ͍

Slide 31

Slide 31 text

໦ߏ଄Λϑϥοτͳ഑ྻͰදݱ Y<> Y<> Y<> ಛ௃ϕΫτϧͷΠϯσΫε ෼ׂ৚݅ͷ஋༿ϊʔυͷ஋

Slide 32

Slide 32 text

໦ߏ଄Λϑϥοτͳ഑ྻͰදݱ • ࢠϊʔυΛͨͲΔํ๏ • ਌ϊʔυͷ഑ྻ্ͷΠϯσΫεΛ i ͱͨ͠ͱ͖ɺࠨͷࢠ ϊʔυ͸ 2 * (i + 1) - 1ɺӈͷࢠϊʔυ͸ 2 * (i + 1) Ͱࢀ রͰ͖Δ • φΠʔϒͳ໦ߏ଄ΑΓ͸ΩϟογϡϝϞϦతʹ༏͍ͩ͠Ζ ͏… • ͳ͓πϦʔ͸ৗʹ׬શೋ෼໦Ͱ͋Δͱ͸ݶΒͳ͍ • Ұ෦ͷαϒπϦʔ͕ࣃൈ͚ʹͳΓ͏Δ͜ͱΛ  ߟྀ͢Δඞཁ͕͋Δ

Slide 33

Slide 33 text

ωετͨ͠ if - else Ͱίʔυදݱ double predict_tree1(double[] x) { if (x[3] < 5) { if (x[8] < 2) return 9; else return 5; } else { if (x[7] < 3) return 4; else return 7; } }

Slide 34

Slide 34 text

ωετͨ͠ if - else Ͱίʔυදݱ • ωΠςΟϒίʔυʹม׵Ͱ͖Δ / ͞ΕΔݴޠͳΒߴ଎ͳॲཧ͕ ظ଴Ͱ͖Δ • Java ͷ৔߹ͷ͓࿩ • ASM ͳͲΛ࢖ͬͯ GBDT ͷ༧ଌϞσϧΛόΠτίʔυʹ  ม׵ & ಈతΫϥεϩʔυ → JIT ίϯύΠϧ͕ظ଴Ͱ͖Δ • ҰํͰɺπϦʔͷਂ࣍͞ୈͰ͸ϝιουͷόΠτίʔυαΠ ζ͕େ͖͗ͯ͢ΠϯϥΠϯల։Λ્֐ͯ͠͠·͏͜ͱ΋…

Slide 35

Slide 35 text

ಛ௃ϕΫτϧͷ දݱํ๏Λ޻෉͢Δ

Slide 36

Slide 36 text

ಛ௃ϕΫτϧʹٻΊΒΕΔಛੑ • GBDT Ͱ͸ɺಛ௃ϕΫτϧͷ֤ཁૉͷࢀর͸ϥϯμϜ ΞΫηεʹͳΔ • ϩδεςΟοΫճؼ΍ factorization machines ͱ͸ େ͖͘ҟͳΔ • ΩϟογϡϝϞϦతʹͳ͔ͳ͔͓ਏ͍ײ͡ • ಛ௃ྔΛอ࣋͢Δۭؒޮ཰ͱࢀর࣌ͷॲཧ଎౓ͱͷτ ϨʔυΦϑʹͳΔ

Slide 37

Slide 37 text

ૉ๿ʹ഑ྻͰදݱ͢Δ x = { 3, 0, NaN, 7, 2, NaN, 4 }; OB OB

Slide 38

Slide 38 text

ૉ๿ʹ഑ྻͰදݱ͢Δ • ۭؒޮ཰͸ྑ͘ͳ͍͕ɺࢀরੑೳ͸ߴ͍ • ద͍ͯ͠Δέʔε • ಛ௃ϕΫτϧͷ࣍ݩ͕ൺֱతখ͍͞ • ಛ௃ϕΫτϧ͕ൺֱత ີ (dense) Ͱ͋Δ • Missing value ͷѻ͍ʹؾΛ͚ͭΔඞཁ͕͋Δ

Slide 39

Slide 39 text

࿈૝഑ྻͰදݱ͢Δ x = { 0: 3, 1: 0, 3: 7, 4: 2, 6: 4 }

Slide 40

Slide 40 text

࿈૝഑ྻͰදݱ͢Δ • ࿈૝഑ྻ: ฏߧ໦΍ϋογϡදʹΑΔ࣮૷ • ۭؒޮ཰͸ͦΕͳΓʹྑ͍ҰํͰɺࢀর࣌ͷΦʔόʔϔο υ͕ؾʹͳΓ͕ͪ • ద͍ͯ͠Δέʔε • ඇৗʹߴ࣍ݩͷಛ௃ϕΫτϧΛѻΘͳ͚Ε͹ͳΒͳ͍ • ͦͷҰํͰɺ࣮ࡍͷಛ௃ϕΫτϧ͸ ૄ (sparse) Ͱ͋Δ

Slide 41

Slide 41 text

AssocVector తͳදݱΛ༻͍Δ J W J W J W J W J W

Slide 42

Slide 42 text

AssocVector తͳදݱΛ༻͍Δ • ಛ௃ϕΫτϧ্ͷΠϯσΫεͱ஋ͷ૊Λɺ  ΠϯσΫεͰιʔτͯ͠഑ྻͰอ࣋͢Δ • ࢀর͸ೋ෼୳ࡧͰߦ͏ → O(log N) • ۭؒޮ཰͸࿈૝഑ྻΑΓ༏Ε͍ͯΔ • ΩϟογϡϝϞϦʹ΋༏͍͠දݱͳͷͰ͸…

Slide 43

Slide 43 text

͞ΒͳΔ޻෉

Slide 44

Slide 44 text

༧ଌͷॲཧखॱʹӨڹ͢Δ޻෉ • ޿ࠂΫϦοΫ཰ͷ༧ଌ͸ 1 Ϣʔβ 1 ޿ࠂ͚ͩͰࡁΉ Α͏ͳ΋ͷͰ͸ͳ͍ • ͢΂ͯͷ޿ࠂʹ͍ͭͯΫϦοΫ཰Λ༧ଌ͍ͨ͠ • ͦΕͧΕͷ޿ࠂ͝ͱʹஞ࣍తʹ༧ଌͯ͠΋͍͍͕ɺ Ұׅͯ͠༧ଌ͢Δ͜ͱΛલఏͱͯ͠ޮ཰Խ͢Δํ๏ ΋ଘࡏ͢Δ • ͨͩ͠ɺ͔ͳΓखͷࠐΜ࣮ͩ૷ʹͳΓ͕ͪ

Slide 45

Slide 45 text

Ұ࿈ͷ༧ଌʹͯෆมͷಛ௃ྔͰࢬמΓ͢Δ ੑผஉੑʁ ޿ࠂΧςΰϦήʔϜ ޿ࠂΧςΰϦԽহ඼ ZFT OP Ϣʔβͷಛ௃ྔஉੑ ࡀ ʜ ͪ͜Β͸ઈରʹḷΒͳ͍

Slide 46

Slide 46 text

Ұ࿈ͷ༧ଌʹͯෆมͷಛ௃ྔͰࢬמΓ͢Δ • ྫ͑͹Ϣʔβͷಛ௃ྔ͸ɺ޿ࠂ͝ͱͷ༧ଌʹ͓͍ ͯ͸ෆมͰ͋Δ • ෆมͷಛ௃ྔʹର͢Δ৚݅ΛຖճධՁ͢Δͷ͸ ແବ • ༧ଌͷલॲཧͱͯ͠ɺͦͷΑ͏ͳ಺෦ϊʔυΛε Ωοϓͯ͠πϦʔΛίϯύΫτʹ͓ͯ͘͠ • લॲཧ෼ͷΦʔόʔϔου͕ੜ͡Δ

Slide 47

Slide 47 text

QuickScorer • ޱ಄Ͱઆ໌͢Δͷ͸ແཧͳͷͰɺ໊લ͚ͩ  ֮͑ͯޙͰ͙͙͍ͬͯͩ͘͞ • ಛڐग़ئதΒ͍͠ͷͰɺ͝ར༻ʹ͸͝஫ҙΛ

Slide 48

Slide 48 text

ଞʹ΋͍Ζ͍Ζ͋Γ·͕͢… • Google Scholar Ͱ QuickScorer ΛҾ༻ͯ͠ ͍Δ࿦จ΍ؔ࿈هࣄΛ୳͢ͷΛ͓͢͢Ί͠ ·͢

Slide 49

Slide 49 text

ँࣙɾࢀߟจݙ

Slide 50

Slide 50 text

ँࣙɾࢀߟจݙ • ࠓճ͝঺հͨ͠΋ͷͷଟ͘͸ɺಙӬ͞Μ (@tkng) ΑΓΞυόΠε͍͍ͨͩͨ΋ͷʹͳΓ·͢ • Evaluating boosted decision trees for billions of users • https://code.facebook.com/posts/ 975025089299409/evaluating-boosted- decision-trees-for-billions-of-users/

Slide 51

Slide 51 text

Thank you!