$30 off During Our Annual Pro Sale. View Details »

GBDT によるクリック率予測を高速化したい #オレシカナイト vol.4

GBDT によるクリック率予測を高速化したい #オレシカナイト vol.4

Gradient boosting decision trees (GBDT) を使ってクリック率を予測する場合に使えそうな高速化方法についてお話しました。

KOMIYA Atsushi

November 22, 2017
Tweet

More Decks by KOMIYA Atsushi

Other Decks in Programming

Transcript

  1. GBDT ʹΑΔ
    ΫϦοΫ཰༧ଌΛߴ଎Խ͍ͨ͠
    ΦϨγΧφΠτ vol.4 2017-11-22
    KOMIYA Atsushi

    View Slide

  2. ޿ࠂ഑৴࠷దԽ͓͡͞Μۀ

    View Slide

  3. @komiya_atsushi

    View Slide

  4. ࠓ೔ͷ͓࿩

    View Slide

  5. ͍ΘΏΔ “50ms or die” ͱ
    ݴΘΕ͍ͯΔք۾ʹ͓͚Δ

    View Slide

  6. ޿ࠂΫϦοΫ཰ͷ
    ΦϯϥΠϯɾϦΞϧλΠϜ༧ଌ

    View Slide

  7. ※ ػցֶशدΓͷ
    ࠐΈೖͬͨ࿩͸͠·ͤΜ
    (ػցֶशΏΔ;Θ੎ͳͷͰͰ͖·ͤΜ)

    View Slide

  8. ΫϦοΫ཰Λ༧ଌ͢Δํ๏ (Ұྫ)
    • ϩδεςΟοΫճؼ
    • (Field-aware) factorization machines
    • Gradient boosting decision trees (GBDT)

    View Slide

  9. GBDT ʹΑΔ޿ࠂΫϦοΫ཰༧ଌ
    • Practical Lessons from Predicting Clicks on Ads at
    Facebook
    • https://research.fb.com/publications/practical-
    lessons-from-predicting-clicks-on-ads-at-facebook/
    • Using boosted trees for click-through rate prediction
    for sponsored search (Yandex)
    • http://wan.poly.edu/KDD2012/forms/workshop/
    ADKDD12/doc/a3.pdf

    View Slide

  10. GBDT is Կʁ

    View Slide

  11. ܾఆ໦
    IUUQYHCPPTUSFBEUIFEPDTJPFOMBUFTUNPEFMIUNM

    View Slide

  12. GBDT
    IUUQYHCPPTUSFBEUIFEPDTJPFOMBUFTUNPEFMIUNM

    View Slide

  13. (ͱͬͯ΋ࡶʹදݱ͢Δͱ)

    ܾఆ໦Λԣʹͨ͘͞Μ
    ฒ΂ͨ΋ͷ

    View Slide

  14. ୅දతͳ GBDT ͷ࣮૷
    • XGBoost (DMLC ⭐9,661)
    • LightGBM (Microsoft ⭐4,184)
    • CatBoost (Yandex ⭐2,129)

    View Slide

  15. GBDT ͰΫϦοΫ཰༧ଌ

    ͏Ε͍͜͠ͱɾ൵͍͜͠ͱ

    View Slide

  16. Կ͕͏Ε͍͠ͷʁ
    • ༧ଌਫ਼౓͸ͦΕͳΓʹ͍͍Α
    • n ࣍ͷಛ௃ྔͷަޓ࡞༻Λѻ͑ΔΑ
    • ʮੑผ x ޿ࠂΧςΰϦʯΈ͍ͨͳಛ௃ྔͷ
    ૊߹ΘͤΛ༻ҙ͓ͯ͘͠ඞཁ͕ͳ͍
    • (n ͸πϦʔͷਂ͞)

    View Slide

  17. ޿ࠂΫϦοΫ཰༧ଌͷ࣮ࡍ
    Ϣʔβ"

    ޿ࠂ
    ޿ࠂ
    ޿ࠂ
    ޿ࠂO
    ʴ
    Ϣʔβͷಛ௃ྔ
    ޿ࠂ͝ͱͷಛ௃ྔ
    ༧ଌϞσϧ





    ޿ࠂ͝ͱʹ
    ༧ଌ

    View Slide

  18. ޿ࠂΫϦοΫ཰༧ଌͷ࣮ࡍ
    Ϣʔβ"

    ޿ࠂ
    ޿ࠂ
    ޿ࠂ
    ޿ࠂO
    ʴ
    Ϣʔβͷಛ௃ྔ
    ޿ࠂ͝ͱͷಛ௃ྔ
    ༧ଌϞσϧ





    ޿ࠂ͝ͱʹ
    ༧ଌ
    ผʑʹଘࡏ͢ΔϢʔβͷಛ௃ྔͱ
    ޿ࠂͷಛ௃ྔΛΦϯϥΠϯͰ݁߹ KPJO

    ͯ͠ಛ௃ϕΫτϧΛ࡞Δ

    View Slide

  19. ޿ࠂΫϦοΫ཰༧ଌͷ࣮ࡍ
    Ϣʔβ"

    ޿ࠂ
    ޿ࠂ
    ޿ࠂ
    ޿ࠂO
    ʴ
    Ϣʔβͷಛ௃ྔ
    ޿ࠂ͝ͱͷಛ௃ྔ
    ༧ଌϞσϧ





    ޿ࠂ͝ͱʹ
    ༧ଌ
    ݁߹ͨ͠ಛ௃ϕΫτϧΛ
    ༧ଌϞσϧʹ༩͑ͯ݁ՌΛಘΔ

    View Slide

  20. Կ͕൵͍͠ͷʁ
    • ֶश͸ͦΕͳΓʹ͕͔͔࣌ؒΔΑ
    • ༧ଌ΋ͦΕͳΓʹ͕͔͔࣌ؒΔΑ
    • πϦʔ਺Λ૿΍͢΄Ͳ (աֶशͷϦεΫ͸͋
    Δ͚Ͳ) ਫ਼౓͕ߴ͘ͳΔ
    • ҰํͰɺ༧ଌʹ͔͔Δ࣌ؒܭࢉྔ͸πϦʔ
    ਺ʹൺྫͯ͠૿Ճ͢Δ

    View Slide

  21. ༧ଌਫ਼౓ͱॲཧ଎౓͸
    τϨʔυΦϑؔ܎ʹͳΓ͕ͪ

    View Slide

  22. ༧ଌਫ਼౓ΛߴΊΔͨΊʹ΋

    ༧ଌͷॲཧ଎౓ΛߴΊ͍ͨ❗

    View Slide

  23. GBDT ʹΑΔΫϦοΫ཰༧ଌͷ
    ߴ଎ԽΛਤΔ

    View Slide


  24. View Slide

  25. ͝஫ҙ
    • ࠓճ͝঺հ͢ΔɺGBDT ʹ͓͚Δ༧ଌॲཧͷ
    ߴ଎ԽϙΠϯτ͸ɺ޿ࠂࡏݿͷྔɺར༻͢Δ
    ϋʔυ΢ΣΞ΍ݴޠɺಛ௃ϕΫτϧͷେ͖͞
    ͳͲΑͬͯੑೳͷྑ͠ѱ͕͠େ͖͘มΘΔՄ
    ೳੑ͕͋Γ·͢
    • ※ඞͣߴ଎ԽͰ͖Δ͜ͱΛอূ͢Δ΋ͷͰ͸͋
    Γ·ͤΜ

    View Slide

  26. GBDT ༧ଌॲཧͷߴ଎ԽϙΠϯτ
    • GBDT ͷ֤πϦʔͷϝϞϦ্Ͱͷදݱํ๏Λ
    ޻෉͢Δ
    • ಛ௃ϕΫτϧͷදݱํ๏Λ޻෉͢Δ
    • ༧ଌͷॲཧखॱʹେ͖͘ӨڹΛ༩͑͏Δ

    ͞ΒͳΔ޻෉Λ͢Δ

    View Slide

  27. πϦʔͷϝϞϦ্Ͱͷ
    දݱํ๏Λ޻෉͢Δ

    View Slide

  28. ϕʔεϥΠϯɿφΠʔϒͳ໦ߏ଄
    Y<>

    Y<>

    Y<>

    View Slide

  29. ϕʔεϥΠϯɿφΠʔϒͳ໦ߏ଄
    Y<>

    Y<>

    Y<>
    ɾಛ௃ϕΫτϧ্ͷΠϯσΫε
    ɾ෼ׂ৚݅ͷ஋
    ɾ಺෦ϊʔυPS༿ϊʔυʁ
    ɾࠨͷࢠϊʔυ΁ͷࢀর
    ɾӈͷࢠϊʔυ΁ͷࢀর

    View Slide

  30. ϕʔεϥΠϯɿφΠʔϒͳ໦ߏ଄
    • ࠨӈͷࢠϊʔυ΁ͷࢀরΛ࣋ͨͳ͚Ε͹

    ͍͚ͳͯ͘৑௕Ͱ͋Δ
    • ΩϟογϡϝϞϦతʹܾͯ͠๬·͍͠දݱͰ
    ͸ͳ͍

    View Slide

  31. ໦ߏ଄Λϑϥοτͳ഑ྻͰදݱ
    Y<>

    Y<>

    Y<>




    ಛ௃ϕΫτϧͷΠϯσΫε
    ෼ׂ৚݅ͷ஋༿ϊʔυͷ஋

    View Slide

  32. ໦ߏ଄Λϑϥοτͳ഑ྻͰදݱ
    • ࢠϊʔυΛͨͲΔํ๏
    • ਌ϊʔυͷ഑ྻ্ͷΠϯσΫεΛ i ͱͨ͠ͱ͖ɺࠨͷࢠ
    ϊʔυ͸ 2 * (i + 1) - 1ɺӈͷࢠϊʔυ͸ 2 * (i + 1) Ͱࢀ
    রͰ͖Δ
    • φΠʔϒͳ໦ߏ଄ΑΓ͸ΩϟογϡϝϞϦతʹ༏͍ͩ͠Ζ
    ͏…
    • ͳ͓πϦʔ͸ৗʹ׬શೋ෼໦Ͱ͋Δͱ͸ݶΒͳ͍
    • Ұ෦ͷαϒπϦʔ͕ࣃൈ͚ʹͳΓ͏Δ͜ͱΛ

    ߟྀ͢Δඞཁ͕͋Δ

    View Slide

  33. ωετͨ͠ if - else Ͱίʔυදݱ
    double predict_tree1(double[] x) {
    if (x[3] < 5) {
    if (x[8] < 2)
    return 9;
    else
    return 5;
    } else {
    if (x[7] < 3)
    return 4;
    else
    return 7;
    }
    }

    View Slide

  34. ωετͨ͠ if - else Ͱίʔυදݱ
    • ωΠςΟϒίʔυʹม׵Ͱ͖Δ / ͞ΕΔݴޠͳΒߴ଎ͳॲཧ͕
    ظ଴Ͱ͖Δ
    • Java ͷ৔߹ͷ͓࿩
    • ASM ͳͲΛ࢖ͬͯ GBDT ͷ༧ଌϞσϧΛόΠτίʔυʹ

    ม׵ & ಈతΫϥεϩʔυ → JIT ίϯύΠϧ͕ظ଴Ͱ͖Δ
    • ҰํͰɺπϦʔͷਂ࣍͞ୈͰ͸ϝιουͷόΠτίʔυαΠ
    ζ͕େ͖͗ͯ͢ΠϯϥΠϯల։Λ્֐ͯ͠͠·͏͜ͱ΋…

    View Slide

  35. ಛ௃ϕΫτϧͷ
    දݱํ๏Λ޻෉͢Δ

    View Slide

  36. ಛ௃ϕΫτϧʹٻΊΒΕΔಛੑ
    • GBDT Ͱ͸ɺಛ௃ϕΫτϧͷ֤ཁૉͷࢀর͸ϥϯμϜ
    ΞΫηεʹͳΔ
    • ϩδεςΟοΫճؼ΍ factorization machines ͱ͸
    େ͖͘ҟͳΔ
    • ΩϟογϡϝϞϦతʹͳ͔ͳ͔͓ਏ͍ײ͡
    • ಛ௃ྔΛอ࣋͢Δۭؒޮ཰ͱࢀর࣌ͷॲཧ଎౓ͱͷτ
    ϨʔυΦϑʹͳΔ

    View Slide

  37. ૉ๿ʹ഑ྻͰදݱ͢Δ
    x = { 3, 0, NaN, 7, 2, NaN, 4 };

    OB OB

    View Slide

  38. ૉ๿ʹ഑ྻͰදݱ͢Δ
    • ۭؒޮ཰͸ྑ͘ͳ͍͕ɺࢀরੑೳ͸ߴ͍
    • ద͍ͯ͠Δέʔε
    • ಛ௃ϕΫτϧͷ࣍ݩ͕ൺֱతখ͍͞
    • ಛ௃ϕΫτϧ͕ൺֱత ີ (dense) Ͱ͋Δ
    • Missing value ͷѻ͍ʹؾΛ͚ͭΔඞཁ͕͋Δ

    View Slide

  39. ࿈૝഑ྻͰදݱ͢Δ
    x = {
    0: 3,
    1: 0,
    3: 7,
    4: 2,
    6: 4
    }

    View Slide

  40. ࿈૝഑ྻͰදݱ͢Δ
    • ࿈૝഑ྻ: ฏߧ໦΍ϋογϡදʹΑΔ࣮૷
    • ۭؒޮ཰͸ͦΕͳΓʹྑ͍ҰํͰɺࢀর࣌ͷΦʔόʔϔο
    υ͕ؾʹͳΓ͕ͪ
    • ద͍ͯ͠Δέʔε
    • ඇৗʹߴ࣍ݩͷಛ௃ϕΫτϧΛѻΘͳ͚Ε͹ͳΒͳ͍
    • ͦͷҰํͰɺ࣮ࡍͷಛ௃ϕΫτϧ͸ ૄ (sparse) Ͱ͋Δ

    View Slide

  41. AssocVector తͳදݱΛ༻͍Δ
    J
    W
    J
    W
    J
    W
    J
    W
    J
    W

    View Slide

  42. AssocVector తͳදݱΛ༻͍Δ
    • ಛ௃ϕΫτϧ্ͷΠϯσΫεͱ஋ͷ૊Λɺ

    ΠϯσΫεͰιʔτͯ͠഑ྻͰอ࣋͢Δ
    • ࢀর͸ೋ෼୳ࡧͰߦ͏ → O(log N)
    • ۭؒޮ཰͸࿈૝഑ྻΑΓ༏Ε͍ͯΔ
    • ΩϟογϡϝϞϦʹ΋༏͍͠දݱͳͷͰ͸…

    View Slide

  43. ͞ΒͳΔ޻෉

    View Slide

  44. ༧ଌͷॲཧखॱʹӨڹ͢Δ޻෉
    • ޿ࠂΫϦοΫ཰ͷ༧ଌ͸ 1 Ϣʔβ 1 ޿ࠂ͚ͩͰࡁΉ
    Α͏ͳ΋ͷͰ͸ͳ͍
    • ͢΂ͯͷ޿ࠂʹ͍ͭͯΫϦοΫ཰Λ༧ଌ͍ͨ͠
    • ͦΕͧΕͷ޿ࠂ͝ͱʹஞ࣍తʹ༧ଌͯ͠΋͍͍͕ɺ
    Ұׅͯ͠༧ଌ͢Δ͜ͱΛલఏͱͯ͠ޮ཰Խ͢Δํ๏
    ΋ଘࡏ͢Δ
    • ͨͩ͠ɺ͔ͳΓखͷࠐΜ࣮ͩ૷ʹͳΓ͕ͪ

    View Slide

  45. Ұ࿈ͷ༧ଌʹͯෆมͷಛ௃ྔͰࢬמΓ͢Δ
    ੑผஉੑʁ

    ޿ࠂΧςΰϦήʔϜ

    ޿ࠂΧςΰϦԽহ඼
    ZFT OP
    Ϣʔβͷಛ௃ྔஉੑ ࡀ ʜ
    ͪ͜Β͸ઈରʹḷΒͳ͍

    View Slide

  46. Ұ࿈ͷ༧ଌʹͯෆมͷಛ௃ྔͰࢬמΓ͢Δ
    • ྫ͑͹Ϣʔβͷಛ௃ྔ͸ɺ޿ࠂ͝ͱͷ༧ଌʹ͓͍
    ͯ͸ෆมͰ͋Δ
    • ෆมͷಛ௃ྔʹର͢Δ৚݅ΛຖճධՁ͢Δͷ͸
    ແବ
    • ༧ଌͷલॲཧͱͯ͠ɺͦͷΑ͏ͳ಺෦ϊʔυΛε
    Ωοϓͯ͠πϦʔΛίϯύΫτʹ͓ͯ͘͠
    • લॲཧ෼ͷΦʔόʔϔου͕ੜ͡Δ

    View Slide

  47. QuickScorer
    • ޱ಄Ͱઆ໌͢Δͷ͸ແཧͳͷͰɺ໊લ͚ͩ

    ֮͑ͯޙͰ͙͙͍ͬͯͩ͘͞
    • ಛڐग़ئதΒ͍͠ͷͰɺ͝ར༻ʹ͸͝஫ҙΛ

    View Slide

  48. ଞʹ΋͍Ζ͍Ζ͋Γ·͕͢…
    • Google Scholar Ͱ QuickScorer ΛҾ༻ͯ͠
    ͍Δ࿦จ΍ؔ࿈هࣄΛ୳͢ͷΛ͓͢͢Ί͠
    ·͢

    View Slide

  49. ँࣙɾࢀߟจݙ

    View Slide

  50. ँࣙɾࢀߟจݙ
    • ࠓճ͝঺հͨ͠΋ͷͷଟ͘͸ɺಙӬ͞Μ (@tkng)
    ΑΓΞυόΠε͍͍ͨͩͨ΋ͷʹͳΓ·͢
    • Evaluating boosted decision trees for billions of
    users
    • https://code.facebook.com/posts/
    975025089299409/evaluating-boosted-
    decision-trees-for-billions-of-users/

    View Slide

  51. Thank you!

    View Slide