GBDT によるクリック率予測を高速化したい #オレシカナイト vol.4

Gradient boosting decision trees (GBDT) を使ってクリック率を予測する場合に使えそうな高速化方法についてお話しました。

Transcript

1. GBDT ʹΑΔ
   ΫϦοΫ཰༧ଌΛߴ଎Խ͍ͨ͠
   ΦϨγΧφΠτ vol.4 2017-11-22
   KOMIYA Atsushi
   

   View Slide

2. ޿ࠂ഑৴࠷దԽ͓͡͞Μۀ
   

   View Slide

3. @komiya_atsushi
   

   View Slide

4. ࠓ೔ͷ͓࿩
   

   View Slide

5. ͍ΘΏΔ “50ms or die” ͱ
   ݴΘΕ͍ͯΔք۾ʹ͓͚Δ
   

   View Slide

6. ޿ࠂΫϦοΫ཰ͷ
   ΦϯϥΠϯɾϦΞϧλΠϜ༧ଌ
   

   View Slide

7. ※ ػցֶशدΓͷ
   ࠐΈೖͬͨ࿩͸͠·ͤΜ
   (ػցֶशΏΔ;Θ੎ͳͷͰͰ͖·ͤΜ)
   

   View Slide

8. ΫϦοΫ཰Λ༧ଌ͢Δํ๏ (Ұྫ)
   • ϩδεςΟοΫճؼ
   • (Field-aware) factorization machines
   • Gradient boosting decision trees (GBDT)
   

   View Slide

9. GBDT ʹΑΔ޿ࠂΫϦοΫ཰༧ଌ
   • Practical Lessons from Predicting Clicks on Ads at
   Facebook
   • https://research.fb.com/publications/practical-
   lessons-from-predicting-clicks-on-ads-at-facebook/
   • Using boosted trees for click-through rate prediction
   for sponsored search (Yandex)
   • http://wan.poly.edu/KDD2012/forms/workshop/
   ADKDD12/doc/a3.pdf
   

   View Slide

10. GBDT is Կʁ
    

    View Slide

11. ܾఆ໦
    IUUQYHCPPTUSFBEUIFEPDTJPFOMBUFTUNPEFMIUNM
    

    View Slide

12. GBDT
    IUUQYHCPPTUSFBEUIFEPDTJPFOMBUFTUNPEFMIUNM
    

    View Slide

13. (ͱͬͯ΋ࡶʹදݱ͢Δͱ)

    ܾఆ໦Λԣʹͨ͘͞Μ
    ฒ΂ͨ΋ͷ
    

    View Slide

14. ୅දతͳ GBDT ͷ࣮૷
    • XGBoost (DMLC ⭐9,661)
    • LightGBM (Microsoft ⭐4,184)
    • CatBoost (Yandex ⭐2,129)
    

    View Slide

15. GBDT ͰΫϦοΫ཰༧ଌ

    ͏Ε͍͜͠ͱɾ൵͍͜͠ͱ
    

    View Slide

16. Կ͕͏Ε͍͠ͷʁ
    • ༧ଌਫ਼౓͸ͦΕͳΓʹ͍͍Α
    • n ࣍ͷಛ௃ྔͷަޓ࡞༻Λѻ͑ΔΑ
    • ʮੑผ x ޿ࠂΧςΰϦʯΈ͍ͨͳಛ௃ྔͷ
    ૊߹ΘͤΛ༻ҙ͓ͯ͘͠ඞཁ͕ͳ͍
    • (n ͸πϦʔͷਂ͞)
    

    View Slide

17. ޿ࠂΫϦοΫ཰༧ଌͷ࣮ࡍ
    Ϣʔβ"
    ộ
    ޿ࠂ
    ޿ࠂ
    ޿ࠂ
    ޿ࠂO
    ʴ
    Ϣʔβͷಛ௃ྔ
    ޿ࠂ͝ͱͷಛ௃ྔ
    ༧ଌϞσϧ
    
    
    
    
    ộ
    ޿ࠂ͝ͱʹ
    ༧ଌ
    

    View Slide

18. ޿ࠂΫϦοΫ཰༧ଌͷ࣮ࡍ
    Ϣʔβ"
    ộ
    ޿ࠂ
    ޿ࠂ
    ޿ࠂ
    ޿ࠂO
    ʴ
    Ϣʔβͷಛ௃ྔ
    ޿ࠂ͝ͱͷಛ௃ྔ
    ༧ଌϞσϧ
    
    
    
    
    ộ
    ޿ࠂ͝ͱʹ
    ༧ଌ
    ผʑʹଘࡏ͢ΔϢʔβͷಛ௃ྔͱ
    ޿ࠂͷಛ௃ྔΛΦϯϥΠϯͰ݁߹ KPJO
    
    ͯ͠ಛ௃ϕΫτϧΛ࡞Δ
    

    View Slide

19. ޿ࠂΫϦοΫ཰༧ଌͷ࣮ࡍ
    Ϣʔβ"
    ộ
    ޿ࠂ
    ޿ࠂ
    ޿ࠂ
    ޿ࠂO
    ʴ
    Ϣʔβͷಛ௃ྔ
    ޿ࠂ͝ͱͷಛ௃ྔ
    ༧ଌϞσϧ
    
    
    
    
    ộ
    ޿ࠂ͝ͱʹ
    ༧ଌ
    ݁߹ͨ͠ಛ௃ϕΫτϧΛ
    ༧ଌϞσϧʹ༩͑ͯ݁ՌΛಘΔ
    

    View Slide

20. Կ͕൵͍͠ͷʁ
    • ֶश͸ͦΕͳΓʹ͕͔͔࣌ؒΔΑ
    • ༧ଌ΋ͦΕͳΓʹ͕͔͔࣌ؒΔΑ
    • πϦʔ਺Λ૿΍͢΄Ͳ (աֶशͷϦεΫ͸͋
    Δ͚Ͳ) ਫ਼౓͕ߴ͘ͳΔ
    • ҰํͰɺ༧ଌʹ͔͔Δ࣌ؒܭࢉྔ͸πϦʔ
    ਺ʹൺྫͯ͠૿Ճ͢Δ
    

    View Slide

21. ༧ଌਫ਼౓ͱॲཧ଎౓͸
    τϨʔυΦϑؔ܎ʹͳΓ͕ͪ
    

    View Slide

22. ༧ଌਫ਼౓ΛߴΊΔͨΊʹ΋

    ༧ଌͷॲཧ଎౓ΛߴΊ͍ͨ❗
    

    View Slide

23. GBDT ʹΑΔΫϦοΫ཰༧ଌͷ
    ߴ଎ԽΛਤΔ
    

    View Slide

24. ⚠
    

    View Slide

25. ͝஫ҙ
    • ࠓճ͝঺հ͢ΔɺGBDT ʹ͓͚Δ༧ଌॲཧͷ
    ߴ଎ԽϙΠϯτ͸ɺ޿ࠂࡏݿͷྔɺར༻͢Δ
    ϋʔυ΢ΣΞ΍ݴޠɺಛ௃ϕΫτϧͷେ͖͞
    ͳͲΑͬͯੑೳͷྑ͠ѱ͕͠େ͖͘มΘΔՄ
    ೳੑ͕͋Γ·͢
    • ※ඞͣߴ଎ԽͰ͖Δ͜ͱΛอূ͢Δ΋ͷͰ͸͋
    Γ·ͤΜ
    

    View Slide

26. GBDT ༧ଌॲཧͷߴ଎ԽϙΠϯτ
    • GBDT ͷ֤πϦʔͷϝϞϦ্Ͱͷදݱํ๏Λ
    ޻෉͢Δ
    • ಛ௃ϕΫτϧͷදݱํ๏Λ޻෉͢Δ
    • ༧ଌͷॲཧखॱʹେ͖͘ӨڹΛ༩͑͏Δ

    ͞ΒͳΔ޻෉Λ͢Δ
    

    View Slide

27. πϦʔͷϝϞϦ্Ͱͷ
    දݱํ๏Λ޻෉͢Δ
    

    View Slide

28. ϕʔεϥΠϯɿφΠʔϒͳ໦ߏ଄
    Y>
    
    Y>
    
    Y>
    

    View Slide

29. ϕʔεϥΠϯɿφΠʔϒͳ໦ߏ଄
    Y>
    
    Y>
    
    Y>
    ɾಛ௃ϕΫτϧ্ͷΠϯσΫε
    ɾ෼ׂ৚݅ͷ஋
    ɾ಺෦ϊʔυPS༿ϊʔυʁ
    ɾࠨͷࢠϊʔυ΁ͷࢀর
    ɾӈͷࢠϊʔυ΁ͷࢀর
    

    View Slide

30. ϕʔεϥΠϯɿφΠʔϒͳ໦ߏ଄
    • ࠨӈͷࢠϊʔυ΁ͷࢀরΛ࣋ͨͳ͚Ε͹

    ͍͚ͳͯ͘৑௕Ͱ͋Δ
    • ΩϟογϡϝϞϦతʹܾͯ͠๬·͍͠දݱͰ
    ͸ͳ͍
    

    View Slide

31. ໦ߏ଄Λϑϥοτͳ഑ྻͰදݱ
    Y>
    
    Y>
    
    Y>
    
    
    
    
    ಛ௃ϕΫτϧͷΠϯσΫε
    ෼ׂ৚݅ͷ஋༿ϊʔυͷ஋
    

    View Slide

32. ໦ߏ଄Λϑϥοτͳ഑ྻͰදݱ
    • ࢠϊʔυΛͨͲΔํ๏
    • ਌ϊʔυͷ഑ྻ্ͷΠϯσΫεΛ i ͱͨ͠ͱ͖ɺࠨͷࢠ
    ϊʔυ͸ 2 * (i + 1) - 1ɺӈͷࢠϊʔυ͸ 2 * (i + 1) Ͱࢀ
    রͰ͖Δ
    • φΠʔϒͳ໦ߏ଄ΑΓ͸ΩϟογϡϝϞϦతʹ༏͍ͩ͠Ζ
    ͏…
    • ͳ͓πϦʔ͸ৗʹ׬શೋ෼໦Ͱ͋Δͱ͸ݶΒͳ͍
    • Ұ෦ͷαϒπϦʔ͕ࣃൈ͚ʹͳΓ͏Δ͜ͱΛ

    ߟྀ͢Δඞཁ͕͋Δ
    

    View Slide

33. ωετͨ͠ if - else Ͱίʔυදݱ
    double predict_tree1(double[] x) {
    if (x[3] < 5) {
    if (x[8] < 2)
    return 9;
    else
    return 5;
    } else {
    if (x[7] < 3)
    return 4;
    else
    return 7;
    }
    }
    

    View Slide

34. ωετͨ͠ if - else Ͱίʔυදݱ
    • ωΠςΟϒίʔυʹม׵Ͱ͖Δ / ͞ΕΔݴޠͳΒߴ଎ͳॲཧ͕
    ظ଴Ͱ͖Δ
    • Java ͷ৔߹ͷ͓࿩
    • ASM ͳͲΛ࢖ͬͯ GBDT ͷ༧ଌϞσϧΛόΠτίʔυʹ

    ม׵ & ಈతΫϥεϩʔυ → JIT ίϯύΠϧ͕ظ଴Ͱ͖Δ
    • ҰํͰɺπϦʔͷਂ࣍͞ୈͰ͸ϝιουͷόΠτίʔυαΠ
    ζ͕େ͖͗ͯ͢ΠϯϥΠϯల։Λ્֐ͯ͠͠·͏͜ͱ΋…
    

    View Slide

35. ಛ௃ϕΫτϧͷ
    දݱํ๏Λ޻෉͢Δ
    

    View Slide

36. ಛ௃ϕΫτϧʹٻΊΒΕΔಛੑ
    • GBDT Ͱ͸ɺಛ௃ϕΫτϧͷ֤ཁૉͷࢀর͸ϥϯμϜ
    ΞΫηεʹͳΔ
    • ϩδεςΟοΫճؼ΍ factorization machines ͱ͸
    େ͖͘ҟͳΔ
    • ΩϟογϡϝϞϦతʹͳ͔ͳ͔͓ਏ͍ײ͡
    • ಛ௃ྔΛอ࣋͢Δۭؒޮ཰ͱࢀর࣌ͷॲཧ଎౓ͱͷτ
    ϨʔυΦϑʹͳΔ
    

    View Slide

37. ૉ๿ʹ഑ྻͰදݱ͢Δ
    x = { 3, 0, NaN, 7, 2, NaN, 4 };
    
    OB OB
    

    View Slide

38. ૉ๿ʹ഑ྻͰදݱ͢Δ
    • ۭؒޮ཰͸ྑ͘ͳ͍͕ɺࢀরੑೳ͸ߴ͍
    • ద͍ͯ͠Δέʔε
    • ಛ௃ϕΫτϧͷ࣍ݩ͕ൺֱతখ͍͞
    • ಛ௃ϕΫτϧ͕ൺֱత ີ (dense) Ͱ͋Δ
    • Missing value ͷѻ͍ʹؾΛ͚ͭΔඞཁ͕͋Δ
    

    View Slide

39. ࿈૝഑ྻͰදݱ͢Δ
    x = {
    0: 3,
    1: 0,
    3: 7,
    4: 2,
    6: 4
    }
    

    View Slide

40. ࿈૝഑ྻͰදݱ͢Δ
    • ࿈૝഑ྻ: ฏߧ໦΍ϋογϡදʹΑΔ࣮૷
    • ۭؒޮ཰͸ͦΕͳΓʹྑ͍ҰํͰɺࢀর࣌ͷΦʔόʔϔο
    υ͕ؾʹͳΓ͕ͪ
    • ద͍ͯ͠Δέʔε
    • ඇৗʹߴ࣍ݩͷಛ௃ϕΫτϧΛѻΘͳ͚Ε͹ͳΒͳ͍
    • ͦͷҰํͰɺ࣮ࡍͷಛ௃ϕΫτϧ͸ ૄ (sparse) Ͱ͋Δ
    

    View Slide

41. AssocVector తͳදݱΛ༻͍Δ
    J
    W
    J
    W
    J
    W
    J
    W
    J
    W
    

    View Slide

42. AssocVector తͳදݱΛ༻͍Δ
    • ಛ௃ϕΫτϧ্ͷΠϯσΫεͱ஋ͷ૊Λɺ

    ΠϯσΫεͰιʔτͯ͠഑ྻͰอ࣋͢Δ
    • ࢀর͸ೋ෼୳ࡧͰߦ͏ → O(log N)
    • ۭؒޮ཰͸࿈૝഑ྻΑΓ༏Ε͍ͯΔ
    • ΩϟογϡϝϞϦʹ΋༏͍͠දݱͳͷͰ͸…
    

    View Slide

43. ͞ΒͳΔ޻෉
    

    View Slide

44. ༧ଌͷॲཧखॱʹӨڹ͢Δ޻෉
    • ޿ࠂΫϦοΫ཰ͷ༧ଌ͸ 1 Ϣʔβ 1 ޿ࠂ͚ͩͰࡁΉ
    Α͏ͳ΋ͷͰ͸ͳ͍
    • ͢΂ͯͷ޿ࠂʹ͍ͭͯΫϦοΫ཰Λ༧ଌ͍ͨ͠
    • ͦΕͧΕͷ޿ࠂ͝ͱʹஞ࣍తʹ༧ଌͯ͠΋͍͍͕ɺ
    Ұׅͯ͠༧ଌ͢Δ͜ͱΛલఏͱͯ͠ޮ཰Խ͢Δํ๏
    ΋ଘࡏ͢Δ
    • ͨͩ͠ɺ͔ͳΓखͷࠐΜ࣮ͩ૷ʹͳΓ͕ͪ
    

    View Slide

45. Ұ࿈ͷ༧ଌʹͯෆมͷಛ௃ྔͰࢬמΓ͢Δ
    ੑผஉੑʁ
    
    ޿ࠂΧςΰϦήʔϜ
    
    ޿ࠂΧςΰϦԽহ඼
    ZFT OP
    Ϣʔβͷಛ௃ྔஉੑ ࡀ ʜ
    ͪ͜Β͸ઈରʹḷΒͳ͍
    

    View Slide

46. Ұ࿈ͷ༧ଌʹͯෆมͷಛ௃ྔͰࢬמΓ͢Δ
    • ྫ͑͹Ϣʔβͷಛ௃ྔ͸ɺ޿ࠂ͝ͱͷ༧ଌʹ͓͍
    ͯ͸ෆมͰ͋Δ
    • ෆมͷಛ௃ྔʹର͢Δ৚݅ΛຖճධՁ͢Δͷ͸
    ແବ
    • ༧ଌͷલॲཧͱͯ͠ɺͦͷΑ͏ͳ಺෦ϊʔυΛε
    Ωοϓͯ͠πϦʔΛίϯύΫτʹ͓ͯ͘͠
    • લॲཧ෼ͷΦʔόʔϔου͕ੜ͡Δ
    

    View Slide

47. QuickScorer
    • ޱ಄Ͱઆ໌͢Δͷ͸ແཧͳͷͰɺ໊લ͚ͩ

    ֮͑ͯޙͰ͙͙͍ͬͯͩ͘͞
    • ಛڐग़ئதΒ͍͠ͷͰɺ͝ར༻ʹ͸͝஫ҙΛ
    

    View Slide

48. ଞʹ΋͍Ζ͍Ζ͋Γ·͕͢…
    • Google Scholar Ͱ QuickScorer ΛҾ༻ͯ͠
    ͍Δ࿦จ΍ؔ࿈هࣄΛ୳͢ͷΛ͓͢͢Ί͠
    ·͢
    

    View Slide

49. ँࣙɾࢀߟจݙ
    

    View Slide

50. ँࣙɾࢀߟจݙ
    • ࠓճ͝঺հͨ͠΋ͷͷଟ͘͸ɺಙӬ͞Μ (@tkng)
    ΑΓΞυόΠε͍͍ͨͩͨ΋ͷʹͳΓ·͢
    • Evaluating boosted decision trees for billions of
    users
    • https://code.facebook.com/posts/
    975025089299409/evaluating-boosted-
    decision-trees-for-billions-of-users/
    

    View Slide

51. Thank you!
    

    View Slide