論文紹介：Gradient Boosted Feature Selection

2017/10/6の論文紹介で使用したスライドです．

論文：
Z. Xu, G. Huang, K. Q. Weinberger, and A. X. Zheng, “Gradient boosted feature selection,” in Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD ’14, pp. 522-531, 2014.

Transcript

1. (SBEJFOU#PPTUFE
   'FBUVSF4FMFDUJPO
   ;9V ()VBOH ,28FJOCFSHFS BOE"9;IFOH
   JO,%%ˏ QQˊ
   橆㞮䞔㜠㷕䏍
   䞔㜠ًر؍،橆㞮㷕㼔余
   䞔㜠ًر؍،㷕؝٦أ
   涯䊛灇瑔㹓 ⥜㡦䎃
   俕谏 绤實
   䎃剢傈
   

   View Slide

2. 俑柃䞔㜠٥鼅׿׌椚歋
   • ;9V ()VBOH ,28FJOCFSHFS BOE"9;IFOH
   ˑ(SBEJFOUCPPTUFEGFBUVSFTFMFDUJPO ˒JO1SPDFFEJOHTPG
   UIFUI"$.4*(,%%JOUFSOBUJPOBMDPOGFSFODFPO
   ,OPXMFEHFEJTDPWFSZBOEEBUBNJOJOH ,%%ˏ
   QQ
   • -"440הכ殯ז׷暴䗙鼅䫛ך،فٗ٦ثד
   莆㄂׾䭯׏׋׋׭
   • (SBEJFOU#PPTUJOH׉ך׮ךח׮
   莆㄂ָ֮׏׋׋׭
   • LBHHMFדכ㢳欽ׁ׸גְ׷䩛岀׵׃ְ
   
   

   View Slide

3. 嚊銲
   暴䗙鼅䫛،ٕ؞ٔؤيָ弫׋ׅץֹ勴⟝
   • ꟼ鸬ׅ׷暴䗙׾然㹋ח鼅䫛〳腉
   • ꬊ简䕎暴䗙ך湱✼⡲欽׾陎ⴽ〳腉
   • 暴䗙侧如⯋侧ח㼎׃鎘皾ꆀָ简䕎ח㢌⻉
   • 毙ז圓鸡׾䭯אر٦ةח׮⢪欽〳腉
   
   ꬊ简䕎⻉
   鎘皾ꆀָ㣐
   鎘皾ꆀָ㼭
   ؕ٦طٕ岀
   #PPTUJOH
   暴䗙侧ָ㟓ִ׷ה
   

   View Slide

4. 嚊銲
   
   #PPTUJOH
   䓲ⴓ겲㐻׾穈׫さ׻ׇג
   מהאךⴓ겲㐻ה׃ג欽ְ׷䩛岀
   (SBEJFOU#PPTUJOH
   䓲ⴓ겲㐻ח㔐䌓加׾⢪欽׃
   湡涸ꟼ侧ך⺟ꂁַ׵䓲ⴓ겲㐻׀הך
   ꅾ׫׾锃侭ׅ׷䩛岀
   (SBEJFOU#PPTUJOH׾暴䗙鼅䫛ח⢪欽
   

   View Slide

5. 暴䗙鼅䫛׾遤ֲה何㊣ׁ׸׷挿
   • 㷕统׃ג䖤׋ٌرٕך〳铣䚍
   • 㷕统װذأز儗ךًٌٔ⢪欽ꆀ٥鎘皾儗꟦
   • ٌرٕך害⻉䚍腉
   ➿邌涸ז䩛岀
   ꬊ简䕎暴䗙鼅䫛
   *OUSPEVDUJPO
   
   简䕎暴䗙鼅䫛
   • -"34
   • -"440
   • 3BOEPN'PSFTU
   • ؕ٦طٕ岀ך㼪Ⰵ
   

   View Slide

6. *OUSPEVDUJPO
   
   暴䗙鼅䫛㉏겗ךؔ٦فٝفٗـٖي
   ꬊ简䕎זꟼ鸬׾罋䣁׃אא
   暴䗙䬄⳿חַַ׷鎘皾ꆀ㟓⸇׾䫇ִ׷
   简䕎暴䗙鼅䫛 • 简䕎ٌرٕ׾⟎㹀
   • 暴䗙꟦ךꬊ简䕎זꟼ鸬ָ罋䣁♶〳
   ꬊ简䕎暴䗙鼅䫛 • ꬊ简䕎זꟼ鸬׮罋䣁〳腉
   • ر٦ة侧ח㼎׃鎘皾ꆀָꬊ简䕎㢌⻉
   

   View Slide

7. *OUSPEVDUJPO
   
   (SBEJFOU#PPTUFE'FBUVSF4FMFDUJPO (#'4
   
   简䕎陎ⴽ㐻
   -
   ظٕي
   (SBEJFOU
   #PPTUFE5SFF
   (#'4׮-"440הずֻׄ
   &NCFEEFE"QQSPBDIךא
   ꬊ简䕎㼎䘔
   㷕统ה暴䗙鼅䫛׾
   ず儗ח遤ֲ単穈׫
   

   View Slide

8. 3FMBUFE8PSL
   
   • -"440
   • 湡涸ꟼ侧ח-
   ظٕيח״׷姻⵱⻉׾鷄⸇
   • )4*$-"440
   • אך暴䗙׀הח醱侧ךؕ٦طٕ׾黝欽ׅ׷
   -"440
   • N3.3
   • 暴䗙꟦ך湱✼䞔㜠ꆀח㛇בֻ暴䗙鼅䫛
   

   View Slide

9. #BDLHSPVOE
   
   邌鎸חꟼׅ׷《׶寸׭
   • Ⰵ⸂ر٦ةإحز
   • ⳿⸂ٓكٕإحز
   • ✳⦼ⴓ겲ח搊挿׾䔲ג הׅ׷
   {
   x1
   , . . . ,
   xn} 2 Rd
   {y1, . . . , yn} 2 Y
   Y 2 {+1, 1}
   

   View Slide

10. -"440חֶֽ׷剑黝⻉㉏겗
    䖤׵׸׷⸬卓
    • 䖤׵׸׷鍑ָأػ٦أ暴䗙鼅䫛
    • 害⻉䚍腉ךぢ♳姻⵱⻉
    'FBUVSF4FMFDUJPOXJUIUIF-
    OPSN
    
    log(1
    +
    exp(
    yi
    wT
    xi
    ))
    䴦㣟ꟼ侧
    min
    w
    X
    (xi,yi
    )
    l
    (xi, yi,
    w)
    + k
    w
    k
    1
    -
    ظٕي
    |w1| + · · · + |wd|
    אך⸬卓ךغٓٝأכוֲ锃侭ׅ׷
    

    View Slide

11. 5IFDBQQFE-
    OPSN
    
    ˟ ך㹀纏ָ꟦麩׏גְ׷孡ָׅ׷㢳ⴓ如䒭דכ
    ♧䏝鼅䫛ׁ׸׋暴䗙חכ׉׸⟃♳
    لشٕذ؍׾Ⰵ׸זְ״ֲח何葺
    q✏
    (w) = min(|w1|, ✏) + · · · + min(|wd|, ✏)
    q✏
    (w)
    ぐ銲稆ך窫㼎⦼חٍؗحفׅ׷ꟼ侧
    ׾欽ְג
    q✏
    (w) = [q✏
    (w1
    ), . . . , q✏
    (wd
    )]
    q✏
    (wi
    ) = min(|wi|, ✏)
    min
    w
    X
    (xi,yi
    )
    l
    (xi, yi,
    w)
    + k
    w
    k
    1
    + µq✏(w)
    -
    ظٕيך
    葺ְ鵚⡂
    

    View Slide

12. (SBEJFOU#PPTUFE'FBUVSF4FMFDUJPO
    
    㔐䌓加׾醱侧欽ְגⰅ⸂׾ꬊ简䕎⻉
    • ⢪欽〳腉ז㔐䌓加ך꧊さ
    • ⢪欽ׅ׷㔐䌓加׾寸׭׷كؙزٕ
    • Ⰵ⸂ח㼎ׅ׷ꬊ简䕎ⱖ⫷
    5أذحف湡חֶֽ׷ⴓ겲㐻ך✮庠⦼
    {h1, . . . , h|H|
    } 2 H
    2 R|H|
    (x)
    =
    [
    h
    1(x)
    , . . . , h|H|
    (x)]T
    2 R|H|
    H
    (x)
    =
    T
    X
    t=
    1
    t
    ht
    (x)
    简䕎ⴓ겲㐻חֶֽ׷
    ✮庠⦼ ח㼎䘔
    wT
    x
    

    View Slide

13. (SBEJFOU#PPTUFE'FBUVSF4FMFDUJPO
    
    $BQQFE-
    ظٕيך《׶䪔ְ
    • ֮׷暴䗙G׾欽ְ׷加ח㼎ׅ׷ꅾ׫ךさ鎘⦼
    • さ鎘⦼׾暴䗙׀החꟼ侧 חⰅ׸ג筨ㄤ
    T
    X
    t=1
    |Fft t| Fft 2 {0, 1}
    ׋׌׃
    q✏
    min l( ) + k k1 + µ
    d
    X
    f=1
    q✏
    0
    B
    B
    B
    B
    B
    @
    T
    X
    t=1
    |Fft t|
    1
    C
    C
    C
    C
    C
    A
    (#'4ך剑黝⻉㉏겗ꬊ⳻ַא䗍ⴓ♶〳
    

    View Slide

14. 0QUJNJ[BUJPO
    
    (SBEJFOU#PPTUJOHד剑黝⻉㉏겗׾鍑ֻ
    • 嫣أذحف׀הח剑䚈倯ぢח鹌׬如⯋׾刿倜
    • 㔐䌓加 ח㼎׃ ך㶷㖈׾鏩ׇל
    頾ך⺟ꂁ倯ぢח⹛ַֿׅהד䌢ח כ㟓⸇
    • 剑䚈倯ぢח鹌׬如⯋ ך䱱稊
    h 2 H h 2 H
    t⇤
    t⇤ = arg min
    t
    rL( )t
    䏟垥鯥ח屟׏ג剑䚈倯ぢח刿倜
    䏟垥꣬♴岀$%岀
    
    

    View Slide

15. 0QUJNJ[BUJPO
    
    -
    姻⵱⻉ה&BSMZ4UPQQJOH
    • أذحف֮׋׶ך刿倜ꆀĔךאך如⯋ח
    㼎׃أذحف؟؎ؤē׌ֽ㟓⸇
    • 5أذحف䖓ך-
    ظٕي
    • -
    姻⵱⻉5أذحف湡דך&BSMZ4UPQQJOH
    • ⟃䖓-
    ظٕيך➿׻׶ח ׾⢪欽
    | |1 = ↵T
    ↵T
    

    View Slide

16. 0QUJNJ[BUJPO
    
    أذحف湡חֶֽ׷湡涸ꟼ侧ך⺟ꂁ
    T0 + 1
    rL( )t =
    @l
    @ t
    + µ
    d
    X
    f=1
    rq✏
    0
    B
    B
    B
    B
    B
    B
    @
    T0
    X
    t=1
    Fft t
    1
    C
    C
    C
    C
    C
    C
    A
    T0
    X
    t=1
    Fft t < ✏
    T0
    X
    t=1
    Fft t ✏
    Fft
    0
    if
    if
    f
    Fft ה邌鎸
    f
    暴䗙Gָ鼅䫛
    PUIFSXJTF
    
    

    View Slide

17. 0QUJNJ[BUJPO
    
    如⯋ ׾䱱稊̔Ⰻגך加׾嫣أذحف䱱稊
    • ㉏겗׾չ䴦㣟׾剑㼭⻉ׅ׷加ך䱱稊պח鯄䳔
    • ֿך㉏겗כ$"35ח״׏ג⸬桦涸ח鍑ֽ׷
    • $"35׾黝欽ׅ׷׋׭ח⺟ꂁ׾㢌䕎
    t⇤
    rL
    ( )t =
    n
    X
    i=
    1
    @l
    @H
    (xi
    )
    @H
    (xi
    )
    @ t
    + µ
    d
    X
    f=
    1
    f
    Fft
    ⺟ꂁ ה縧ֻ
    gi ht
    (xi
    )
    

    View Slide

18. 加ך䱱稊㉏겗פך㢌䳔
    • 加כⰋג姻鋉⻉ׁ׸גְ׷ה⟎㹀
    • ⺟ꂁח如ךאך㹀侧갪׾⸇ִג侭椚
    • 侭椚׃׋穠卓ָ加ך䱱稊㉏겗ך㹀䒭⻉
    ✳⛦铎䊴  姻⵱⻉갪
    0QUJNJ[BUJPO
    
    X
    i
    h2
    t (xi
    )
    =
    1
    1
    2
    X
    i
    h2
    t (xi
    )
    , 1
    2
    X
    i
    g2
    i
    ht =
    arg min
    ht
    2H
    1
    2
    n
    X
    i=
    1
    (
    gi
    hi
    (xi))2 + µ
    d
    X
    f=
    1
    f
    Fft
    

    View Slide

19. 0QUJNJ[BUJPO
    
    (#'4ך亻⡂؝٦س
    

    View Slide

20. 4USVDUVSFE'FBUVSF4FMFDUJPO
    
    圓鸡⻉ׁ׸׋ر٦ةפך㼎䘔
    • 暴䗙鼅䫛ؿؚٓ ׾չ鼅䫛ׁ׸׋ؚٕ٦فח
    㾩׃גְ׷ז׵պה㢌刿
    • 暴㹀ךؚٕ٦فך暴䗙ָ鼅䫛ׁ׸׷ה
    ׉ךؚٕ٦فך暴䗙׾䱖➭涸涸ח⢪欽
    • 歗⫷ד׮չ㽷䨽걄㚖׾ꅾ挿涸ח鼅䫛ׇׁ׷պ
    הְ׏׋⢪欽倯岀ָ〳腉
    f
    

    View Slide

21. 3FTVMU
    
    • 㹋꿀갪湡
    • ➂䊨涸זさ䧭ر٦ة
    • 㹋⚅歲ךر٦ةإحز
    • 圓鸡⻉ׁ׸׋ر٦ة穠艌ָ׿ر٦ةإحز
    • كٝثو٦ؙر٦ةإحز
    • 㹋꿀橆㞮
    • *OUFMJ ()[
    (#3". -JOVY04CJU
    

    View Slide

22. 4ZOUIFUJD%BUB
    
    如⯋ך➂䊨ر٦ةח㼎ׅ׷✳⦼ⴓ겲
    • 简䕎ⴓꨄכ♶〳腉
    • 暴䗙[כ➭ךאך暴䗙ך简䕎穠さ
    ♶銲ז暴䗙
    • 鎮箺ر٦ةذأزر٦ةח
    ٓٝتيדⴓֽג㹋꿀
    • (#'4ה-姻⵱⻉➰ֹٗآأذ؍حؙ㔐䌓׾
    嫰鯰
    

    View Slide

23. 4ZOUIFUJD%BUB
    
    

    View Slide

24. 4ZOUIFUJD%BUB
    
    㹋꿀穠卓
    • -姻⵱⻉➰ֹٗآأذ؍حؙ㔐䌓 --3
    ָ鼅
    䫛׃׋暴䗙כ暴䗙Zך׫
    • --3דכꬊ简䕎ر٦ة׾ⴓꨄדֹגְזְ
    • (#'4דכ䗳銲ז暴䗙׾麓♶駈זֻ鼅䫛
    • (#'4כꬊ简䕎זꟼ⤘׾䭯׏גְג׮ⴓꨄ〳腉
    

    View Slide

25. 4USVDUVSFE'FBUVSF4FMFDUJPO
    
    ָ׿הꟼ鸬ך֮׷鼋⠗㶨ؚٕ٦ف׾暴㹀
    • $PMPOر٦ةإحز
    • ؟ٝفٕ侧؟ٝفٕ
    ָ׿⻉؟ٝفٕ⨳䌢؟ٝفٕ
    • 暴䗙侧
    • 暴䗙כאךؙٓأة CBH
    ח׻ֽ׵׸גְ׷
    

    View Slide

26. 4USVDUVSFE'FBUVSF4FMFDUJPO
    
    

    View Slide

27. 4USVDUVSFE'FBUVSF4FMFDUJPO
    
    㹋꿀穠卓
    • (#'4כָ׿חꟼ鸬ׅ׷ؚٕ٦ف׾
    אח穾׶鴥׬ֿהָדֹגְ׷
    • ずׄؒٓ٦桦׾䭯א3BOEPN'PSFTUכ圫ղז
    CBHַ׵暴䗙׾鼅䫛׃גֶ׶鍑ꅸָ㔭ꨇ
    • (#'4ך㜥さ鼅䫛׃׋暴䗙חכ⡭ⴓז
    لشٕذ؍׾♷ִזְ׋׭✮庠礵䏝ָ넝ְ
    • -"440ך㜥さ鼅䫛ׅץֹCBHך暴䗙ח׮
    姻⵱⻉ח״׷⡭ⴓזلشٕذ؍ָⰅ׏גְ׷
    

    View Slide

28. #FODINBSL%BUBTFUT
    
    ر٦ة侧ך㢳ְر٦ةإحزדך✳⦼ⴓ겲
    • Ⰻג✳⦼ⴓ겲ךةأؙח窟♧
    • 暴䗙侧״׶׮ر٦ة侧ך倯ָ㢳ְ״ֲז㉏겗
    鏣㹀׾䟝㹀
    

    View Slide

29. #FODINBSL%BUBTFUT
    
    

    View Slide

30. #FODINBSL%BUBTFUT
    
    㹋꿀穠卓
    • (#'4ה3BOEPN'PSFTUכر٦ة؟؎ؤָ
    殯ז׏גְג׮暴䗙鼅䫛ָ〳腉
    • )4*$-"440כر٦ةإحزך㣐ֹׁח佴䠬
    • (#'4ה3BOEPN'PSFTUך䚍腉כקרずׄ
    • (#'4ךⵃ挿כ暴䗙鼅䫛הず儗חٌرٕך㷕统
    ׮遤׏גְ׷׋׭넝鸞ד֮׷הְֲ挿
    • 3BOEPN'PSFTUכ8SBQQFS"QQSPBDI
    

    View Slide

31. #FODINBSL%BUBTFUT
    
    䊬㣐ر٦ةإحز
    • 3BOEPN'PSFTUדכ
    鹈꟦⟃♳ַַ׷
    • (#'4כ侧鑐遤ך
    穠卓׾فٗحز
    • (#'4דכ鼅䫛ׁ׸׋
    暴䗙׮㼰זֻ
    ؒٓ٦桦׮何㊣
    

    View Slide

32. #FODINBSL%BUBTFUT
    
    鼅䫛׃׋暴䗙ך颵
    • 鼅䫛׃׋暴䗙ד
    3#'47.׾㷕统
    • 㼰זְ暴䗙侧ח
    ְֶג(#'4ך
    ؒٓ٦桦ָ⡚ְ
    • ״׶ꅾ銲ז暴䗙׾鼅
    䫛דֹגְ׷
    • 暴䗙侧׾㟓װׅה
    קרずׄؒٓ٦桦
    

    View Slide

33. #FODINBSL%BUBTFUT
    
    暴䗙ָꬊ䌢ח㢳ְ؛٦أ
    • 㔐ך䎂㖱׾邌爙
    • 鎮箺ذأز 
    חٓٝتيⴓⶴ
    • (#'4כ➭ך䩛岀״׶׮
    ⡚ְؒٓ٦桦׾麦䧭
    暴䗙侧
    ر٦ة侧
    

    View Slide

34. #FODINBSL%BUBTFUT
    
    儗꟦鎘皾ꆀך嫰鯰
    • N3.3װ)4*$-"440 PS
    • 3BOEPN'PSFTU
    • (#'4
    • (#'4כأذحف侧װ加ך帾ׁכⵖꣲ〳腉
    • (#'4כ,%%$VQך״ֲז䊬㣐זر٦ة
    إحزד넝鸞ח⹛⡲
    O(d2) O(n2)
    O
    (
    p
    dn
    log
    n
    )
    O(dn)
    

    View Slide

35. %JTDVTTJPO
    
    תה׭ה'VUVSF8PSL
    • (SBEJFOU#PPTUJOH׾欽ְג暴䗙䬄⳿הٌرٕ
    ך㷕统׾ず儗ח遤ֲ،ٕ؞ٔؤي׾䲿周
    • 儗꟦鎘皾ꆀכ暴䗙侧הر٦ةꆀך琎
    $"35ך儗꟦鎘皾ꆀ
    • 暴䗙侧חꟼׅ׷儗꟦鎘皾ꆀ׾何㊣ׅ׷䗳銲䚍
    • 3BOEPN'PSFTUה殯ז׶暴䗙ך鼅䫛然桦ח
    غ؎،أַַָ׷ךד♶銲ז暴䗙׾䱖ꤐ〳腉
    

    View Slide