$30 off During Our Annual Pro Sale. View Details »

論文紹介:Gradient Boosted Feature Selection

S.Shota
October 06, 2017

論文紹介:Gradient Boosted Feature Selection

2017/10/6の論文紹介で使用したスライドです.

論文:
Z. Xu, G. Huang, K. Q. Weinberger, and A. X. Zheng, “Gradient boosted feature selection,” in Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD ’14, pp. 522-531, 2014.

S.Shota

October 06, 2017
Tweet

More Decks by S.Shota

Other Decks in Technology

Transcript

  1. (SBEJFOU#PPTUFE
    'FBUVSF4FMFDUJPO
    ;9V ()VBOH ,28FJOCFSHFS BOE"9;IFOH
    JO,%%ˏ QQˊ
    橆㞮䞔㜠㷕䏍
    䞔㜠ًر؍،橆㞮㷕㼔余
    䞔㜠ًر؍،㷕؝٦أ
    涯䊛灇瑔㹓 ⥜㡦䎃
    俕谏 绤實
    䎃剢傈

    View Slide

  2. 俑柃䞔㜠٥鼅׿׌椚歋
    • ;9V ()VBOH ,28FJOCFSHFS BOE"9;IFOH
    ˑ(SBEJFOUCPPTUFEGFBUVSFTFMFDUJPO ˒JO1SPDFFEJOHTPG
    UIFUI"$.4*(,%%JOUFSOBUJPOBMDPOGFSFODFPO
    ,OPXMFEHFEJTDPWFSZBOEEBUBNJOJOH ,%%ˏ
    QQ
    • -"440הכ殯ז׷暴䗙鼅䫛ך،فٗ٦ثד
    莆㄂׾䭯׏׋׋׭
    • (SBEJFOU#PPTUJOH׉ך׮ךח׮
    莆㄂ָ֮׏׋׋׭
    • LBHHMFדכ㢳欽ׁ׸גְ׷䩛岀׵׃ְ

    View Slide

  3. 嚊銲
    暴䗙鼅䫛،ٕ؞ٔؤيָ弫׋ׅץֹ勴⟝
    • ꟼ鸬ׅ׷暴䗙׾然㹋ח鼅䫛〳腉
    • ꬊ简䕎暴䗙ך湱✼⡲欽׾陎ⴽ〳腉
    • 暴䗙侧如⯋侧ח㼎׃鎘皾ꆀָ简䕎ח㢌⻉
    • 毙ז圓鸡׾䭯אر٦ةח׮⢪欽〳腉

    ꬊ简䕎⻉
    鎘皾ꆀָ㣐
    鎘皾ꆀָ㼭
    ؕ٦طٕ岀
    #PPTUJOH
    暴䗙侧ָ㟓ִ׷ה

    View Slide

  4. 嚊銲

    #PPTUJOH
    䓲ⴓ겲㐻׾穈׫さ׻ׇג
    מהאךⴓ겲㐻ה׃ג欽ְ׷䩛岀
    (SBEJFOU#PPTUJOH
    䓲ⴓ겲㐻ח㔐䌓加׾⢪欽׃
    湡涸ꟼ侧ך⺟ꂁַ׵䓲ⴓ겲㐻׀הך
    ꅾ׫׾锃侭ׅ׷䩛岀
    (SBEJFOU#PPTUJOH׾暴䗙鼅䫛ח⢪欽

    View Slide

  5. 暴䗙鼅䫛׾遤ֲה何㊣ׁ׸׷挿
    • 㷕统׃ג䖤׋ٌرٕך〳铣䚍
    • 㷕统װذأز儗ךًٌٔ⢪欽ꆀ٥鎘皾儗꟦
    • ٌرٕך害⻉䚍腉
    ➿邌涸ז䩛岀
    ꬊ简䕎暴䗙鼅䫛
    *OUSPEVDUJPO

    简䕎暴䗙鼅䫛
    • -"34
    • -"440
    • 3BOEPN'PSFTU
    • ؕ٦طٕ岀ך㼪Ⰵ

    View Slide

  6. *OUSPEVDUJPO

    暴䗙鼅䫛㉏겗ךؔ٦فٝفٗـٖي
    ꬊ简䕎זꟼ鸬׾罋䣁׃אא
    暴䗙䬄⳿חַַ׷鎘皾ꆀ㟓⸇׾䫇ִ׷
    简䕎暴䗙鼅䫛 • 简䕎ٌرٕ׾⟎㹀
    • 暴䗙꟦ךꬊ简䕎זꟼ鸬ָ罋䣁♶〳
    ꬊ简䕎暴䗙鼅䫛 • ꬊ简䕎זꟼ鸬׮罋䣁〳腉
    • ر٦ة侧ח㼎׃鎘皾ꆀָꬊ简䕎㢌⻉

    View Slide

  7. *OUSPEVDUJPO

    (SBEJFOU#PPTUFE'FBUVSF4FMFDUJPO (#'4

    简䕎陎ⴽ㐻
    -
    ظٕي
    (SBEJFOU
    #PPTUFE5SFF
    (#'4׮-"440הずֻׄ
    &NCFEEFE"QQSPBDIךא
    ꬊ简䕎㼎䘔
    㷕统ה暴䗙鼅䫛׾
    ず儗ח遤ֲ単穈׫

    View Slide

  8. 3FMBUFE8PSL

    • -"440
    • 湡涸ꟼ侧ח-
    ظٕيח״׷姻⵱⻉׾鷄⸇
    • )4*$-"440
    • אך暴䗙׀הח醱侧ךؕ٦طٕ׾黝欽ׅ׷
    -"440
    • N3.3
    • 暴䗙꟦ך湱✼䞔㜠ꆀח㛇בֻ暴䗙鼅䫛

    View Slide

  9. #BDLHSPVOE

    邌鎸חꟼׅ׷《׶寸׭
    • Ⰵ⸂ر٦ةإحز
    • ⳿⸂ٓكٕإحز
    • ✳⦼ⴓ겲ח搊挿׾䔲ג הׅ׷
    {
    x1
    , . . . ,
    xn} 2 Rd
    {y1, . . . , yn} 2 Y
    Y 2 {+1, 1}

    View Slide

  10. -"440חֶֽ׷剑黝⻉㉏겗
    䖤׵׸׷⸬卓
    • 䖤׵׸׷鍑ָأػ٦أ暴䗙鼅䫛
    • 害⻉䚍腉ךぢ♳姻⵱⻉
    'FBUVSF4FMFDUJPOXJUIUIF-
    OPSN

    log(1
    +
    exp(
    yi
    wT
    xi
    ))
    䴦㣟ꟼ侧
    min
    w
    X
    (xi,yi
    )
    l
    (xi, yi,
    w)
    + k
    w
    k
    1
    -
    ظٕي
    |w1| + · · · + |wd|
    אך⸬卓ךغٓٝأכוֲ锃侭ׅ׷

    View Slide

  11. 5IFDBQQFE-
    OPSN

    ˟ ך㹀纏ָ꟦麩׏גְ׷孡ָׅ׷㢳ⴓ如䒭דכ
    ♧䏝鼅䫛ׁ׸׋暴䗙חכ׉׸⟃♳
    لشٕذ؍׾Ⰵ׸זְ״ֲח何葺
    q✏
    (w) = min(|w1|, ✏) + · · · + min(|wd|, ✏)
    q✏
    (w)
    ぐ銲稆ך窫㼎⦼חٍؗحفׅ׷ꟼ侧
    ׾欽ְג
    q✏
    (w) = [q✏
    (w1
    ), . . . , q✏
    (wd
    )]
    q✏
    (wi
    ) = min(|wi|, ✏)
    min
    w
    X
    (xi,yi
    )
    l
    (xi, yi,
    w)
    + k
    w
    k
    1
    + µq✏(w)
    -
    ظٕيך
    葺ְ鵚⡂

    View Slide

  12. (SBEJFOU#PPTUFE'FBUVSF4FMFDUJPO

    㔐䌓加׾醱侧欽ְגⰅ⸂׾ꬊ简䕎⻉
    • ⢪欽〳腉ז㔐䌓加ך꧊さ
    • ⢪欽ׅ׷㔐䌓加׾寸׭׷كؙزٕ
    • Ⰵ⸂ח㼎ׅ׷ꬊ简䕎ⱖ⫷
    5أذحف湡חֶֽ׷ⴓ겲㐻ך✮庠⦼
    {h1, . . . , h|H|
    } 2 H
    2 R|H|
    (x)
    =
    [
    h
    1(x)
    , . . . , h|H|
    (x)]T
    2 R|H|
    H
    (x)
    =
    T
    X
    t=
    1
    t
    ht
    (x)
    简䕎ⴓ겲㐻חֶֽ׷
    ✮庠⦼ ח㼎䘔
    wT
    x

    View Slide

  13. (SBEJFOU#PPTUFE'FBUVSF4FMFDUJPO

    $BQQFE-
    ظٕيך《׶䪔ְ
    • ֮׷暴䗙G׾欽ְ׷加ח㼎ׅ׷ꅾ׫ךさ鎘⦼
    • さ鎘⦼׾暴䗙׀החꟼ侧 חⰅ׸ג筨ㄤ
    T
    X
    t=1
    |Fft t| Fft 2 {0, 1}
    ׋׌׃
    q✏
    min l( ) + k k1 + µ
    d
    X
    f=1
    q✏
    0
    B
    B
    B
    B
    B
    @
    T
    X
    t=1
    |Fft t|
    1
    C
    C
    C
    C
    C
    A
    (#'4ך剑黝⻉㉏겗ꬊ⳻ַא䗍ⴓ♶〳

    View Slide

  14. 0QUJNJ[BUJPO

    (SBEJFOU#PPTUJOHד剑黝⻉㉏겗׾鍑ֻ
    • 嫣أذحف׀הח剑䚈倯ぢח鹌׬如⯋׾刿倜
    • 㔐䌓加 ח㼎׃ ך㶷㖈׾鏩ׇל
    頾ך⺟ꂁ倯ぢח⹛ַֿׅהד䌢ח כ㟓⸇
    • 剑䚈倯ぢח鹌׬如⯋ ך䱱稊
    h 2 H h 2 H
    t⇤
    t⇤ = arg min
    t
    rL( )t
    䏟垥鯥ח屟׏ג剑䚈倯ぢח刿倜
    䏟垥꣬♴岀$%岀

    View Slide

  15. 0QUJNJ[BUJPO

    -
    姻⵱⻉ה&BSMZ4UPQQJOH
    • أذحف֮׋׶ך刿倜ꆀĔךאך如⯋ח
    㼎׃أذحف؟؎ؤē׌ֽ㟓⸇
    • 5أذحف䖓ך-
    ظٕي
    • -
    姻⵱⻉5أذحف湡דך&BSMZ4UPQQJOH
    • ⟃䖓-
    ظٕيך➿׻׶ח ׾⢪欽
    | |1 = ↵T
    ↵T

    View Slide

  16. 0QUJNJ[BUJPO

    أذحف湡חֶֽ׷湡涸ꟼ侧ך⺟ꂁ
    T0 + 1
    rL( )t =
    @l
    @ t
    + µ
    d
    X
    f=1
    rq✏
    0
    B
    B
    B
    B
    B
    B
    @
    T0
    X
    t=1
    Fft t
    1
    C
    C
    C
    C
    C
    C
    A
    T0
    X
    t=1
    Fft t < ✏
    T0
    X
    t=1
    Fft t ✏
    Fft
    0
    if
    if
    f
    Fft ה邌鎸
    f
    暴䗙Gָ鼅䫛
    PUIFSXJTF

    View Slide

  17. 0QUJNJ[BUJPO

    如⯋ ׾䱱稊̔Ⰻגך加׾嫣أذحف䱱稊
    • ㉏겗׾չ䴦㣟׾剑㼭⻉ׅ׷加ך䱱稊պח鯄䳔
    • ֿך㉏겗כ$"35ח״׏ג⸬桦涸ח鍑ֽ׷
    • $"35׾黝欽ׅ׷׋׭ח⺟ꂁ׾㢌䕎
    t⇤
    rL
    ( )t =
    n
    X
    i=
    1
    @l
    @H
    (xi
    )
    @H
    (xi
    )
    @ t
    + µ
    d
    X
    f=
    1
    f
    Fft
    ⺟ꂁ ה縧ֻ
    gi ht
    (xi
    )

    View Slide

  18. 加ך䱱稊㉏겗פך㢌䳔
    • 加כⰋג姻鋉⻉ׁ׸גְ׷ה⟎㹀
    • ⺟ꂁח如ךאך㹀侧갪׾⸇ִג侭椚
    • 侭椚׃׋穠卓ָ加ך䱱稊㉏겗ך㹀䒭⻉
    ✳⛦铎䊴  姻⵱⻉갪
    0QUJNJ[BUJPO

    X
    i
    h2
    t (xi
    )
    =
    1
    1
    2
    X
    i
    h2
    t (xi
    )
    , 1
    2
    X
    i
    g2
    i
    ht =
    arg min
    ht
    2H
    1
    2
    n
    X
    i=
    1
    (
    gi
    hi
    (xi))2 + µ
    d
    X
    f=
    1
    f
    Fft

    View Slide

  19. 0QUJNJ[BUJPO

    (#'4ך亻⡂؝٦س

    View Slide

  20. 4USVDUVSFE'FBUVSF4FMFDUJPO

    圓鸡⻉ׁ׸׋ر٦ةפך㼎䘔
    • 暴䗙鼅䫛ؿؚٓ ׾չ鼅䫛ׁ׸׋ؚٕ٦فח
    㾩׃גְ׷ז׵պה㢌刿
    • 暴㹀ךؚٕ٦فך暴䗙ָ鼅䫛ׁ׸׷ה
    ׉ךؚٕ٦فך暴䗙׾䱖➭涸涸ח⢪欽
    • 歗⫷ד׮չ㽷䨽걄㚖׾ꅾ挿涸ח鼅䫛ׇׁ׷պ
    הְ׏׋⢪欽倯岀ָ〳腉
    f

    View Slide

  21. 3FTVMU

    • 㹋꿀갪湡
    • ➂䊨涸זさ䧭ر٦ة
    • 㹋⚅歲ךر٦ةإحز
    • 圓鸡⻉ׁ׸׋ر٦ة穠艌ָ׿ر٦ةإحز
    • كٝثو٦ؙر٦ةإحز
    • 㹋꿀橆㞮
    • *OUFMJ ()[
    (#3". -JOVY04CJU

    View Slide

  22. 4ZOUIFUJD%BUB

    如⯋ך➂䊨ر٦ةח㼎ׅ׷✳⦼ⴓ겲
    • 简䕎ⴓꨄכ♶〳腉
    • 暴䗙[כ➭ךאך暴䗙ך简䕎穠さ
    ♶銲ז暴䗙
    • 鎮箺ر٦ةذأزر٦ةח
    ٓٝتيדⴓֽג㹋꿀
    • (#'4ה-姻⵱⻉➰ֹٗآأذ؍حؙ㔐䌓׾
    嫰鯰

    View Slide

  23. 4ZOUIFUJD%BUB

    View Slide

  24. 4ZOUIFUJD%BUB

    㹋꿀穠卓
    • -姻⵱⻉➰ֹٗآأذ؍حؙ㔐䌓 --3
    ָ鼅
    䫛׃׋暴䗙כ暴䗙Zך׫
    • --3דכꬊ简䕎ر٦ة׾ⴓꨄדֹגְזְ
    • (#'4דכ䗳銲ז暴䗙׾麓♶駈זֻ鼅䫛
    • (#'4כꬊ简䕎זꟼ⤘׾䭯׏גְג׮ⴓꨄ〳腉

    View Slide

  25. 4USVDUVSFE'FBUVSF4FMFDUJPO

    ָ׿הꟼ鸬ך֮׷鼋⠗㶨ؚٕ٦ف׾暴㹀
    • $PMPOر٦ةإحز
    • ؟ٝفٕ侧؟ٝفٕ
    ָ׿⻉؟ٝفٕ⨳䌢؟ٝفٕ
    • 暴䗙侧
    • 暴䗙כאךؙٓأة CBH
    ח׻ֽ׵׸גְ׷

    View Slide

  26. 4USVDUVSFE'FBUVSF4FMFDUJPO

    View Slide

  27. 4USVDUVSFE'FBUVSF4FMFDUJPO

    㹋꿀穠卓
    • (#'4כָ׿חꟼ鸬ׅ׷ؚٕ٦ف׾
    אח穾׶鴥׬ֿהָדֹגְ׷
    • ずׄؒٓ٦桦׾䭯א3BOEPN'PSFTUכ圫ղז
    CBHַ׵暴䗙׾鼅䫛׃גֶ׶鍑ꅸָ㔭ꨇ
    • (#'4ך㜥さ鼅䫛׃׋暴䗙חכ⡭ⴓז
    لشٕذ؍׾♷ִזְ׋׭✮庠礵䏝ָ넝ְ
    • -"440ך㜥さ鼅䫛ׅץֹCBHך暴䗙ח׮
    姻⵱⻉ח״׷⡭ⴓזلشٕذ؍ָⰅ׏גְ׷

    View Slide

  28. #FODINBSL%BUBTFUT

    ر٦ة侧ך㢳ְر٦ةإحزדך✳⦼ⴓ겲
    • Ⰻג✳⦼ⴓ겲ךةأؙח窟♧
    • 暴䗙侧״׶׮ر٦ة侧ך倯ָ㢳ְ״ֲז㉏겗
    鏣㹀׾䟝㹀

    View Slide

  29. #FODINBSL%BUBTFUT

    View Slide

  30. #FODINBSL%BUBTFUT

    㹋꿀穠卓
    • (#'4ה3BOEPN'PSFTUכر٦ة؟؎ؤָ
    殯ז׏גְג׮暴䗙鼅䫛ָ〳腉
    • )4*$-"440כر٦ةإحزך㣐ֹׁח佴䠬
    • (#'4ה3BOEPN'PSFTUך䚍腉כקרずׄ
    • (#'4ךⵃ挿כ暴䗙鼅䫛הず儗חٌرٕך㷕统
    ׮遤׏גְ׷׋׭넝鸞ד֮׷הְֲ挿
    • 3BOEPN'PSFTUכ8SBQQFS"QQSPBDI

    View Slide

  31. #FODINBSL%BUBTFUT

    䊬㣐ر٦ةإحز
    • 3BOEPN'PSFTUדכ
    鹈꟦⟃♳ַַ׷
    • (#'4כ侧鑐遤ך
    穠卓׾فٗحز
    • (#'4דכ鼅䫛ׁ׸׋
    暴䗙׮㼰זֻ
    ؒٓ٦桦׮何㊣

    View Slide

  32. #FODINBSL%BUBTFUT

    鼅䫛׃׋暴䗙ך颵
    • 鼅䫛׃׋暴䗙ד
    3#'47.׾㷕统
    • 㼰זְ暴䗙侧ח
    ְֶג(#'4ך
    ؒٓ٦桦ָ⡚ְ
    • ״׶ꅾ銲ז暴䗙׾鼅
    䫛דֹגְ׷
    • 暴䗙侧׾㟓װׅה
    קרずׄؒٓ٦桦

    View Slide

  33. #FODINBSL%BUBTFUT

    暴䗙ָꬊ䌢ח㢳ְ؛٦أ
    • 㔐ך䎂㖱׾邌爙
    • 鎮箺ذأز 
    חٓٝتيⴓⶴ
    • (#'4כ➭ך䩛岀״׶׮
    ⡚ְؒٓ٦桦׾麦䧭
    暴䗙侧
    ر٦ة侧

    View Slide

  34. #FODINBSL%BUBTFUT

    儗꟦鎘皾ꆀך嫰鯰
    • N3.3װ)4*$-"440 PS
    • 3BOEPN'PSFTU
    • (#'4
    • (#'4כأذحف侧װ加ך帾ׁכⵖꣲ〳腉
    • (#'4כ,%%$VQך״ֲז䊬㣐זر٦ة
    إحزד넝鸞ח⹛⡲
    O(d2) O(n2)
    O
    (
    p
    dn
    log
    n
    )
    O(dn)

    View Slide

  35. %JTDVTTJPO

    תה׭ה'VUVSF8PSL
    • (SBEJFOU#PPTUJOH׾欽ְג暴䗙䬄⳿הٌرٕ
    ך㷕统׾ず儗ח遤ֲ،ٕ؞ٔؤي׾䲿周
    • 儗꟦鎘皾ꆀכ暴䗙侧הر٦ةꆀך琎
    $"35ך儗꟦鎘皾ꆀ
    • 暴䗙侧חꟼׅ׷儗꟦鎘皾ꆀ׾何㊣ׅ׷䗳銲䚍
    • 3BOEPN'PSFTUה殯ז׶暴䗙ך鼅䫛然桦ח
    غ؎،أַַָ׷ךד♶銲ז暴䗙׾䱖ꤐ〳腉

    View Slide