[読み会] TabNet: Attentive Interpretable Tabular Learning

読み会資料
TabNet: Attentive Interpretable Tabular Learning(ICLR, 2020, rejected)

Transcript

1. TabNet: Attentive Interpretable Tabular Learning ಡΈձ@2021/01/05 ༶໌఩
2.  • ஶऀ • Sercan O. Arik, Tomas Pfister • Google Cloud AI • ग़య: ArxivͷPreprint • ICLR 2020ͰϦδΣΫτ͞Εͨ࿦จ ࿦จ৘ใ
3.  • ςʔϒϧσʔλ޲͚ͷDNNϞσϧ • ܾఆ໦ͱNNϞσϧͷ͍͍ͱ͜औΓΛ໨ࢦͨ͠ख๏ • ղऍੑ + ਫ਼౓ ͷ޲্͕ୡ੒Ͱ͖ͨɽ ֓ཁ ͲΜͳ࿦จʁ
4.  • DNNͷϞσϧ͕ಛʹը૾,ݴޠ,Ի੠ͷ෼໺ͰSOTAͰ͋Δɽ • KaggleͳͲͷ෼ੳίϯϖͰ͸ॳΊʹܾఆ໦ϕʔεͷख๏͕ओྲྀ • ղऍੑ͕ߴ͍͔Β ং࿦ ݚڀഎܠ
5.  • ͳΜͰςʔϒϧσʔλʹରͯ͠ɼਂ૚ֶशΛऔΓೖΕ͍ͨͷ͔ʁ • େن໛ͳσʔληοτʹ͍ͨͯ͠͸ɼਂ૚ֶशʹΑͬͯ޲্͕ظ଴Ͱ͖Δ ͔Β • Deep Learning Scaling is Predictable, Empirically.(Hestness et al., 2017) ং࿦ ݚڀഎܠ
6.  • ςʔϒϧσʔλʹରͯ͠NNϞσϧΛ࢖͏3ͭͷϝϦοτ 1. ෳ਺ͷσʔλΛޮ཰Α͘ΤϯίʔσΟϯάͰ͖Δ 2. ಛ௃ྔΤϯδχΞϦϯάͷखؒΛݮΒͤΔ 3. End-to-endͰѻ͏͜ͱ͕Ͱ͖Δɽ ং࿦ ݚڀഎܠ
7.  • σʔλͷલॲཧΛߦΘͣʹend-to-endͰͷֶशΛߦ͑Δɽ • ஞ࣍஫ҙΛ༻͍Δ͜ͱͰղऍੑͷߴ͍Ϟσϧʹͳ͍ͬͯΔɽ • Local interpretability: ೖྗಛ௃ͷॏཁ౓ • Global interpretability: ֤ಛ௃ྔ͕Ϟσϧʹରͯ͠Ͳͷ͘Β͍Өڹ͔ͨ͠ ং࿦ ఏҊख๏ͷߩݙ
8.  • DNN+DT • ஞ࣍஫ҙΛ༻͍ͯɼಛ௃બ୒Λߦ͍ಛ௃ΛೖΕࠐΜͰ͍Δɽ • Tree-based learning • ಛ௃બ୒ʹDNNΛ༻͍͍ͯΔɽ • Feature Selection • ίϯύΫτͳදݱ͕Ͱ͖ͨɽ ؔ࿈ݚڀ
9.  • Attentive transformer • ಛ௃ྔʹରͯ͠࢖͏MaskͷֶशΛߦ͏ɽ • Feature transformer • ಛ௃ྔͷม׵ɼ࣍εςοϓʹ࢖͏΋ͷΛܾΊΔɽ ఏҊख๏ ॏཁͳύʔπ
10.  • ͜ΕҎ߱ग़ͯ͘Δ ͸εςοϓ1,2,...ʹରԠ͍ͯ͠Δ i ఏҊख๏ શମͷߏ଄
11.  • • : աڈͷMͰ࢖ΘΕ͍ͯΔ͔ʁʹΑͬͯ มΘΔॏΈ(࣮૷Ͱ͸ར༻੍ݶΈ͍ͨͳ΋ͷ) • Sparsemax: softmaxʹࣅͨ׆ੑԽؔ਺ M[i] = sparsemax(P[i] ⋅ hi (a[i − 1])) P[i] ఏҊख๏ Attentive Transformer: ϚεΫͷֶशΛߦ͏ɽ
12.  • SoftmaxΑΓ΋ૄʹͳΓ΍͍͢ ͔Βɼॏཁͳಛ௃ྔΛऔΓग़ ͠΍͍͢ ίϥϜ SparseMax (Andre et al., 2016)
13.  • ɼa͸࣍ͷεςοϓʹճ͞ΕΔ [d[i], a[i]] = fi (M[i] ⋅ f) ఏҊख๏ Feature Transformer: ೖྗΛม׵͠ɼ࣍ʹ࢖͏΋ͷΛܾΊΔ
14.  • ֤εςοϓ Λूܭ ͯ͠࠷ऴతͳ༧ଌʹ ༻͍Δ d[i] ఏҊख๏ ࠷ऴ༧ଌ
15.  • ಛ௃ྔͷॏཁ౓͸ϚεΫΛ࢖ͬͯܭࢉ͢Δ • ؆୯ʹܭࢉ͢ΔͨΊɼϚεΫͰ͸ͳ͘ಛ௃ྔΛ༻͍Δ ɹɹɹ ɹˠͲͷαϯϓϧ͕ॏཁ͔ʁ • → ಛ௃ྔͷॏཁ౓ ηb [i] = Nd ∑ c ReLU(db,c [i]) Magg−b,j = ∑Ns teps i=1 ηb [i]Mb,j [i] ∑D j=1 ∑Nsteps i=1 ηb [i]Mb,j [i] ఏҊख๏ ղऍੑʹ͍ͭͯ
16.  • Feature selection͕֤εςοϓʹରԠ ఏҊख๏ ಛ௃ྔબ୒ͷΠϝʔδ
17.  • ֤ϚεΫʹΑͬͯ࡞ΒΕΔಛ௃ྔ͕෼ذʹରԠ͍ͯ͠Δɽ ఏҊख๏ Ͳ͕ܾ͜ఆ໦Ά͍ͷʁ
18.  • ର߅ख๏: • ޯ഑ϒʔεςΟϯάܥ: LightGBM, XGBoost, CatBoost • NNϞσϧ • ͳʹͰൺ΂Δ͔ʁ • ςετσʔλʹର͢Δaccuracy • ϞσϧͷαΠζ ࣮ݧ ࣮ݧઃఆ
19.  • ࣮σʔλ(ForestCoverType)Ͱ͸ର߅ख๏ΑΓ΋ਫ਼౓͕ྑ͔ͬͨɽ ࣮ݧ݁Ռ ਫ਼౓ʹؔͯ͠
20.  • ϞσϧαΠζ͕ܰྔͰ΋ਫ਼౓͕͍͍ɽ ࣮ݧ݁Ռ ϞσϧαΠζʹؔͯ͠
21.  ࣮ݧ݁Ռ ղऍੑʹ͍ͭͯ • ͷ݁ՌΛՄࢹԽ • ߦ͕αϯϓϧɼྻ͕ಛ௃ྔ • ന͍ͱ͜Ζ͕ಛ௃ྔͱͯ͠ॏཁ ͱ൑அͨ͠ͱ͜Ζ ηb [i]
22.  • ஞ࣍஫ҙΛߦ͏͜ͱͰɼॏཁͳಛ௃ྔબ୒Λߦͳ͍ͬͯΔɽ • ϚεΫΛ༻͍Δ͜ͱͰղऍੑͷߴ͍Ϟσϧʹͳͬͨɽ • ༷ʑͳྖҬͷςʔϒϧσʔλͰ΋ੑೳΛൃشͰ͖Δ͜ͱΛࣔ͠ ͨɽ ·ͱΊ
23.  • Accuracy: 0.81, ROC-AUC: 0.78 ͓·͚ TitanicσʔληοτͰTabNetΛ༡ΜͰΈͨɽ
24.  ͓·͚ LightGBM vs NN model vs TabNet LightGBM NN model • TabNet: Accuracy: 0.81, ROC-AUC: 0.78 https://github.com/mei28/playground_python/blob/main/notebooks/titanic.ipynb ϋΠύϥ͸ॳظ஋ͷ··Ͱ νϡʔχϯάΛߦͳ͍ͬͯͳ͍