Share
Embed
A Stacking Ensemble Model for Prediction of Multi-type Tweet Engagements(RecSys Challenge 2020 3rd place solution)
RecSys2020論文読み会(オンライン)
https://connpass.com/event/189192/
Shuhei Goda
October 17, 2020
Transcript
22 slides
-
©2020 Wantedly, Inc. RecSys Challenge 2020 3rd place solution RecSys2020จಡΈձ
17.Oct.2020 - Shuhei Goda, Naomichi Agata, Yuya Matsumura for Prediction of Multi-type Tweet Engagements A Stacking Ensemble Model -
©2020 Wantedly, Inc. ߹ా पฏ @jy_msc Team Wantedly ០ ಓ
@agatan_ দଜ ༏ @yu__ya4 -
©2020 Wantedly, Inc. Twitter ʹ͓͚ΔϢʔβͷΤϯήʔδϝϯτΛ༧ଌ͢ΔλεΫ • (tweet ID, engaging user
ID) ʹର֤ͯ͠Τϯήʔδϝϯτͷ༗ແΛ༧ଌ͢Δ Multi-label binary classification • ༧ଌରͷϥϕϧ4छྨ: Like, Reply, RT and RT with comment ධՁࢦඪ2छྨ 1. PR-AUC 2. RCE (Relative Cross Entropy) CHALLENGE TASK φΠʔϒͳฏۉ༧ଌ͔Βͷ૬ରతͳ CE ͷྑ͞Λද͢ࢦඪ. ͕ߴ͍΄Ͳྑ͍. -
©2020 Wantedly, Inc. Challenge Ͱ༻͞Εͨσʔληοτใ • Tweet ʹؔ͢Δใ: tweet ID,
timestamp, text token, etc. • Τϯήʔδϝϯτ͢ΔϢʔβͷใ: engaging user ID, follower count, etc. • Τϯήʔδϝϯτ͞ΕΔϢʔβͷใ: engaged user ID, follower count, etc. • Τϯήʔδϝϯτใ(λʔήοτ): timestamps of the engagements ධՁσʔλͷׂํ๏ DATASET DESCRIPTION Training Data ( ~ 120 millions samples ) Testing Data Validation Data 1 week 1 week -
©2020 Wantedly, Inc. Pseudo Negative Label ʹΑΔෛྫ࡞ • ʮπΠʔτΛݟ͕ͨengagement͠ͳ͍ʯͱ͍͏ωΨςΟϒͳใͷެ։Λ͙. •
ෛྫΛϑΥϩʔάϥϑ͔ΒϥϯμϜαϯϓϦϯά͢Δ͜ͱͰ, ҎԼͷαϯϓϧΛࠞͥΔ. 1. πΠʔτΛݟ͕ͨengagement͠ͳ͔ͬͨαϯϓϧ 2. πΠʔτΛݟ͍ͯͳ͍͔Βengagement͠ͳ͔ͬͨαϯϓϧ GDPR ʹΑͬͯ༻ෆՄͷσʔλ͕ʑ૿͍͑ͯ͘ • GDPR ʹ४ڌ͍ͯ͠ΔͨΊ, ಛఆͷσʔλֶ͕शɾධՁڞʹ༻ෆՄͱͳΔ. • ܇࿅σʔληοτͷαΠζ, ։͔࢝Βऴྃ·Ͱͷ 3 ϲ݄Ͱ 1.6 ԯ -> 1.2 ԯ ͷݮগ. Privacy-Preserving -
©2020 Wantedly, Inc. σʔληοτͷαΠζ • ܇࿅σʔληοτ: 1.6ԯϨίʔυ (࠷ऴతʹ1.2ԯϨίʔυ) • ධՁ༻σʔληοτ:
1,500ສϨίʔυ (࠷ऴతʹ1,200ສϨίʔυ) • float32ͷsubmissionϑΝΠϧͷ߹ܭ͕ 4GB ͙Β͍ ҎԼͷܭࢉϦιʔεͰରԠ • Google BigQuery • Google Dataflow • Google Compute Engine (vCPUs: 64, Memory: 600GB) DATASET CHARACTERISTICS (1) -
©2020 Wantedly, Inc. ϥϕϧͷෆۉߧ • Like > RT > Reply
> RT with Comment ͷॱʹਖ਼ྫͷׂ߹͕খ͘͞ͳΔ. • Like 43% ͕ਖ਼ྫ͕ͩ, RT with Comment 0.7% ͔͠ਖ਼ྫΛ࣋ͨͳ͍. DATASET CHARACTERISTICS (2) -
©2020 Wantedly, Inc. ΤϯήʔδϝϯτؒͰͷߴ͍ڞىੑ • Ϣʔβ1ͭͷTweetʹରͯ͠ෳछྨͷΤϯήʔδϝϯτΛߦ͏߹͕͋Δ. • ಛఆͷΤϯήʔδϝϯτؒͰߴ͍ڞىੑΛ࣋ͭ. • e.g.
RT and Like , RT and RT with comment DATASET CHARACTERISTICS (3) -
©2020 Wantedly, Inc. OVERVIEW OF OUR SOLUTION Model Architecture •
Stacking LightGBMs Features • Categorical Features • Network Features • Text Features • Meta Features • etc. Training Process • Bagging with negative under sampling • Stratified K-Folds over Retweet with Comment -
©2020 Wantedly, Inc. MODEL ARCHITECTURE The First Stage Models The
Second Stage Models Like Models Reply Models RT Models RT with Comment Models Target Independent Features Target Dependent Features Like Models Reply Models RT Models RT with Comment Models Like Predictions Reply Predictions RT Predictions RT with Comment Predictions Meta Features -
©2020 Wantedly, Inc. The First Stage Models The Second Stage
Models Like Models Reply Models RT Models RT with Comment Models Target Independent Features Target Dependent Features Like Models Reply Models RT Models RT with Comment Models Like Predictions Reply Predictions RT Predictions RT with Comment Predictions Meta Features ୯ҰͷΤϯήʔδϝϯτΛ༧ଌ ͢ΔϞσϧΛ, Τϯήʔδϝϯτ ͷछྨ͚ͩ࡞͢Δ (1st stage models) 1st Stage MODEL ARCHITECTURE -
©2020 Wantedly, Inc. The First Stage Models The Second Stage
Models Like Models Reply Models RT Models RT with Comment Models Target Independent Features Target Dependent Features Like Models Reply Models RT Models RT with Comment Models Like Predictions Reply Predictions RT Predictions RT with Comment Predictions Meta Features 2nd Stage 1st stage modelsͷ༧ଌΛೖྗ ʹՃͨ͠ϞσϧΛ, Τϯήʔδ ϝϯτͷछྨ͚ͩ࡞͢Δ (2nd stage models) MODEL ARCHITECTURE -
©2020 Wantedly, Inc. ༷ʑͳΤϯίʔσΟϯάख๏ʹΑΔΧςΰϦมͷಛྔͷ࡞ • ΧʔσΟφϦςΟͷখ͍͞ΧςΰϦมʹ Label Encoding • e.g.
language, tweet type • ΧʔσΟφϦςΟͷେ͖͍ΧςΰϦมʹ Frequency Encoding & Target Encoding • e.g. tweet ID, user ID ΧςΰϦมͷΈ߹ΘͤΛ৽͍͠ΧςΰϦͱΈͳͨ͠ಛྔͷ࡞ • ΧςΰϦมؒͷෳࡶͳؔੑΛଊ͑Δ͜ͱ͕Ͱ͖Δ • e.g. Hashtag engaging user ID × Categorical Features FEATURES -
©2020 Wantedly, Inc. Graph Features FEATURES Follow͔ؔΒ෮ݩͨ͠ιʔγϟϧάϥϑ • ϢʔβؒͷؔੑͱਓؾΛදݱ͢Δ. •
Ϣʔβؒͷ1࣍ɾ2࣍ͷܨ͕Γͷ༗ແ ͔Βੜͨ͠ಛྔ • ϖʔδϥϯΫ ͔Βੜͨ͠ಛྔ LikeΛΤοδͱ͢Δάϥϑ • ϢʔβͷᅂੑͷྨࣅΛදݱ͢Δ. • Like Graph ্ Ͱ Random Walk with Restarts Λߦͬͨ࣌ͷ๚ճ -
©2020 Wantedly, Inc. Engaging User ͷίϯςϯπʹର͢Δؔ৺ΛςΩετ͔Βਪఆ͢Δ • Engaging User ͷ2छྨͷؔ৺Λಛྔͱͯ͠࠾༻͢Δ.
• Tweet ͷίϯςϯπʹର͢Δؔ৺ • Engaged User ʹର͢Δؔ৺ • ֤ϕΫτϧͷΈ߹ΘͤͷੵΛऔͬͯؔ৺ͱͯ͠දݱ͢Δ. • Tweet ͷϕΫτϧ: pre-trained multi-lingual Bert ͷதؒग़ྗ • Engaging User ͷϕΫτϧ: ΠϯλϥΫγϣϯͨ͠πΠʔτͷϕΫτϧฏۉ • Engaged User ͷϕΫτϧ: πΠʔτͨ͠ Tweet ͷϕΫτϧฏۉ Text Features FEATURES -
©2020 Wantedly, Inc. 1st stage models ͷ༧ଌΛಛྔͱͯ͠༻͢Δ • Τϯήʔδϝϯτؒͷڞىੑ͕ߴ͍ͨΊ, ༧ଌରͱͳΔΤϯήʔδϝϯτҎ֎ͷ
ֶशʹͦͷଞͷΤϯήʔδϝϯτใ͕ॏཁͱͳΔ. • Τϯήʔδϝϯτͷ༧ଌ݁ՌΛ user ID tweet ID ͳͲͷΧςΰϦͰू͢Δ. • user ID tweet ID ͷΤϯήʔδϝϯτ͢͠͞ / ͞Ε͢͞ Λߴ͍දݱྗͰ ѻ͏͜ͱ͕Ͱ͖Δ. Meta Features FEATURES -
©2020 Wantedly, Inc. Bagging with Negative Under-Samplingɹ • ϞσϧΛޮతʹֶश͢ΔͨΊʹ, খσʔληοτΛෳ࡞ͯ͠ෳͷϞσϧΛ
࡞͢Δ Bagging Λ࠾༻͢Δ. • ҎԼͷΑ͏ͳαϯϓϦϯάํ๏Λ࠾༻ͨ͠. 1. Negative User-Sampling Λద༻ͯ͠, σʔλαΠζΛখ͘͢͞Δ. 2. Like Retweet ͳͲͷΤϯήʔδϝϯτґવσʔλαΠζ͕େ͖͍ͷͰ, ࢦఆͨ͠ αΠζʹͳΔΑ͏ Random Sampling ͰߋʹσʔλαΠζΛখ͘͢͞Δ. TRAINING PROCESS Sampling Process -
©2020 Wantedly, Inc. ֤foldͷ RT with comment ͷਖ਼ྫ͕͘͠ͳΔΑ͏ͳ Stratified K-Folds
Λ࠾༻͢Δ • λʔήοτͷछྨຖʹڞ௨ͷׂઃఆΛར༻͢Δ. • λʔήοτຖʹҟͳΔׂઃఆΛ࠾༻͢Δͱ, 2nd stage models Ͱ͍ͬͯΔ Meta Features Target Encoding ʹΑΔ Leakage ͷӨڹ͕େ͖͘ͳΔͨΊ.ɹ • ͜Ε ֤छͷλʔήοτ͕શʹಠཱͰͳ͍͜ͱ͕ݪҼͰ͋Δ. • ܭࢉ࣌ؒΛߟྀͯ͠, fold 3 ʹઃఆ. TRAINING PROCESS Validation Strategy -
©2020 Wantedly, Inc. EXPERIMENTS Final Results 1st stage models ͱ
2nd stage models ͷൺֱ • ͲͪΒͷࢦඪͰ, 2nd stage models 1st stage models ΑΓྑ͍݁Ռ. • ͜ΕΒͷࠩ 2nd stage models Ͱ࠾༻ͨ͠ stacking ʹΑΔޮՌྔΛද͢. • ܇࿅είΞͱݕূείΞͷ͕ࠩେ͖͘ͳ͍ͬͯΔ͕, ྆ࢦඪͱ 1st stage ΑΓվળͯ͠ ͍ΔͨΊ, ͜ͷߏͰͳ͍ͱஅ -
©2020 Wantedly, Inc. EXPERIMENTS Training Data Size 2nd stage models
Ͱ, ܇࿅σʔλΛ૿͢΄ͲݕূείΞ͕ѱ͘ͳΔ • Meta Features ͱ Target Encoding ʹΑΔ Leakage ͕ݪҼͱਪଌ. • ܇࿅σʔλΛϋΠύʔύϥϝʔλͱͯ͠ݕূείΞͰ࠷దԽͨ͠. • Like 100,000, ͦͷଞͷλʔήοτ 1,000,000 ͱͨ͠. -
©2020 Wantedly, Inc. CONCLUSION • RecSys Challenge 2020 , Twitter
ʹ͓͚ΔϢʔβͷΤϯήʔδϝϯτ༧ଌ • Team Wantedly Ͱ, 2nd stage stacking ͷϞσϧߏΛ࠾༻͠, ҟͳΔΤ ϯήʔδϝϯτؒͷڞىੑΛޮతʹଊ͑ΒΕΔΑ͏ͳऔΓΈΛߦͬͨ. • https://github.com/wantedly/recsys2020-challenge -
©2020 Wantedly, Inc. Pseudo Negative Label ʹΑͬͯੜͨ͡ಛ ϑΥϩʔؔΛಛఆ͢Δ͜ͱ͕Ͱ͖Δ • ʮෛྫΛϑΥϩʔάϥϑ͔ΒϥϯμϜαϯϓϦϯάʯ͍ͯ͠ΔͷͰ,
ෛྫʮΤϯήʔδϯά ͢ΔϢʔβʯ->ʮΤϯήʔδϯά͞ΕͨϢʔβʯͷ͕ؔΓཱͭ. • ༩͑ΒΕ͍ͯΔؔ, ʮΤϯήʔδϯά͞ΕΔϢʔβʯ->ʮΤϯήʔδϯάͨ͠Ϣʔβʯ ͷΈ • શͳϑΥϩʔάϥϑ͕͋Ε, 1࣍ͷܨ͕ΓͰͳ͍Ϩίʔυਖ਼ྫͱ֬ఆ͢Δ. ࣌ؒํͷਖ਼ྫසͱෛྫසʹζϨ͕ੜ͡Δ • ʮෛྫΛϑΥϩʔάϥϑ͔ΒϥϯμϜαϯϓϦϯάʯ͍ͯ͠ΔͷͰ, Ϣʔβͷෛྫͷ࣌ؒํ ͷසʮϢʔβͷimpʯͰͳ͘ʮϑΥϩʔ͍ͯ͠ΔϢʔβ܈ͷπΠʔτʯʹґଘͯ͠ ͍ͯ, ਖ਼ྫͱෛྫͷ͕ͣΕΔՄೳੑ͕͋Δ. • ͜ΕʹΑΓ, ࣌ؒࠩͳͲͷಛྔΛՃ͢Δ͜ͱͰ݁ߏվળ͢Δ.