Counterfactual learning to rank: introduction

Transcript

1. Unbiased Learning to Rank May 7, 2020

2. Learning to rank ໰୊ઃఆ Supervised LTR Pointwise loss Pairwise loss

   Listtwise loss Counterfactual Learning to Rank Counterfactual Evaluation Inverse Propensity Scoring Propensity-weighted Learning to Rank 2

3. Learning to rank: ໰୊ઃఆ ೖྗɿ จॻͷू߹ D ग़ྗɿ จॻͷॱҐ R

   = (R1; R2; R3:::) ͨͩ͠ɺ֤จॻʹ͸Ϟσϧ f„ ʹΑͬͯείΞ͕͍͍ͭͯͯ f„ (R1) – f„ (R2) – f„ (R3) ::: ͱͳ͍ͬͯΔɻ(ߴ͍είΞ͕෇͚ΒΕΔ΄ͲॱҐ͕ߴ͍) Learning to Rank (LTR) ͷ໨త͸࠷దͳॱҐΛग़ྗ͢ΔϞσϧ f„ ͷύϥϝʔλ „ Λ σʔλ͔ΒٻΊΔ͜ͱɻ 3

4. Supervised LTR ڭࢣ͋Γ LTR Ͱ͸ɺ › ݕࡧΫΤϦ › จॻू߹ ›

   ॱҐͷϥϕϧ ΛؚΉσʔληοτΛ࢖ͬͯϞσϧύϥϝʔλΛٻΊΔɻ ڭࢣ͋Γ LTR Ͱ༻͍ΒΕΔଛࣦ͸ओʹ 3 ͭɿ › Pointwise loss › Pairwise loss › Listwise loss y (d) ʹΑͬͯɺจॻ d ͷݕࡧΫΤϦ΁ͷؔ࿈౓Λද͢ͱ͢Δɻ(େ͖͍΄ͲॱҐͷ্Ґʹ ͖ͯཉ͍͠) 4

5. Pointwise loss Pointwise loss ͸ɺॱҐͷਪఆΛ෼ྨɾճؼ໰୊ͱͯ͠ղ͘ɻྫ͑͹ɺ௨ৗͷճؼଛࣦ (squared loss) ͱͯ͠ҎԼͷΑ͏ʹ༩͑Δɿ Lpointwise :=

   1 N N X i=1 (f„ (di) ` y (di))2 Pointwise loss ͷ໰୊఺͸ɺϞσϧͷग़ྗΛॱҐͱͯ͠࢖͏͜ͱΛߟྀʹೖΕ͍ͯͳ͍͜ ͱɻLTR Ͱ͸ग़ྗͱͯ͠ಘΒΕΔείΞΛฒͼସ͑ͯಘΒΕΔॱҐʹͷΈؔ৺͕͋Δɻ 5

6. Pairwise loss Pairwise loss Ͱ͸ɺ2 ͭͷจॻؒͷ૬ରతͳείΞͷେখΛߟྀʹ͍ΕΔɻྫ͑͹ɺҎԼ ͷΑ͏ͳ hinge-loss Λ༩͑Δʀ Lpairwise

   := X y(di)>y(dj) max (0; 1 ` (f„ (di) ` f„ (di))): ॱҐ͕૬ରతʹߴ͍จॻ͸είΞ͕ߴ͘ɺॱҐ͕௿͍จॻ͸είΞΛ௿͘͢Δؾ࣋ͪɻ Pairwise loss ͷ໰୊఺͸ɺશͯͷهࣄϖΞΛಉ༷ʹѻ͏͜ͱɻ࣮ͦͯ͠༻্ top100 ͱ top10 ͸ޙऀͷํ͕ॏࢹ͞ΕΔ͜ͱɻPairwise loss Ͱ͸ top100 ͷԼͷํͷॱҐΛվળ ͤ͞ΔͨΊʹ্ҐͷॱҐΛ٘ਜ਼ʹ͢Δ͜ͱ͕͋Γ͑ͯ͠·͏ɻ 6

7. Listwise loss Listwise loss Ͱ͸ॱҐࢦඪΛ௚઀࠷దԽ͢Δɻ՝୊͸ɺॱҐࢦඪ͕ඍ෼ՄೳͰͳ͍͜ͱɻ ྫ͑͹ɺDCG ͸ɿ DCG = N

   X i=1 y (di) log2 (rank (di) + 1) Ͱ͋Δ͕ɺlog2 (rank (di) + 1) ͸ඍ෼ෆՄೳͰ͋Δɻ ͦͷͨΊʹ֬཰తۙࣅΛ༻͍Δํ๏ (ListNetɺListMLE) ΍ɺώϡʔϦεςΟοΫ΍ॱҐ ࢦඪͷό΢ϯυΛ࠷దԽ͢Δख๏͕͋Δɻ(LambdaRankɺLambdaLoss) ྫ͑͹ɺ LambdaRank ͷଛࣦ͸ DCG ͷό΢ϯυͱͳ͍ͬͯΔɿ LLambdaRank := X y(di)>y(dj) log (1 + exp (f„ (dj) ` f„ (di))) j´DCGj 7

8. ҼՌධՁ ໨తɿ৽͍͠ϥϯΩϯάؔ਺ f„ ΛɺผͷϥϯΩϯάؔ਺ fdeploy ͷԼͰूΊΒΕͨաڈ ͷσʔλ (ΫϦοΫσʔλͳͲ) Λ࢖ͬͯධՁ͍ͨ͠ɻ ҎԼͷ

   2 ͭͷ৔߹ʹ͍ͭͯߟ͑Δɻ › શͯͷจॻʹ͍ͭͯਅͷؔ࿈౓ y (di) ͕ط஌Ͱ͋Δ࣌ › y (di) ͸Θ͔Βͳ͍͕ɺΫϦοΫ৘ใͳͲͷ҉໧తͳϑΟʔυόοΫͷΈར༻Մೳͳ࣌ 8

9. ҼՌධՁɿϥϕϧ͕ط஌ͳΒ͹׬શʹධՁ͕Ͱ͖Δ શͯͷจॻʹ͍ͭͯਅͷϥϕϧ y (di) ͕ط஌Ͱ͋Δ࣌ɺIR(৘ใݕࡧ) ࢦඪΛܭࢉͰ͖Δɿ ´ (f„; D; y)

   = X di2D – (rank (di j f„; D)) ´ y (di) ͜͜Ͱɺ– ͸ॱҐॏΈ෇͚ؔ਺Ͱ͋ͬͯɺྫ͑͹ɿ APR: – (r) = r DCG: – (r) = 1 log2 (1+r) ͳͲ͕༻͍ΒΕΔɻ 9

10. ҼՌධՁ y (di) ͸Θ͔Βͳ͍͕ɺΫϦοΫ৘ใͳͲͷ҉໧తͳϑΟʔυόοΫͷΈར༻Մೳͳ࣌ɿ › ͋Δจॻʹର͢ΔΫϦοΫ͸ɺͦͷจॻ͕ؔ࿈͍ͯ͠Δ͜ͱΛࣔ͢ɺόΠΞεɾϊΠζ ͖ͭͷࢦඪʹͳ͍ͬͯΔɻ › ΫϦοΫ͞Εͳ͔͔ͬͨΒͱ͍ͬͯͦͷจॻ͕ؔ܎ͳ͍Θ͚Ͱ͸ͳ͍ɻ(จॻ͕ؔ܎ͳ ͍ɾϢʔβ͕จॻΛ؍ଌ͍ͯ͠ͳ͍ɾϥϯμϜཁૉʹΑΔ΋ͷ)

    ଟ͘ͷ؍ଌσʔλʹ͍ͭͯฏۉΛऔΕ͹ϊΠζ͸আڈͰ͖Δͱߟ͑ΒΕΔ͕ɺόΠΞε͸আ ڈͰ͖ͳ͍ɻ 10

11. ҼՌධՁɿ؍ଌɾΫϦοΫϞσϧ Ϣʔβͷ؍ଌٴͼจॻͷؔ࿈౓ͷΈΛߟྀʹೖΕΔͱɺϢʔβͷΫϦοΫ͸ҎԼͷΑ͏ʹϞ σϦϯάͰ͖ͦ͏ɿ › ϥϯΩϯά R ʹ͓͍ͯจॻ di ͕؍ଌ͞ΕΔ (oi

    = 1 Ͱද͢) ֬཰͸ɺ P (oi = 1 j R; di) (؍ଌ͞ΕΔ֬཰͸ؔ࿈౓ʹؔ܎ͳ͍ͱԾఆ͍ͯ͠Δɻ) › ؔ࿈౓ y (di) ͱ؍ଌ oi ͕༩͑ΒΕͨ࣌ͷɺจॻ di ͕ΫϦοΫ͞ΕΔ֬཰ (ci = 1 Ͱද͢) ͸ɺ P (ci = 1 j oi; y (di)) › ΫϦοΫ͸؍ଌ͞Εͨจॻʹ͔͠ى͜Βͳ͍ͨΊɺϥϯΩϯά R ʹ͓͍ͯΫϦοΫ͞ ΕΔ֬཰͸ɿ P (ci = 1 ^ oi = 1 j y (di) ; R) = P (ci = 1 j oi = 1; y (di)) ´ P (oi = 1 j R; di) 11

12. ҼՌධՁɿ´ (f„; D; y) ͷφΠʔϒਪఆ ´ (f„; D; y) ΛφΠʔϒʹਪఆ͢Δʹ͸ɺΫϦοΫͷ৘ใ

    (ci) Λਅͷؔ࿈౓ϥϕϧ (y (di)) ͷ୅ΘΓʹ࢖͑͹Αͯ͘ɺ ´NAIVE (f„; D; c) := X di2D – (rank (di j f„; D)) ´ ci ͱͳΔɻ ΫϦοΫʹϊΠζ͕৐͍ͬͯͳ͍࣌ɺͭ·Γ P (ci = 1 j oi = 1; y (di)) = y (di) Ͱ͋Δ࣌Ͱ͑͞ɺφΠʔϒਪఆ͸؍ଌόΠΞεΛड͚͍ͯΔɿ Eo ˆ´NAIVE (f„; D; c)˜ = Eo 2 4 X di2D – (rank (di j f„; D)) ´ ci 3 5 = Eo 2 6 4 X di:oi=1^y(di)=1 – (rank (di j f„; D)) 3 7 5 = X di:y(di)=1 P (oi = 1 j R; di)– (rank (di j f„; D)) = X di2D P (oi = 1 j R; di)– (rank (di j f„; D)) ´ y (di) 12

13. ҼՌධՁɿ´ (f„; D; y) ͷφΠʔϒਪఆ φΠʔϒਪఆɿ Eo ˆ´NAIVE (f„; D;

    c)˜ = X di:y(di)=1 P (oi = 1 j R; di)– (rank (di j f„; D)) Ͱ͸ɺͦΕͧΕͷจॻͷɺϩάऩू࣌ͷϥϯΩϯά R Ͱͷ؍ଌ֬཰ͰॏΈ෇ͨ͠ਪఆʹͳͬ ͯ͠·͏ɻ ϥϯΩϯάͰ͸ɺߴॱҐͷจॻ΄Ͳ؍ଌ͞Ε΍͍͢ɿ͜ΕΛ position bias ͱݺͿɻϩάऩ ूͷࡍʹߴॱҐʹදࣔ͞Εͨจॻ͸ਅͷؔ࿈౓ΑΓ΋ؔ࿈͕͋ΔɺͱόΠΞεΛड͚ͯ͠· ͏ɻ όΠΞεΛআڈ͢ΔͨΊʹ͸ɺP (oi = 1 j R; di) Λਪఆ͠ɺิਖ਼ͯ͋͛͠Ε͹ྑͦ͞͏ ! ܏޲είΞʹΑΔόΠΞεআڈ 13

14. ܏޲είΞΛ༻͍ͨόΠΞεআڈ Inverse Propensity Scoring(IPS) ʹΑͬͯόΠΞεΛআڈ͢Δɿ ´IPS (f„; D; c) :=

    X di2D – (rank (di j f„; D)) P (oi = 1 j R; di) ´ ci ͜͜ͰɺP (oi = 1 j R; di) ͸ϩάऩूதʹදࣔ͞ΕͨϥϯΩϯά R Ͱจॻ di ͕؍ଌ͞ ΕΔ֬཰Ͱ͋Δɻ´IPS (f„; D; c) ͸ΫϦοΫϊΠζ͕ͳ͍৔߹ɺͭ·Γ P (ci = 1 j oi = 1; y (di)) = y (di) Ͱ͋Δ࣌ʹ ´ (f„; D; y) ͷෆภਪఆྔͰ͋Δɿ Eo ˆ´IPS (f„; D; c)˜ = Eo 2 4 X di2D – (rank (di j f„; D)) P (oi = 1 j R; di) ´ ci 3 5 = Eo 2 6 4 X di:oi=1^y(di)=1 – (rank (di j f„; D)) P (oi = 1 j R; di) 3 7 5 = X di:y(di)=1 P (oi = 1 j R; di) ´ – (rank (di j f„; D)) P (oi = 1 j R; di) = X di2D – (rank (di j f„; D)) ´ y (di) = ´ (f„; D; y) : 14

15. Propensity-weighted LTR IPS ͸ ´ (f„; D; y) ͷෆภਪఆͰ͋ͬͨɻΑͬͯɺ࠷దͳϞσϧύϥϝʔλ „

    ͸ IPS Λ ࠷దԽ͢Δ͜ͱͰٻΊΔ͜ͱ͕Ͱ͖ΔɻIPS Λ࠷దԽ͢ΔࡍɺϥϯΩϯάࢦඪ – (r) ͷඍ෼ ෆՄೳੑʹରॲ͢ΔͨΊɺ– (r) ͷ bound Λར༻͢Δɻ Propensity-weighted LTR ͷྲྀΕɿ › ΫϦοΫͷ܏޲είΞΛਪఆɿ P (oi = 1 j R; di) › ෆภਪఆྔ ´IPS (f„; D; c) ͷ bound ʹ͍ͭͯඍ෼஋Λܭࢉɿ „0 = r„ "– (rank (di j f„; D)) P (oi = 1 j R; di) # › ϞσϧύϥϝʔλΛߋ৽ „new „old ` „0 15

16. References › https://ilps.github.io/webconf2020-tutorial-unbiased-ltr/ 16