[読み会] Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference

読み会資料
Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference (NeurIPS 2020)

Transcript

1. Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference ಡΈձ@2021/10/1 2 ༶ ໌఩
2.  ஶऀ: Disi Ji1, Padhraic Smyth1, Mark Steyvers 2 ॴଐ: University of California, 1 Department of Computer Science
 2 Department of Cognitive Science s બΜͩཧ༝ : ެฏੑʹؔ͢ΔධՁͱϕΠζతΞϓϩʔνΛֶͿͨΊ ࿦จ৘ใ
3.  ໨త : ϥϕϧ෇͖αϯϓϧ͕গͳ͍͕ɼϥϕϧͳ͠αϯϓϧ͸ͨ͘͞Μ ͋Δ࣌ʹɼάϧʔϓͷެฏੑΛਖ਼֬ʹධՁ͍ͨ͠ ߩݙ: ϥϕϧ෇͖͚ͩͷํ๏ΑΓϥϕϧͳ͠σʔλͰิڧͯ͠ɺΑΓਖ਼ ֬Ͱ෼ࢄͷগͳ͍ਪఆ஋Λੜ੒Ͱ͖ΔΑ͏ʹͳͬͨ ͲΜͳ࿦จ?
4.  ػցֶश͕ҙࢥܾఆʹ༻͍ΒΕΔ →ηϯγςΟϒଐੑʹରͯ͠ภͬͨग़ྗ͕໰୊ ެฏੑ഑ྀܕػցֶशͰओʹऔΓ૊·ΕͯΔ΋ͷ 1. ػցֶशจ຺ʹ͓͚Δެฏੑͷఆٛ 2. ެฏੑΛߟྀͨ͠ΞϧΰϦζϜͷઃܭ എܠ ެฏੑ഑ྀܕػցֶशʹ͍ͭͯ
5.  ݶΒΕͨϥϕϧ෇͖αϯϓϧ͕༩͑ΒΕͨதͰͷϞσϧͷެฏੑ Λਖ਼͘͠ධՁ ಛʹೋ஋෼ྨʹ͓͚ΔूஂެฏੑΛѻ͏ ूஂެฏੑ άϧʔϓ(ੑผ,ਓछ…)಺Ͱͷࢦඪ(TPR, Accuracy…)͕౳͍͜͠ͱ͕ެฏ Πϯτϩ ର৅ͱͳΔެฏੑ໰୊
6.  ϥϕϧ෇͖σʔλ͕গྔͷͱ͖ɼ͜ΕΒͷެฏੑධՁࢦඪͷਪఆ஋͕͹Βͭ͘ ඪຊ෼ࢄ͸ ͷ଎͞Ͱখ͘͞ͳΔ͕ɼൺֱతΏͬ͘Γ 1/n Πϯτϩ ެฏੑࢦඪͷ໰୊
7.  ϥϕϧ෼෍͕ෆۉߧͳ৔߹ɼTPR΍FPRͷάϧʔϓࠩͷਪఆ෼ࢄ͸ѱԽ͢Δɽ ؆୯ͳγϛϡϨʔγϣϯ : உੑ:ঁੑ=8:2, άϧʔϓ಺ͷਖ਼ྫ͕20%ɼϞσϧͷάϧʔϓ͝ͱͷਅͷTPR͕0.95, 0.90ͷ࣌(ެฏੑ ࢦඪ: 0.95-0.90=0.05)ɽ ਪఆެฏੑ͕[0.04,0.06]ͷதʹ͋Δ͜ͱΛ৴པ۠ؒ95%ʹ͢Δʹ͸αϯϓϧ͕গͳ͘ͱ΋96,000ݸඞ ཁʹͳΔ → ࣮ੈքͷσʔληοτ͸͜ΕΑΓ΋খ͍͜͞ͱ͕΄ͱΜͲͰɼ
 ͔ͭ͜ͷΑ͏ʹσʔλ෼෍͕ภ͍ͬͯΔͷ͸௝͍͜͠ͱͰ͸ͳ͍ Πϯτϩ ϥϕϧ෼෍͕ෆۉߧͳ࣌
8.  ࣮ੈքʹ͋Δσʔληοτ͚ͩͰެฏੑධՁΛ
 ৴པ͢Δ͜ͱ͸ࠔ೉ ͞Βʹެฏੑ͕ඞཁͳঢ়گ(ҩྍ΍࢘๏)Ͱ͸ɼ
 σʔληοτ͕͋Δ͕ϥϕϧ֫ಘ͕ࠔ೉ Πϯτϩ ݱ࣮ͷσʔληοτ
9.  ఏҊ େྔͷϥϕϧͳ͠+গྔͷϥϕϧ͋ΓʹΑͬͯɼ෼ࢄ͕খ͍͞ਪఆΛੜ੒ ߩݙ ެฏੑࢦඪͷਪఆΛϕΠζతʹѻ͏ ϕΠζʹΑΔΩϟϦϒϨʔγϣϯΛఏҊ গྔͷϥϕϧ͋ΓͰ΋ਪఆޡࠩΛখ͘͞Ͱ͖Δ͜ͱΛ࣮ূ Πϯτϩ ຊݚڀͰ΍Δ͜ͱ
10.  Ϟσϧͷग़ྗ=positive ʹͳΔ֬཰ ͱ͍͍ͯ͠ ͷ͔ ? ΩϟϒϨʔγϣϯʹ Αͬͯमਖ਼͢Δ ΩϟϦϒϨʔγϣϯͱ͸ Ϟσϧͷग़ྗΛ֤Ϋϥεʹଐ͢Δ֬཰ʹ͚ۙͮΔ
11.  : ֶशͨ͠܇࿅Ϟσϧ, : ೖྗ, : Ϋϥεϥϕϧ Ϟσϧੜ੒είΞ: →Ϟσϧͷ༧ଌ֬཰ : Ϟσϧͷ༧ଌϥϕϧ ʹԠͯ͡ϥϕϧ͕ܾఆ͢Δ ෼ྨث͕ΩϟϦϒϨʔγϣϯ͞ΕΔ → είΞsͷ஋ͷ֬཰Ͱ༧ଌ͕߹͍ͬͯΔͱߟ͑ΒΕΔ M x y ∈ {0,1} s = PM (y = 1|x) ∈ [0,1] ̂ y s P( ̂ y = y|s) = s ४උ දه
12.  : ର৅ͷूஂ (e.g. ਓछɼੑผ… ) : ूஂgʹ͓͚ΔԿ͔͠Βͷࢦඪ (e.g. Accuracy, TPR, FPR… ) : ެฏੑࢦඪ, ࠓճ͸ Ͱߟ͍͑ͯΔ : ͦΕͧΕϥϕϧ͋Γɼϥϕϧͳ͠σʔληοτ
 Ͱ͋Δঢ়گΛߟ͍͑ͯΔ g ∈ {0,1,...,G − 1} θg Δ = θ0 − θ1 g ∈ {0,1} nL , nU nL ≪ nU ४උ දه (ެฏੑ)
13.  ϥϕϧͳ͠σʔληοτͷϥϕϧ͸ɼείΞ Λ༻͍ٖͯࣅతʹར ༻ αϯϓϧ( )͸฼ूஂ ΋͘͠͸ ͔ΒIIDʹαϯϓϦ ϯά͞Ε͍ͯΔͱߟ͑Δɽ ·ͨ ͷ΋ͷ͸୯ʹ ΍ ͔Βੜ੒͞Ε͍ͯΔͱߟ͑Δ s x, s, y P(x, y) P(s, y) nU P(x) P(s) ४උ ฼ूஂ
14.  ఏҊख๏Ͱ͸2ͭͷσʔληοτΛ૊Έ߹ΘͤΔ ϥϕϧ͋Γ →Beta-Binomial Estimatio n ϥϕϧͳ͠ →Bayesian Calibration Model ఏҊख๏ ֓ཁ
15.  άϧʔϓ͝ͱͷࢦඪ ɹ
 Ͱߟ͑Δ ༧ଌϞσϧͷਖ਼ޡ ެฏੑࢦඪ ͱͨ͋͠ͱMCMCαϯϓϦϯάʹΑͬͯࣄޙ෼෍ Λ֫ಘ ҰԠਪఆͰ͖Δ͕ɼσʔλ਺ʹਫ਼౓͕ґଘ͢Δͷ͕໰୊ θg = P( ̂ y = 1|y = 1,g) θg ∼ Beta(αg , βg ) αg = βg = 1 Ii = I( ̂ (yi = yi ),1 ≤ i ≤ nL Ii ∼ Bernoulli(θg ) Δ = θ1 − θ0 P(Δ|DL ) ఏҊख๏: ४උ Beta-Binomial Estimation
16.  ϥϕϧͳ͠σʔληοτ ʹରͯ͠είΞ ΋͠Ϟσϧ͕׬શʹΩϟϦϒϨʔγϣϯ͞Ε͍ͯΔͳΒείΞΛ ͦͷ··༧ଌʹ༻͍Δ͜ͱ͕Ͱ͖Δ Ͱάϧʔϓ͝ͱͷධՁࢦඪΛఆٛͰ͖Δ nU sj = PM (yj = 1|xj ) ̂ θg = (1/nU,g)∑ j∈g sj I (sj ≥ 0.5) + (1 − sj) I (sj < 0.5) ఏҊख๏: ४උ Leveraging Unlabeled Data with a Bayesian Calibration Model
17.  ϞσϧείΞʢΩϟϦϒϨʔγϣϯ͞Ε͍ͯͳ͍ʣʹΑΔ༧ଌ͸ ෳࡶͳϞσϧͰ͸େ͖ؒ͘ҧ͑ͯ͠·͏ɽ ຊख๏ͷΞϓϩʔνͱͯ͠ɼ
 ϥϕϧ෇͖σʔλΛ༻͍ͯΩϟϦϒϨʔγϣϯΛͯ͠
 ਫ਼౓ͷภΓΛͳ͍ͯ͘͘͠ ఏҊख๏: ४උ ϞσϧείΞΛͦͷ··༻͍Δ໰୊
18.  ͱఆٛ
 → είΞ ͕༩͑Εͨ࣌ͷϞσϧͷਫ਼౓ɼ͜ΕΛαϯϓϧ͝ͱͷ જࡏม਺ͱͯ͠ར༻ zj = E[I( ̂ yj = yj )] = P(yj = ̂ yj |sj ) sj ఏҊख๏ જࡏม਺Λͭ͘Δ
19.  1. Λ༻͍ͯάϧʔϓ͝ͱͷΩϟϦϒϨʔγϣϯؔ਺ Λਪఆ 2. ͔Β ͷࣄޙ෼෍ Λ֫ಘ 3. ͱϥϕϧ෇͖σʔληοτΛ༻͍ͯ Λਪఆ nL ϕg ϕg zj Pϕg (zj |DL , sj ) zj θg , Δ ఏҊख๏ ਪఆ
20.  ϥϕϧͷͳ͍αϯϓϧͷείΞͱϥ ϕϧ෇͖ͷσʔληοτΛ૊Έ߹Θ ͤΔ͜ͱͰ, ͷਪఆΛߦ͏͜ͱ͕Ͱ ͖Δɽ θt g θt g = 1 nL,g + nU,g ∑ i:i∈a I ( ̂ yi = yi) + ∑ i⋅j∈a zt j ఏҊख๏ άϥϑΟΧϧϞσϧ
21.  ΩϟϦϒϨʔγϣϯؔ਺ ͷֶशΛߟ͍͑ͯ͘ ϞσϧείΞͷΩϟϦϒϨʔγϣϯͱͯ࣍ࣜ͠Λ༻͍Δ ֤άϧʔϓʹରͯ͠ΩϟϦϒϨʔγϣϯύϥϝʔλ Λ֫ಘ͍͖͍ͯͨ͠ ϕg f(s; a, b, c) = 1 1 + e−c−a log s+b log(1−s) ϕg = {ag , bg , cg } ఏҊख๏ ֊૚ϕΠζΩϟϦϒϨʔγϣϯ
22.  Ϟσϧʹద༻͢ΔͨΊʹ࣍ࣜͰਖ਼ղϥϕϧ͕ੜ੒͞ΕΔͱԾఆ άϧʔϓ͝ͱͷύϥϝʔλ͸ͦΕͧΕڞ௨ͷ෼෍͔Βੜ੒͞ΕΔ ϋΠύϥ ͸੾அਖ਼ن෼෍͔Βੜ੒ yi ∼ Bernoulli (f (si ; agj , bqi , cgi )) logag ∼ N(μa , σa ), logbg ∼ N(μb , σb ), logcg ∼ N(μc , σc )whereπ = {μa,b,c , σa,b,c } π ఏҊख๏ ΩϟϦϒϨʔγϣϯؔ਺ͷϞσϧԽ
23.  ຊݚڀͰ͸ϋΠύϥΛઃఆ͍ͯ͠Δ ͕ɼ͜Ε͸ύϥϝʔλͱͯ͠ଥ౰ͳ ஋Ͱઃఆ
 →͢΂ͯಉ͡ઃఆͰ࣮ݧɼఏҊख๏ ͕ؤ݈Ͱ͋Δ͜ͱΛࣔ͢ ఏҊख๏
24.  ໨త:
 ݶΒΕͯͨϥϕϧ෇͖σʔλΛ༻͍ͯɼ༷ʑͳਪఆͷਫ਼౓ΛධՁ ͢Δ͜ͱ σʔληοτ : Adult, German Credit, Ricci, Compa s ਪఆ๏: logisticճؼ, MLP, Random Forest, Gaussian Naive Bayes ࣮ݧ
25.  MLPͰάϧʔϓ͝ͱͷaccuracyࠩʹ͍ͭͯධՁ͢Δ ස౓ओٛ: ͷ஋Λςετηοτ͢΂͔ͯΒܭࢉɼਅͷ஋ ϕʔλ-ೋ߲๏ (BB): ϥϕϧ෇͖σʔληοτͷΈʹΞΫηεՄೳ ϕΠζΩϟϦϒϨʔγϣϯ(BC): ϥϕϧͳ͠ʹ΋ΞΫηεՄೳ Δ ࣮ݧ ਪఆਫ਼౓ͷൺֱ
26.  20ճ࣮ߦɼ95%ࣄޙ৴པ۠ؒͱࣄޙฏۉΛ͍ࣔͯ͠Δ ࣮ݧ BBͱBCͷਪఆਫ਼౓ͷൺֱ
27.  ϥϕϧ෇͖σʔλͷαΠζΛมԽͤͨ͞ͱ͖ ࣮ݧ σʔληοταΠζʹΑΔൺֱ
28.  ςετηοτશͯͷධՁࢦඪͷMA E ͷ࣌΄ͱΜͲಉ༷ͷ݁Ռ nL = 200 ࣮ݧ ҟͳΔϞσϧͰͷਪఆਫ਼౓
29.  αϯϓϧαΠζ͕গͳ͍ͱ͖ͷެฏੑධՁ͸ෆ࣮֬ͩͱࢦఠ ϥϕϧͳ͠ɼ͋ΓΛ૊Έ߹ΘͤͯϕΠζΩϟϦϒϨʔγϣϯΛ༻ ͍ͯਪఆ෼ࢄΛখ͘͢͞Δख๏ΛఏҊ ࠓճͷϑϨʔϜϫʔΫ͸ɼख๏ʹద༻͢Δͷ͕༰қͷͨΊɼެฏ ੑධՁʹ༻͍Δͱ͍͍͔΋ ·ͱΊ