[読み会] Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference

Slide 1

Slide 1 text

Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference ಡΈձ@2021/10/1 2 ༶ ໌఩

Slide 2

Slide 2 text

ஶऀ: Disi Ji1, Padhraic Smyth1, Mark Steyvers 2 ॴଐ: University of California, 1 Department of Computer Science  2 Department of Cognitive Science s બΜͩཧ༝ : ެฏੑʹؔ͢ΔධՁͱϕΠζతΞϓϩʔνΛֶͿͨΊ ࿦จ৘ใ

Slide 3

Slide 3 text

໨త : ϥϕϧ෇͖αϯϓϧ͕গͳ͍͕ɼϥϕϧͳ͠αϯϓϧ͸ͨ͘͞Μ ͋Δ࣌ʹɼάϧʔϓͷެฏੑΛਖ਼֬ʹධՁ͍ͨ͠ ߩݙ: ϥϕϧ෇͖͚ͩͷํ๏ΑΓϥϕϧͳ͠σʔλͰิڧͯ͠ɺΑΓਖ਼ ֬Ͱ෼ࢄͷগͳ͍ਪఆ஋Λੜ੒Ͱ͖ΔΑ͏ʹͳͬͨ ͲΜͳ࿦จ?

Slide 4

Slide 4 text

ػցֶश͕ҙࢥܾఆʹ༻͍ΒΕΔ →ηϯγςΟϒଐੑʹରͯ͠ภͬͨग़ྗ͕໰୊ ެฏੑ഑ྀܕػցֶशͰओʹऔΓ૊·ΕͯΔ΋ͷ 1. ػցֶशจ຺ʹ͓͚Δެฏੑͷఆٛ 2. ެฏੑΛߟྀͨ͠ΞϧΰϦζϜͷઃܭ എܠ ެฏੑ഑ྀܕػցֶशʹ͍ͭͯ

Slide 5

Slide 5 text

ݶΒΕͨϥϕϧ෇͖αϯϓϧ͕༩͑ΒΕͨதͰͷϞσϧͷެฏੑ Λਖ਼͘͠ධՁ ಛʹೋ஋෼ྨʹ͓͚ΔूஂެฏੑΛѻ͏ ूஂެฏੑ άϧʔϓ(ੑผ,ਓछ…)಺Ͱͷࢦඪ(TPR, Accuracy…)͕౳͍͜͠ͱ͕ެฏ Πϯτϩ ର৅ͱͳΔެฏੑ໰୊

Slide 6

Slide 6 text

ϥϕϧ෇͖σʔλ͕গྔͷͱ͖ɼ͜ΕΒͷެฏੑධՁࢦඪͷਪఆ஋͕͹Βͭ͘ ඪຊ෼ࢄ͸ ͷ଎͞Ͱখ͘͞ͳΔ͕ɼൺֱతΏͬ͘Γ 1/n Πϯτϩ ެฏੑࢦඪͷ໰୊

Slide 7

Slide 7 text

ϥϕϧ෼෍͕ෆۉߧͳ৔߹ɼTPR΍FPRͷάϧʔϓࠩͷਪఆ෼ࢄ͸ѱԽ͢Δɽ ؆୯ͳγϛϡϨʔγϣϯ : உੑ:ঁੑ=8:2, άϧʔϓ಺ͷਖ਼ྫ͕20%ɼϞσϧͷάϧʔϓ͝ͱͷਅͷTPR͕0.95, 0.90ͷ࣌(ެฏੑ ࢦඪ: 0.95-0.90=0.05)ɽ ਪఆެฏੑ͕[0.04,0.06]ͷதʹ͋Δ͜ͱΛ৴པ۠ؒ95%ʹ͢Δʹ͸αϯϓϧ͕গͳ͘ͱ΋96,000ݸඞ ཁʹͳΔ → ࣮ੈքͷσʔληοτ͸͜ΕΑΓ΋খ͍͜͞ͱ͕΄ͱΜͲͰɼ  ͔ͭ͜ͷΑ͏ʹσʔλ෼෍͕ภ͍ͬͯΔͷ͸௝͍͜͠ͱͰ͸ͳ͍ Πϯτϩ ϥϕϧ෼෍͕ෆۉߧͳ࣌

Slide 8

Slide 8 text

࣮ੈքʹ͋Δσʔληοτ͚ͩͰެฏੑධՁΛ  ৴པ͢Δ͜ͱ͸ࠔ೉ ͞Βʹެฏੑ͕ඞཁͳঢ়گ(ҩྍ΍࢘๏)Ͱ͸ɼ  σʔληοτ͕͋Δ͕ϥϕϧ֫ಘ͕ࠔ೉ Πϯτϩ ݱ࣮ͷσʔληοτ

Slide 9

Slide 9 text

ఏҊ େྔͷϥϕϧͳ͠+গྔͷϥϕϧ͋ΓʹΑͬͯɼ෼ࢄ͕খ͍͞ਪఆΛੜ੒ ߩݙ ެฏੑࢦඪͷਪఆΛϕΠζతʹѻ͏ ϕΠζʹΑΔΩϟϦϒϨʔγϣϯΛఏҊ গྔͷϥϕϧ͋ΓͰ΋ਪఆޡࠩΛখ͘͞Ͱ͖Δ͜ͱΛ࣮ূ Πϯτϩ ຊݚڀͰ΍Δ͜ͱ

Slide 10

Slide 10 text

Ϟσϧͷग़ྗ=positive ʹͳΔ֬཰ ͱ͍͍ͯ͠ ͷ͔ ? ΩϟϒϨʔγϣϯʹ Αͬͯमਖ਼͢Δ ΩϟϦϒϨʔγϣϯͱ͸ Ϟσϧͷग़ྗΛ֤Ϋϥεʹଐ͢Δ֬཰ʹ͚ۙͮΔ

Slide 11

Slide 11 text

: ֶशͨ͠܇࿅Ϟσϧ, : ೖྗ, : Ϋϥεϥϕϧ Ϟσϧੜ੒είΞ: →Ϟσϧͷ༧ଌ֬཰ : Ϟσϧͷ༧ଌϥϕϧ ʹԠͯ͡ϥϕϧ͕ܾఆ͢Δ ෼ྨث͕ΩϟϦϒϨʔγϣϯ͞ΕΔ → είΞsͷ஋ͷ֬཰Ͱ༧ଌ͕߹͍ͬͯΔͱߟ͑ΒΕΔ M x y ∈ {0,1} s = PM (y = 1|x) ∈ [0,1] ̂ y s P( ̂ y = y|s) = s ४උ දه

Slide 12

Slide 12 text

: ର৅ͷूஂ (e.g. ਓछɼੑผ… ) : ूஂgʹ͓͚ΔԿ͔͠Βͷࢦඪ (e.g. Accuracy, TPR, FPR… ) : ެฏੑࢦඪ, ࠓճ͸ Ͱߟ͍͑ͯΔ : ͦΕͧΕϥϕϧ͋Γɼϥϕϧͳ͠σʔληοτ  Ͱ͋Δঢ়گΛߟ͍͑ͯΔ g ∈ {0,1,...,G − 1} θg Δ = θ0 − θ1 g ∈ {0,1} nL , nU nL ≪ nU ४උ දه (ެฏੑ)

Slide 13

Slide 13 text

ϥϕϧͳ͠σʔληοτͷϥϕϧ͸ɼείΞ Λ༻͍ٖͯࣅతʹར ༻ αϯϓϧ( )͸฼ूஂ ΋͘͠͸ ͔ΒIIDʹαϯϓϦ ϯά͞Ε͍ͯΔͱߟ͑Δɽ ·ͨ ͷ΋ͷ͸୯ʹ ΍ ͔Βੜ੒͞Ε͍ͯΔͱߟ͑Δ s x, s, y P(x, y) P(s, y) nU P(x) P(s) ४උ ฼ूஂ

Slide 14

Slide 14 text

ఏҊख๏Ͱ͸2ͭͷσʔληοτΛ૊Έ߹ΘͤΔ ϥϕϧ͋Γ →Beta-Binomial Estimatio n ϥϕϧͳ͠ →Bayesian Calibration Model ఏҊख๏ ֓ཁ

Slide 15

Slide 15 text

άϧʔϓ͝ͱͷࢦඪ ɹ  Ͱߟ͑Δ ༧ଌϞσϧͷਖ਼ޡ ެฏੑࢦඪ ͱͨ͋͠ͱMCMCαϯϓϦϯάʹΑͬͯࣄޙ෼෍ Λ֫ಘ ҰԠਪఆͰ͖Δ͕ɼσʔλ਺ʹਫ਼౓͕ґଘ͢Δͷ͕໰୊ θg = P( ̂ y = 1|y = 1,g) θg ∼ Beta(αg , βg ) αg = βg = 1 Ii = I( ̂ (yi = yi ),1 ≤ i ≤ nL Ii ∼ Bernoulli(θg ) Δ = θ1 − θ0 P(Δ|DL ) ఏҊख๏: ४උ Beta-Binomial Estimation

Slide 16

Slide 16 text

ϥϕϧͳ͠σʔληοτ ʹରͯ͠είΞ ΋͠Ϟσϧ͕׬શʹΩϟϦϒϨʔγϣϯ͞Ε͍ͯΔͳΒείΞΛ ͦͷ··༧ଌʹ༻͍Δ͜ͱ͕Ͱ͖Δ Ͱάϧʔϓ͝ͱͷධՁࢦඪΛఆٛͰ͖Δ nU sj = PM (yj = 1|xj ) ̂ θg = (1/nU,g)∑ j∈g sj I (sj ≥ 0.5) + (1 − sj) I (sj < 0.5) ఏҊख๏: ४උ Leveraging Unlabeled Data with a Bayesian Calibration Model

Slide 17

Slide 17 text

ϞσϧείΞʢΩϟϦϒϨʔγϣϯ͞Ε͍ͯͳ͍ʣʹΑΔ༧ଌ͸ ෳࡶͳϞσϧͰ͸େ͖ؒ͘ҧ͑ͯ͠·͏ɽ ຊख๏ͷΞϓϩʔνͱͯ͠ɼ  ϥϕϧ෇͖σʔλΛ༻͍ͯΩϟϦϒϨʔγϣϯΛͯ͠  ਫ਼౓ͷภΓΛͳ͍ͯ͘͘͠ ఏҊख๏: ४උ ϞσϧείΞΛͦͷ··༻͍Δ໰୊

Slide 18

Slide 18 text

ͱఆٛ  → είΞ ͕༩͑Εͨ࣌ͷϞσϧͷਫ਼౓ɼ͜ΕΛαϯϓϧ͝ͱͷ જࡏม਺ͱͯ͠ར༻ zj = E[I( ̂ yj = yj )] = P(yj = ̂ yj |sj ) sj ఏҊख๏ જࡏม਺Λͭ͘Δ

Slide 19

Slide 19 text

1. Λ༻͍ͯάϧʔϓ͝ͱͷΩϟϦϒϨʔγϣϯؔ਺ Λਪఆ 2. ͔Β ͷࣄޙ෼෍ Λ֫ಘ 3. ͱϥϕϧ෇͖σʔληοτΛ༻͍ͯ Λਪఆ nL ϕg ϕg zj Pϕg (zj |DL , sj ) zj θg , Δ ఏҊख๏ ਪఆ

Slide 20

Slide 20 text

ϥϕϧͷͳ͍αϯϓϧͷείΞͱϥ ϕϧ෇͖ͷσʔληοτΛ૊Έ߹Θ ͤΔ͜ͱͰ, ͷਪఆΛߦ͏͜ͱ͕Ͱ ͖Δɽ θt g θt g = 1 nL,g + nU,g ∑ i:i∈a I ( ̂ yi = yi) + ∑ i⋅j∈a zt j ఏҊख๏ άϥϑΟΧϧϞσϧ

Slide 21

Slide 21 text

ΩϟϦϒϨʔγϣϯؔ਺ ͷֶशΛߟ͍͑ͯ͘ ϞσϧείΞͷΩϟϦϒϨʔγϣϯͱͯ࣍ࣜ͠Λ༻͍Δ ֤άϧʔϓʹରͯ͠ΩϟϦϒϨʔγϣϯύϥϝʔλ Λ֫ಘ͍͖͍ͯͨ͠ ϕg f(s; a, b, c) = 1 1 + e−c−a log s+b log(1−s) ϕg = {ag , bg , cg } ఏҊख๏ ֊૚ϕΠζΩϟϦϒϨʔγϣϯ

Slide 22

Slide 22 text

Ϟσϧʹద༻͢ΔͨΊʹ࣍ࣜͰਖ਼ղϥϕϧ͕ੜ੒͞ΕΔͱԾఆ άϧʔϓ͝ͱͷύϥϝʔλ͸ͦΕͧΕڞ௨ͷ෼෍͔Βੜ੒͞ΕΔ ϋΠύϥ ͸੾அਖ਼ن෼෍͔Βੜ੒ yi ∼ Bernoulli (f (si ; agj , bqi , cgi )) logag ∼ N(μa , σa ), logbg ∼ N(μb , σb ), logcg ∼ N(μc , σc )whereπ = {μa,b,c , σa,b,c } π ఏҊख๏ ΩϟϦϒϨʔγϣϯؔ਺ͷϞσϧԽ

Slide 23

Slide 23 text

ຊݚڀͰ͸ϋΠύϥΛઃఆ͍ͯ͠Δ ͕ɼ͜Ε͸ύϥϝʔλͱͯ͠ଥ౰ͳ ஋Ͱઃఆ  →͢΂ͯಉ͡ઃఆͰ࣮ݧɼఏҊख๏ ͕ؤ݈Ͱ͋Δ͜ͱΛࣔ͢ ఏҊख๏

Slide 24

Slide 24 text

໨త:  ݶΒΕͯͨϥϕϧ෇͖σʔλΛ༻͍ͯɼ༷ʑͳਪఆͷਫ਼౓ΛධՁ ͢Δ͜ͱ σʔληοτ : Adult, German Credit, Ricci, Compa s ਪఆ๏: logisticճؼ, MLP, Random Forest, Gaussian Naive Bayes ࣮ݧ

Slide 25

Slide 25 text

MLPͰάϧʔϓ͝ͱͷaccuracyࠩʹ͍ͭͯධՁ͢Δ ස౓ओٛ: ͷ஋Λςετηοτ͢΂͔ͯΒܭࢉɼਅͷ஋ ϕʔλ-ೋ߲๏ (BB): ϥϕϧ෇͖σʔληοτͷΈʹΞΫηεՄೳ ϕΠζΩϟϦϒϨʔγϣϯ(BC): ϥϕϧͳ͠ʹ΋ΞΫηεՄೳ Δ ࣮ݧ ਪఆਫ਼౓ͷൺֱ

Slide 26

Slide 26 text

20ճ࣮ߦɼ95%ࣄޙ৴པ۠ؒͱࣄޙฏۉΛ͍ࣔͯ͠Δ ࣮ݧ BBͱBCͷਪఆਫ਼౓ͷൺֱ

Slide 27

Slide 27 text

ϥϕϧ෇͖σʔλͷαΠζΛมԽͤͨ͞ͱ͖ ࣮ݧ σʔληοταΠζʹΑΔൺֱ

Slide 28

Slide 28 text

ςετηοτશͯͷධՁࢦඪͷMA E ͷ࣌΄ͱΜͲಉ༷ͷ݁Ռ nL = 200 ࣮ݧ ҟͳΔϞσϧͰͷਪఆਫ਼౓

Slide 29

Slide 29 text

αϯϓϧαΠζ͕গͳ͍ͱ͖ͷެฏੑධՁ͸ෆ࣮֬ͩͱࢦఠ ϥϕϧͳ͠ɼ͋ΓΛ૊Έ߹ΘͤͯϕΠζΩϟϦϒϨʔγϣϯΛ༻ ͍ͯਪఆ෼ࢄΛখ͘͢͞Δख๏ΛఏҊ ࠓճͷϑϨʔϜϫʔΫ͸ɼख๏ʹద༻͢Δͷ͕༰қͷͨΊɼެฏ ੑධՁʹ༻͍Δͱ͍͍͔΋ ·ͱΊ