Share
Embed
When Relaxations Go Bad: “Differentially-Private” Machine Learning
Bargav Jayaraman's talk at the DC-Area Anonymity, Privacy, and Security Seminar
Monday, 25 February 2019
College Park, Maryland
Associated paper is: https://arxiv.org/abs/1902.08874
David Evans
February 25, 2019
Transcript
40 slides
-
When Relaxations Go Bad: “Differentially-Private” Machine Learning Bargav Jayaraman PhD
in Computer Science University of Virginia -
Quick Overview Data Machine Learning M Differential Privacy Low Budget
Relaxed DP More Privacy Leakage! -
Machine Learning with Data Privacy? Data Machine Learning M Blackbox
/ Whitebox Access Can the adversary gain any information about the training data? Membership of a record? Value of a sensitive attribute? Recurring patterns in the data set? Latent statistics of the data set? -
What is Differential Privacy? A randomized mechanism M is -DP
if for two neighboring datasets D and D’ Pr[M(D) ∈ S] Pr[M(D′) ∈ S] ≤ eϵ+δ (ϵ, δ) *Image taken from “Differential Privacy and Pan-Private Algorithms” slides by Cynthia Dwork -
Applying DP to Machine Learning (ϵ, δ) ϵ For -DP
For -DP β ∼ Lap( |S| ϵ ) β ∼ ( 2 log(1/δ) |S| ϵ ) -
Existing Works on Practical Implementation 2006 2008 2010 2012 2014
2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced -
Existing Works on Practical Implementation 2006 2008 2010 2012 2014
2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced Output Perturbation -
Existing Works on Practical Implementation 2006 2008 2010 2012 2014
2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced Objective Perturbation Output Perturbation -
Existing Works on Practical Implementation 2006 2008 2010 2012 2014
2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced Objective Perturbation Output Perturbation Gradient Perturbation -
Existing Works on Practical Implementation 2006 2008 2010 2012 2014
2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 Complex tasks requiring high value ϵ Multi-class ERM Online ERM -
Existing Works on Practical Implementation 2006 2008 2010 2012 2014
2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 Complex tasks requiring high value ϵ Deep Learning Deep Learning Multi-class ERM Online ERM -
Motivation for Relaxed Definitions If each iteration is -DP By
composition, model: -DP ϵ Tϵ -
Motivation for Relaxed Definitions If each iteration is -DP By
composition, model: -DP ϵ Tϵ Advanced composition theorem “If we only care about expected privacy loss” Model is: (Tϵ(eϵ − 1) + ϵ 2T log(1/δ), δ) -DP -
Relaxed Definitions - Bounding the Expected Privacy Loss Concentrated DP
Zero Concentrated DP Renyi DP Moments Accountant [Dwork et al. (2016)] [Bun & Steinke (2016)] [Abadi et al. (2016)] [Mironov (2017)] Pure DP Notion Relaxed DP Notions max D,D′ log( Pr[M(D) ∈ S] Pr[M(D′) ∈ S]) ≤ ϵ ED,D′,d∼M(D) log( Pr[M(D) = d] Pr[M(D′) = d] ) ≤ μ -
Existing Works on Practical Implementation 2006 2008 2010 2012 2014
2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 [JWEG18] [HHGC18] [PFCW16] [L17] [GSC17] ϵ = 0.5 ϵ = 0.5 ϵ = 0.5 ϵ = 1.6 ϵ = 0.1 Works using relaxed DP notions ERM ERM -
Existing Works on Practical Implementation 2006 2008 2010 2012 2014
2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 [JWEG18] [HHGC18] [BDFKR18] [HCS18] [YLPGT19] [PFCW16] [L17] [GSC17] [GKN17] [ACGMMTZ16] [PAEGT16] ϵ = 0.5 ϵ = 0.5 ϵ = 3 ϵ = 4 ϵ = 0.5 ϵ = 8 ϵ = 8 ϵ = 21.5 ϵ = 1.6 ϵ = 0.1 ϵ = 8 Works using relaxed DP notions ERM ERM DL DL -
Our Objective To evaluate the privacy leakage of relaxed notions
Pure DP Notion Relaxed DP Notions max D,D′ log( Pr[M(D) ∈ S] Pr[M(D′) ∈ S]) ≤ ϵ ED,D′,d∼M(D) log( Pr[M(D) = d] Pr[M(D′) = d] ) ≤ μ Leakage is quantified in terms of inference attacks -
Membership Inference Attack Data M Black-box attack of Shokri et
al. (2017) M1 Mk : : A Member / Non-member D1 Dk -
Membership Inference Attack Data M Black-box attack of Shokri et
al. (2017) M1 Mk : : A Member / Non-member Key Intuition: Confidence score of model is high for members, due to overfitting on training set. D1 Dk -
Membership Inference Attack Data M White-box attack of Yeom et
al. (2018) Attacker has: and M L = 1 |D| |D| ∑ i=1 ℓ(di ) At inference, given record d, attacker classifies it as member if: ℓ(d) ≤ L -
Membership Inference Attack Data M White-box attack of Yeom et
al. (2018) Attacker has: and M L = 1 |D| |D| ∑ i=1 ℓ(di ) At inference, given record d, attacker classifies it as member if: ℓ(d) ≤ L Key Intuition: Sample loss of training instance is lower than that of non-member, due to generalization gap. -
Experiments We train logistic regression and neural network models over
CIFAR-100 and Purchase-100 data sets, and measure model utility and privacy leakage Accuracy loss w.r.t. non-private model Attack advantage = (TPR - FPR) -
Logistic Regression Results (CIFAR-100) Naive Composition Naive Composition Advanced Composition
Advanced Composition RDP RDP zCDP zCDP We train L2 regularized logistic regression models -
Members Revealed by Logistic Regression (CIFAR-100) Naive Composition Advanced Composition
zCDP RDP Budget Loss 1% FPR Loss 1% FPR Loss 1% FPR Loss 1% FPR 0.1 0.94 0 0.93 0 0.94 0 0.94 0 0.5 0.03 0 0.94 0 0.93 0 0.93 0 1.0 0.94 0 0.93 0 0.92 0 0.93 0 5.0 0.94 0 0.92 0 0.91 0 0.92 0 10.0 0.93 0 0.92 0 0.90 0 0.89 0 50.0 0.92 0 0.81 0 0.65 6 0.66 4 100.0 0.89 0 0.62 1 0.43 28 0.47 19 500.0 0.30 23 0.07 103 0.06 109 0.06 101 1000.0 0.11 54 0.04 106 0.04 115 0.04 105 No Privacy 0.00 145 0.00 145 0.00 145 0.00 145 -
Neural Network Results (CIFAR-100) Naive Composition Advanced Composition RDP zCDP
Naive Composition Advanced Composition RDP zCDP We train 2-layer neural network models with 256 neurons per layer -
Members Revealed by Neural Network (CIFAR-100) Naive Composition Advanced Composition
zCDP RDP Budget Loss 1% FPR Loss 1% FPR Loss 1% FPR Loss 1% FPR 0.1 0.95 0 0.95 0 0.94 0 0.93 0 0.5 0.94 0 0.94 0 0.93 0 0.93 0 1.0 0.94 0 0.94 0 0.92 0 0.91 0 5.0 0.94 0 0.93 0 0.83 0 0.83 0 10.0 0.94 0 0.87 0 0.81 0 0.80 0 50.0 0.95 0 0.73 0 0.64 0 0.64 0 100.0 0.93 0 0.61 1 0.49 30 0.48 11 500.0 0.93 0 0.06 26 0.00 54 0.00 40 1000.0 0.59 0 0.06 13 0.00 28 0.07 22 No Privacy 0.00 155 0.00 155 0.00 155 0.00 155 -
Conclusion Relaxed definitions make the privacy budget look small, but
may leak more For complex learning tasks, leakage increases with increase in utility For simple tasks, the existing attacks don’t seem to be effective -
Conclusion Relaxed definitions make the privacy budget look small, but
may leak more For complex learning tasks, leakage increases with increase in utility For simple tasks, the existing attacks don’t seem to be effective Future Directions: Protection against property inference attacks Exploring stronger adversaries with more background knowledge -
Attribute Inference Attack Data M White-box attack of Yeom et
al. (2018) Attacker has: and M L = 1 |D| |D| ∑ i=1 ℓ(di ) At inference, given record d, attacker plugs in different values of sensitive attribute and outputs the value for which: is maximum. Pr(ℓ(d), L) Key Intuition: Sample loss of training instance with the correct value of sensitive attribute has the maximum probability estimate. sensitive attribute -
Relaxed Definitions - Bounding the Expected Privacy Loss Concentrated DP
Zero Concentrated DP Renyi DP Moments Accountant [Dwork et al. (2016)] [Bun & Steinke (2016)] [Abadi et al. (2016)] [Mironov (2017)] “Privacy Loss RV is Sub-Gaussian” “Privacy Loss RV is strictly distributed around zero mean” “Renyi divergence of Privacy Loss RV is bounded” “Higher order moments of Privacy Loss RV is bounded” DsubG(M(D)||M(D′)) ≤ (μ, τ) Dα (M(D)||M(D′)) ≤ ζ + ρα; ∀α ∈ (1,∞) Dα (M(D)||M(D′)) ≤ ϵ λDλ+1 (M(D)||M(D′)) ≤ αM (λ) -
Members Revealed by Logistic Regression (CIFAR-100) Non-private model leaks 145,
265 and 704 members for 1%, 2% and 5% FPR respectively. -
Members Revealed by Logistic Regression (CIFAR-100) Non-private model leaks 145,
265 and 704 members for 1%, 2% and 5% FPR respectively. -
Members Revealed by Neural Network (CIFAR-100) Non-private model leaks 155,
425 and 2667 members for 1%, 2% and 5% FPR respectively. -
Members Revealed by Neural Network (CIFAR-100) Non-private model leaks 155,
425 and 2667 members for 1%, 2% and 5% FPR respectively.