When Relaxations Go Bad: “Differentially-Private” Machine Learning Bargav Jayaraman PhD in Computer Science University of Virginia

Quick Overview Data Machine Learning M Differential Privacy High Budget

Quick Overview Data Machine Learning M Differential Privacy Low Budget Relaxed DP

Quick Overview Data Machine Learning M Differential Privacy Low Budget Relaxed DP More Privacy Leakage!

Machine Learning with Data Privacy? Data Machine Learning M Blackbox / Whitebox 
 Access Can the adversary gain any information about the training data? Membership of a record? Value of a sensitive attribute? Recurring patterns in the data set? Latent statistics of the data set?

What is Differential Privacy? A randomized mechanism M is -DP if for two neighboring datasets D and D’ Pr[M(D) ∈ S] Pr[M(D′) ∈ S] ≤ eϵ+δ (ϵ, δ) *Image taken from “Differential Privacy and Pan-Private Algorithms” slides by Cynthia Dwork

Applying DP to Machine Learning (ϵ, δ) ϵ For -DP For -DP β ∼ Lap( |S| ϵ ) β ∼ ( 2 log(1/δ) |S| ϵ )

Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced

Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced Output Perturbation

Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced Objective Perturbation Output Perturbation

Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced Objective Perturbation Output Perturbation Gradient Perturbation

Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 Complex tasks requiring high value ϵ Multi-class ERM Online ERM

Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 Complex tasks requiring high value ϵ Deep Learning Deep Learning Multi-class ERM Online ERM

Motivation for Relaxed Definitions If each iteration is -DP By composition, model: -DP ϵ Tϵ

Motivation for Relaxed Definitions If each iteration is -DP By composition, model: -DP ϵ Tϵ Advanced composition theorem “If we only care about 
 expected privacy loss” Model is: (Tϵ(eϵ − 1) + ϵ 2T log(1/δ), δ) -DP

Relaxed Definitions - Bounding the Expected Privacy Loss Concentrated DP Zero Concentrated DP Renyi DP Moments Accountant [Dwork et al. (2016)] [Bun & Steinke (2016)] [Abadi et al. (2016)] [Mironov (2017)] Pure DP Notion Relaxed DP Notions max D,D′ log( Pr[M(D) ∈ S] Pr[M(D′) ∈ S]) ≤ ϵ ED,D′,d∼M(D) log( Pr[M(D) = d] Pr[M(D′) = d] ) ≤ μ

Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 [JWEG18] [HHGC18] [PFCW16] [L17] [GSC17] ϵ = 0.5 ϵ = 0.5 ϵ = 0.5 ϵ = 1.6 ϵ = 0.1 Works using relaxed DP notions ERM ERM

Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 [JWEG18] [HHGC18] [BDFKR18] [HCS18] [YLPGT19] [PFCW16] [L17] [GSC17] [GKN17] [ACGMMTZ16] [PAEGT16] ϵ = 0.5 ϵ = 0.5 ϵ = 3 ϵ = 4 ϵ = 0.5 ϵ = 8 ϵ = 8 ϵ = 21.5 ϵ = 1.6 ϵ = 0.1 ϵ = 8 Works using relaxed DP notions ERM ERM DL DL

Our Objective To evaluate the privacy leakage of relaxed notions Pure DP Notion Relaxed DP Notions max D,D′ log( Pr[M(D) ∈ S] Pr[M(D′) ∈ S]) ≤ ϵ ED,D′,d∼M(D) log( Pr[M(D) = d] Pr[M(D′) = d] ) ≤ μ Leakage is quantified in terms of inference attacks

Membership Inference Attack Data M

Membership Inference Attack Data M Black-box attack of Shokri et al. (2017) M1 Mk : : A Member / Non-member D1 Dk

Membership Inference Attack Data M Black-box attack of Shokri et al. (2017) M1 Mk : : A Member / Non-member Key Intuition:
 Confidence score of model is 
 high for members, due to overfitting on training set. D1 Dk

Membership Inference Attack Data M

Membership Inference Attack Data M White-box attack of Yeom et al. (2018) Attacker has: and M L = 1 |D| |D| ∑ i=1 ℓ(di ) At inference, given record d, attacker classifies it as member if: ℓ(d) ≤ L

Membership Inference Attack Data M White-box attack of Yeom et al. (2018) Attacker has: and M L = 1 |D| |D| ∑ i=1 ℓ(di ) At inference, given record d, attacker classifies it as member if: ℓ(d) ≤ L Key Intuition:
 Sample loss of training instance
 is lower than that of non-member,
 due to generalization gap.

Experiments We train logistic regression and neural network models over CIFAR-100 and Purchase-100 data sets, and measure model utility and privacy leakage Accuracy loss w.r.t. non-private model Attack advantage = (TPR - FPR)

Logistic Regression Results (CIFAR-100) Naive Composition Naive Composition Advanced Composition Advanced Composition RDP RDP zCDP zCDP We train L2 regularized logistic regression models

Members Revealed by Logistic Regression (CIFAR-100) Naive Composition Advanced Composition zCDP RDP Budget Loss 1% FPR Loss 1% FPR Loss 1% FPR Loss 1% FPR 0.1 0.94 0 0.93 0 0.94 0 0.94 0 0.5 0.03 0 0.94 0 0.93 0 0.93 0 1.0 0.94 0 0.93 0 0.92 0 0.93 0 5.0 0.94 0 0.92 0 0.91 0 0.92 0 10.0 0.93 0 0.92 0 0.90 0 0.89 0 50.0 0.92 0 0.81 0 0.65 6 0.66 4 100.0 0.89 0 0.62 1 0.43 28 0.47 19 500.0 0.30 23 0.07 103 0.06 109 0.06 101 1000.0 0.11 54 0.04 106 0.04 115 0.04 105 No Privacy 0.00 145 0.00 145 0.00 145 0.00 145

Neural Network Results (CIFAR-100) Naive Composition Advanced Composition RDP zCDP Naive Composition Advanced Composition RDP zCDP We train 2-layer neural network models with 256 neurons per layer

Members Revealed by Neural Network (CIFAR-100) Naive Composition Advanced Composition zCDP RDP Budget Loss 1% FPR Loss 1% FPR Loss 1% FPR Loss 1% FPR 0.1 0.95 0 0.95 0 0.94 0 0.93 0 0.5 0.94 0 0.94 0 0.93 0 0.93 0 1.0 0.94 0 0.94 0 0.92 0 0.91 0 5.0 0.94 0 0.93 0 0.83 0 0.83 0 10.0 0.94 0 0.87 0 0.81 0 0.80 0 50.0 0.95 0 0.73 0 0.64 0 0.64 0 100.0 0.93 0 0.61 1 0.49 30 0.48 11 500.0 0.93 0 0.06 26 0.00 54 0.00 40 1000.0 0.59 0 0.06 13 0.00 28 0.07 22 No Privacy 0.00 155 0.00 155 0.00 155 0.00 155

Conclusion Relaxed definitions make the privacy budget look small, but may leak more For complex learning tasks, leakage increases with increase in utility For simple tasks, the existing attacks don’t seem to be effective

Conclusion Relaxed definitions make the privacy budget look small, but may leak more For complex learning tasks, leakage increases with increase in utility For simple tasks, the existing attacks don’t seem to be effective Future Directions: Protection against property inference attacks Exploring stronger adversaries with more background knowledge

Questions? Thank You!

Extra Slides

Attribute Inference Attack Data M White-box attack of Yeom et al. (2018) Attacker has: and M L = 1 |D| |D| ∑ i=1 ℓ(di ) At inference, given record d, attacker plugs in different values of
 sensitive attribute and outputs the value for which: is maximum. Pr(ℓ(d), L) Key Intuition:
 Sample loss of training instance
 with the correct value of sensitive
 attribute has the maximum 
 probability estimate. sensitive attribute

Relaxed Definitions - Bounding the Expected Privacy Loss Concentrated DP Zero Concentrated DP Renyi DP Moments Accountant [Dwork et al. (2016)] [Bun & Steinke (2016)] [Abadi et al. (2016)] [Mironov (2017)] “Privacy Loss RV is Sub-Gaussian” “Privacy Loss RV is strictly 
 distributed around zero mean” “Renyi divergence of 
 Privacy Loss RV is bounded” “Higher order moments of 
 Privacy Loss RV is bounded” DsubG(M(D)||M(D′)) ≤ (μ, τ) Dα (M(D)||M(D′)) ≤ ζ + ρα; ∀α ∈ (1,∞) Dα (M(D)||M(D′)) ≤ ϵ λDλ+1 (M(D)||M(D′)) ≤ αM (λ)

Members Revealed by Logistic Regression (CIFAR-100) Non-private model leaks 145, 265 and 704 members for 1%, 2% and 5% FPR respectively.

Members Revealed by Logistic Regression (CIFAR-100) Non-private model leaks 145, 265 and 704 members for 1%, 2% and 5% FPR respectively.

Members Revealed by Neural Network (CIFAR-100) Non-private model leaks 155, 425 and 2667 members for 1%, 2% and 5% FPR respectively.

Members Revealed by Neural Network (CIFAR-100) Non-private model leaks 155, 425 and 2667 members for 1%, 2% and 5% FPR respectively.