When Relaxations Go Bad: “Differentially-Private” Machine Learning

Bargav Jayaraman's talk at the DC-Area Anonymity, Privacy, and Security Seminar
Monday, 25 February 2019
College Park, Maryland

Associated paper is: https://arxiv.org/abs/1902.08874

Transcript

1. When Relaxations Go Bad: “Differentially-Private” Machine Learning Bargav Jayaraman PhD in Computer Science University of Virginia
2. Quick Overview Data Machine Learning M Differential Privacy High Budget
3. Quick Overview Data Machine Learning M Differential Privacy Low Budget Relaxed DP
4. Quick Overview Data Machine Learning M Differential Privacy Low Budget Relaxed DP More Privacy Leakage!
5. Machine Learning with Data Privacy? Data Machine Learning M Blackbox / Whitebox 
 Access Can the adversary gain any information about the training data? Membership of a record? Value of a sensitive attribute? Recurring patterns in the data set? Latent statistics of the data set?
6. What is Differential Privacy? A randomized mechanism M is -DP if for two neighboring datasets D and D’ Pr[M(D) ∈ S] Pr[M(D′) ∈ S] ≤ eϵ+δ (ϵ, δ) *Image taken from “Differential Privacy and Pan-Private Algorithms” slides by Cynthia Dwork
7. Applying DP to Machine Learning (ϵ, δ) ϵ For -DP For -DP β ∼ Lap( |S| ϵ ) β ∼ ( 2 log(1/δ) |S| ϵ )
8. Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced
9. Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced Output Perturbation
10. Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced Objective Perturbation Output Perturbation
11. Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced Objective Perturbation Output Perturbation Gradient Perturbation
12. Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 Complex tasks requiring high value ϵ Multi-class ERM Online ERM
13. Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 Complex tasks requiring high value ϵ Deep Learning Deep Learning Multi-class ERM Online ERM
14. Motivation for Relaxed Definitions If each iteration is -DP By composition, model: -DP ϵ Tϵ
15. Motivation for Relaxed Definitions If each iteration is -DP By composition, model: -DP ϵ Tϵ Advanced composition theorem “If we only care about 
 expected privacy loss” Model is: (Tϵ(eϵ − 1) + ϵ 2T log(1/δ), δ) -DP
16. Relaxed Definitions - Bounding the Expected Privacy Loss Concentrated DP Zero Concentrated DP Renyi DP Moments Accountant [Dwork et al. (2016)] [Bun & Steinke (2016)] [Abadi et al. (2016)] [Mironov (2017)] Pure DP Notion Relaxed DP Notions max D,D′ log( Pr[M(D) ∈ S] Pr[M(D′) ∈ S]) ≤ ϵ ED,D′,d∼M(D) log( Pr[M(D) = d] Pr[M(D′) = d] ) ≤ μ
17. Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 [JWEG18] [HHGC18] [PFCW16] [L17] [GSC17] ϵ = 0.5 ϵ = 0.5 ϵ = 0.5 ϵ = 1.6 ϵ = 0.1 Works using relaxed DP notions ERM ERM
18. Existing Works on Practical Implementation 2006 2008 2010 2012 2014 2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 [JWEG18] [HHGC18] [BDFKR18] [HCS18] [YLPGT19] [PFCW16] [L17] [GSC17] [GKN17] [ACGMMTZ16] [PAEGT16] ϵ = 0.5 ϵ = 0.5 ϵ = 3 ϵ = 4 ϵ = 0.5 ϵ = 8 ϵ = 8 ϵ = 21.5 ϵ = 1.6 ϵ = 0.1 ϵ = 8 Works using relaxed DP notions ERM ERM DL DL
19. Our Objective To evaluate the privacy leakage of relaxed notions Pure DP Notion Relaxed DP Notions max D,D′ log( Pr[M(D) ∈ S] Pr[M(D′) ∈ S]) ≤ ϵ ED,D′,d∼M(D) log( Pr[M(D) = d] Pr[M(D′) = d] ) ≤ μ Leakage is quantified in terms of inference attacks
20. Membership Inference Attack Data M
21. Membership Inference Attack Data M Black-box attack of Shokri et al. (2017) M1 Mk : : A Member / Non-member D1 Dk
22. Membership Inference Attack Data M Black-box attack of Shokri et al. (2017) M1 Mk : : A Member / Non-member Key Intuition:
 Confidence score of model is 
 high for members, due to overfitting on training set. D1 Dk
23. Membership Inference Attack Data M
24. Membership Inference Attack Data M White-box attack of Yeom et al. (2018) Attacker has: and M L = 1 |D| |D| ∑ i=1 ℓ(di ) At inference, given record d, attacker classifies it as member if: ℓ(d) ≤ L
25. Membership Inference Attack Data M White-box attack of Yeom et al. (2018) Attacker has: and M L = 1 |D| |D| ∑ i=1 ℓ(di ) At inference, given record d, attacker classifies it as member if: ℓ(d) ≤ L Key Intuition:
 Sample loss of training instance
 is lower than that of non-member,
 due to generalization gap.
26. Experiments We train logistic regression and neural network models over CIFAR-100 and Purchase-100 data sets, and measure model utility and privacy leakage Accuracy loss w.r.t. non-private model Attack advantage = (TPR - FPR)
27. Logistic Regression Results (CIFAR-100) Naive Composition Naive Composition Advanced Composition Advanced Composition RDP RDP zCDP zCDP We train L2 regularized logistic regression models
28. Members Revealed by Logistic Regression (CIFAR-100) Naive Composition Advanced Composition zCDP RDP Budget Loss 1% FPR Loss 1% FPR Loss 1% FPR Loss 1% FPR 0.1 0.94 0 0.93 0 0.94 0 0.94 0 0.5 0.03 0 0.94 0 0.93 0 0.93 0 1.0 0.94 0 0.93 0 0.92 0 0.93 0 5.0 0.94 0 0.92 0 0.91 0 0.92 0 10.0 0.93 0 0.92 0 0.90 0 0.89 0 50.0 0.92 0 0.81 0 0.65 6 0.66 4 100.0 0.89 0 0.62 1 0.43 28 0.47 19 500.0 0.30 23 0.07 103 0.06 109 0.06 101 1000.0 0.11 54 0.04 106 0.04 115 0.04 105 No Privacy 0.00 145 0.00 145 0.00 145 0.00 145
29. Neural Network Results (CIFAR-100) Naive Composition Advanced Composition RDP zCDP Naive Composition Advanced Composition RDP zCDP We train 2-layer neural network models with 256 neurons per layer
30. Members Revealed by Neural Network (CIFAR-100) Naive Composition Advanced Composition zCDP RDP Budget Loss 1% FPR Loss 1% FPR Loss 1% FPR Loss 1% FPR 0.1 0.95 0 0.95 0 0.94 0 0.93 0 0.5 0.94 0 0.94 0 0.93 0 0.93 0 1.0 0.94 0 0.94 0 0.92 0 0.91 0 5.0 0.94 0 0.93 0 0.83 0 0.83 0 10.0 0.94 0 0.87 0 0.81 0 0.80 0 50.0 0.95 0 0.73 0 0.64 0 0.64 0 100.0 0.93 0 0.61 1 0.49 30 0.48 11 500.0 0.93 0 0.06 26 0.00 54 0.00 40 1000.0 0.59 0 0.06 13 0.00 28 0.07 22 No Privacy 0.00 155 0.00 155 0.00 155 0.00 155
31. Conclusion Relaxed definitions make the privacy budget look small, but may leak more For complex learning tasks, leakage increases with increase in utility For simple tasks, the existing attacks don’t seem to be effective
32. Conclusion Relaxed definitions make the privacy budget look small, but may leak more For complex learning tasks, leakage increases with increase in utility For simple tasks, the existing attacks don’t seem to be effective Future Directions: Protection against property inference attacks Exploring stronger adversaries with more background knowledge
33. Questions? Thank You!
34. Extra Slides
35. Attribute Inference Attack Data M White-box attack of Yeom et al. (2018) Attacker has: and M L = 1 |D| |D| ∑ i=1 ℓ(di ) At inference, given record d, attacker plugs in different values of
 sensitive attribute and outputs the value for which: is maximum. Pr(ℓ(d), L) Key Intuition:
 Sample loss of training instance
 with the correct value of sensitive
 attribute has the maximum 
 probability estimate. sensitive attribute
36. Relaxed Definitions - Bounding the Expected Privacy Loss Concentrated DP Zero Concentrated DP Renyi DP Moments Accountant [Dwork et al. (2016)] [Bun & Steinke (2016)] [Abadi et al. (2016)] [Mironov (2017)] “Privacy Loss RV is Sub-Gaussian” “Privacy Loss RV is strictly 
 distributed around zero mean” “Renyi divergence of 
 Privacy Loss RV is bounded” “Higher order moments of 
 Privacy Loss RV is bounded” DsubG(M(D)||M(D′)) ≤ (μ, τ) Dα (M(D)||M(D′)) ≤ ζ + ρα; ∀α ∈ (1,∞) Dα (M(D)||M(D′)) ≤ ϵ λDλ+1 (M(D)||M(D′)) ≤ αM (λ)
37. Members Revealed by Logistic Regression (CIFAR-100) Non-private model leaks 145, 265 and 704 members for 1%, 2% and 5% FPR respectively.
38. Members Revealed by Logistic Regression (CIFAR-100) Non-private model leaks 145, 265 and 704 members for 1%, 2% and 5% FPR respectively.
39. Members Revealed by Neural Network (CIFAR-100) Non-private model leaks 155, 425 and 2667 members for 1%, 2% and 5% FPR respectively.
40. Members Revealed by Neural Network (CIFAR-100) Non-private model leaks 155, 425 and 2667 members for 1%, 2% and 5% FPR respectively.