Upgrade to Pro — share decks privately, control downloads, hide ads and more …

When Relaxations Go Bad: “Differentially-Private” Machine Learning

40e37c08199ed4d3866ce6e1ff0be06d?s=47 David Evans
February 25, 2019

When Relaxations Go Bad: “Differentially-Private” Machine Learning

Bargav Jayaraman's talk at the DC-Area Anonymity, Privacy, and Security Seminar
Monday, 25 February 2019
College Park, Maryland

Associated paper is: https://arxiv.org/abs/1902.08874

40e37c08199ed4d3866ce6e1ff0be06d?s=128

David Evans

February 25, 2019
Tweet

Transcript

  1. When Relaxations Go Bad: “Differentially-Private” Machine Learning Bargav Jayaraman PhD

    in Computer Science University of Virginia
  2. Quick Overview Data Machine Learning M Differential Privacy High Budget

  3. Quick Overview Data Machine Learning M Differential Privacy Low Budget

    Relaxed DP
  4. Quick Overview Data Machine Learning M Differential Privacy Low Budget

    Relaxed DP More Privacy Leakage!
  5. Machine Learning with Data Privacy? Data Machine Learning M Blackbox

    / Whitebox 
 Access Can the adversary gain any information about the training data? Membership of a record? Value of a sensitive attribute? Recurring patterns in the data set? Latent statistics of the data set?
  6. What is Differential Privacy? A randomized mechanism M is -DP

    if for two neighboring datasets D and D’ Pr[M(D) ∈ S] Pr[M(D′) ∈ S] ≤ eϵ+δ (ϵ, δ) *Image taken from “Differential Privacy and Pan-Private Algorithms” slides by Cynthia Dwork
  7. Applying DP to Machine Learning (ϵ, δ) ϵ For -DP

    For -DP β ∼ Lap( |S| ϵ ) β ∼ ( 2 log(1/δ) |S| ϵ )
  8. Existing Works on Practical Implementation 2006 2008 2010 2012 2014

    2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced
  9. Existing Works on Practical Implementation 2006 2008 2010 2012 2014

    2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced Output Perturbation
  10. Existing Works on Practical Implementation 2006 2008 2010 2012 2014

    2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced Objective Perturbation Output Perturbation
  11. Existing Works on Practical Implementation 2006 2008 2010 2012 2014

    2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.2 [WLKCJN17] ϵ = 0.05 ERM Algorithms using ϵ ≤ 1 DP introduced Objective Perturbation Output Perturbation Gradient Perturbation
  12. Existing Works on Practical Implementation 2006 2008 2010 2012 2014

    2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 Complex tasks requiring high value ϵ Multi-class ERM Online ERM
  13. Existing Works on Practical Implementation 2006 2008 2010 2012 2014

    2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 Complex tasks requiring high value ϵ Deep Learning Deep Learning Multi-class ERM Online ERM
  14. Motivation for Relaxed Definitions If each iteration is -DP By

    composition, model: -DP ϵ Tϵ
  15. Motivation for Relaxed Definitions If each iteration is -DP By

    composition, model: -DP ϵ Tϵ Advanced composition theorem “If we only care about 
 expected privacy loss” Model is: (Tϵ(eϵ − 1) + ϵ 2T log(1/δ), δ) -DP
  16. Relaxed Definitions - Bounding the Expected Privacy Loss Concentrated DP

    Zero Concentrated DP Renyi DP Moments Accountant [Dwork et al. (2016)] [Bun & Steinke (2016)] [Abadi et al. (2016)] [Mironov (2017)] Pure DP Notion Relaxed DP Notions max D,D′ log( Pr[M(D) ∈ S] Pr[M(D′) ∈ S]) ≤ ϵ ED,D′,d∼M(D) log( Pr[M(D) = d] Pr[M(D′) = d] ) ≤ μ
  17. Existing Works on Practical Implementation 2006 2008 2010 2012 2014

    2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 [JWEG18] [HHGC18] [PFCW16] [L17] [GSC17] ϵ = 0.5 ϵ = 0.5 ϵ = 0.5 ϵ = 1.6 ϵ = 0.1 Works using relaxed DP notions ERM ERM
  18. Existing Works on Practical Implementation 2006 2008 2010 2012 2014

    2016 2018 [D06] [DMNS06] [CM09] [CMS11] [PRR10] [ZZXYW12] [JT13] [JT14] [SCS13] [WLKCJN17] [WFWJN15] [HCB16] ϵ = 0.2 ϵ = 0.2 ϵ = 0.2 ϵ = 0.8 ϵ = 0.5 ϵ = 0.1 ϵ = 1 ϵ = 1 ϵ = 0.05 ϵ = 0.2 [SS15] [ZZWCWZ18] [JKT12] [INSTTW19] ϵ = 10 ϵ = 10 ϵ = 100 ϵ = 369,200 [JWEG18] [HHGC18] [BDFKR18] [HCS18] [YLPGT19] [PFCW16] [L17] [GSC17] [GKN17] [ACGMMTZ16] [PAEGT16] ϵ = 0.5 ϵ = 0.5 ϵ = 3 ϵ = 4 ϵ = 0.5 ϵ = 8 ϵ = 8 ϵ = 21.5 ϵ = 1.6 ϵ = 0.1 ϵ = 8 Works using relaxed DP notions ERM ERM DL DL
  19. Our Objective To evaluate the privacy leakage of relaxed notions

    Pure DP Notion Relaxed DP Notions max D,D′ log( Pr[M(D) ∈ S] Pr[M(D′) ∈ S]) ≤ ϵ ED,D′,d∼M(D) log( Pr[M(D) = d] Pr[M(D′) = d] ) ≤ μ Leakage is quantified in terms of inference attacks
  20. Membership Inference Attack Data M

  21. Membership Inference Attack Data M Black-box attack of Shokri et

    al. (2017) M1 Mk : : A Member / Non-member D1 Dk
  22. Membership Inference Attack Data M Black-box attack of Shokri et

    al. (2017) M1 Mk : : A Member / Non-member Key Intuition:
 Confidence score of model is 
 high for members, due to overfitting on training set. D1 Dk
  23. Membership Inference Attack Data M

  24. Membership Inference Attack Data M White-box attack of Yeom et

    al. (2018) Attacker has: and M L = 1 |D| |D| ∑ i=1 ℓ(di ) At inference, given record d, attacker classifies it as member if: ℓ(d) ≤ L
  25. Membership Inference Attack Data M White-box attack of Yeom et

    al. (2018) Attacker has: and M L = 1 |D| |D| ∑ i=1 ℓ(di ) At inference, given record d, attacker classifies it as member if: ℓ(d) ≤ L Key Intuition:
 Sample loss of training instance
 is lower than that of non-member,
 due to generalization gap.
  26. Experiments We train logistic regression and neural network models over

    CIFAR-100 and Purchase-100 data sets, and measure model utility and privacy leakage Accuracy loss w.r.t. non-private model Attack advantage = (TPR - FPR)
  27. Logistic Regression Results (CIFAR-100) Naive Composition Naive Composition Advanced Composition

    Advanced Composition RDP RDP zCDP zCDP We train L2 regularized logistic regression models
  28. Members Revealed by Logistic Regression (CIFAR-100) Naive Composition Advanced Composition

    zCDP RDP Budget Loss 1% FPR Loss 1% FPR Loss 1% FPR Loss 1% FPR 0.1 0.94 0 0.93 0 0.94 0 0.94 0 0.5 0.03 0 0.94 0 0.93 0 0.93 0 1.0 0.94 0 0.93 0 0.92 0 0.93 0 5.0 0.94 0 0.92 0 0.91 0 0.92 0 10.0 0.93 0 0.92 0 0.90 0 0.89 0 50.0 0.92 0 0.81 0 0.65 6 0.66 4 100.0 0.89 0 0.62 1 0.43 28 0.47 19 500.0 0.30 23 0.07 103 0.06 109 0.06 101 1000.0 0.11 54 0.04 106 0.04 115 0.04 105 No Privacy 0.00 145 0.00 145 0.00 145 0.00 145
  29. Neural Network Results (CIFAR-100) Naive Composition Advanced Composition RDP zCDP

    Naive Composition Advanced Composition RDP zCDP We train 2-layer neural network models with 256 neurons per layer
  30. Members Revealed by Neural Network (CIFAR-100) Naive Composition Advanced Composition

    zCDP RDP Budget Loss 1% FPR Loss 1% FPR Loss 1% FPR Loss 1% FPR 0.1 0.95 0 0.95 0 0.94 0 0.93 0 0.5 0.94 0 0.94 0 0.93 0 0.93 0 1.0 0.94 0 0.94 0 0.92 0 0.91 0 5.0 0.94 0 0.93 0 0.83 0 0.83 0 10.0 0.94 0 0.87 0 0.81 0 0.80 0 50.0 0.95 0 0.73 0 0.64 0 0.64 0 100.0 0.93 0 0.61 1 0.49 30 0.48 11 500.0 0.93 0 0.06 26 0.00 54 0.00 40 1000.0 0.59 0 0.06 13 0.00 28 0.07 22 No Privacy 0.00 155 0.00 155 0.00 155 0.00 155
  31. Conclusion Relaxed definitions make the privacy budget look small, but

    may leak more For complex learning tasks, leakage increases with increase in utility For simple tasks, the existing attacks don’t seem to be effective
  32. Conclusion Relaxed definitions make the privacy budget look small, but

    may leak more For complex learning tasks, leakage increases with increase in utility For simple tasks, the existing attacks don’t seem to be effective Future Directions: Protection against property inference attacks Exploring stronger adversaries with more background knowledge
  33. Questions? Thank You!

  34. Extra Slides

  35. Attribute Inference Attack Data M White-box attack of Yeom et

    al. (2018) Attacker has: and M L = 1 |D| |D| ∑ i=1 ℓ(di ) At inference, given record d, attacker plugs in different values of
 sensitive attribute and outputs the value for which: is maximum. Pr(ℓ(d), L) Key Intuition:
 Sample loss of training instance
 with the correct value of sensitive
 attribute has the maximum 
 probability estimate. sensitive attribute
  36. Relaxed Definitions - Bounding the Expected Privacy Loss Concentrated DP

    Zero Concentrated DP Renyi DP Moments Accountant [Dwork et al. (2016)] [Bun & Steinke (2016)] [Abadi et al. (2016)] [Mironov (2017)] “Privacy Loss RV is Sub-Gaussian” “Privacy Loss RV is strictly 
 distributed around zero mean” “Renyi divergence of 
 Privacy Loss RV is bounded” “Higher order moments of 
 Privacy Loss RV is bounded” DsubG(M(D)||M(D′)) ≤ (μ, τ) Dα (M(D)||M(D′)) ≤ ζ + ρα; ∀α ∈ (1,∞) Dα (M(D)||M(D′)) ≤ ϵ λDλ+1 (M(D)||M(D′)) ≤ αM (λ)
  37. Members Revealed by Logistic Regression (CIFAR-100) Non-private model leaks 145,

    265 and 704 members for 1%, 2% and 5% FPR respectively.
  38. Members Revealed by Logistic Regression (CIFAR-100) Non-private model leaks 145,

    265 and 704 members for 1%, 2% and 5% FPR respectively.
  39. Members Revealed by Neural Network (CIFAR-100) Non-private model leaks 155,

    425 and 2667 members for 1%, 2% and 5% FPR respectively.
  40. Members Revealed by Neural Network (CIFAR-100) Non-private model leaks 155,

    425 and 2667 members for 1%, 2% and 5% FPR respectively.