Privacy granted by maths

Since March 2019 the TensorFlow family counts a new member TensorFlow Privacy. What is it about? What are the mathematical theories that guarantee the privacy of deep learning models and how they are implemented?

Presented at DevFest Nantes 2019.
Format: 20 minutes Q&A included

Transcript

1. Privacy granted by maths
   @Giuliabianchl
   October 22, 2019
   

   View Slide

2. Giulia Bianchi
   ‍ data scientist
   @Giuliabianchl
   

   View Slide

3. Table of contents
   Privacy leaks
   Differential Privacy
   TensorFlow Privacy
   
   

   View Slide

4. https://www.cs.cmu.edu/~mfredrik/papers/fjr2015ccs.pdf
   Privacy leaks
   

   View Slide

5. https://xkcd.com/2169/
   https://arxiv.org/abs/1802.08232
   Privacy leaks
   

   View Slide

6. https://arxiv.org/abs/1802.08232
   Privacy leaks
   "Unintended memorization occurs when trained neural networks may reveal the
   presence of out-of-distribution training data -- i.e., training data that is irrelevant to
   the learning task [...].
   Let's meet by
   the docks at
   midnight on
   june 28, come
   alone.
   Long live the
   revolution.
   Our next
   meeting will
   be at the
   docks...
   

   View Slide

7. Differential privacy addresses the paradox of learning nothing about an
   individual while learning useful information about a population
   "
   Roughly, an algorithm is differentially private if an observer seeing its output
   cannot tell if a particular individual's information was used in the computation
   "
   Differential privacy
   https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf
   

   View Slide

8. Differential privacy
   Y
   X
   2 databases X, Y
   Y = X + 1 entry
   If M(X) is sufficiently close to M(Y)
   then M is differentially private
   "
   Analyst ‍♀
   The analyst queries X
   and Y and obtains
   M(X) and M(Y)
   Algorithm M Probability of M(X)≅Probability of M(Y)
   

   View Slide

9. "Differential privacy will provide privacy by process; in particular it will introduce
   randomness.
   "
   Differential privacy
   A randomized algorithm M with domain N|X| is (ε, δ)-differentially private if
   for all S ⊆ Range(M) and for all x, y ∈ N|X| such that ∥x − y∥
   1
   ≤ 1:
   Pr[M(x) ∈ S] ≤ exp(ε) Pr[M(y) ∈ S] + δ
   If δ = 0, we say that M is ε-differentially private.
   https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf
   

   View Slide

10. ● adjacent databases x, y ∈ N|X|
    ○ N|x| input set
    ○ x, y differs for 1 entry
    "
    Differential privacy
    A randomized algorithm M with domain N|X| is (ε, δ)-differentially private if
    for all S ⊆ Range(M) and for all x, y ∈ N|X| such that ∥x − y∥
    1
    ≤ 1:
    Pr[M(x) ∈ S] ≤ exp(ε) Pr[M(y) ∈ S] + δ
    If δ = 0, we say that M is ε-differentially private.
    https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf
    

    View Slide

11. ● randomized algorithm M: N|x| → Range(M)
    ○ N|x| input set
    ○ Range(M) output set
    ○ S ⊆ Range(M) → S is an output
    "
    Differential privacy
    A randomized algorithm M with domain N|X| is (ε, δ)-differentially private if
    for all S ⊆ Range(M) and for all x, y ∈ N|X| such that ∥x − y∥
    1
    ≤ 1:
    Pr[M(x) ∈ S] ≤ exp(ε) Pr[M(y) ∈ S] + δ
    If δ = 0, we say that M is ε-differentially private.
    https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf
    

    View Slide

12. ● δ=0
    ○ Probability[M(x) ∈ S] ≤ exp(ε) Probability[M(y) ∈ S]
    ● ε≅0 ⇒ exp(ε)≅1
    ○ Probability[M(x) ∈ S] / Probability[M(y) ∈ S] ≤ 1
    "
    Differential privacy
    A randomized algorithm M with domain N|X| is (ε, δ)-differentially private if
    for all S ⊆ Range(M) and for all x, y ∈ N|X| such that ∥x − y∥
    1
    ≤ 1:
    Pr[M(x) ∈ S] ≤ exp(ε) Pr[M(y) ∈ S] + δ
    If δ = 0, we say that M is ε-differentially private.
    https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf
    

    View Slide

13. "
    Differential privacy
    A randomized algorithm M with domain N|X| is (ε, δ)-differentially private if
    for all S ⊆ Range(M) and for all x, y ∈ N|X| such that ∥x − y∥
    1
    ≤ 1:
    Pr[M(x) ∈ S] ≤ exp(ε) Pr[M(y) ∈ S] + δ
    If δ = 0, we say that M is ε-differentially private.
    ● The probability of an output of a randomized (ε, δ)-differentially private
    algorithm on two adjacent databases is pretty much the same
    ○ ε small enough
    ○ δ < 1/|X|
    https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf
    

    View Slide

14. *Easy way to approximate a deterministic function with a differentially private
    algorithm
    *
    Differential privacy
    Privacy guarantee achieved is quantifiable
    *Differential privacy has properties that makes it useful in machine learning
    (composability, group privacy, robustness to auxiliary information)
    

    View Slide

15. *Easy way to approximate a deterministic function with a differentially private
    algorithm
    Differential privacy
    

    View Slide

16. *Easy way to approximate a deterministic function with a differentially private
    algorithm
    Differential privacy
    By adding NOISE
    *Gaussian mechanism G
    σ
    f(x) ≝ f(x) + N(0, σ2)
    https://en.wikipedia.org/wiki/Additive_noise_mechanisms
    

    View Slide

17. Deep Learning with Differential Privacy
    https://arxiv.org/abs/1607.00133
    Differentially private Stochastic Gradient
    Descent
    *Differential privacy is introduced in deep learning algorithms by modifying the
    stochastic gradient descent
    At each iteration:
    1. clip gradient
    2. add noise
    Clipping gradient is common practice to avoid
    overfitting even when privacy is not a matter.
    It is required to prove the differential privacy
    guarantee
    

    View Slide

18. Differentially private sgd
    Deep Learning with Differential Privacy
    https://arxiv.org/abs/1607.00133
    

    View Slide

19. TensorFlow Privacy
    https://github.com/tensorflow/privacy
    Latest release 0.1.0
    2 oct 2019
    First release 0.0.1
    23 aug 2019
    

    View Slide

20. Without DP
    With DP
    TensorFlow Privacy
    https://github.com/tensorflow/privacy/blob/master/tutorials/mnist_dpsgd_tutorial_keras.py
    if train_with_differential_privacy == True:
    optimizer = DPGradientDescentGaussianOptimizer(
    l2_norm_clip=l2_norm_clip,
    noise_multiplier=noise_multiplier,
    num_microbatches=microbatches,
    learning_rate=learning_rate)
    # Compute vector of per-example loss rather than its mean over a minibatch.
    loss = tf.keras.losses.CategoricalCrossentropy(
    from_logits=True, reduction=tf.losses.Reduction.NONE)
    else:
    optimizer = GradientDescentOptimizer(learning_rate=learning_rate)
    loss = tf.keras.losses.CategoricalCrossentropy(from_logits=True)
    model.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])
    

    View Slide

21. if train_with_differential_privacy == True:
    optimizer = DPGradientDescentGaussianOptimizer(
    l2_norm_clip=l2_norm_clip,
    noise_multiplier=noise_multiplier,
    num_microbatches=microbatches,
    learning_rate=learning_rate)
    # Compute vector of per-example loss rather than its mean over a minibatch.
    loss = tf.keras.losses.CategoricalCrossentropy(
    from_logits=True, reduction=tf.losses.Reduction.NONE)
    else:
    optimizer = GradientDescentOptimizer(learning_rate=FLAGS.learning_rate)
    loss = tf.keras.losses.CategoricalCrossentropy(from_logits=True)
    model.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])
    TensorFlow Privacy
    https://github.com/tensorflow/privacy/blob/master/tutorials/mnist_dpsgd_tutorial_keras.py
    privacy.optimizers.dp_optimizer.
    DPGradientDescentGaussianOptimizer
    tf.optimizers.SGD
    DPAdamGaussianOptimizer
    DPAdagradGaussianOptimizer
    DPGradientDescentGaussianOptimizer
    

    View Slide

22. Privacy Analysis
    *Differential privacy guarantee is expressed by epsilon and delta
    ● epsilon: upper bound on how much the probability of a particular model output can vary by adding
    or removing a single training point
    ● delta: bounds the probability of our privacy guarantee not holding
    *TensorFlow Privacy provides methods to compute them
    

    View Slide

23. Take aways
    Differential privacy
    Privacy leaks TensorFlow Privacy
    Y
    X
    Pr [M(X)]
    ≅
    Pr [M(Y)]
    M
    M
    DPAdamGaussianOptimizer
    DPAdagradGaussianOptimizer
    DPGradientDescentGaussianOptimizer
    

    View Slide

24. Thank you
    
    @Giuliabianchl
    

    View Slide