19th International School on Foundations of Security Analysis and Design
Mini-course on "Trustworthy Machine Learning"
https://jeffersonswheel.org/fosad2019 David Evans
Trustworthy Machine Learning David Evans University of Virginia jeffersonswheel.org Bertinoro, Italy 26 August 2019 19th International School on Foundations of Security Analysis and Design 3: Privacy
Machine Learning Pipeline 2 Data Subjects Data Collection Data Owner Data Collection Model Training Trained Model Deployed Model Hyperparameters User Machine Learning Service
Potential Privacy Goals 3 Data Subjects Data Collection Data Owner Data Collection Model Training Trained Model Deployed Model Hyperparameters User Machine Learning Service Data Subject Privacy API User
Potential Privacy Goals 4 Data Subjects Data Collection Data Owner Data Collection Model Training Trained Model Deployed Model Hyperparameters User Machine Learning Service Data Subject Privacy Distributed (Federated) Learning API User
5 Data Subjects Data Collection Data Owner Data Collection Model Training Trained Model Deployed Model Hyperparameters User Machine Learning Service Data Subject Privacy Distributed (Federated) Learning Inference Attack API User
6 Data Subjects Data Collection Data Owner Data Collection Model Training Trained Model Deployed Model Hyperparameters User Machine Learning Service Data Subject Privacy Distributed (Federated) Learning Inference Attack API User
7 Data Subjects Data Collection Data Owner Data Collection Model Training Trained Model Deployed Model Hyperparameters User Machine Learning Service Data Subject Privacy Distributed (Federated) Learning Inference Attack API User Model Stealing Attack
8 Data Subjects Data Collection Data Owner Data Collection Model Training Trained Model Deployed Model Hyperparameters User Machine Learning Service Data Subject Privacy Distributed (Federated) Learning Inference Attack API User Model Stealing Attack Hyperparameter Stealing Attack
9 Data Subjects Data Collection Data Owner Data Collection Model Training Trained Model Deployed Model Hyperparameters User Machine Learning Service Data Subject Privacy Distributed (Federated) Learning Inference Attack API User Model Stealing Attack Hyperparameter Stealing Attack Note: only considering confidentiality; lots of integrity attacks also (poisoning, evasion, …)
Privacy Mechanisms: Encryption 10 Data Subjects Data Collection Data Owner Data Collection Model Training Trained Model Deployed Model Hyperparameters User API User Randomized Response, Local Differential Privacy Output Perturbation Objective Perturbation Gradient Perturbation Distributed Learning (Federated Learning)
Privacy Mechanisms: Encryption 11 Data Subjects Data Collection Data Owner Data Collection Model Training Trained Model Deployed Model Hyperparameters User API User Randomized Response, Local Differential Privacy Output Perturbation Objective Perturbation Gradient Perturbation Distributed Learning (Federated Learning) Oblivious Model Execution
Privacy Mechanisms: Noise 12 Data Subjects Data Collection Data Owner Data Collection Model Training Trained Model Deployed Model Hyperparameters User Machine Learning Service API User Randomized Response, Local Differential Privacy Output Perturbation Objective Perturbation Gradient Perturbation
Secure Two-Party Computation Can Alice and Bob compute a function on private data, without exposing anything about their data besides the result? ! = #(%, ') Alice’s Secret Input: % Bob’s Secret Input: ' 14
Secure Two-Party Computation Can Alice and Bob compute a function on private data, without exposing anything about their data besides the result? ! = #(%, ') Alice’s Secret Input: % Bob’s Secret Input: ' “private” and “correct” 15
“Obfuscated” Logic Inputs Output a b ! "# $# %# "# $& %# "& $# %# "& $& %& ' ( ! AND ") , $) , %) are random values, chosen by generator but meaningless to evaluator. 19
Yao’s GC Protocol Alice (generator) Sends tables, her input labels (!" ) Bob (evaluator) Picks random values for ! #,% . ' #,% , ( #,% Evaluates circuit, decrypting one row of each garbled gate ( ) Decodes output ) Generates garbled tables 23 *+,,-, ((# ) *+,,-0 ((# ) *+0,-, ((# ) *+0,-0 ((% ) How does the Bob learn his own input wire labels?
Building Computing Systems Digital Electronic Circuits Garbled Circuits Operate on known data Operate on encrypted wire labels 32-bit logical operation requires moving some electrons a few nm One-bit AND requires four encryptions Reuse is great! Reuse is not allowed! ! "# #,"# % ('( )) ! "% #,"# % ('( )) … 27
Federated Learning 35 Central Aggregator and Controler ! ! 1. Server sends candidate models to local devices 2. Local devices train models on their local data 3. Devices send back gradient updates (for some parameters) 4. Server aggregated updates, produces new model "# "$
Distributed Learning 37 Data Subjects Data Collection Data Owner Data Collection Model Training Trained Model Output Model Hyperparameters Output Perturbation Objective Perturbation Gradient Perturbation Distributed/Federated Learning Inference Attack
No Inference Protection 38 Data Subjects Data Collection Data Owner Data Collection Model Training Trained Model Deployed Model Hyperparameters User API User Distributed Learning (Federated Learning) Inference Attack
Limiting Inference 44 Data Collection Data Collection Model Training Trained Model Deployed Model Hyperparameters Output Perturbation Objective Perturbation Gradient Perturbation Inference Attack Local DP
Limiting Inference 45 Data Collection Data Collection Model Training Trained Model Deployed Model Hyperparameters Output Perturbation Objective Perturbation Gradient Perturbation Inference Attack Local DP Trust Boundary
Limiting Inference 46 Data Collection Data Collection Model Training Trained Model Deployed Model Hyperparameters Output Perturbation Objective Perturbation Gradient Perturbation Inference Attack Trust Boundary Preventing inference requires adding noise to the deployed model: how much noise and where to add it?
Differential Privacy Definition 48 A randomized mechanism ! satisfies (#)-Differential Privacy if for any two neighboring datasets % and %’: “Neighboring” datasets differ in at most one entry. Pr[! % ∈ +] Pr[! %′ ∈ +] ≤ /0
53 Differential privacy describes a promise, made by a data holder, or curator, to a data subject: “You will not be affected, adversely or otherwise, by allowing your data to be used in any study or analysis, no matter what other studies, data sets, or information sources, are available.”
Limiting Inference 54 Data Collection Data Collection Model Training Trained Model Deployed Model Hyperparameters Output Perturbation Objective Perturbation Gradient Perturbation Inference Attack Trust Boundary
Differential Privacy for Complex Learning To achieve DP, need to know the sensitivity: Pr[$ % ∈ '] Pr[$ %′ ∈ '] ≤ +, + . max2,24, 2 524 6 78 ℳ % − ℳ %; < how much a difference in the input could impact the output.
Differential Privacy for Complex Learning To achieve DP, need to know the sensitivity: Pr[$ % ∈ '] Pr[$ %′ ∈ '] ≤ +, + . max2,24, 2 524 6 78 ℳ % − ℳ %; < how much a difference in the input could impact the output.
72 Accuracy Loss Privacy Budget ! Rènyi DP has 0.1 accuracy loss at ! ≈ 10 Naïve Composion has 0.1 accuracy loss at ! ≈ 500 Logistic Regression on CIFAR-100
Experimentally Measuring Leakage 73 Data Subjects Data Collection Data Owner Data Collection Model Training Trained Model Deployed Model User Inference Attack Gradient Perturbation
How can adversary guess membership? 76 Test error Training error Accuracy on CIFAR-10 Generalization Gap Overfitting: Model is “more confident” in predictions for training examples
Membership Inference Attack: Shokri+ 77 Reza Shokri, Marco Stronati, Congzheng Song, Vitaly Shmatikov [S&P 2017] !" !# Assumption: adversary has access to similar training data 1. Train several local models Intuition: Confidence score of model is high for members, due to overfitting on training set. !$ ...
Membership Inference Attack: Shokri+ 78 Reza Shokri, Marco Stronati, Congzheng Song, Vitaly Shmatikov [S&P 2017] !" !# A Assumption: adversary has access to similar training data 1. Train several local models 2. Train a binary classifier model on local model outputs to distinguish member/non- member Intuition: Confidence score of model is high for members, due to overfitting on training set. !$ ...
Membership Inference Attack: Yeom+ 79 Samuel Yeom, Irene Giacomelli, Matt Fredrikson, Somesh Jha [CSF 2018] Attack: At inference, given record !, attacker classifies it as member if ℓ(!) ≤ & Intuition: Sample loss of training instance is lower than that of non-member, due to generalization gap. Assumption: adversary knows expected training loss of target model & = 1 ) * +,- . ℓ/ !+
Open Problems Close gap between theory and meaningful privacy: - Tighter theoretical bounds - Better attacks - Theory for non-worst-case What properties put a record at risk of exposure? Understanding tradeoffs between model capacity and privacy 90
Other Security Faculty at the University of Virginia 95 Yonghwi Kwon Systems security Cyberforensics Yuan Tian IoT Security ML Security and Privacy Yixin Sun [Joining Jan 2020] Network Security & Privacy Mohammad Mahmoody Theoretical Cryptography David Wu Applied Cryptography Collaborators in Machine Learning, Computer Vision, Natural Language Processing, Software Engineering
Visit Opportunities PhD Student Post-Doc Year/Semester/Summer Undergraduate, Graduate, Faculty 96 Please contact me if you are interested even if in another area
evansuva
fullfull
yasulab
taiheioki
wingnus
kfujita
codeforeveryone
signer
juliawrobel
gihoz_support
paulrobertlloyd
jponch
ufuk
eileencodes
moore
keathley
jcasabona
geeforr
roundedbygravity
bluesmoon
akmur
panda_program