Membership Inference [Shokri+ 2017] ‣ Data Privacy is an increasingly important issue
‣ Membership Inference Problem: Given a blackbox machine learning model, guess if data was in the training data 2 [Shokri+ 2017] “Membership Inference Attacks against Machine Learning Models”
Membership Inference [Shokri+ 2017] ‣ Data Privacy is an increasingly important issue
‣ Membership Inference Problem: Given a blackbox machine learning model, guess if data was in the training data 2 [Shokri+ 2017] “Membership Inference Attacks against Machine Learning Models” Service Provider Training Data
Membership Inference [Shokri+ 2017] ‣ Data Privacy is an increasingly important issue
‣ Membership Inference Problem: Given a blackbox machine learning model, guess if data was in the training data 2 [Shokri+ 2017] “Membership Inference Attacks against Machine Learning Models” Service Provider Machine Learning as a Service Training Data
Membership Inference [Shokri+ 2017] ‣ Data Privacy is an increasingly important issue
‣ Membership Inference Problem: Given a blackbox machine learning model, guess if data was in the training data 2 [Shokri+ 2017] “Membership Inference Attacks against Machine Learning Models” Service Provider Machine Learning as a Service Blackbox Training Model Training Data Training API
Membership Inference [Shokri+ 2017] ‣ Data Privacy is an increasingly important issue
‣ Membership Inference Problem: Given a blackbox machine learning model, guess if data was in the training data 2 [Shokri+ 2017] “Membership Inference Attacks against Machine Learning Models” Service Provider Machine Learning as a Service Blackbox Training Model User / Attacker Training Data Private Data Training API
Membership Inference [Shokri+ 2017] ‣ Data Privacy is an increasingly important issue
‣ Membership Inference Problem: Given a blackbox machine learning model, guess if data was in the training data 2 [Shokri+ 2017] “Membership Inference Attacks against Machine Learning Models” Service Provider Machine Learning as a Service Blackbox Training Model User / Attacker Training Data Private Data Result Training API Prediction API
Membership Inference [Shokri+ 2017] ‣ Data Privacy is an increasingly important issue
‣ Membership Inference Problem: Given a blackbox machine learning model, guess if data was in the training data 2 [Shokri+ 2017] “Membership Inference Attacks against Machine Learning Models” Service Provider Machine Learning as a Service Blackbox Training Model User / Attacker Training Data Private Data Result Training API Prediction API ? ? Is user’s private data in model training set?
Attack with “Shadow Models” 3 ‣ Assume attacker has access to the training API (or knows the model detail)
‣ Synthesis data similar to the target training data, and train “shadow models” Service Provider Training API ML as a Service Training Data Target Model Attacker Shadow Set 1 Shadow Set 2 Shadow Model 1 Shadow Model 2 … …
Train “IN or OUT” Classifier for Attack 4 ML as a Service Shadow Training Data Shadow Model Prediction API Result IN Binary Classifier for Membership Inference Some Other Data Prediction API Result OUT ‣ Shadow model mimics the target, and attacker knows its training data
Machine Translation (MT) as An Example ‣ Given black-box access to an MT model, is it possible to determine whether a particular sentence pair was in the training set? 7 Blackbox MT Translation API only ? “Hello” ⁶ “Bonjour” Attacker
Possible Scenarios 8 Bitext Data Provider MT Conference Organizer “MT as a Service” Provider ‣ Attacker may not necessarily be the “bad guy” ‣ Check license violation in published models License License License ‣ Annual bakeoff (e.g., WMT)
‣ Confirm participants are not using test sets Participant Participant Participant ‣ Customized models for users
‣ Attack its own model: Provide privacy guarantee that user data not used elsewhere User Attack Attack Provide & Attack User User
Experiment: Data and Splits ‣ Formulate a fair and reproducible setup for both Alice and Bob 10 Alice data * Actual experiment details more complicated: Please refer to the paper. She uses this to train her model
Experiment: Data and Splits ‣ Formulate a fair and reproducible setup for both Alice and Bob 10 Alice data Bob Data * Actual experiment details more complicated: Please refer to the paper. She uses this to train her model Subset of Alice data: He can use this in whatever way he desires for attacks
Experiment: Data and Splits ‣ Formulate a fair and reproducible setup for both Alice and Bob 10 Alice data Bob Data * Actual experiment details more complicated: Please refer to the paper. She uses this to train her model Subset of Alice data: He can use this in whatever way he desires for attacks IN probes OUT probes Samples for evaluation: IN and OUT of training
Evaluation Procedure 11 Alice data Bob data Target MT model Shadow MT models IN probes OUT probes Attack classifier ‣ If Bob can get attack accuracy above 50%, privacy leak suggested
‣ Alice / Bob model difference
‣ Bob’s attack accuracy on his own model is likely the optimistic upper-bound on the real attack Translate Infer Membership
Attack Classifier for Membership Inference ‣ Binary Classification
‣ “IN” or “OUT” of the model training data?
‣ Features
‣ Modified 1-4 gram precisions
‣ Sentence-level BLEU scores
‣ Optional: MT Model score - extra information for the attacker 12 Intuition: If output is a “good” translation (i.e. similar to the reference translation), the model might have seen it in training time and memorized it
Results: Attacks Not Successful ‣ Around 50%: same as by chance
‣ BLEU and N-gram precision: not enough information to distinguish
‣ Using MT model score did not help either 13 Alice Bob:train Bob:valid Bob:test 50.4 51.5 51.1 51.2 Attack Accuracy of Different Probes Accuracy low even for Classifier in-sample data → Overfitting is not the problem * Even with external resources (MT Quality Estimation model or BERT), the results were the same.
sorami
thc1006
datalabs
uepon73
ryou0634
kuanyi_n_n
sumauma
kosuken
kei18
koheishinden
yukinobaba
yuyay
yukizenimoto
vanstee
shlominoach
maltzj
bermonpainter
colly
hannesfritz
brad_frost
soudai
smashingmag
andyhume
scottboms
morganepeng