Adversarial Machine Learning: Are We Playing the Wrong Game? David Evans University of Virginia work mostly with Weilin Xu and Yanjun Qi evadeML.org Center for IT-Security, Privacy and Accountability, Universität des Saarlandes 10 July 2017
Labelled Training Data ML Algorithm Feature Extraction Vectors Deployment Malicious / Benign Operational Data Trained Classifier Training (supervised learning)
Labelled Training Data ML Algorithm Feature Extraction Vectors Deployment Malicious / Benign Operational Data Trained Classifier Training (supervised learning) Assumption: Training Data is Representative
Goal of Machine Learning Classifier 11 Metric Space 1: Target Classifier Metric Space 2: “Oracle” Model and visualization based on work by Beilun Wang, Ji Gao and Yanjun Qi (ICLR 2017 Workshop)
Well-Trained Classifier 12 Metric Space 1: Target Classifier Metric Space 2: “Oracle” Model and visualization based on work by Beilun Wang, Ji Gao and Yanjun Qi (ICLR 2017 Workshop)
Adversarial Examples 13 Metric Space 1: Target Classifier Metric Space 2: “Oracle” Model and visualization based on work by Beilun Wang, Ji Gao and Yanjun Qi (ICLR 2017 Workshop)
Adversarial Examples 14 Metric Space 1: Target Classifier Metric Space 2: “Oracle” Adversary’s goal: find a small perturbation that changes class for classifier, but imperceptible to oracle.
Misleading Visualization 15 Metric Space 1: Target Classifier Cartoon Reality 2 dimensions thousands of dimensions few samples near boundaries all samples near boundaries every sample near 1-3 classes every sample near all classes
AdversarialDNN Playground 25 Andrew Norton and Yanjun Qi Live demo: https://evadeML.org/playground Will be integrated with EvadeML-Zoo models and attacks soon! L
26 Given seed sample, , find 0 where: 0 ≠ () Class is different or 0 = Class is target class ∆ , 0 ≤ Difference below threshold Is this the right game?
New Idea: Detect Adversarial Examples 29 Given seed sample, , find 0 where: 0 ≠ () Class is different ∆ , 0 ≤ Difference below threshold Deployed classifier only sees 0 - can we search for “”?
30 Model Model Model Filter 1 Filter 2 Prediction Prediction′ Prediction′′ Compare Predictions Difference exceeds threshold Reject Prediction Ok Input Need filters that do not affect predictions on normal inputs, but that reverse malicious perturbations.
Squeezing Images 34 Reduce Color Depth Median Smoothing 8-bit greyscale 1-bit monochrome 3x3 smoothing: Replace with median of pixels and its neighbors
MNIST Results: Accuracy 35 Original (8) 7 6 5 4 3 2 1 .9930 .9930 .9930 .9930 .9930 .9928 .9926 .9924 Reducing bit depth (all the way to 1) barely reduces model accuracy! Correct on original image, wrong on 1-bit filtered image (19) Wrong on original image, correct on 1-bit filtered image (13) (out of 10 000 MNIST test images) Both wrong, but differently
Carlini/Wagner Untargeted Attacks 41 Data Set Attack Accuracy on Adversarial Examples MNIST C 0.0 D 0.0 9 0.0 CIFAR-10 C 0.0 D 0.0 9 0.0 Nicholas Carlini, David Wagner. Oakland 2017 (Best Student Paper) Adversary suceeds 100% of the time with very small perturbations “Our D attacks on ImageNet are so successful that we can change the classification of an image to any desired label by only flipping the lowest bit of each pixel, a change that would be impossible to detect visually.”
Squeezing Results (2x2 Median Smoothing) 42 Weilin Xu, David Evans, Yanjun Qi. https://arxiv.org/1705.10686 Data Set Attack Accuracy on Adversarial Examples Original Squeezed MNIST C 0.0 0.904 D 0.0 0.942 9 0.0 0.817 CIFAR-10 C 0.0 0.682 D 0.0 0.661 9 0.0 0.706
Results on Carlini/Wagner Untargeted Attacks 43 Weilin Xu, David Evans, Yanjun Qi. https://arxiv.org/1705.10686 Data Set Attack Accuracy on Adversarial Examples Original Squeezed MNIST C 0.0 0.904 D 0.0 0.942 9 0.0 0.817 CIFAR-10 C 0.0 0.682 D 0.0 0.661 9 0.0 0.706 Accuracy on legitimate examples: 0.783
Results on Carlini/Wagner Targeted Attacks 44 Weilin Xu, David Evans, Yanjun Qi. https://arxiv.org/1705.10686 Data Set Attack Adversary Success Rate Original Squeezed MNIST C 0.999 0.022 D 1.0 0.011 9 1.0 0.057 CIFAR-10 C 1.0 0.033 D 1.0 0.037 9 1.0 0.037
Detecting Adversarial Examples 45 Model Model Model Squeeze Bit Depth Median Smoothing Prediction Prediction′ Prediction′′ Compare Predictions (> distance) Difference exceeds threshold Adversarial Normal Ok Input
Raising the Bar or Changing the Game? 52 Metric Space 1: Target Classifier Metric Space 2: “Oracle” Before: find a small perturbation that changes class for classifier, but imperceptible to oracle.
Raising the Bar or Changing the Game? 53 Metric Space 1: Target Classifier Metric Space 2: “Oracle” Before: find a small perturbation that changes class for classifier, but imperceptible to oracle. Now: change class for both original and squeezed classifier, but imperceptible to oracle.
“Feature Squeezing” Conjecture For any distance-limited adversarial method, there exists some feature squeezer that accurately detects its adversarial examples. 54 Intuition: if the perturbation is small (in some simple metric space), there is some squeezer that coalesces original and adversarial example into same sample.
Entropy Advantage Model Model Model Randomized Squeezer #1 Prediction Prediction′ Prediction′′ Compare Predictions (> distance) Difference exceeds threshold Adversarial Normal Ok Input Randomized Squeezer #2 Squeezers can be selected randomly, and behave randomly different for each feature
Changing the Game Option 1: Find distance-limited adversarial methods for which it is intractable to find effective feature squeezer. Option 2: Redefine adversarial examples so distance is not limited (in simple metric space). 56 focus of rest of the talk
PDF Malware Classifiers Random Forest Random Forest Support Vector Machine Features Object counts, lengths, positions, … Object structural paths Very robust against “strongest conceivable mimicry attack”. Automated Features Manual Features PDFrate [ACSA 2012] Hidost16 [JIS 2016] Hidost13 [NDSS 2013]
Oracle Execute candidate in vulnerable Adobe Reader in virtual environment Behavioral signature: malicious if signature matches https://github.com/cuckoosandbox Simulated network: INetSim Cuckoo HTTP_URL + HOST extracted from API traces Advantage: we know the target malware behavior
Fitness Function Assumes lost malicious behavior will not be recovered = m .5 − classifier_score if oracle = "malicious" −∞ otherwise classifier_score ≥ 0.5: labeled malicious
0 100 200 300 400 500 0 100 200 300 Seeds Evaded (out of 500) PDFRate Number of Mutations Hidost Works on 162/500 seeds Some seeds required complex transformations
Possible Defense: Adjust Threshold Charles Smutz, Angelos Stavrou. When a Tree Falls: Using Diversity in Ensemble Classifiers to Identify Evasion in Malware Detectors. NDSS 2016.
Labelled Training Data ML Algorithm Feature Extraction Vectors Deployment Malicious / Benign Operational Data Trained Classifier Training (supervised learning) Retrain Classifier
Hopeful Conclusions Domain Knowledge is not Dead • Classifiers trained without understanding vulnerable • Adversaries can exploit unnecessary features Trust Requires Understanding • Good results against test data do not apply to adaptive adversaries but there is hope for building robust ML models!
Credits Funding: National Science Foundation, Air Force Office of Scientific Research, Google, Microsoft, Amazon Weilin Xu Security Research Group Yanjun Qi
evansuva
dscs
doradora09
nkmr_rl
lcolladotor
s1ok69oo
ssii
deepflow
wakamatsu_takumu
aronwalsh
ianozsvald
miyayou
edds
geoffreycrofte
aki_iinuma
holman
myddelton
akmur
dotmariusz
eileencodes
jeffersonlam
gr2m
maggiecrowley
bcantrill