Is "adversarial examples" an Adversarial Example?

Is "adversarial
examples" an
Adversarial
Example?
David Evans
University of Virginia
evadeML.org
Deep Learning and
Security Workshop
24 May 2018
San Francisco, CA

View Slide

GDPR in
effect May 25
(tomorrow)!

View Slide

GDPR in
effect now!

View Slide

GDPR in Effect 00:37:34
Response Due 71:22:26
Maximum Fine (Google) $2,120,889,281
“Manager’s nightmare, but a researcher’s paradise!”
– David Basin
GDPR in Effect
Response Due
Maximum Fine (Google)
GDPR in
effect now!

View Slide

Article 22

View Slide

Is “adversarial
examples” an
Adversarial
Example?

View Slide

6
Papers on “Adversarial Examples”
(Google Scholar)
675
0
200
400
600
800
1000
1200
2018
(5/22)
2017
2016
2015
2014
2013
1241.5 papers
expected in 2018!

View Slide

Adversarial Examples before Deep Learning
7

View Slide

Adversarial Examples “before ML”
Péter Ször (1970-2013)

View Slide

Adversarial Examples before “Oakland”
9

View Slide

Adversarial Examples before “Oakland”
10
The crowd, uncertain, was split by opposing opinions.
Then Laocoön rushes down eagerly from the heights of
the citadel, to confront them all, a large crowd with
him, and shouts from far off: ‘O unhappy citizens, what
madness? ... Do you think the enemy’s sailed away? Or
do you think any Greek gift’s free of treachery? Is that
Ulysses’s reputation? Either there are Greeks in hiding,
concealed by the wood, or it’s been built as a machine
to use against our walls, or spy on our homes, or fall
on the city from above, or it hides some other trick:
Trojans, don’t trust this horse. Whatever it is, I’m afraid
of Greeks even those bearing gifts.’
Virgil, The Aenid (Book II)

View Slide

11
How should we define
“adversarial example”?

View Slide

How should we define
“adversarial example”?
12
“Adversarial examples are inputs to
machine learning models that an
attacker has intentionally designed to
cause the model to make a mistake.”
Ian Goodfellow, earlier today

View Slide

Adversarial Examples across Domains
13
Domain Classifier Space “Reality” Space
Trojan Wars
Judgment of Trojans
!(#) = “gift”
Physical Reality
!∗(#) = invading army
Malware
Malware Detector
!(#) = “benign”
Victim’s Execution
!∗(#) = malicious behavior
Image
Classification
DNN Classifier
!(#) = )
Human Perception
!∗(#) = *
Later Next Not DL

View Slide

Malware Adversarial Examples
14
Classifier Space
Oracle Space
actual program execution
https://github.com/cuckoosandbox
Cuckoo

View Slide

“Oracle” Definition
15
Given seed sample, !, !" is an adversarial example iff:
# !" = % Class is % (for malware, %= “benign”)
ℬ !′) = ℬ(! Behavior we care about is the same
Malware: evasive variant preserves malicious
behavior of seed, but is classified as benign
No requirement that ! ~ !′ except through ℬ.

View Slide

Definitions suggest Attacks
16
# !" = % Class is % (for malware, %= “benign”)
ℬ !′) = ℬ(! Behavior we care about is the same
Generic attack: heuristically explore input
space for !′ that satisfies definition.

View Slide

Variants
Evolutionary Search
Clone
Benign PDFs
Malicious PDF
Mutation
01011001101
Variants
Variants
Select
Variants
✓
✓
✗
✓
Found
Evasive?
Benign
Oracle
Weilin Xu Yanjun Qi
Fitness
Selection
Mutant
Generation

View Slide

Variants
Generating Variants
Clone
Benign PDFs
Malicious PDF
Mutation
01011001101
Variants
Variants
Select
Variants
✓
✓
✗
✓
Found
Evasive?
Fitness
Selection
Mutant
Generation

View Slide

Variants
Generating Variants
Clone
Benign PDFs
Malicious PDF
Mutation
01011001101
Variants
Variants
Select
Variants
✓
✓
✗
✓
Found
Evasive?
Found
Evasive
?
0
/JavaScript
eval(‘…’);
/Root
/Catalog
/Pages
Select random node
Randomly transform: delete, insert, replace

View Slide

Variants
Generating Variants
Clone
Benign PDFs
Malicious PDF
Mutation
01011001101
Variants
Variants
Select
Variants
Found
Evasive?
Found
Evasive
?
Select random node
Randomly transform: delete, insert, replace
Nodes from
Benign PDFs
0
/JavaScript
eval(‘…’);
/Root
/Catalog
/Pages
128
546
7
63
128

View Slide

Variants
Selecting Promising Variants
Clone
Benign PDFs
Malicious PDF
Mutation
01011001101
Variants
Variants
Select
Variants
✓
✓
✗
✓
Found
Evasive?
Fitness
Selection
Mutant
Generation

View Slide

Variants
Selecting Promising Variants
Clone
Benign PDFs
Malicious PDF
Mutation
01011001101
Variants
Variants
Select
Variants
✓
✓
✗
✓
Found
Evasive?
Fitness Function
Candidate Variant
!(#$%&'()
, #'(&++
)
Score
Malicious
0
/JavaScript
eval(‘…’);
/Root
/Catalog
/Pages
128
Oracle
Target Classifier

View Slide

Oracle: ℬ "′) = ℬ(" ?
Execute candidate in
vulnerable Adobe Reader in
virtual environment
Behavioral signature:
malicious if signature matches
https://github.com/cuckoosandbox
Simulated network: INetSim
Cuckoo
HTTP_URL + HOST
extracted from API traces

View Slide

Fitness Function
Assumes lost malicious behavior will not be
recovered
!itness '′ = *
1 − classi!ier_score '3 if ℬ '′) = ℬ('
−∞ otherwise

View Slide

0
100
200
300
400
500
0 100 200 300
Seeds Evaded
(out of 500)
PDFRate
Number of Mutations
Hidost

View Slide

0
100
200
300
400
500
0 100 200 300
Seeds Evaded
(out of 500)
PDFRate
Number of Mutations
Hidost
Simple
transformations
often worked

View Slide

0
100
200
300
400
500
0 100 200 300
Seeds Evaded
(out of 500)
PDFRate
Number of Mutations
Hidost
(insert, /Root/Pages/Kids,
3:/Root/Pages/Kids/4/Kids/5/)
Works on 162/500 seeds

View Slide

0
100
200
300
400
500
0 100 200 300
Seeds Evaded
(out of 500)
PDFRate
Number of Mutations
Hidost
Some seeds
required complex
transformations

View Slide

Attacks suggest Defenses*
29

View Slide

Attacks suggest Defenses*
30
* That only work against a very particular instantiation of that attack.
Maginot Line
Enigma Plugboard

View Slide

Malicious Label
Threshold
Original Malicious Seeds
Evading
PDFrate
Classification Score
Malware Seed (sorted by original score)
Discovered Evasive Variants

View Slide

Discovered Evasive Variants
Malicious Label
Threshold
Original Malicious Seeds
Adjust threshold?
Charles Smutz, Angelos
Stavrou. When a Tree Falls:
Using Diversity in Ensemble
Classifiers to Identify
Evasion in Malware
Detectors. NDSS 2016.

View Slide

Variants found with threshold = 0.25
Variants found with threshold = 0.50
Adjust threshold?

View Slide

Variants
Hide the Classifier Score?
Clone
Benign PDFs
Malicious PDF
Mutation
01011001101
Variants
Variants
Select
Variants
✓
✓
✗
✓
Found
Evasive?
Fitness Function
Candidate Variant
!(#$%&'()
, #'(&++
)
Score
Malicious
0
/JavaScript
eval(‘…’);
/Root
/Catalog
/Pages
128
Oracle
Target Classifier

View Slide

Variants
Binary Classifier Output is Enough
Clone
Benign PDFs
Malicious PDF
Mutation
01011001101
Variants
Variants
Select
Variants
✓
✓
✗
✓
Found
Evasive?
Fitness Function
Candidate Variant
!(#$%&'()
, #'(&++
)
Score
Malicious
0
/JavaScript
eval(‘…’);
/Root
/Catalog
/Pages
128
Oracle
Target Classifier
ACM CCS 2017

View Slide

36
Defenses should be designed around clear
definitions of adversary goals and capabilities,
not around thwarting particular attacks.
(The second oldest principle in security.)

View Slide

37
Trojan Wars
Judgment of Trojans
!(#) = “gift”
Physical Reality
Malware
Malware Detector
!(#) = “benign”
Image
Classification
DNN Classifier
!(#) = )
Human Perception
!∗(#) = *
Next Done Not DL

View Slide

38
Trojan Wars
Judgment of Trojans
!(#) = “gift”
Physical Reality
Malware
Malware Detector
!(#) = “benign”
Image
Classification
DNN Classifier
!(#) = )
Human Perception
!∗(#) = *
Fixing (Breaking?) the Definition

View Slide

Image
Classification
DNN Classifier
!(#) = &
Human Perception
!∗(#) = (
39
Fixing (Breaking?) the Definition

View Slide

Well-Trained Classifier
40
Model and visualization based on work by Beilun Wang, Ji Gao and Yanjun Qi (ICLR 2017 Workshop)
Classifier Space
(DNN Model)
“Oracle” Space
(human perception)

View Slide

Adversarial Examples
41
Model and visualization based on work by Beilun Wang, Ji Gao and Yanjun Qi (ICLR 2017 Workshop)
Classifier Space
(DNN Model)
“Oracle” Space
(human perception)

View Slide

Misleading Visualization
42
Cartoon Reality
2 dimensions thousands of
dimensions
few samples near
boundaries
all samples near
boundaries
every sample
near 1-3 classes
every sample
near all classes
Classifier Space
(DNN Model)

View Slide

Adversarial Examples
43
Adversary’s goal: find a small perturbation that changes class for classifier, but imperceptible to oracle.
Classifier Space
(DNN Model)
“Oracle” Space
(human perception)

View Slide

44
Battista Biggio, et al. ECML-KDD 2013

View Slide

“Biggio” Definition
45
Assumption (to map to earlier definition):
small perturbation does not change class in “Reality Space”
# !" = % Class is % (targeted)
∆ !, !" ≤ ) Difference below threshold
∆ !, !" is defined in some (simple!) metric space:
*+
norm (# different), *,
, *-
norm (“Euclidean distance”), *.

View Slide

“Biggio” Definition
46
*+
, *-
Problem #1: Every model with boundaries has adversarial examples.
Problem #2: Very unnatural limit on adversary strength.
Problem #3: Values all adversarial examples equally.

View Slide

DSML Papers
47
Biggio
Definition
(6)
No
Version
On-Line
(5)
Oracle
Definition
(3)
KFS, YKLALYP, RG
AHHO, CW, GLSQ,
HD, MW, SBC
Building
Classifiers (5)
AMNKV, CSS,
DAF, SHWS,
ZCPS,
Software
(2)
BGS, XLZX

View Slide

Impact of Adversarial Perturbations
48
Distance between layer output and its output for original seed
FGSM
! = 0.0245
CIFAR-10
DenseNet
95th percentile
5th percentile

View Slide

49
FGSM
! = 0.0245
CIFAR-10
DenseNet
95th percentile
5th percentile Mainuddin
Jonas

View Slide

50
Random noise
(same amount)
FGSM
! = 0.0245
CIFAR-10
DenseNet

View Slide

51
Random noise
(same amount)
Carlini-
Wagner L2
CIFAR-10
DenseNet

View Slide

Definitions Suggest Defenses
52
*+
, *-
Suggested Defense: given an input !∗, see how the model behaves
on 0(!∗) where 0(3) reverses transformations in ∆-space.

View Slide

Model
Model
Squeezer
1
Prediction0
Prediction1
"($%&'(
, $%&'*
, … , $%&',
)
Input
Adversarial
Legitimate
Model’
Squeezer
k
…
Predictionk
Feature Squeezing Detection Framework
Weilin Xu Yanjun Qi

View Slide

Model
Model
Squeezer
1
Prediction0
Prediction1
"($%&'(
, $%&'*
, … , $%&',
)
Input
Adversarial
Legitimate
Model’
Squeezer
k
…
Predictionk
Feature Squeezing Detection Framework
Feature Squeezer coalesces similar inputs into one point:
• Barely change legitimate inputs.
• Destruct adversarial perturbations.

View Slide

Coalescing by Feature Squeezing
55
Metric Space 1: Target Classifier Metric Space 2: “Oracle”
Before: find a small perturbation that changes class for classifier, but imperceptible to oracle.
Now: change class for both original and squeezed classifier, but imperceptible to oracle.

View Slide

Example Squeezer: Bit Depth Reduction
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
3-bit
1-bit
8-bit
Input
Output
56
Signal Quantization

View Slide

Example Squeezer: Bit Depth Reduction
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
3-bit
1-bit
8-bit
Input
Output
57
Signal Quantization
Seed
1 1 4 2 2
1 1 1 1 1
CW
2
CW
∞
BIM
FGSM

View Slide

Other Potential Squeezers
58
C Xie, et al. Mitigating Adversarial Effects Through Randomization, ICLR 2018.
J Buckman, et al. Thermometer Encoding: One Hot Way To Resist Adversarial
Examples, ICLR 2018.
D Meng and H Chen, MagNet: a Two-Pronged Defense against Adversarial
Examples, CCS 2017; A Prakash, et al., Deflecting Adversarial Attacks with Pixel
Deflection, CVPR 2018;...
Thermometer Encoding (learnable bit depth reduction)
Image denoising using autoencoder, wavelet, JPEG, etc.
Image resizing
...
Spatial Smoothers: median filter, non-local means

View Slide

“Feature Squeezing” (Vacuous) Conjecture
For any distance-limited adversarial method,
there exists some feature squeezer that
accurately detects its adversarial examples.
59
Intuition: if the perturbation is small (in some simple
metric space), there is some squeezer that coalesces
original and adversarial example into same sample.

View Slide

Feature Squeezing Detection
Model
(7-layer
CNN)
Model
Model
Bit Depth-
1
Median
2×2
Prediction0
Prediction1
Prediction2
Yes
Input
Adversarial
No
Legitimate
max '(
)*
, )(
, '(
)*
, )2 > -

View Slide

Detecting Adversarial Examples
Distance between original input and its squeezed version
Adversarial
inputs
(CW attack)
Legitimate
inputs

View Slide

62
0
200
400
600
800
0.0 0.4 0.8 1.2 1.6 2.0
Number of Examples
Legitimate
Adversarial
Maximum !"
distance between original and squeezed input
threshold = 0.0029
detection: 98.2%, FP < 4%
Training a detector
(MNIST)
set the detection threshold to keep
false positive rate below target

View Slide

ImageNet Configuration
Model
(MobileNet)
Model
Model
Bit Depth-
5
Median
2×2
Prediction0
Prediction1
Prediction2
Yes
Input
Adversarial
No
Legitimate
max(()
(*+
, {*)
, *.
, */
}) > 3
Model
Non-local
Mean
Prediction3

View Slide

64
0
20
40
60
80
100
120
140
0.0 0.4 0.8 1.2 1.6 2.0
Legitimate
Adversarial
Maximum !"
distance between original and squeezed input
threshold = 1.24
detection: 85%, FP < 5%
Training a detector
(ImageNet)

View Slide

How should we
evaluate defenses?
65

View Slide

Threat Models
Oblivious attack: The adversary has full knowledge of
the target model, but is not aware of the detector.
Adaptive attack: The adversary has full knowledge of
the target model and the detector.
66

View Slide

(Generic) Adaptive Adversary
Adaptive CW
2
attack, unbounded adversary:
Warren He, James Wei, Xinyun Chen, Nicholas Carlini,
Dawn Song, Adversarial Example Defense: Ensembles of
Weak Defenses are not Strong, USENIX WOOT’17.
!"#"!"$% & '( − * + , ∗ Δ ', '( + 0 ∗ 12
3456%('′)
67
Misclassification term Distance term Detection term

View Slide

Adaptive Adversarial Examples
68
No successful adversarial examples were found
for images originally labeled as 3 or 8.
Mean L2
2.80
4.14
4.67
Attack
Untargeted
Targeted
(next)
Targeted
(least likely)

View Slide

Adaptive Adversary Success Rates
69
0.68
0.06
0.01
0.44
0.01
0.24
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Adversary’s Success Rate
Clipped ε
Targeted
(Next)
Targeted
(LL)
Untargeted
Unbounded
Typical !

View Slide

Revisiting Attacker’s Goal
Find one adversarial example Find many adversarial examples
Suya Yuan Tian

View Slide

Attacker Visibility
“White-box attacker”
Knows model architecture and all parameters
“Black-box attacker”
Interacts with model through API
Limited number of interactions
Output is vector
decision-based: output is just class
“bird”, 0.09
“horse”, 0.84
...

View Slide

Black-Box Cost Variance
72
CIFAR-10
MNIST
Number of Queries
Number of Adversarial Examples
Pin-Yu Chen, Huan Zhang,
Yash Sharma, Jinfeng Yi,
Cho-Jui Hsieh. ZOO: Zeroth
Order Optimization Based
Black-box Attacks to Deep
Neural Networks without
Training Substitute
Models. AISec 2017.
(1000 queries per
iteration, 256 max iters)
fails for 14
117,820 Average for MNIST
60,378
CIFAR-10 overall average
15,795 (26%)
CIFAR-10 lowest-cost 20 ave:
50,240
Average for
lowest 20
Target:
least-likely class
max !
2
= 3
256,000

View Slide

Easy and Hard Examples
“Easy”
“Hard”
“Easy” images: 5 with fewest number of queries needed to find adversarial example
“Hard” images: 5 with highest number of queries (failed)
256,000 query attempts without success
14,592
0 → (least likely) 1
43,008 43,776 49,152 49,920

View Slide

Easy and Hard Examples
“Easy”
“Hard”
“Easy” images: 5 with fewest number of queries needed to find adversarial example
“Hard” images: 5 with highest number of queries (failed)
14,592
0 → (least likely) 1 “airplane” → “frog”
43,008 43,776 49,152 49,920 9,728 10,496 10,752 12,288 13,824

View Slide

White-Box Cost Variance
75
CIFAR-10
MNIST
Number of Iterations
Carlini-Wagner L
2
Attack
82 CIFAR-10 average
Target:
least-likely class
MNIST: max !
2
= 3.0
CIFAR-10: max !
2
= 1.0
2000
566 Average for MNIST
174
Average for
lowest 20

View Slide

White-Box Cost Variance
76
CIFAR-10
MNIST
Number of Iterations
Carlini-Wagner L
2
Attack
Target:
least-likely class
MNIST: max !
2
= 3.0
CIFAR-10: max !
2
= 1.0
2000
566
174
Average for
lowest 20
CIFAR-10 lowest 20 (average: 3.6)
82 CIFAR-10 average

View Slide

How does
cost-variance
impact attack
cost?
77

View Slide

78
CIFAR-10
Average queries per AE found (× 10$)
Random target selection
Greedy heuristic
Oracle Optimal
Simple Greedy Search Works Well
MNIST
ZOO Black-Box Attack
Target: 20 MNIST CIFAR
Greedy/Optimal 1.50 1.30
Random/Optimal 2.37 3.86

View Slide

White-Box Batch Attack Cost
79
Random target
selection
Greedy heuristic
Oracle
Optimal
CIFAR-10
Average iterations per AE found
MNIST
CW L
2
Attack

View Slide

Madry Defense
80
Accuracy
“9”
“7”
“0”
Batch (10 samples, sorted by initial distance)
MNIST
airplane
cars
deer
CIFAR-10
Aleksander Madry,
Aleksandar Makelov,
Ludwig Schmidt, Dimitris
Tsipras, Adrian Vladu.
Towards Deep Learning
Models Resistant to
Adversarial Attacks.
https://github.com/Madr
yLab/mnist_challenge

View Slide

History of the
destruction
of Troy, 1498
Conclusion

View Slide

Security State-of-the-Art
Attack success
probability
Threat models Proofs
Cryptography !"#!$
information
theoretic, resource
bounded
required
System Security !"%!
capabilities,
motivations,
rationality
common
Adversarial
Machine Learning
!&; !"#*
white-box,
black-box
making
progress?
82

View Slide

83
Ali Rahimi
NIPS Test-of-Time Award Speech
(Dec 2017)
”If you're building photo-
sharing systems alchemy
is okay but we're beyond
that; now we're building
systems that govern
healthcare and mediate
our civic dialogue”

View Slide

84
Ali Rahimi
NIPS Test-of-Time Award Speech
(Dec 2017)
”If you're building photo-
sharing systems alchemy
is okay but we're beyond
that; now we're building
systems that govern
healthcare and mediate
our civic dialogue”

View Slide

Alchemy (~700 − 1660)
Well-defined, testable goal (turn
lead into gold)
Established theory (four elements:
earth, fire, water, air)
Methodical experiments and lab
techniques (Jabir ibn Hayyan in
8th century)
Wrong and ultimately unsuccessful,
but led to modern chemistry.

View Slide

86
Trojan Wars
Judgment of Trojans
!(#) = “gi=”
Physical Reality
Malware
Malware Detector
!(#) = “benign”
Image
Classification
DNN Classifier
!(#) = )
Human Perception
!∗(#) = *

View Slide

87
Trojan Wars
Judgment of Trojans
!(#) = “gift”
Physical Reality
Malware
Malware Detector
!(#) = “benign”
Image
Classification
DNN Classifier
!(#) = )
Human Perception
!∗(#) = *
Academic
Research
Conferences, Fun
!(+,s) = “awesome”
Systems, Society, Ideas
!∗ +,s = ?

View Slide

David Evans
[email protected]
EvadeML.org
Weilin Xu Yanjun Qi Fnu Suya Yuan Tian Mainuddin Jonas
Funding: NSF, Intel

View Slide

89

View Slide

David Evans
[email protected]
EvadeML.org
Weilin Xu Yanjun Qi Fnu Suya Yuan Tian Mainuddin Jonas
Funding: NSF, Intel

View Slide

91
@_youhadonejob1

View Slide

92

View Slide

Is "adversarial examples" an Adversarial Example?

Is "adversarial examples" an Adversarial Example?

David Evans

More Decks by David Evans

Other Decks in Research

Featured

Transcript