Mutually Assured Destruction and the Impending AI Apocalypse

Slide 1

Slide 1 text

Mutually Assured Destruction and the Impending AI Apocalypse David Evans University of Virginia evadeML.org USENIX Workshop on Offensive Technologies 13 August 2018 Baltimore, MD

Slide 2

Slide 2 text

AI Arms Races and How to End Them David Evans University of Virginia evadeML.org USENIX Workshop on Offensive Technologies 13 August 2018 Baltimore, MD

Slide 3

Slide 3 text

2 All technologies are (potentially) offensive Artificial Intelligence is an encompassing, disruptive technology

Slide 4

Slide 4 text

Plan for Talk 1. What is AI? Definitions 2. What should (and shouldn’t) we be afraid of? Harmful use of AI 3. What can we learn from previous arms races? Evasive malware 4. What (if anything) can we do? 3

Slide 5

Slide 5 text

What is Artificial Intelligence? 4

Slide 6

Slide 6 text

5 Doesn’t distinguish from computing in general Unclear target

Slide 7

Slide 7 text

6 Cognitive Task Human Machine (2018) Adding 4-digit numbers ü Adding 5-digit numbers ü ... ü Adding 8923-digit numbers ü Spelling ü Sorting alphabetically ü Sorting numerically ü Factoring big numbers ü Playing chess ü Playing poker ü Playing go ü Face recognition ü

Slide 8

Slide 8 text

7 Cognitive Task Human Machine (2018) Adding 4-digit numbers ü Adding 5-digit numbers ü ... ü Adding 8923-digit numbers ü Spelling ü Sorting alphabetically ü Sorting numerically ü Factoring big numbers ü Playing chess ü Playing poker ü Playing go ü Face recognition ü Giving talks at WOOT ?

Slide 9

Slide 9 text

Preparation for 1st Grade 8

Slide 10

Slide 10 text

Cognitive Tasks 9 Typical 6-Year Old

Slide 11

Slide 11 text

Cognitive Tasks 10 Typical 6-Year Old Typical Adult

Slide 12

Slide 12 text

Cognitive Tasks 11 Typical 6-Year Old Typical Adult Median WOOT Attendee

Slide 13

Slide 13 text

Cognitive Tasks 12 Typical 6-Year Old Typical Adult Any Human Alive Median WOOT Attendee

Slide 14

Slide 14 text

Humanity Cognitive Tasks 13

Slide 15

Slide 15 text

Humanity Cognitive Tasks 14 Machines (2018)

Slide 16

Slide 16 text

Humanity Cognitive Tasks 15 Machines (2018) Machines (202x)

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

More Ambition 18 “The human race will have a new kind of instrument which will increase the power of the mind much more than optical lenses strengthen the eyes and which will be as far superior to microscopes or telescopes as reason is superior to sight.”

Slide 20

Slide 20 text

More Ambition 19 “The human race will have a new kind of instrument which will increase the power of the mind much more than optical lenses strengthen the eyes and which will be as far superior to microscopes or telescopes as reason is superior to sight.” Gottfried Wilhelm Leibniz (1679)

Slide 21

Slide 21 text

20 Gottfried Wilhelm Leibniz (Universitat Altdorf, 1666) who advised: Jacob Bernoulli (Universitdt Basel, 1684) who advised: Johann Bernoulli (Universitdt Basel, 1694) who advised: Leonhard Euler (Universitat Basel, 1726) who advised: Joseph Louis Lagrange who advised: Simeon Denis Poisson who advised: Michel Chasles (Ecole Polytechnique, 1814) who advised: H. A. (Hubert Anson) Newton (Yale, 1850) who advised: E. H. Moore (Yale, 1885) who advised: Oswald Veblen (U. of Chicago, 1903) who advised: Philip Franklin (Princeton 1921) who advised: Alan Perlis (MIT Math PhD 1950) who advised: Jerry Feldman (CMU Math 1966) who advised: Jim Horning (Stanford CS PhD 1969) who advised: John Guttag (U. of Toronto CS PhD 1975) who advised: David Evans (MIT CS PhD 2000) my academic great- great-great-great- great-great-great- great-great-great- great-great-great- great-great- grandparent!

Slide 22

Slide 22 text

More Precision 21 “The human race will have a new kind of instrument which will increase the power of the mind much more than optical lenses strengthen the eyes and which will be as far superior to microscopes or telescopes as reason is superior to sight.” Gottfried Wilhelm Leibniz (1679) Normal computing amplifies (quadrillions of times faster) and aggregates (enables millions of humans to work together) human cognitive abilities; AI goes beyond what humans can do.

Slide 23

Slide 23 text

22 (Cover story by Steve Levy) May 5, 1997

Slide 24

Slide 24 text

23 The history of computer chess is the history of artificial intelligence. After their disappointments in trying to reverse- engineer the brain, computer scientists narrowed their sights. Abandoning their pursuit of human-like intelligence, they began to concentrate on accomplishing sophisticated, but limited, analytical tasks by capitalizing on the inhuman speed of the modern computer’s calculations. This less ambitious but more pragmatic approach has paid off in areas ranging from medical diagnosis to self-driving cars. Computers are replicating the results of human thought without replicating thought itself. Nicolas Carr, A Brutal Intelligence: AI, Chess, and the Human Mind, 2017

Slide 25

Slide 25 text

Slide 26

Slide 26 text

25 Claude Shannon, 1948 Reinforcement Learning Image: Mark Chang, AlphaGo in Depth

Slide 27

Slide 27 text

Operational Definition “Artificial Intelligence” means making computers do things their programmers don’t understand well enough to program explicitly. 26 If it is explainable, its not AI!

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Making Predictions 28 Paul Gascoigne I never predict anything

Slide 30

Slide 30 text

Making Predictions 29 Paul Gascoigne I never predict anything, and I never will.

Slide 31

Slide 31 text

Harmful AI Benign developers and operators AI out of control AI inadvertently causes harm Malicious operators Build AI to do harm 30

Slide 32

Slide 32 text

Harmful AI Benign developers and operators AI out of control AI inadvertently causes harm Malicious operators Build AI to do harm 31

Slide 33

Slide 33 text

Out-of-Control AI 32 HAL, 2001: A Space Odyssey SkyNet, The Terminator

Slide 34

Slide 34 text

Alignment Problem 33 Bostrom’s Paperclip Maximizer

Slide 35

Slide 35 text

Harmful AI Benign developers and operators AI out of control AI inadvertently causes harm to humanity Malicious operators Build AI to do harm 34

Slide 36

Slide 36 text

Lost Jobs and Dignity 35

Slide 37

Slide 37 text

Lost Jobs and Dignity 36

Slide 38

Slide 38 text

37 On Robots Joe Berger and Pascal Wyse (The Guardian, 21 July 2018) Human Jobs of the Future

Slide 39

Slide 39 text

Inadvertent Bias and Discrimination 38

Slide 40

Slide 40 text

Inadvertent Bias and Discrimination 39

Slide 41

Slide 41 text

Harmful AI Benign developers AI out of control AI causes harm (without creators objecting) Malicious developers Using AI to do harm 40 Malice is (often) in the eye of the beholder (e.g., mass surveillance, pop-up ads, etc.)

Slide 42

Slide 42 text

41 “The future has arrived — it’s just not evenly distributed yet.” (William Gibson, 1990s) Photo: Christopher J. Morris/Corbis

Slide 43

Slide 43 text

42 “The future has arrived — it’s just not evenly distributed yet.” (William Gibson, 1990s) Expanding victims: Attacks that are only cost-effective for high-value, easy-compromise targets, become cost-effective against everyone Expanding adversaries: Attacks only available to nation-state level adversaries, become accessible to everyone

Slide 44

Slide 44 text

Malicious Uses of AI 43 Malware Automated Vulnerability Finding, Exploit Generation Social Engineering Mass-market Spear Phishing Fake content generation Virtual-physical attacks

Slide 45

Slide 45 text

Software Vulnerabilities and Exploits 44 IEEE S&P 2013 DARPA Cyber Grand Challenge 2016 1996

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Strategy 1: Deception Arms Race! 46

Slide 48

Slide 48 text

Strategy 2: Build Less Vulnerable Systems 47 Rust Project Everest We actually know how to build much less vulnerable software, it just costs too much for everyday use.

Slide 49

Slide 49 text

Malicious Uses of AI 48 Malware Automated Vulnerability Finding, Exploit Generation Social Engineering Mass-market Spear Phishing Fake content generation Virtual-physical attacks

Slide 50

Slide 50 text

49 WEIS 2012 Automated, low cost: sending out initial scam email Human, high effort: conversing with potential victims What happens when the conversing with potential victims part is automated also?

Slide 51

Slide 51 text

Automated Spear Phishing 50 “It’s slightly less effective [than manually generated] but it’s dramatically more efficient” (John Seymour)

Slide 52

Slide 52 text

Asymmetry of Automated Spear Phishing 51 AI Classifier “99.9% accurate” AI Spear Phishing Generator + Botnet ... Victim

Slide 53

Slide 53 text

(Long-Term) Solution to Spear Phishing 52 Better Authentication Mechanisms Better Software

Slide 54

Slide 54 text

Malicious Uses of AI 53 Malware Automated Vulnerability Finding, Exploit Generation Social Engineering Mass-market Spear Phishing Fake content generation Virtual-physical attacks

Slide 55

Slide 55 text

Fake Content 54 Deep Video Portraits (SIGGRAPH 2018)

Slide 56

Slide 56 text

Fake Content 55 Deep Video Portraits (SIGGRAPH 2018)

Slide 57

Slide 57 text

Detection-Generation Arms Race 56 Forgery Technique Detection Classifier Forgery Technique Detection Classifier If you know the forgery technique, detection (by machines) has advantage.

Slide 58

Slide 58 text

Slide 59

Slide 59 text

Trojan Horse Arms Race 58 Or do you think any Greek gift’s free of treachery? Is that Ulysses’s reputation? Either there are Greeks in hiding, concealed by the wood, or it’s been built as a machine to use against our walls, or spy on our homes, or fall on the city from above, or it hides some other trick: Trojans, don’t trust this horse. Whatever it is, I’m afraid of Greeks even those bearing gifts.’ Virgil, The Aenid (Book II)

Slide 60

Slide 60 text

Evasive Malware Péter Ször (1970-2013)

Slide 61

Slide 61 text

Adversarial Examples before Deep Learning 60

Slide 62

Slide 62 text

Labelled Training Data ML Algorithm Feature Extraction Vectors Deployment Malicious / Benign Operational Data Trained Classifier Training (supervised learning) Assumption: Training Data is Representative

Slide 63

Slide 63 text

Deployment Adversaries Don’t Cooperate Assumption: Training Data is Representative Training Poisoning

Slide 64

Slide 64 text

Adversaries Don’t Cooperate Assumption: Training Data is Representative Evading Deployment Training

Slide 65

Slide 65 text

Domain: PDF Malware Classifiers

Slide 66

Slide 66 text

PDF Malware Classifiers Random Forest Features Object counts, lengths, positions, … Manual Features PDFrate [ACSA 2012]

Slide 67

Slide 67 text

PDF Malware Classifiers Random Forest Random Forest Support Vector Machine Features Object counts, lengths, positions, … Object structural paths Very robust against “strongest conceivable mimicry attack”. Automated Features Manual Features PDFrate [ACSA 2012] Hidost16 [JIS 2016] Hidost13 [NDSS 2013]

Slide 68

Slide 68 text

Adversarial Examples across Domains 67 Domain Classifier Space “Reality” Space Trojan Wars Judgment of Trojans !(#) = “gift” Physical Reality !∗(#) = invading army Malware Malware Detector !(#) = “benign” Victim’s Execution !∗(#) = malicious behavior Image Classification, Detection DNN Classifier !(#) = ) Human Perception !∗(#) = * Next Next 2 talks!

Slide 69

Slide 69 text

“Oracle” Definition 68 Given seed sample, !, !" is an adversarial example iff: # !" = % Class is % (for malware, %= “benign”) ℬ !′) = ℬ(! Behavior we care about is the same Malware: evasive variant preserves malicious behavior of seed, but is classified as benign No requirement that ! ~ !′ except through ℬ.

Slide 70

Slide 70 text

Finding Evasive Malware 69 Given seed sample, !, !" is an adversarial example iff: # !" = % Class is % (for malware, %= “benign”) ℬ !′) = ℬ(! Behavior we care about is the same Generic attack: heuristically explore input space for !′ that satisfies definition.

Slide 71

Slide 71 text

Variants Evolutionary Search Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Benign Oracle Weilin Xu Yanjun Qi Fitness Selection Mutant Generation

Slide 72

Slide 72 text

Variants Generating Variants Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Selection Mutant Generation

Slide 73

Slide 73 text

PDF Structure

Slide 74

Slide 74 text

Variants Generating Variants Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Selection Mutant Generation

Slide 75

Slide 75 text

Variants Generating Variants Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Found Evasive ? 0 /JavaScript eval(‘…’); /Root /Catalog /Pages Select random node Randomly transform: delete, insert, replace

Slide 76

Slide 76 text

Variants Generating Variants Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants Found Evasive? Found Evasive ? Select random node Randomly transform: delete, insert, replace Nodes from Benign PDFs 0 /JavaScript eval(‘…’); /Root /Catalog /Pages 128 546 7 63 128

Slide 77

Slide 77 text

Variants Selecting Promising Variants Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Selection Mutant Generation

Slide 78

Slide 78 text

Variants Selecting Promising Variants Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Function Candidate Variant !(#$%&'() , #'(&++ ) Score Malicious 0 /JavaScript eval(‘…’); /Root /Catalog /Pages 128 Oracle Target Classifier

Slide 79

Slide 79 text

Oracle: ℬ "′) = ℬ(" ? Execute candidate in vulnerable Adobe Reader in virtual environment Behavioral signature: malicious if signature matches https://github.com/cuckoosandbox Simulated network: INetSim Cuckoo HTTP_URL + HOST extracted from API traces

Slide 80

Slide 80 text

Fitness Function Assumes lost malicious behavior will not be recovered !itness '′ = * 1 − classi!ier_score '3 if ℬ '′) = ℬ(' −∞ otherwise

Slide 81

Slide 81 text

0 100 200 300 400 500 0 100 200 300 Seeds Evaded (out of 500) PDFRate Number of Mutations Hidost

Slide 82

Slide 82 text

0 100 200 300 400 500 0 100 200 300 Seeds Evaded (out of 500) PDFRate Number of Mutations Hidost Simple transformations often worked

Slide 83

Slide 83 text

0 100 200 300 400 500 0 100 200 300 Seeds Evaded (out of 500) PDFRate Number of Mutations Hidost (insert, /Root/Pages/Kids, 3:/Root/Pages/Kids/4/Kids/5/) Works on 162/500 seeds

Slide 84

Slide 84 text

0 100 200 300 400 500 0 100 200 300 Seeds Evaded (out of 500) PDFRate Number of Mutations Hidost Some seeds required complex transformations

Slide 85

Slide 85 text

Malicious Label Threshold Original Malicious Seeds Evading PDFrate Classification Score Malware Seed (sorted by original score) Discovered Evasive Variants

Slide 86

Slide 86 text

Discovered Evasive Variants Malicious Label Threshold Original Malicious Seeds Adjust threshold? Charles Smutz, Angelos Stavrou. When a Tree Falls: Using Diversity in Ensemble Classifiers to Identify Evasion in Malware Detectors. NDSS 2016. Classification Score Malware Seed (sorted by original score)

Slide 87

Slide 87 text

Variants found with threshold = 0.25 Variants found with threshold = 0.50 Adjust threshold? Classification Score Malware Seed (sorted by original score)

Slide 88

Slide 88 text

Variants Hide the Classifier Score? Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Function Candidate Variant !(#$%&'() , #'(&++ ) Score Malicious 0 /JavaScript eval(‘…’); /Root /Catalog /Pages 128 Oracle Target Classifier

Slide 89

Slide 89 text

Variants Binary Classifier Output is Enough Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Function Candidate Variant !(#$%&'() , #'(&++ ) Score Malicious 0 /JavaScript eval(‘…’); /Root /Catalog /Pages 128 Oracle Target Classifier ACM CCS 2017

Slide 90

Slide 90 text

Labelled Training Data ML Algorithm Feature Extraction Vectors Deployment Malicious / Benign Operational Data Trained Classifier Training (supervised learning) Retrain Classifier

Slide 91

Slide 91 text

Labelled Training Data ML Algorithm Feature Extraction Vectors Training (supervised learning) Clone 01011001 101 EvadeML Deployment

Slide 92

Slide 92 text

0 100 200 300 400 500 0 200 400 600 800 Seeds Evaded (out of 500) Generations Hidost16 Original classifier: Takes 614 generations to evade all seeds

Slide 93

Slide 93 text

0 100 200 300 400 500 0 200 400 600 800 HidostR1 Seeds Evaded (out of 500) Generations Hidost16

Slide 94

Slide 94 text

0 100 200 300 400 500 0 200 400 600 800 HidostR1 Seeds Evaded (out of 500) Generations Hidost16

Slide 95

Slide 95 text

0 100 200 300 400 500 0 200 400 600 800 HidostR1 HidostR2 Seeds Evaded (out of 500) Generations Hidost16

Slide 96

Slide 96 text

0 100 200 300 400 500 0 200 400 600 800 HidostR1 HidostR2 Seeds Evaded (out of 500) Generations Hidost16

Slide 97

Slide 97 text

0 100 200 300 400 500 0 200 400 600 800 Hidost16 Genome Contagio Benign Hidost16 0.00 0.00 HidostR1 0.78 0.30 HidostR2 0.85 0.53 False Positive Rates HidostR1 Seeds Evaded (out of 500) Generations HidostR2

Slide 98

Slide 98 text

97 Only 8/6987 robust features (Hidost) Robust classifier High false positives /Names /Names /JavaScript /Names /JavaScript /Names /Names /JavaScript /JS /OpenAction /OpenAction /JS /OpenAction /S /Pages

Slide 99

Slide 99 text

AI Arms Races AI-based defenses are at-best temporary 98 “Artificial Intelligence” means making computers do things their programmers don’t understand well enough to program explicitly. Can be effective against current adversaries Asymmetries benefit attackers Motivated adversary with any access to defense can learn to thwart it

Slide 100

Slide 100 text

AI Arms Races AI-based defenses are at-best temporary 99 “Artificial Intelligence” means making computers do things their programmers don’t understand well enough to program explicitly. Can be effective against current adversaries Asymmetries benefit attackers Motivated adversary with any access to defense can learn to thwart it Can only work reliably, if we are using robust features that are strong signals – but then, don’t need AI!

Slide 101

Slide 101 text

Real Solution to Malicious PDFs 100 Better Software

Slide 102

Slide 102 text

Slide 103

Slide 103 text

102 https://maliciousaireport.com/

Slide 104

Slide 104 text

103 https://maliciousaireport.com/

Slide 105

Slide 105 text

AI-Based Attacks Low-cost, low-risk automation of attacks New types of attacks Humans will be easily fooled 104

Slide 106

Slide 106 text

105 In defense of Luddites?

Slide 107

Slide 107 text

106 In defense of Luddites?

Slide 108

Slide 108 text

107

Slide 109

Slide 109 text

“Made by Human” Labels Certified: Human Made

Slide 110

Slide 110 text

Google’s Duplex Demo

Slide 111

Slide 111 text

110 @_youhadonejob1

Slide 112

Slide 112 text

David Evans University of Virginia evans@virginia.edu EvadeML.org