Mutually Assured Destruction and the Impending AI Apocalypse

USENIX Workshop on Offensive Technologies 2018
Opening Keynote
Baltimore, Maryland
13 August 2018

The history of security includes a long series of arms races, where a new technology emerges and is subsequently developed and exploited by both defenders and attackers. Over the past few years, "Artificial Intelligence" has re-emerged as a potentially transformative technology, and deep learning in particular has produced a barrage of amazing results. We are in the very early stages of understanding the potential of this technology in security, but more worryingly, seeing how it may be exploited by malicious individuals and powerful organizations. In this talk, I'll look at what lessons might be learned from previous security arms races, consider how asymmetries in AI may be exploited by attackers and defenders, touch on some recent work in adversarial machine learning, and hopefully help progress-loving Luddites figure out how to survive in a world overrun by AI doppelgängers, GAN gangs, and gibbon-impersonating pandas.

David Evans is a Professor of Computer Science at the University of Virginia where he leads the Security Research Group. He is the author of an open computer science textbook and a children's book on combinatorics and computability. He won the Outstanding Faculty Award from the State Council of Higher Education for Virginia, and was Program Co-Chair for the 24th ACM Conference on Computer and Communications Security (CCS 2017) and the 30th (2009) and 31st (2010) IEEE Symposia on Security and Privacy. He has SB, SM and PhD degrees in Computer Science from MIT and has been a faculty member at the University of Virginia since 1999.

Transcript

1. Mutually Assured Destruction and the Impending AI Apocalypse David Evans University of Virginia evadeML.org USENIX Workshop on Offensive Technologies 13 August 2018 Baltimore, MD
2. AI Arms Races and How to End Them David Evans University of Virginia evadeML.org USENIX Workshop on Offensive Technologies 13 August 2018 Baltimore, MD
3. 2 All technologies are (potentially) offensive Artificial Intelligence is an encompassing, disruptive technology
4. Plan for Talk 1. What is AI? Definitions 2. What should (and shouldn’t) we be afraid of? Harmful use of AI 3. What can we learn from previous arms races? Evasive malware 4. What (if anything) can we do? 3
5. What is Artificial Intelligence? 4
6. 5 Doesn’t distinguish from computing in general Unclear target
7. 6 Cognitive Task Human Machine (2018) Adding 4-digit numbers ü Adding 5-digit numbers ü ... ü Adding 8923-digit numbers ü Spelling ü Sorting alphabetically ü Sorting numerically ü Factoring big numbers ü Playing chess ü Playing poker ü Playing go ü Face recognition ü
8. 7 Cognitive Task Human Machine (2018) Adding 4-digit numbers ü Adding 5-digit numbers ü ... ü Adding 8923-digit numbers ü Spelling ü Sorting alphabetically ü Sorting numerically ü Factoring big numbers ü Playing chess ü Playing poker ü Playing go ü Face recognition ü Giving talks at WOOT ?
9. Preparation for 1st Grade 8
10. Cognitive Tasks 9 Typical 6-Year Old
11. Cognitive Tasks 10 Typical 6-Year Old Typical Adult
12. Cognitive Tasks 11 Typical 6-Year Old Typical Adult Median WOOT Attendee
13. Cognitive Tasks 12 Typical 6-Year Old Typical Adult Any Human Alive Median WOOT Attendee
14. Humanity Cognitive Tasks 13
15. Humanity Cognitive Tasks 14 Machines (2018)
16. Humanity Cognitive Tasks 15 Machines (2018) Machines (202x)
19. More Ambition 18 “The human race will have a new kind of instrument which will increase the power of the mind much more than optical lenses strengthen the eyes and which will be as far superior to microscopes or telescopes as reason is superior to sight.”
20. More Ambition 19 “The human race will have a new kind of instrument which will increase the power of the mind much more than optical lenses strengthen the eyes and which will be as far superior to microscopes or telescopes as reason is superior to sight.” Gottfried Wilhelm Leibniz (1679)
21. 20 Gottfried Wilhelm Leibniz (Universitat Altdorf, 1666) who advised: Jacob Bernoulli (Universitdt Basel, 1684) who advised: Johann Bernoulli (Universitdt Basel, 1694) who advised: Leonhard Euler (Universitat Basel, 1726) who advised: Joseph Louis Lagrange who advised: Simeon Denis Poisson who advised: Michel Chasles (Ecole Polytechnique, 1814) who advised: H. A. (Hubert Anson) Newton (Yale, 1850) who advised: E. H. Moore (Yale, 1885) who advised: Oswald Veblen (U. of Chicago, 1903) who advised: Philip Franklin (Princeton 1921) who advised: Alan Perlis (MIT Math PhD 1950) who advised: Jerry Feldman (CMU Math 1966) who advised: Jim Horning (Stanford CS PhD 1969) who advised: John Guttag (U. of Toronto CS PhD 1975) who advised: David Evans (MIT CS PhD 2000) my academic great- great-great-great- great-great-great- great-great-great- great-great-great- great-great- grandparent!
22. More Precision 21 “The human race will have a new kind of instrument which will increase the power of the mind much more than optical lenses strengthen the eyes and which will be as far superior to microscopes or telescopes as reason is superior to sight.” Gottfried Wilhelm Leibniz (1679) Normal computing amplifies (quadrillions of times faster) and aggregates (enables millions of humans to work together) human cognitive abilities; AI goes beyond what humans can do.
23. 22 (Cover story by Steve Levy) May 5, 1997
24. 23 The history of computer chess is the history of artificial intelligence. After their disappointments in trying to reverse- engineer the brain, computer scientists narrowed their sights. Abandoning their pursuit of human-like intelligence, they began to concentrate on accomplishing sophisticated, but limited, analytical tasks by capitalizing on the inhuman speed of the modern computer’s calculations. This less ambitious but more pragmatic approach has paid off in areas ranging from medical diagnosis to self-driving cars. Computers are replicating the results of human thought without replicating thought itself. Nicolas Carr, A Brutal Intelligence: AI, Chess, and the Human Mind, 2017
26. 25 Claude Shannon, 1948 Reinforcement Learning Image: Mark Chang, AlphaGo in Depth
27. Operational Definition “Artificial Intelligence” means making computers do things their programmers don’t understand well enough to program explicitly. 26 If it is explainable, its not AI!
28. Plan for Talk 1. What is AI? Definitions 2. What should (and shouldn’t) we be afraid of? Harmful use of AI 3. What can we learn from previous arms races? Evasive malware 4. What (if anything) can we do? 27
29. Making Predictions 28 Paul Gascoigne I never predict anything
30. Making Predictions 29 Paul Gascoigne I never predict anything, and I never will.
31. Harmful AI Benign developers and operators AI out of control AI inadvertently causes harm Malicious operators Build AI to do harm 30
32. Harmful AI Benign developers and operators AI out of control AI inadvertently causes harm Malicious operators Build AI to do harm 31
33. Out-of-Control AI 32 HAL, 2001: A Space Odyssey SkyNet, The Terminator
34. Alignment Problem 33 Bostrom’s Paperclip Maximizer
35. Harmful AI Benign developers and operators AI out of control AI inadvertently causes harm to humanity Malicious operators Build AI to do harm 34
36. Lost Jobs and Dignity 35
37. Lost Jobs and Dignity 36
38. 37 On Robots Joe Berger and Pascal Wyse (The Guardian, 21 July 2018) Human Jobs of the Future
39. Inadvertent Bias and Discrimination 38
40. Inadvertent Bias and Discrimination 39
41. Harmful AI Benign developers AI out of control AI causes harm (without creators objecting) Malicious developers Using AI to do harm 40 Malice is (often) in the eye of the beholder (e.g., mass surveillance, pop-up ads, etc.)
42. 41 “The future has arrived — it’s just not evenly distributed yet.” (William Gibson, 1990s) Photo: Christopher J. Morris/Corbis
43. 42 “The future has arrived — it’s just not evenly distributed yet.” (William Gibson, 1990s) Expanding victims: Attacks that are only cost-effective for high-value, easy-compromise targets, become cost-effective against everyone Expanding adversaries: Attacks only available to nation-state level adversaries, become accessible to everyone
44. Malicious Uses of AI 43 Malware Automated Vulnerability Finding, Exploit Generation Social Engineering Mass-market Spear Phishing Fake content generation Virtual-physical attacks
45. Software Vulnerabilities and Exploits 44 IEEE S&P 2013 DARPA Cyber Grand Challenge 2016 1996
47. Strategy 1: Deception Arms Race! 46
48. Strategy 2: Build Less Vulnerable Systems 47 Rust Project Everest We actually know how to build much less vulnerable software, it just costs too much for everyday use.
49. Malicious Uses of AI 48 Malware Automated Vulnerability Finding, Exploit Generation Social Engineering Mass-market Spear Phishing Fake content generation Virtual-physical attacks
50. 49 WEIS 2012 Automated, low cost: sending out initial scam email Human, high effort: conversing with potential victims What happens when the conversing with potential victims part is automated also?
51. Automated Spear Phishing 50 “It’s slightly less effective [than manually generated] but it’s dramatically more efficient” (John Seymour)
52. Asymmetry of Automated Spear Phishing 51 AI Classifier “99.9% accurate” AI Spear Phishing Generator + Botnet ... Victim
53. (Long-Term) Solution to Spear Phishing 52 Better Authentication Mechanisms Better Software
54. Malicious Uses of AI 53 Malware Automated Vulnerability Finding, Exploit Generation Social Engineering Mass-market Spear Phishing Fake content generation Virtual-physical attacks
55. Fake Content 54 Deep Video Portraits (SIGGRAPH 2018)
56. Fake Content 55 Deep Video Portraits (SIGGRAPH 2018)
57. Detection-Generation Arms Race 56 Forgery Technique Detection Classifier Forgery Technique Detection Classifier If you know the forgery technique, detection (by machines) has advantage.
58. Plan for Talk 1. What is AI? Definitions 2. What should (and shouldn’t) we be afraid of? Harmful use of AI 3. What can we learn from previous arms races? Evasive malware 4. What (if anything) can we do? 57
59. Trojan Horse Arms Race 58 Or do you think any Greek gift’s free of treachery? Is that Ulysses’s reputation? Either there are Greeks in hiding, concealed by the wood, or it’s been built as a machine to use against our walls, or spy on our homes, or fall on the city from above, or it hides some other trick: Trojans, don’t trust this horse. Whatever it is, I’m afraid of Greeks even those bearing gifts.’ Virgil, The Aenid (Book II)
60. Evasive Malware Péter Ször (1970-2013)
61. Adversarial Examples before Deep Learning 60
62. Labelled Training Data ML Algorithm Feature Extraction Vectors Deployment Malicious / Benign Operational Data Trained Classifier Training (supervised learning) Assumption: Training Data is Representative
63. Deployment Adversaries Don’t Cooperate Assumption: Training Data is Representative Training Poisoning
64. Adversaries Don’t Cooperate Assumption: Training Data is Representative Evading Deployment Training
65. Domain: PDF Malware Classifiers
66. PDF Malware Classifiers Random Forest Features Object counts, lengths, positions, … Manual Features PDFrate [ACSA 2012]
67. PDF Malware Classifiers Random Forest Random Forest Support Vector Machine Features Object counts, lengths, positions, … Object structural paths Very robust against “strongest conceivable mimicry attack”. Automated Features Manual Features PDFrate [ACSA 2012] Hidost16 [JIS 2016] Hidost13 [NDSS 2013]
68. Adversarial Examples across Domains 67 Domain Classifier Space “Reality” Space Trojan Wars Judgment of Trojans !(#) = “gift” Physical Reality !∗(#) = invading army Malware Malware Detector !(#) = “benign” Victim’s Execution !∗(#) = malicious behavior Image Classification, Detection DNN Classifier !(#) = ) Human Perception !∗(#) = * Next Next 2 talks!
69. “Oracle” Definition 68 Given seed sample, !, !" is an adversarial example iff: # !" = % Class is % (for malware, %= “benign”) ℬ !′) = ℬ(! Behavior we care about is the same Malware: evasive variant preserves malicious behavior of seed, but is classified as benign No requirement that ! ~ !′ except through ℬ.
70. Finding Evasive Malware 69 Given seed sample, !, !" is an adversarial example iff: # !" = % Class is % (for malware, %= “benign”) ℬ !′) = ℬ(! Behavior we care about is the same Generic attack: heuristically explore input space for !′ that satisfies definition.
71. Variants Evolutionary Search Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Benign Oracle Weilin Xu Yanjun Qi Fitness Selection Mutant Generation
72. Variants Generating Variants Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Selection Mutant Generation
73. PDF Structure
74. Variants Generating Variants Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Selection Mutant Generation
75. Variants Generating Variants Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Found Evasive ? 0 /JavaScript eval(‘…’); /Root /Catalog /Pages Select random node Randomly transform: delete, insert, replace
76. Variants Generating Variants Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants Found Evasive? Found Evasive ? Select random node Randomly transform: delete, insert, replace Nodes from Benign PDFs 0 /JavaScript eval(‘…’); /Root /Catalog /Pages 128 546 7 63 128
77. Variants Selecting Promising Variants Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Selection Mutant Generation
78. Variants Selecting Promising Variants Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Function Candidate Variant !(#$%&'() , #'(&++ ) Score Malicious 0 /JavaScript eval(‘…’); /Root /Catalog /Pages 128 Oracle Target Classifier
79. Oracle: ℬ "′) = ℬ(" ? Execute candidate in vulnerable Adobe Reader in virtual environment Behavioral signature: malicious if signature matches https://github.com/cuckoosandbox Simulated network: INetSim Cuckoo HTTP_URL + HOST extracted from API traces
80. Fitness Function Assumes lost malicious behavior will not be recovered !itness '′ = * 1 − classi!ier_score '3 if ℬ '′) = ℬ(' −∞ otherwise
81. 0 100 200 300 400 500 0 100 200 300 Seeds Evaded (out of 500) PDFRate Number of Mutations Hidost
82. 0 100 200 300 400 500 0 100 200 300 Seeds Evaded (out of 500) PDFRate Number of Mutations Hidost Simple transformations often worked
83. 0 100 200 300 400 500 0 100 200 300 Seeds Evaded (out of 500) PDFRate Number of Mutations Hidost (insert, /Root/Pages/Kids, 3:/Root/Pages/Kids/4/Kids/5/) Works on 162/500 seeds
84. 0 100 200 300 400 500 0 100 200 300 Seeds Evaded (out of 500) PDFRate Number of Mutations Hidost Some seeds required complex transformations
85. Malicious Label Threshold Original Malicious Seeds Evading PDFrate Classification Score Malware Seed (sorted by original score) Discovered Evasive Variants
86. Discovered Evasive Variants Malicious Label Threshold Original Malicious Seeds Adjust threshold? Charles Smutz, Angelos Stavrou. When a Tree Falls: Using Diversity in Ensemble Classifiers to Identify Evasion in Malware Detectors. NDSS 2016. Classification Score Malware Seed (sorted by original score)
87. Variants found with threshold = 0.25 Variants found with threshold = 0.50 Adjust threshold? Classification Score Malware Seed (sorted by original score)
88. Variants Hide the Classifier Score? Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Function Candidate Variant !(#$%&'() , #'(&++ ) Score Malicious 0 /JavaScript eval(‘…’); /Root /Catalog /Pages 128 Oracle Target Classifier
89. Variants Binary Classifier Output is Enough Clone Benign PDFs Malicious PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Function Candidate Variant !(#$%&'() , #'(&++ ) Score Malicious 0 /JavaScript eval(‘…’); /Root /Catalog /Pages 128 Oracle Target Classifier ACM CCS 2017
90. Labelled Training Data ML Algorithm Feature Extraction Vectors Deployment Malicious / Benign Operational Data Trained Classifier Training (supervised learning) Retrain Classifier
91. Labelled Training Data ML Algorithm Feature Extraction Vectors Training (supervised learning) Clone 01011001 101 EvadeML Deployment
92. 0 100 200 300 400 500 0 200 400 600 800 Seeds Evaded (out of 500) Generations Hidost16 Original classifier: Takes 614 generations to evade all seeds
93. 0 100 200 300 400 500 0 200 400 600 800 HidostR1 Seeds Evaded (out of 500) Generations Hidost16
94. 0 100 200 300 400 500 0 200 400 600 800 HidostR1 Seeds Evaded (out of 500) Generations Hidost16
95. 0 100 200 300 400 500 0 200 400 600 800 HidostR1 HidostR2 Seeds Evaded (out of 500) Generations Hidost16
96. 0 100 200 300 400 500 0 200 400 600 800 HidostR1 HidostR2 Seeds Evaded (out of 500) Generations Hidost16
97. 0 100 200 300 400 500 0 200 400 600 800 Hidost16 Genome Contagio Benign Hidost16 0.00 0.00 HidostR1 0.78 0.30 HidostR2 0.85 0.53 False Positive Rates HidostR1 Seeds Evaded (out of 500) Generations HidostR2
98. 97 Only 8/6987 robust features (Hidost) Robust classifier High false positives /Names /Names /JavaScript /Names /JavaScript /Names /Names /JavaScript /JS /OpenAction /OpenAction /JS /OpenAction /S /Pages
99. AI Arms Races AI-based defenses are at-best temporary 98 “Artificial Intelligence” means making computers do things their programmers don’t understand well enough to program explicitly. Can be effective against current adversaries Asymmetries benefit attackers Motivated adversary with any access to defense can learn to thwart it
100. AI Arms Races AI-based defenses are at-best temporary 99 “Artificial Intelligence” means making computers do things their programmers don’t understand well enough to program explicitly. Can be effective against current adversaries Asymmetries benefit attackers Motivated adversary with any access to defense can learn to thwart it Can only work reliably, if we are using robust features that are strong signals – but then, don’t need AI!
101. Real Solution to Malicious PDFs 100 Better Software
102. Plan for Talk 1. What is AI? Definitions 2. What should (and shouldn’t) we be afraid of? Harmful use of AI 3. What can we learn from previous arms races? Evasive malware 4. What (if anything) can we do? 101
103. 102 https://maliciousaireport.com/
104. 103 https://maliciousaireport.com/
105. AI-Based Attacks Low-cost, low-risk automation of attacks New types of attacks Humans will be easily fooled 104
106. 105 In defense of Luddites?
107. 106 In defense of Luddites?
109. “Made by Human” Labels Certified: Human Made
110. Google’s Duplex Demo
111. 110 @_youhadonejob1
112. David Evans University of Virginia evans@virginia.edu EvadeML.org