Pro Yearly is on sale from $80 to $50! »

Mutually Assured Destruction and the Impending AI Apocalypse

40e37c08199ed4d3866ce6e1ff0be06d?s=47 David Evans
August 13, 2018

Mutually Assured Destruction and the Impending AI Apocalypse

USENIX Workshop on Offensive Technologies 2018
Opening Keynote
Baltimore, Maryland
13 August 2018

The history of security includes a long series of arms races, where a new technology emerges and is subsequently developed and exploited by both defenders and attackers. Over the past few years, "Artificial Intelligence" has re-emerged as a potentially transformative technology, and deep learning in particular has produced a barrage of amazing results. We are in the very early stages of understanding the potential of this technology in security, but more worryingly, seeing how it may be exploited by malicious individuals and powerful organizations. In this talk, I'll look at what lessons might be learned from previous security arms races, consider how asymmetries in AI may be exploited by attackers and defenders, touch on some recent work in adversarial machine learning, and hopefully help progress-loving Luddites figure out how to survive in a world overrun by AI doppelgängers, GAN gangs, and gibbon-impersonating pandas.

David Evans is a Professor of Computer Science at the University of Virginia where he leads the Security Research Group. He is the author of an open computer science textbook and a children's book on combinatorics and computability. He won the Outstanding Faculty Award from the State Council of Higher Education for Virginia, and was Program Co-Chair for the 24th ACM Conference on Computer and Communications Security (CCS 2017) and the 30th (2009) and 31st (2010) IEEE Symposia on Security and Privacy. He has SB, SM and PhD degrees in Computer Science from MIT and has been a faculty member at the University of Virginia since 1999.

40e37c08199ed4d3866ce6e1ff0be06d?s=128

David Evans

August 13, 2018
Tweet

Transcript

  1. Mutually Assured Destruction and the Impending AI Apocalypse David Evans

    University of Virginia evadeML.org USENIX Workshop on Offensive Technologies 13 August 2018 Baltimore, MD
  2. AI Arms Races and How to End Them David Evans

    University of Virginia evadeML.org USENIX Workshop on Offensive Technologies 13 August 2018 Baltimore, MD
  3. 2 All technologies are (potentially) offensive Artificial Intelligence is an

    encompassing, disruptive technology
  4. Plan for Talk 1. What is AI? Definitions 2. What

    should (and shouldn’t) we be afraid of? Harmful use of AI 3. What can we learn from previous arms races? Evasive malware 4. What (if anything) can we do? 3
  5. What is Artificial Intelligence? 4

  6. 5 Doesn’t distinguish from computing in general Unclear target

  7. 6 Cognitive Task Human Machine (2018) Adding 4-digit numbers ü

    Adding 5-digit numbers ü ... ü Adding 8923-digit numbers ü Spelling ü Sorting alphabetically ü Sorting numerically ü Factoring big numbers ü Playing chess ü Playing poker ü Playing go ü Face recognition ü
  8. 7 Cognitive Task Human Machine (2018) Adding 4-digit numbers ü

    Adding 5-digit numbers ü ... ü Adding 8923-digit numbers ü Spelling ü Sorting alphabetically ü Sorting numerically ü Factoring big numbers ü Playing chess ü Playing poker ü Playing go ü Face recognition ü Giving talks at WOOT ?
  9. Preparation for 1st Grade 8

  10. Cognitive Tasks 9 Typical 6-Year Old

  11. Cognitive Tasks 10 Typical 6-Year Old Typical Adult

  12. Cognitive Tasks 11 Typical 6-Year Old Typical Adult Median WOOT

    Attendee
  13. Cognitive Tasks 12 Typical 6-Year Old Typical Adult Any Human

    Alive Median WOOT Attendee
  14. Humanity Cognitive Tasks 13

  15. Humanity Cognitive Tasks 14 Machines (2018)

  16. Humanity Cognitive Tasks 15 Machines (2018) Machines (202x)

  17. 16

  18. 17

  19. More Ambition 18 “The human race will have a new

    kind of instrument which will increase the power of the mind much more than optical lenses strengthen the eyes and which will be as far superior to microscopes or telescopes as reason is superior to sight.”
  20. More Ambition 19 “The human race will have a new

    kind of instrument which will increase the power of the mind much more than optical lenses strengthen the eyes and which will be as far superior to microscopes or telescopes as reason is superior to sight.” Gottfried Wilhelm Leibniz (1679)
  21. 20 Gottfried Wilhelm Leibniz (Universitat Altdorf, 1666) who advised: Jacob

    Bernoulli (Universitdt Basel, 1684) who advised: Johann Bernoulli (Universitdt Basel, 1694) who advised: Leonhard Euler (Universitat Basel, 1726) who advised: Joseph Louis Lagrange who advised: Simeon Denis Poisson who advised: Michel Chasles (Ecole Polytechnique, 1814) who advised: H. A. (Hubert Anson) Newton (Yale, 1850) who advised: E. H. Moore (Yale, 1885) who advised: Oswald Veblen (U. of Chicago, 1903) who advised: Philip Franklin (Princeton 1921) who advised: Alan Perlis (MIT Math PhD 1950) who advised: Jerry Feldman (CMU Math 1966) who advised: Jim Horning (Stanford CS PhD 1969) who advised: John Guttag (U. of Toronto CS PhD 1975) who advised: David Evans (MIT CS PhD 2000) my academic great- great-great-great- great-great-great- great-great-great- great-great-great- great-great- grandparent!
  22. More Precision 21 “The human race will have a new

    kind of instrument which will increase the power of the mind much more than optical lenses strengthen the eyes and which will be as far superior to microscopes or telescopes as reason is superior to sight.” Gottfried Wilhelm Leibniz (1679) Normal computing amplifies (quadrillions of times faster) and aggregates (enables millions of humans to work together) human cognitive abilities; AI goes beyond what humans can do.
  23. 22 (Cover story by Steve Levy) May 5, 1997

  24. 23 The history of computer chess is the history of

    artificial intelligence. After their disappointments in trying to reverse- engineer the brain, computer scientists narrowed their sights. Abandoning their pursuit of human-like intelligence, they began to concentrate on accomplishing sophisticated, but limited, analytical tasks by capitalizing on the inhuman speed of the modern computer’s calculations. This less ambitious but more pragmatic approach has paid off in areas ranging from medical diagnosis to self-driving cars. Computers are replicating the results of human thought without replicating thought itself. Nicolas Carr, A Brutal Intelligence: AI, Chess, and the Human Mind, 2017
  25. 24

  26. 25 Claude Shannon, 1948 Reinforcement Learning Image: Mark Chang, AlphaGo

    in Depth
  27. Operational Definition “Artificial Intelligence” means making computers do things their

    programmers don’t understand well enough to program explicitly. 26 If it is explainable, its not AI!
  28. Plan for Talk 1. What is AI? Definitions 2. What

    should (and shouldn’t) we be afraid of? Harmful use of AI 3. What can we learn from previous arms races? Evasive malware 4. What (if anything) can we do? 27
  29. Making Predictions 28 Paul Gascoigne I never predict anything

  30. Making Predictions 29 Paul Gascoigne I never predict anything, and

    I never will.
  31. Harmful AI Benign developers and operators AI out of control

    AI inadvertently causes harm Malicious operators Build AI to do harm 30
  32. Harmful AI Benign developers and operators AI out of control

    AI inadvertently causes harm Malicious operators Build AI to do harm 31
  33. Out-of-Control AI 32 HAL, 2001: A Space Odyssey SkyNet, The

    Terminator
  34. Alignment Problem 33 Bostrom’s Paperclip Maximizer

  35. Harmful AI Benign developers and operators AI out of control

    AI inadvertently causes harm to humanity Malicious operators Build AI to do harm 34
  36. Lost Jobs and Dignity 35

  37. Lost Jobs and Dignity 36

  38. 37 On Robots Joe Berger and Pascal Wyse (The Guardian,

    21 July 2018) Human Jobs of the Future
  39. Inadvertent Bias and Discrimination 38

  40. Inadvertent Bias and Discrimination 39

  41. Harmful AI Benign developers AI out of control AI causes

    harm (without creators objecting) Malicious developers Using AI to do harm 40 Malice is (often) in the eye of the beholder (e.g., mass surveillance, pop-up ads, etc.)
  42. 41 “The future has arrived — it’s just not evenly

    distributed yet.” (William Gibson, 1990s) Photo: Christopher J. Morris/Corbis
  43. 42 “The future has arrived — it’s just not evenly

    distributed yet.” (William Gibson, 1990s) Expanding victims: Attacks that are only cost-effective for high-value, easy-compromise targets, become cost-effective against everyone Expanding adversaries: Attacks only available to nation-state level adversaries, become accessible to everyone
  44. Malicious Uses of AI 43 Malware Automated Vulnerability Finding, Exploit

    Generation Social Engineering Mass-market Spear Phishing Fake content generation Virtual-physical attacks
  45. Software Vulnerabilities and Exploits 44 IEEE S&P 2013 DARPA Cyber

    Grand Challenge 2016 1996
  46. 45

  47. Strategy 1: Deception Arms Race! 46

  48. Strategy 2: Build Less Vulnerable Systems 47 Rust Project Everest

    We actually know how to build much less vulnerable software, it just costs too much for everyday use.
  49. Malicious Uses of AI 48 Malware Automated Vulnerability Finding, Exploit

    Generation Social Engineering Mass-market Spear Phishing Fake content generation Virtual-physical attacks
  50. 49 WEIS 2012 Automated, low cost: sending out initial scam

    email Human, high effort: conversing with potential victims What happens when the conversing with potential victims part is automated also?
  51. Automated Spear Phishing 50 “It’s slightly less effective [than manually

    generated] but it’s dramatically more efficient” (John Seymour)
  52. Asymmetry of Automated Spear Phishing 51 AI Classifier “99.9% accurate”

    AI Spear Phishing Generator + Botnet ... Victim
  53. (Long-Term) Solution to Spear Phishing 52 Better Authentication Mechanisms Better

    Software
  54. Malicious Uses of AI 53 Malware Automated Vulnerability Finding, Exploit

    Generation Social Engineering Mass-market Spear Phishing Fake content generation Virtual-physical attacks
  55. Fake Content 54 Deep Video Portraits (SIGGRAPH 2018)

  56. Fake Content 55 Deep Video Portraits (SIGGRAPH 2018)

  57. Detection-Generation Arms Race 56 Forgery Technique Detection Classifier Forgery Technique

    Detection Classifier If you know the forgery technique, detection (by machines) has advantage.
  58. Plan for Talk 1. What is AI? Definitions 2. What

    should (and shouldn’t) we be afraid of? Harmful use of AI 3. What can we learn from previous arms races? Evasive malware 4. What (if anything) can we do? 57
  59. Trojan Horse Arms Race 58 Or do you think any

    Greek gift’s free of treachery? Is that Ulysses’s reputation? Either there are Greeks in hiding, concealed by the wood, or it’s been built as a machine to use against our walls, or spy on our homes, or fall on the city from above, or it hides some other trick: Trojans, don’t trust this horse. Whatever it is, I’m afraid of Greeks even those bearing gifts.’ Virgil, The Aenid (Book II)
  60. Evasive Malware Péter Ször (1970-2013)

  61. Adversarial Examples before Deep Learning 60

  62. Labelled Training Data ML Algorithm Feature Extraction Vectors Deployment Malicious

    / Benign Operational Data Trained Classifier Training (supervised learning) Assumption: Training Data is Representative
  63. Deployment Adversaries Don’t Cooperate Assumption: Training Data is Representative Training

    Poisoning
  64. Adversaries Don’t Cooperate Assumption: Training Data is Representative Evading Deployment

    Training
  65. Domain: PDF Malware Classifiers

  66. PDF Malware Classifiers Random Forest Features Object counts, lengths, positions,

    … Manual Features PDFrate [ACSA 2012]
  67. PDF Malware Classifiers Random Forest Random Forest Support Vector Machine

    Features Object counts, lengths, positions, … Object structural paths Very robust against “strongest conceivable mimicry attack”. Automated Features Manual Features PDFrate [ACSA 2012] Hidost16 [JIS 2016] Hidost13 [NDSS 2013]
  68. Adversarial Examples across Domains 67 Domain Classifier Space “Reality” Space

    Trojan Wars Judgment of Trojans !(#) = “gift” Physical Reality !∗(#) = invading army Malware Malware Detector !(#) = “benign” Victim’s Execution !∗(#) = malicious behavior Image Classification, Detection DNN Classifier !(#) = ) Human Perception !∗(#) = * Next Next 2 talks!
  69. “Oracle” Definition 68 Given seed sample, !, !" is an

    adversarial example iff: # !" = % Class is % (for malware, %= “benign”) ℬ !′) = ℬ(! Behavior we care about is the same Malware: evasive variant preserves malicious behavior of seed, but is classified as benign No requirement that ! ~ !′ except through ℬ.
  70. Finding Evasive Malware 69 Given seed sample, !, !" is

    an adversarial example iff: # !" = % Class is % (for malware, %= “benign”) ℬ !′) = ℬ(! Behavior we care about is the same Generic attack: heuristically explore input space for !′ that satisfies definition.
  71. Variants Evolutionary Search Clone Benign PDFs Malicious PDF Mutation 01011001101

    Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Benign Oracle Weilin Xu Yanjun Qi Fitness Selection Mutant Generation
  72. Variants Generating Variants Clone Benign PDFs Malicious PDF Mutation 01011001101

    Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Selection Mutant Generation
  73. PDF Structure

  74. Variants Generating Variants Clone Benign PDFs Malicious PDF Mutation 01011001101

    Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Selection Mutant Generation
  75. Variants Generating Variants Clone Benign PDFs Malicious PDF Mutation 01011001101

    Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Found Evasive ? 0 /JavaScript eval(‘…’); /Root /Catalog /Pages Select random node Randomly transform: delete, insert, replace
  76. Variants Generating Variants Clone Benign PDFs Malicious PDF Mutation 01011001101

    Variants Variants Select Variants Found Evasive? Found Evasive ? Select random node Randomly transform: delete, insert, replace Nodes from Benign PDFs 0 /JavaScript eval(‘…’); /Root /Catalog /Pages 128 546 7 63 128
  77. Variants Selecting Promising Variants Clone Benign PDFs Malicious PDF Mutation

    01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Selection Mutant Generation
  78. Variants Selecting Promising Variants Clone Benign PDFs Malicious PDF Mutation

    01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Function Candidate Variant !(#$%&'() , #'(&++ ) Score Malicious 0 /JavaScript eval(‘…’); /Root /Catalog /Pages 128 Oracle Target Classifier
  79. Oracle: ℬ "′) = ℬ(" ? Execute candidate in vulnerable

    Adobe Reader in virtual environment Behavioral signature: malicious if signature matches https://github.com/cuckoosandbox Simulated network: INetSim Cuckoo HTTP_URL + HOST extracted from API traces
  80. Fitness Function Assumes lost malicious behavior will not be recovered

    !itness '′ = * 1 − classi!ier_score '3 if ℬ '′) = ℬ(' −∞ otherwise
  81. 0 100 200 300 400 500 0 100 200 300

    Seeds Evaded (out of 500) PDFRate Number of Mutations Hidost
  82. 0 100 200 300 400 500 0 100 200 300

    Seeds Evaded (out of 500) PDFRate Number of Mutations Hidost Simple transformations often worked
  83. 0 100 200 300 400 500 0 100 200 300

    Seeds Evaded (out of 500) PDFRate Number of Mutations Hidost (insert, /Root/Pages/Kids, 3:/Root/Pages/Kids/4/Kids/5/) Works on 162/500 seeds
  84. 0 100 200 300 400 500 0 100 200 300

    Seeds Evaded (out of 500) PDFRate Number of Mutations Hidost Some seeds required complex transformations
  85. Malicious Label Threshold Original Malicious Seeds Evading PDFrate Classification Score

    Malware Seed (sorted by original score) Discovered Evasive Variants
  86. Discovered Evasive Variants Malicious Label Threshold Original Malicious Seeds Adjust

    threshold? Charles Smutz, Angelos Stavrou. When a Tree Falls: Using Diversity in Ensemble Classifiers to Identify Evasion in Malware Detectors. NDSS 2016. Classification Score Malware Seed (sorted by original score)
  87. Variants found with threshold = 0.25 Variants found with threshold

    = 0.50 Adjust threshold? Classification Score Malware Seed (sorted by original score)
  88. Variants Hide the Classifier Score? Clone Benign PDFs Malicious PDF

    Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Function Candidate Variant !(#$%&'() , #'(&++ ) Score Malicious 0 /JavaScript eval(‘…’); /Root /Catalog /Pages 128 Oracle Target Classifier
  89. Variants Binary Classifier Output is Enough Clone Benign PDFs Malicious

    PDF Mutation 01011001101 Variants Variants Select Variants ✓ ✓ ✗ ✓ Found Evasive? Fitness Function Candidate Variant !(#$%&'() , #'(&++ ) Score Malicious 0 /JavaScript eval(‘…’); /Root /Catalog /Pages 128 Oracle Target Classifier ACM CCS 2017
  90. Labelled Training Data ML Algorithm Feature Extraction Vectors Deployment Malicious

    / Benign Operational Data Trained Classifier Training (supervised learning) Retrain Classifier
  91. Labelled Training Data ML Algorithm Feature Extraction Vectors Training (supervised

    learning) Clone 01011001 101 EvadeML Deployment
  92. 0 100 200 300 400 500 0 200 400 600

    800 Seeds Evaded (out of 500) Generations Hidost16 Original classifier: Takes 614 generations to evade all seeds
  93. 0 100 200 300 400 500 0 200 400 600

    800 HidostR1 Seeds Evaded (out of 500) Generations Hidost16
  94. 0 100 200 300 400 500 0 200 400 600

    800 HidostR1 Seeds Evaded (out of 500) Generations Hidost16
  95. 0 100 200 300 400 500 0 200 400 600

    800 HidostR1 HidostR2 Seeds Evaded (out of 500) Generations Hidost16
  96. 0 100 200 300 400 500 0 200 400 600

    800 HidostR1 HidostR2 Seeds Evaded (out of 500) Generations Hidost16
  97. 0 100 200 300 400 500 0 200 400 600

    800 Hidost16 Genome Contagio Benign Hidost16 0.00 0.00 HidostR1 0.78 0.30 HidostR2 0.85 0.53 False Positive Rates HidostR1 Seeds Evaded (out of 500) Generations HidostR2
  98. 97 Only 8/6987 robust features (Hidost) Robust classifier High false

    positives /Names /Names /JavaScript /Names /JavaScript /Names /Names /JavaScript /JS /OpenAction /OpenAction /JS /OpenAction /S /Pages
  99. AI Arms Races AI-based defenses are at-best temporary 98 “Artificial

    Intelligence” means making computers do things their programmers don’t understand well enough to program explicitly. Can be effective against current adversaries Asymmetries benefit attackers Motivated adversary with any access to defense can learn to thwart it
  100. AI Arms Races AI-based defenses are at-best temporary 99 “Artificial

    Intelligence” means making computers do things their programmers don’t understand well enough to program explicitly. Can be effective against current adversaries Asymmetries benefit attackers Motivated adversary with any access to defense can learn to thwart it Can only work reliably, if we are using robust features that are strong signals – but then, don’t need AI!
  101. Real Solution to Malicious PDFs 100 Better Software

  102. Plan for Talk 1. What is AI? Definitions 2. What

    should (and shouldn’t) we be afraid of? Harmful use of AI 3. What can we learn from previous arms races? Evasive malware 4. What (if anything) can we do? 101
  103. 102 https://maliciousaireport.com/

  104. 103 https://maliciousaireport.com/

  105. AI-Based Attacks Low-cost, low-risk automation of attacks New types of

    attacks Humans will be easily fooled 104
  106. 105 In defense of Luddites?

  107. 106 In defense of Luddites?

  108. 107

  109. “Made by Human” Labels Certified: Human Made

  110. Google’s Duplex Demo

  111. 110 @_youhadonejob1

  112. David Evans University of Virginia evans@virginia.edu EvadeML.org