Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Automatically Evading Classifiers: A Case Study on PDF Malware Classifiers

David Evans
February 24, 2016

Automatically Evading Classifiers: A Case Study on PDF Malware Classifiers

Weilin Xu's talk at
Network and Distributed Systems Symposium 2016
24 February 2016
San Diego, California

For more details see: http://evademl.org/
and the full paper: http://evademl.org/docs/evademl.pdf

David Evans

February 24, 2016
Tweet

More Decks by David Evans

Other Decks in Research

Transcript

  1. Automatically Evading Classifiers
    A Case Study on PDF Malware Classifiers
    Weilin Xu David Evans Yanjun Qi
    University of Virginia

    View Slide

  2. Machine Learning is Solving Our Problems
    2
    Fake
    Spam IDS Malware
    Fake
    Accounts


    View Slide

  3. 3

    View Slide

  4. 4

    View Slide

  5. Machine Learning is Eating the World
    Data
    Scientist
    Security
    Expert
    5
    ?

    View Slide

  6. Machine Learning is Eating the World
    Data
    Scientist
    Security
    Expert
    6
    No!
    Security is different.

    View Slide

  7. Goal: Understand classifiers under attack.
    Results: Vulnerable to automated evasion.
    Security Tasks are Different:
    Adversary Adapts
    7

    View Slide

  8. Building Machine Learning Classifiers
    8
    Trained Classifier
    Labelled
    Training
    Data
    ML
    Algorithm
    Training
    (Supervised Learning)
    Feature
    Extraction
    Vectors

    View Slide

  9. Assumption: Training Data is Representative
    9
    Labelled
    Training
    Data
    ML
    Algorithm
    Feature
    Extraction
    Vectors
    Deployment
    Malicious / Benign
    Operational Data
    Trained Classifier
    Training
    (Supervised Learning)

    View Slide

  10. Results: Evaded PDF Malware Classifiers
    PDFrate*
    [ACSAC’12]
    Hidost
    [NDSS’13]
    Accuracy 0.9976 0.9996
    False Negative Rate 0.0000 0.0056
    False Negative Rate
    with Adversary
    1.0000 1.0000
    10
    * Mimicus [Oakland ’14], an open source reimplementation of PDFrate.

    View Slide

  11. Results: Evaded PDF Malware Classifiers
    PDFrate*
    [ACSAC’12]
    Hidost
    [NDSS’13]
    Accuracy 0.9976 0.9996
    False Negative Rate 0.0000 0.0056
    False Negative Rate
    with Adversary
    1.0000 1.0000
    11
    Very robust against “strongest
    conceivable mimicry attack”.
    * Mimicus [Oakland ’14], an open source reimplementation of PDFrate.

    View Slide

  12. Variants
    12
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    01011001101
    Variants
    Variants
    Select
    Variants




    Based on Genetic Programming
    Automated Evasion Approach

    View Slide

  13. Variants
    13
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    01011001101
    Variants
    Variants
    Select
    Variants




    Based on Genetic Programming
    Automated Evasion Approach
    /Catalog /Pages
    0
    /JavaScript
    eval(‘…’);
    /Root
    Modified
    Parser
    Extract Me If You Can:
    Abusing PDF Parsers in Malware Detectors
    Curtis Carmony,et al.

    View Slide

  14. Variants
    14
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    01011001101
    Variants
    Variants
    Select
    Variants




    Based on Genetic Programming
    Automated Evasion Approach
    /Catalog /Pages
    0
    /JavaScript
    eval(‘…’);
    /Root
    Mutation
    Variants From
    Benign
    Insert / Replace / Delete

    View Slide

  15. Variants
    15
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    01011001101
    Variants
    Variants
    Select
    Variants




    Based on Genetic Programming
    Automated Evasion Approach
    /Catalog /Pages
    0
    /JavaScript
    eval(‘…’);
    /Root
    Mutation
    Variants From
    Benign
    128
    546
    0
    0
    Insert / Replace / Delete

    View Slide

  16. Variants
    16
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    01011001101
    Variants
    Variants
    Select
    Variants




    Based on Genetic Programming
    Automated Evasion Approach
    /Catalog /Pages
    0
    /JavaScript
    eval(‘…’);
    /Root
    Mutation
    Variants From
    Benign
    128
    546
    0
    0
    Insert / Replace / Delete

    View Slide

  17. Variants
    17
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    01011001101
    Variants
    Variants
    Select
    Variants




    Based on Genetic Programming
    Automated Evasion Approach
    /Catalog /Pages
    0
    /JavaScript
    eval(‘…’);
    /Root
    Mutation
    Variants From
    Benign
    128
    546
    0
    0
    128
    0
    Insert / Replace / Delete

    View Slide

  18. Variants
    18
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    01011001101
    Variants
    Variants
    Select
    Variants




    Based on Genetic Programming
    Automated Evasion Approach
    /Catalog /Pages
    0
    /JavaScript
    eval(‘…’);
    /Root
    Mutation
    Variants From
    Benign
    128
    0
    Insert / Replace / Delete

    View Slide

  19. Variants
    19
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    01011001101
    Variants
    Variants
    Select
    Variants




    Based on Genetic Programming
    Automated Evasion Approach
    /Catalog /Pages
    0
    /JavaScript
    eval(‘…’);
    /Root
    Mutation
    Variants From
    Benign
    128
    0
    Insert / Replace / Delete

    View Slide

  20. Variants
    20
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    01011001101
    Variants
    Variants
    Select
    Variants




    Based on Genetic Programming
    Automated Evasion Approach

    View Slide

  21. Variants
    21
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    01011001101
    Variants
    Variants
    Select
    Variants




    Based on Genetic Programming
    Automated Evasion Approach
    Fitness Function
    Oracle
    Target Classifier
    f(x)
    Malicious?
    Score
    Fitness Score
    Variants

    View Slide

  22. Variants
    22
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    01011001101
    Variants
    Variants
    Select
    Variants




    Based on Genetic Programming
    Automated Evasion Approach
    Fitness Function
    Oracle
    Target Classifier
    f(x)
    Malicious?
    Score
    Fitness Score
    Variants
    Malicious
    Benign

    View Slide

  23. Variants
    23
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    01011001101
    Variants
    Variants
    Select
    Variants




    Based on Genetic Programming
    Automated Evasion Approach

    View Slide

  24. Results: Evaded PDFrate 100%
    24
    Original Malware
    Seeds

    View Slide

  25. Results: Evaded PDFrate 100%
    25
    Original Malware
    Seeds
    Evasive Variants

    View Slide

  26. Evaded PDFrate with Adjusted Threshold
    26
    Original Malware
    Seeds
    Evasive Variants
    Evasive Variants
    with lower threshold

    View Slide

  27. Results: Evaded Hidost 100%
    27
    Original Malware
    Seeds

    View Slide

  28. Results: Evaded Hidost 100%
    28
    Original Malware
    Seeds
    Evasive Variants

    View Slide

  29. 29
    Difficulty varies by seed
    Simple mutations often work
    Complex mutations sometimes
    needed.
    Difficulty varied by targets:
    PDFrate: 6 days to evade all
    Hidost: 2 days to evade all
    Results: Accumulated Evasion Rate

    View Slide

  30. Cross-Evasion Effects
    30
    PDF Malware
    Seeds
    Hidost
    Evasive
    PDF Malware
    (against Hidost)
    Automated Evasion
    PDFrate
    387/500 Evasive
    (77.4%)
    3/500 Evasive
    (0.6%)
    Gmail’s classifier is secure?

    View Slide

  31. Cross-Evasion Effects
    31
    PDF Malware
    Seeds
    Hidost
    Evasive
    PDF Malware
    (against Hidost)
    Automated Evasion
    PDFrate
    387/500 Evasive
    (77.4%)
    3/500 Evasive
    (0.6%)
    Gmail’s classifier is secure? different.

    View Slide

  32. Evading Gmail’s Classifier
    32
    Evasion rate on : 135/380 (35.5%)

    View Slide

  33. Evading Gmail’s Classifier
    33
    Evasion rate on : 179/380 (47.1%)

    View Slide

  34. Conclusion
    34
    Source Code: http://EvadeML.org
    Vs.
    Who will win this arm race?

    View Slide