Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Classifiers Under Attack

David Evans
February 01, 2017

Classifiers Under Attack

Talk at USENIX Enigma 2017
1 February 2017
Oakland, CA

https://evadeML.org

David Evans

February 01, 2017
Tweet

More Decks by David Evans

Other Decks in Science

Transcript

  1. Classifiers
    Under
    Attack
    David Evans
    work with Weilin Xu and Yanjun Qi
    University of Virginia
    [email protected]
    evadeML.org
    1 February 2017

    View Slide

  2. Machine Learning Does
    Amazing Things
    2
    FeatureSmith

    View Slide

  3. … and can solve all Security Problems!
    Fake
    Spam
    IDS
    Malware
    Fake Accounts

    View Slide

  4. Labelled
    Training Data
    ML
    Algorithm
    Feature
    Extraction
    Vectors
    Deployment
    Malicious / Benign
    Operational Data
    Trained Classifier
    Training
    (Supervised Learning)
    Assumption: Training Data is Representative

    View Slide

  5. Training Data Deployment
    Training
    Adversaries Don’t Cooperate
    Assumption: Training Data is Representative

    View Slide

  6. Training Data Deployment
    Training
    Adversaries Don’t Cooperate
    Assumption: Training Data is Representative
    Poisoning

    View Slide

  7. Adversaries Don’t Cooperate
    Assumption: Training Data is Representative
    Training Data Deployment
    Training
    Evading

    View Slide

  8. Focus: Evasion Attacks
    Goal: Automatically simulate adaptive
    adversary against generic classifier
    Purpose: Understand classifier robustness
    Build better classifiers (or give up)

    View Slide

  9. Case study:
    Evading PDF Malware Classifiers

    View Slide

  10. View Slide

  11. 0
    50
    100
    150
    200
    250
    2006
    2007
    2008
    2009
    2010
    2011
    2012
    2013
    2014
    2015
    2016
    2017
    Vulnerabilities reported in
    Adobe Acrobat Reader
    Source: http://www.cvedetails.com/vulnerability-list.php?vendor_id=53&product_id=921
    33 already in Jan 2017!

    View Slide

  12. PDF Malware Classifiers
    PDFrate
    [ACSA 2012]
    Hidost16
    [JIS 2016]
    Hidost13
    [NDSS 2013]
    Random Forest Random Forest
    Support Vector Machine
    Classifier Accuracy
    0.9976 0.9996 0.9996
    * Mimicus [Oakland 2014], an open source reimplementation of PDFrate.

    View Slide

  13. Random Forest
    x
    y w
    0 1 z
    1
    0 1
    r
    q
    0
    z
    0
    0
    y
    0 1
    Generate many
    random decision trees
    Train independently
    Select best trees
    Vote on result

    View Slide

  14. PDF Malware Classifiers
    Random Forest Random Forest
    Support Vector Machine
    Features
    Object counts,
    lengths,
    positions, …
    Object structural paths
    Very robust against “strongest
    conceivable mimicry attack”.
    Automated Features
    Manual Features
    PDFrate
    [ACSA 2012]
    Hidost16
    [JIS 2016]
    Hidost13
    [NDSS 2013]

    View Slide

  15. Automatically Evading Classifiers

    View Slide

  16. Variants
    Automated Classifier Evasion
    Using Genetic Programming
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    Variants
    Variants
    Select
    Variants




    Found
    Evasive?

    View Slide

  17. Variants
    Goal: Find Evasive Variant
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    Variants
    Variants
    Select
    Variants




    Found
    Evasive?
    Benign Simulated attacker’s
    goal: find sample
    classified as benign,
    that exhibits
    malicious behavior.

    View Slide

  18. Variants
    Start with Malicious Seed
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    Variants
    Variants
    Select
    Variants




    Found
    Evasive?
    Benign

    View Slide

  19. PDF Structure

    View Slide

  20. Variants
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    Variants
    Select
    Variants




    Found
    Evasive?
    Modified
    Parser
    0
    /JavaScript
    eval(‘…’);
    /Root
    /Catalog
    /Pages
    “robust” version
    of pdfrw

    View Slide

  21. Variants
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    Variants
    Variants
    Select
    Variants




    Found
    Evasive?
    Generating Variants

    View Slide

  22. Variants
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    Variants
    Variants
    Select
    Variants




    Found
    Evasive?
    0
    /JavaScript
    eval(‘…’);
    /Root
    /Catalog
    /Pages
    Generating Variants
    Select random node

    View Slide

  23. Variants
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    Variants
    Variants
    Select
    Variants




    Found
    Evasive?
    0
    /JavaScript
    eval(‘…’);
    /Root
    /Catalog
    /Pages
    Select random node
    Random transform: delete, insert, replace
    Generating Variants

    View Slide

  24. Variants
    Generating Variants
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    Variants
    Variants
    Select
    Variants




    Found
    Evasive?
    0
    /JavaScript
    eval(‘…’);
    /Root
    /Catalog
    /Pages
    Nodes from
    Benign PDFs
    128
    546
    7
    63
    Random transform: delete, insert, replace
    128
    Select random node

    View Slide

  25. Variants
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    Variants
    Variants
    Select
    Variants




    Found
    Evasive?
    Selecting Promising Variants

    View Slide

  26. Variants
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    Variants
    Variants
    Select
    Variants




    Found
    Evasive?
    Selecting Promising Variants
    Clone
    Generated Variants
    Clone
    Variants Fitness Function
    Candidate Variant
    ($%&'()
    , '(&++
    )
    Score
    Malicious
    Benign PDFs
    Malicious PDF
    Variants
    Benign PDFs
    Malicious PDF
    Variants
    Oracle
    Variant 0
    /JavaScript
    eval(‘…’);
    /Root
    /Catalog
    /Pages
    128
    Oracle
    Target Classifier

    View Slide

  27. Oracle
    Execute candidate in
    vulnerable Adobe Reader in
    virtual environment
    Behavioral signature:
    malicious if signature matches
    https://github.com/cuckoosandbox
    Simulated network: INetSim
    Cuckoo
    HTTP_URL + HOST
    extracted from API traces
    Advantage: we know the target malware behavior

    View Slide

  28. Variants
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    Variants
    Variants
    Select
    Variants




    Found
    Evasive?
    Selecting Promising Variants
    Clone
    Generated Variants
    Clone
    Variants Fitness Function
    Candidate Variant
    ($%&'()
    , '(&++
    )
    Score
    Malicious
    Benign PDFs
    Malicious PDF
    Variants
    Benign PDFs
    Malicious PDF
    Variants
    Oracle
    Variant 0
    /JavaScript
    eval(‘…’);
    /Root
    /Catalog
    /Pages
    128
    Oracle
    Target Classifier

    View Slide

  29. Fitness Function
    Assumes lost malicious behavior will not be recovered
    = 0
    .5 − classifier_score if oracle = "malicious"
    −∞ otherwise
    classifier_score ≥ 0.5: labeled malicious

    View Slide

  30. Experimental Results

    View Slide

  31. Classifier Performance
    PDFrate Hidost
    Accuracy 0.9976 0.9996
    False Negative Rate 0.0000 0.0056
    Results on non-adversarial samples

    View Slide

  32. Classifier Performance
    PDFrate Hidost
    Accuracy 0.9976 0.9996
    False Negative Rate 0.0000 0.0056
    False Negative Rate
    against Adversary
    1.0000 1.0000

    View Slide

  33. 0
    100
    200
    300
    400
    500
    0 100 200 300
    Seeds Evaded
    (out of 500)
    PDFRate
    Number of Mutations
    Hidost

    View Slide

  34. 0
    100
    200
    300
    400
    500
    0 100 200 300
    Seeds Evaded
    (out of 500)
    PDFRate
    Hidost
    Number of Mutations
    Simple
    transformations
    often worked

    View Slide

  35. 0
    100
    200
    300
    400
    500
    0 100 200 300
    Seeds Evaded
    (out of 500)
    PDFRate
    Number of Mutations
    (insert, /Root/Pages/Kids,
    3:/Root/Pages/Kids/4/Kids/5/)
    Works on 162/500 seeds

    View Slide

  36. 0
    100
    200
    300
    400
    500
    0 100 200 300
    Seeds Evaded
    (out of 500)
    PDFRate
    Number of Mutations
    Hidost
    Some seeds
    required complex
    transformations

    View Slide

  37. Insert: Threads, ViewerPreferences/Direction, Metadata,
    Metadata/Length, Metadata/Subtype, Metadata/Type,
    OpenAction/Contents, OpenAction/Contents/Filter,
    OpenAction/Contents/Length, Pages/MediaBox
    Delete: AcroForm, Names/JavaSCript/Names/S,
    AcroForm/DR/Encoding/PDFDocEncoding,
    AcroForm/DR/Encoding/PDFDocEncoding/Differences,
    AcroForm/DR/Encoding/PDFDocEncoding/Type, Pages/Rotate,
    AcroForm/Fields, AcroForm/DA, Outlines/Type, Outlines,
    Outlines/Count, Pages/Resources/ProcSet, Pages/Resources
    85-step mutation trace evading Hidost
    Effective for 198/500 seeds

    View Slide

  38. 0 20 40 60 80 100 120
    Hidost
    PDFrate Oracle
    Execution Cost
    Hours to find all 500 variants on one desktop PC
    Oracle
    Mutation
    Classifier

    View Slide

  39. Possible Defenses

    View Slide

  40. Possible Defense:
    Adjust Threshold
    Charles Smutz, Angelos Stavrou. When a Tree Falls:
    Using Diversity in Ensemble Classifiers to Identify
    Evasion in Malware Detectors. NDSS 2016.

    View Slide

  41. Original Malicious Seeds
    Evading PDFrate
    Malicious Label Threshold

    View Slide

  42. Discovered Evasive Variants
    Adjust threshold?

    View Slide

  43. Adjust threshold?
    Variants found with threshold = 0.25
    Variants found with threshold = 0.50

    View Slide

  44. Hidost16
    Variants found with threshold = 0.25
    Variants found with threshold = 0.50

    View Slide

  45. Possible Defense:
    Retrain Classifier

    View Slide

  46. Labelled
    Training Data
    ML
    Algorithm
    Feature
    Extraction
    Vectors
    Deployment
    Malicious / Benign
    Operational Data
    Trained Classifier
    Training
    (Supervised Learning)
    Retraining Classifier

    View Slide

  47. Labelled
    Training Data
    ML
    Algorithm
    Feature
    Extraction
    Vectors
    Training
    (Supervised Learning)
    Clone
    EvadeML

    View Slide

  48. Labelled
    Training Data
    ML
    Algorithm
    Feature
    Extraction
    Vectors
    Training
    (Supervised Learning)
    Deployment
    Clone
    EvadeML

    View Slide

  49. 0
    100
    200
    300
    400
    500
    0 200 400 600 800
    Seeds Evaded (out of 500)
    Generations
    Hidost16 Original classifier:
    Takes 614 generations
    to evade all seeds

    View Slide

  50. 0
    100
    200
    300
    400
    500
    0 200 400 600 800
    Seeds Evaded (out of 500)
    Generations
    Hidost16
    HidostR1

    View Slide

  51. 0
    100
    200
    300
    400
    500
    0 200 400 600 800
    Seeds Evaded (out of 500)
    Generations
    Hidost16
    HidostR1
    HidostR2

    View Slide

  52. 0
    100
    200
    300
    400
    500
    0 200 400 600 800
    Seeds Evaded (out of 500)
    Generations
    Hidost16
    HidostR1
    HidostR2

    View Slide

  53. 0
    100
    200
    300
    400
    500
    0 200 400 600 800
    Seeds Evaded (out of 500)
    Generations
    Hidost16
    HidostR1 HidostR2
    Genome Contagio Benign
    Hidost16 0.00 0.00
    HidostR1 0.78 0.30
    HidostR2 0.85 0.53
    False Positive Rates

    View Slide

  54. 0
    100
    200
    300
    400
    500
    0 200 400 600 800
    Seeds Evaded (out of 500)
    Generations
    Hidost16
    HidostR1 HidostR2
    Genome Contagio Benign
    Hidost16 0.00 0.00
    HidostR1 0.78 0.30
    HidostR2 0.85 0.53
    False Positive Rates

    View Slide

  55. 0
    100
    200
    300
    400
    500
    0 500 1000 1500 2000
    Retrained using evasive
    variants and all benign samples
    available to adversary
    .11 Evasion Rate
    .07 False Positive
    Generations

    View Slide

  56. Possible Defense:
    Hide Classifier

    View Slide

  57. Variants
    Clone
    Benign PDFs
    Malicious PDF
    Mutation
    Variants
    Variants
    Select
    Variants




    Found
    Evasive?
    Hide Classifier
    “Security Through Obscurity”
    Clone
    Generated Variants
    Clone
    Variants Fitness Function
    Candidate Variant
    ($%&'()
    , '(&++
    )
    Score
    Malicious
    Benign PDFs
    Malicious PDF
    Variants
    Benign PDFs
    Malicious PDF
    Variants
    Oracle
    Variant 0
    /JavaScript
    eval(‘…’);
    /Root
    /Catalog
    /Pages
    128
    Oracle
    Target Classifier

    View Slide

  58. Cross-Evasion Effects
    PDF Malware
    Seeds
    Hidost 13
    Evasive
    PDF Malware
    (against PDFrate)
    Automated Evasion
    PDFrate
    2/500 Evasive
    (0.4% Success)
    Potentially Good News?

    View Slide

  59. Evasive
    PDF Malware
    (against PDFrate)
    Cross-Evasion Effects
    PDF Malware
    Seeds
    Hidost 13
    Automated Evasion
    PDFrate
    2/500 Evasive
    (0.4% Success)
    Evasive
    PDF Malware
    (against Hidost)
    387/500 Evasive
    (77.4% Success)

    View Slide

  60. 387/500 Evasive
    (77.4% Success)
    Cross-Evasion Effects
    PDF Malware
    Seeds
    Hidost 13
    Automated Evasion
    PDFrate
    Evasive
    PDF Malware
    (against Hidost)

    View Slide

  61. Cross-Evasion Effects
    PDF Malware
    Seeds
    Hidost 13
    Automated Evasion
    Evasive
    PDF Malware
    (against Hidost)
    6/500 Evasive
    (0.6% Success)

    View Slide

  62. Evading Gmail’s Classifier
    Evasion rate on Gmail: 179/380 (47.1%)
    for javascript in pdf.all_js:
    javascript.append_code("var enigma=1;“)
    if pdf.get_size() < 7050000:
    pdf.add_padding(7050000 – pdf.get_size())

    View Slide

  63. Conclusion

    View Slide

  64. Conclusion
    Domain Knowledge is Not Dead
    Trust Demands Understanding

    View Slide

  65. David Evans
    University of Virginia
    [email protected]
    EvadeML.org
    source code, papers
    Credits: Weilin Xu, Yanjun Qi

    View Slide