Share
Embed
Transcript
39 slides
-
GVDJ #IDSECCONF2016 security goals ▸ security goals ▸ confidentiality of
information and resources ▸ integrity of information and resources ▸ availability of information and resources ▸ basic definitions ▸ threat: potential violation of a security goal ▸ security: protection from intentional threats ▸ attack: intentional violation of a security goal SOURCE: MACHINE LEARNING FOR COMPUTER SECURITY -
GVDJ #IDSECCONF2016 security mechanisms ▸ security policies and mechanisms ▸
policy: statement of what is and what is not allowed ▸ mechanism: method or tool enforcing a security policy ▸ security is a process, not a product! ▸ strategies for security mechanisms ▸ prevention of attacks, e.g. encryption ▸ detection of attacks, e.g. virus scanner ▸ analysis of attacks, e.g. forensic SOURCE: MACHINE LEARNING FOR COMPUTER SECURITY — https://www.tu-braunschweig.de/sec/teaching/ss16/mlsec -
GVDJ #IDSECCONF2016 prevention is a hard task ▸ continuous discovery
of vulnerabilities ▸ insecure software and hardware ▸ developers unawareness goto fail; goto fail; goto fail (february 2014) heartbleed (april 2014) shellshock (september 2014) SOURCE: MACHINE LEARNING FOR COMPUTER SECURITY — https://www.tu-braunschweig.de/sec/teaching/ss16/mlsec -
GVDJ #IDSECCONF2016 attacks against services ▸ numerous security breaches at
popular web services ▸ identities often include real names, addresses, emails, passwords, etc. ‘;--have i been pwned? 142 pwned websites 1,444,567,928 pwned accounts 39,842 pastes 31,108,929 paste accounts SOURCE: MACHINE LEARNING FOR COMPUTER SECURITY — https://www.tu-braunschweig.de/sec/teaching/ss16/mlsec -
GVDJ #IDSECCONF2016 imbalance of security cycle ▸ increasing imbalance of
security cycle ▸ increasing number of vulnerabilities ▸ high amount of novel attacks ▸ high diversity of malicious software ▸ bottleneck: human analyst in the loop ▸ manual discovery of vulnerabilities ▸ manual generation of attack signatures ▸ manual analysis of malicious software SOURCE: MACHINE LEARNING FOR COMPUTER SECURITY — https://www.tu-braunschweig.de/sec/teaching/ss16/mlsec -
GVDJ #IDSECCONF2016 conventional detection ▸ conventional attack detection using signatures
▸ ineffective against novel and unknown attacks ▸ inherent delay to availability of novel signatures ▸ analysis obstructed by polymorphism and obfuscation HEADER APPLICATION PAYLOAD ... IP TCP GET /scripts/ ..%c1%9c.. /system32/cmd.exe TCP ..%c1%9c.. NIMDA WORM SOURCE: MACHINE LEARNING FOR COMPUTER SECURITY — https://www.tu-braunschweig.de/sec/teaching/ss16/mlsec -
GVDJ #IDSECCONF2016 intelligent defence ▸ construction of intelligent security systems
▸ combining computer security and machine learning ▸ minimum human intervention on prevention, detection, and analysis ▸ challenge in practice ▸ effectivity, efficiency, and robustness ▸ transparency and controlability SOURCE: MACHINE LEARNING FOR COMPUTER SECURITY — https://www.tu-braunschweig.de/sec/teaching/ss16/mlsec -
attack mitigation issues supervised unsupervised rules driven (limited by experiences
and expertise) high rates undetectable attacks (false negatives) delayed response (between detection and prevention) statistical driven (improved detection of new attacks) substantial investigative efforts (false positives) alarm fatigue and distrust (reversion to supervised method) -
GVDJ #IDSECCONF2016 implementation challenges ▸ lack of data: limited or
no history of previous attacks (required by supervised learning model) ▸ evolving attacks: attackers constantly change their behaviours, making current models obsolete ▸ limited resources: relying on security analysts to investigate the attacks can be costly and time consuming -
GVDJ #IDSECCONF2016 components THREAT PREDICTION PLATFORM MODEL ANALYSTS PREDICTION FEATURE
RAW DATA ACTION EVENTS MODELLING CONTEXTUAL MODELLING -
GVDJ #IDSECCONF2016 components ▸ big data processing system: quantifying features
from raw data ▸ outlier detection system: learning a descriptive model using features from unsupervised learning process ▸ feedback mechanism and continuous learning: incorporating analyst input -
GVDJ #IDSECCONF2016 data characteristics 0.1 data sources ▸ common sources:
networking devices and applications log ▸ router, switch, firewall, ids, ips, and load balancer devices ▸ web, database, and micro services ▸ frontend and backend applications ▸ delivered in realtime from widely distributed systems -
GVDJ #IDSECCONF2016 data characteristics 0.2 data dimensions and unique entities
▸ volume of raw data: metrics (GB/TB) or number of lines (≥ tens of millions on a daily basis) ▸ specific to behavioural analytics: IP addresses, users, sessions, etc. 01010101010101001111010111010101 01010001100010010100010011110110 10100100010010010010001010111101 10100111101101001100011110101011 10101110011010111011011101100111 11100000101001100010000011101101 01100001000000011010111110111011 00111001110001000100010011100100 00111011111011110110100100100110 10001010001110111110001001001001 -
GVDJ #IDSECCONF2016 data characteristics 0.3 malicious activity prevalence ▸ under
normal circumstances, malicious activities are extremely rare (generally ≤ 0.1%) ▸ resulting extreme class imbalance in supervised learning ▸ increasing the difficulty of detection processes ▸ unknown and/or unreported attacks introduce noise into data ▸ attack vectors can take a wide variety of shapes -
GVDJ #IDSECCONF2016 big data analytics DAILY WEEKLY MONTHLY RAW DATA
AGGREGATED DATA JIM ✖ ✖ ✖ FEATURES IS NEW USER? LAST CHANGED PASSWORD LAST IP ADDRESS LAST SESSION LENGTH ..... ..... ..... ..... ..... NUMBER OF FAILED LOGIN JIM -
GVDJ #IDSECCONF2016 big data analytics 0.1 behavioural signatures ▸ quantifying
signatures (often comprises the series of attack steps) from raw data ▸ quantitative values can be defined by security analysts ▸ extracting features per-entity and per-time-segment basis -
GVDJ #IDSECCONF2016 big data analytics 0.2 design requirements ▸ capable
of analysing ≥ 10 millions entities in daily basis ▸ capable of updating and retrieving signatures of active entities, on demand and/or in realtime -
GVDJ #IDSECCONF2016 big data analytics 0.3.1 process: activity tracking ▸
absorbing the log stream: identifying the entities and updating corresponding records ▸ in short temporal window: 30 minutes, 1 hour, 12 hours, or 24 hours. ▸ focus on efficient retrieval for feature computation -
GVDJ #IDSECCONF2016 big data analytics 0.3.2 process: activity aggregation ▸
computing behavioural features over an interval of time ▸ retrieving all activity records within given interval ▸ aggregating smaller time unit (minutes, hours, days, weeks) as the feature demands -
GVDJ #IDSECCONF2016 outlier detection ▸ matrix decomposition-based outlier analysis ▸
replicator neural networks ▸ density-based outlier analysis ▸ score interpretation ▸ transforming score to probabilities ▸ detection ensembles MATRIX DECOMPOSITION REPLICATOR NEURAL NETWORKS -
GVDJ #IDSECCONF2016 continuous learning ▸ overcomes limited analyst bandwidth ▸
overcomes weaknesses of unsupervised learning ▸ actively adapts and synthesises new models PREDICT ACT TRAIN -
GVDJ #IDSECCONF2016 example: open network insight leveraging insights from flow
and packet analysis -
GVDJ #IDSECCONF2016 summary ▸ current problems of security ▸ automatisation
of attacks ▸ massive amount of novel malicious code ▸ defences involving manual actions (often ineffective) ▸ machine learning in security ▸ adaptive defences using learning algorithms ▸ automatic detection and analysis of threats -
GVDJ #IDSECCONF2016 references ▸ machine learning for computer security by
konrad rieck, fabian yamaguchi, and alwin maier (institute of system security, tu-braunschweig) https://www.tu-braunschweig.de/sec/teaching/ss16/mlsec ▸ 360º unsupervised anomaly-based intrusion detection by stefano zanero https://www.blackhat.com/presentations/bh-dc-07/Zanero/Presentation/bh-dc-07-Zanero.pdf ▸ mlsec project http://www.mlsecproject.org/ ▸ redefining infosec by combining ai and human intuition https://www.patternex.com/redefining-infosec-by-combining-ai-and-human-intuition-wp