Unearthing Stealthy Program Attacks Buried in Extremely Long Execution Paths

Slides for my CCS '15 paper (same title).

1. Unearthing Stealthy Program Attacks Buried in Extremely Long Execution Paths Xiaokui Shu, Danfeng (Daphne) Yao, Naren Ramakrishnan Department of Computer Science Virginia Tech Blacksburg, Virginia CCS 2015
2. Anomaly Detection Anomalies Our Attack Model Program Anomaly Detection (a.k.a., host-based anomaly detection [Denning 1987]) Introduction 2
3. … sys_ioctl() sys_open() sys_read() sys_setpgid() sys_setsid() sys_fork() … Time n-gram [Forrest 1996] FSA [Sekar 2001, Wagner 2001] Illegal control flow Xj+1 Xj … Xi+1 Xi … X1 X0 Yj+1 Yj … Yi+1 Yi … Y1 Y0 PDA [Feng 2003, Feng 2004, Giffin 2004] Hijacked return sites x = 1 y = x+1 y = x*2 w = x*y Data analysis [Giffin 2006, Bhatkar 2006] Malicious arguments Anomaly Detection Anomalies Our Attack Model Introduction 3
4. An anomaly made up of normal program execution fragments. Existing anomaly detection that focus on local region analysis cannot detect this anomaly. Anomaly Detection Anomalies Our Attack Model Introduction 4
5. a b d a c e a c b e a c c e c f f d c e c c f e d a b d a c c f e d Attack examples: • Workflow violation attack • Non-control data attack • Fragment-based mimicry attack a b b c d e a b b c d . . . b c d e Montage Anomaly Frequency Anomaly Attack examples: • Denial of Service attack • Click fraud • Heap spray Anomaly Detection Anomalies Our Attack Model Introduction N1: N2: N3: A: N: A: 5
6. A sshd flag variable overwritten attack [Chen 2005] void do_authentication(...) { int authenticated = 0; while (!authenticated) { if (auth_password(...)) { memset(...); xfree(...); log_msg(...); authenticated = 1; break; } memset(...); xfree(...); debug(...); break; ... } if (authenticated) break; ... 218 call instructions In between Normal 1 Normal 2 Anomalous No existing solution models general event correlation at a large scale. Anomaly Detection Anomalies Our Attack Model Introduction 6
7. Correlate events far-away 218 function calls away . . . a b . . . d e f . . . Exponential Explosion If we have 100 function calls, the detection space size: 100218 Exponential Explosion Diverse Normal Behaviors Challenges 7
8. Diverse Normal Behaviors Function call 16 Figure: distinct behaviors of libpcre Similar to N1 Similar to N2 Similar to N3 Montage anomaly 0 0.5 1 0 0.05 0.1 Detection rate False positive rate Exponential Explosion Diverse Normal Behaviors Challenges 8
9. … -> bar -> main -> foo -> … 0 24 0 0 0 0 30 0 2 6 89 1 0 0 0 0 main foo bar qux Our solution to exponential explosion handling profile into … Sequence profiles are large. qux bar main foo … Only need the correlation among events and their frequencies. An infinitely long trace: slice into Behavior instance Behavior instance Behavior instance Long trace segments Behavior instance 0 24 0 0 0 0 30 0 2 6 89 1 0 0 0 0 F T F F F F T F T T T T F F F F Matrix Profile Two-Stage Data Mining Clustering Our Approach Event co-occurrence matrix Transition frequency matrix 9
10. Our solution to diverse normal behavior modeling Two-stage data mining approach • Inter-cluster modeling Event co-occurrence analysis • Intra-cluster modeling Occurrence frequency analysis • Inter-cluster detection Against montage anomalies • Intra-cluster detection Against frequency anomalies Montage anomaly Frequency anomaly Matrix Profile Two-Stage Data Mining Clustering Our Approach 10
11. Behavior instance By similar sets of co-occurred events How do we cluster diverse normal behaviors? Behavior instances that share similar event co-occurrence matrices (long trace segment) T T T T T T 0 24 0 0 0 0 30 0 2 6 89 1 0 0 0 0 T T T T T T T T T T T T T T Matrix Profile Two-Stage Data Mining Clustering Our Approach 11
12. What programs did we study? sshd libpcre sendmail What events did we trace? function call trace (Pintool) system call trace (SystemTap script) What research questions to answer? • Detection capability • Accuracy • Performance Setup Attack Detection Accuracy Performance Evaluation 12
13. Normal: authenticated … do_auth > xfree do_auth > log_msg do_auth > packet_start … pwrite > buffer_len do_auth > do_auth … Anomalous: attack … do_auth > debug do_auth > xfree do_auth > packet_start … pwrite > buffer_len do_auth > do_auth … Montage Anomaly detected at inter-cluster detection. sshd function call trace (Pintool) sshd flag variable overwritten attack Normal: wrong password … do_auth > debug do_auth > xfree do_auth > packet_start … pwrite > buffer_len do_auth > pread … Setup Attack Detection Accuracy Performance Evaluation 13
14. 0 0.5 1 0 0.05 0.1 Detection rate False positive rate 0 0.5 1 0 0.05 0.1 Detection rate False positive rate Detection against synthetic montage anomalies (libpcre) Baseline: One-class SVM with kernel function Our two-stage detection approach We also test our approach against synthetic incomplete path, high-frequency, and low-frequency anomalies. Setup Attack Detection Accuracy Performance Evaluation 14
15. Overall Overhead Tracing Overhead Analysis Overhead Over 100% overhead for function call tracing with our Pintool Similar to Trin-Trin [Jalan 2012] • Single behavior instance: 0.1~1.3 ms • Less time used if only first stage is performed • Positively correlated with the size of the matrix Setup Attack Detection Accuracy Performance Evaluation 15
16. Summary Evaluation sshd libpcre sendmail 4800 normal profiles 34511 events avg. 11027 normal profiles 44893 events avg. 6579 normal profiles 1134 events avg. Flag variable overwritten attack ReDoS 3 malicious patterns 8-23 strings to match Directory harvest attack w/ probing batch sizes: 8 to 400 emails • Montage anomaly • Incomplete path anomaly • High-frequency anomaly • Low-frequency anomaly Synthetic anomalies Analysis Overhead 16
17. 0 24 0 0 0 0 30 0 2 6 89 1 0 0 0 0 main foo bar qux Flow-insensitive Data-insensitive … do_authentication > debug do_authentication > xfree do_authentication > packet_start packet_start > buffer_clear packet_start > buffer_append … packet_write_poll > buffer_ptr packet_write_poll > buffer_consume packet_write_wait > packet_to_write packet_to_write > buffer_len do_authentication > do_authenticated … We sacrifice the order of events for feasible long trace event correlation modeling. Limitations 17
18. … sys_ioctl() sys_open() sys_read() sys_setpgid() sys_setsid() sys_fork() … Time n-gram [Forrest 1996] FSA [Sekar 2001, Wagner 2001] Xj+1 Xj … Xi+1 Xi … X1 X0 Yj+1 Yj … Yi+1 Yi … Y1 Y0 PDA [Feng 2003, Feng 2004, Giffin 2004] x = 1 y = x+1 y = x*2 w = x*y Data analysis [Giffin 2006, Bhatkar 2006] Machine learning [Lee 1998, Mutz 2006, Kui 2015] Static Program Analysis Dynamic Program Analysis Hybrid detection [Gao 2004, Liu 2005] + We proposed a uniform framework presenting any PAD [RAID 2015]. Related Work 18
19. Conclusion • An attack made up of normal program execution fragments • Montage anomaly • Frequency anomaly • Two challenges in program anomaly detection • Exponential Explosion • Diverse Normal Behaviors • We propose a two-stage data mining approach • Inter-cluster modeling/detection • Intra-cluster modeling/detection Future Work • Basic block correlation • Incidence response system Conclusion This work has been supported by grants ONR N00014-13-1-0016 and ARO YIP W911NF-14-1-0535. 19
20. THANK YOU! Xiaokui is seeking a postdoc/researcher position. subx@cs.vt.edu http://xshu.net