Improving Host-Based Computer Security Using Secure Active Monitoring and Memory Analysis

(PhD Thesis Defense)

Thirty years ago, research in designing operating systems to defeat malicious software was very popular. The primary technique was to design and implement a small security kernel that could provide security assurances to the rest of the system. However, as operating systems grew in size throughout the 1980's and 1990's, research into security kernels slowly waned. From a security perspective, the story was bleak. Providing security to one of these large operating systems typically required running software within that operating system. This weak security foundation made it relatively easy for attackers to subvert the entire system without detection.

The research presented in this thesis aims to reimagine how we design and deploy computer systems. We show that through careful use of virtualization technology, one can effectively isolate the security critical components in a system from malicious software. Furthermore, we can control this isolation to allow the security software a complete view to monitor the running system. This view includes all of the necessary information for implementing useful security applications including the system memory, storage, hardware events, and network traffic. In addition, we show how to perform both passive and active monitoring securely, using this new system architecture.

Security applications must be redesigned to work within this new monitoring architecture. The data acquired through our monitoring is typically very low-level and difficult to use directly. In this thesis, we describe work that helps bridge this semantic gap by locating data structures within the memory of a running virtual machine. We also describe work that shows a useful and novel security framework made possible through this new monitoring architecture. This framework correlates human interaction with the system to distinguish legitimate and malicious outgoing network traffic.

Transcript

1. Improving Host-Based Computer Security Using Secure Active Monitoring and Memory Analysis Thesis Defense Bryan D. Payne School of Computer Science Georgia Institute of Technology 1 Monday, May 24, 2010
2. Hardware / Firmware Operating System Users / Processes 2 Monday, May 24, 2010
3. Hardware / Firmware Operating System Users / Processes 3 Monday, May 24, 2010
4. Hardware / Firmware Operating System Users / Processes 4 Monday, May 24, 2010
5. Hardware / Firmware Operating System Users / Processes 5 Monday, May 24, 2010
6. Virtual Machine Introspection (VMI) Security VM User VM Hypervisor Hardware 6 Monday, May 24, 2010
7. Thesis Statement This thesis investigates a practical approach to enabling simple, flexible, and comprehensive active monitoring and memory analysis techniques for security software in a virtualized environment. 7 Monday, May 24, 2010
8. Problem Decomposition Secure Access - Passive - Active Memory Analysis - Semantic gap Applications - Feasibility - Performance Hypervisor User VM Security VM Hooks User Processes ... Memory Protector Virtual Machine Introspection Network Traffic Trampoline Hardware Events Hook Events Security Application Memory Analysis Mouse / Keyboard Network Disk Architecture enables secure active monitoring of virtual machines. 8 Monday, May 24, 2010
9. Problem Decomposition Secure Access - Passive - Active Memory Analysis - Semantic gap Applications - Feasibility - Performance Memory analysis techniques to locate data structures across software versions. 9 Monday, May 24, 2010
10. Problem Decomposition Secure Access - Passive - Active Memory Analysis - Semantic gap Applications - Feasibility - Performance Gyrus is the primary focus of todayʼs presentation. Anti-virus Linking user intent to security policy 10 Monday, May 24, 2010
11. The Gyrus Framework 11 Monday, May 24, 2010
12. Using Hardware Events Hi 㱺 e content (Hi) = f (content (e)) [1] [2] User Application h1 = Click h2 = Key(A) h3 = Key(D) h4 = Click h5 = Key(S) H Hi = {h2, h3, h5} content(Hi) = 'ADS' e1 = HTTP GET e2 = EMAIL 'ADS' e3 = HTTP GET content(e2) = 'ADS' Security Monitor E 12 Monday, May 24, 2010
13. Gyrus Framework Hypervisor Security Virtual Machine User VM Network-Based User Application User Kernel User Kernel Transparent Network Redirection Mouse / Keyboard Network Disk Transparent Proxy Enforcement Module Authorization Database User VM Device Model H/W Event 1 2 3 4 5 6 7 Authorization Definition Event Testing — Authorization Creation — Enforcement 1,2 3,4 5,6,7 13 Monday, May 24, 2010
14. Hardware Event Interposition Memory Access Screen Scraping Supporting API For App Modules 14 Monday, May 24, 2010
15. 15 Monday, May 24, 2010
16. 16 Monday, May 24, 2010
17. 17 Monday, May 24, 2010
18. Application Support Email Client Web Browser 18 Monday, May 24, 2010
19. Email Support In Gyrus 19 Monday, May 24, 2010
20. Goal: Stop Spambots Key Insight... Spambots cannot press a key on your keyboard (or use your mouse)! 20 Monday, May 24, 2010
21. Outlook Express Module Design User VM Outlook Express Email Client User Kernel comctl32.dll win32k !"#$%&#'()$**+,"-'.'/012*34& !56"&,77*&#*5-' '89:;"<'////1:=/ !"#$%&#'()$**+,"-'.'/0=34:>=2 !"#$%&#'(?9*@*7#-'.'/0=3A5B&/' '89:;"<'////1/C1' 'D8EF)G !"#$%&#'()$**+,"-'.'/0=/:H53& !56"&,77*&#*5-' '89:;"<'//////:> !"#$%&#'()$**+,"-'.'/0=3A*>5/ !"#$%&#'(?9*@*7#-'.'/0=3A=H2/' '89:;"<'/////&H1' 'DIEJKG !"#$%&#'()$**+,"-'.'/0=/:H5A2 !56"&,77*&#*5-' '89:;"<'/////4:2 !"#$%&#'()$**+,"-'.'/0=/:H5C> !56"&,77*&#*5-' '89:;"<'/////43> !"#$%&#'()$**+,"-'.'/0=3A*>*/ !"#$%&#'(?9*@*7#-'.'/0=3A=H2/' '89:;"<'//////4=' 'DLIEJKG !"#$%&#'()$**+,"-'.'/0=34:452 !"#$%&#'(?9*@*7#-'.'/0=34&3H/' '89:;"<'/////=H=' 'DLJMNG !"#$%&#'()$**+,"-'.'/0=34:C1/ !"#$%&#'(?9*@*7#-'.'/0=34&::/' '89:;"<'//////4=' 'DLJMNG !"#$%&#'()$**+,"-'.'/0=34:H&/ !"#$%&#'(?9*@*7#-'.'/0=34&32/' '89:;"<'/////=H=' 'DL8EF)G !"#$%&#'()$**+,"-'.'/0=34:C// !"#$%&#'(?9*@*7#-'.'/0=34&::/' '89:;"<'//////41' 'DJMNG !"#$%&#'()$**+,"-'.'/0=34:C2/ !"#$%&#'(?9*@*7#-'.'/0=34&:>/' '89:;"<'//////C=' 'DL8EF)G !"#$%&#'()$**+,"-'.'/0=3A*/1/ !"#$%&#'(?9*@*7#-'.'/0=3A=45/' '89:;"<'/////24=' 'DLO)PQG !"#$%&#'()$**+,"-'.'/0=3=&B:2 !"#$%&#'(?9*@*7#-'.'/05=4=2' '89:;"<'//////4=' 'DLG !"#$%&#'()$**+,"-'.'/0=34:4&2 !"#$%&#'(?9*@*7#-'.'/0=34&3H/' '89:;"<'//////H1' 'DJMNG !"#$%&#'()$**+,"-'.'/0=3A*/// !"#$%&#'(?9*@*7#-'.'/0=3A=45/' '89:;"<'/////=H1' 'DO)PQG !"#$%&#'()$**+,"-'.'/0=3=&B32 !"#$%&#'(?9*@*7#-'.'/05=4=2' '89:;"<'//////H1' 'DG !"#$%&#'()$**+,"-'.'/0=3A*>C/ !"#$%&#'(?9*@*7#-'.'/0=3A=4>/' '89:;"<'/////24=' 'DLO?RJG !"#$%&#'()$**+,"-'.'/0=3A*HS2 !"#$%&#'(?9*@*7#-'.'/0=3A1A/2' '89:;"<'/////2C=' 'DLT)KQ?G !"#$%&#'()$**+,"-'.'/0=34:>A2 !"#$%&#'(?9*@*7#-'.'/0=3A5B&/' '89:;"<'//////H=' 'DL8EF)G !"#$%&#'()$**+,"-'.'/0=3A*>// !"#$%&#'(?9*@*7#-'.'/0=3A===/' '89:;"<'/////=4=' 'DLP?)RG !"#$%&#'()$**+,"-'.'/0=3A*AS/ !"#$%&#'(?9*@*7#-'.'/0=3A===/' '89:;"<'//////C1' 'DP?)RG !"#$%&#'()$**+,"-'.'/0=3A*H*2 !"#$%&#'(?9*@*7#-'.'/0=3A1A/2' '89:;"<'/////4C1' 'DT)KQ?G !"#$%&#'()$**+,"-'.'/0=3A*>32 !"#$%&#'(?9*@*7#-'.'/0=3A=HH/' '89:;"<'/////=H1' 'D)M)Q?G !"#$%&#'()$**+,"-'.'/0=3A*>4/ !"#$%&#'(?9*@*7#-'.'/0=3A=4>/' '89:;"<'//////C1' 'DO?RJG !"#$%&#'()$**+,"-'.'/0=3A*>:2 !"#$%&#'(?9*@*7#-'.'/0=3A=HH/' '89:;"<'/////>C=' 'DL)M)Q?G !"#$%&#'()$**+,"-'.'/0=34:>5/ !"#$%&#'(?9*@*7#-'.'/0=3A542/' '89:;"<'//////41' 'DJMNG !"#$%&#'()$**+,"-'.'/0=/:H5>& !56"&,77*&#*5-' '89:;"<'//////B2 !"#$%&#'()$**+,"-'.'/0=34:>*/ !"#$%&#'(?9*@*7#-'.'/0=3A542/' '89:;"<'//////4=' 'DLJMNG !"#$%&#'()$**+,"-'.'/0=34:HB/ !"#$%&#'(?9*@*7#-'.'/0=34&32/' '89:;"<'//////C1' 'D8EF)G !"#$%&#'()$**+,"-'.'/0=34:CC/ !"#$%&#'(?9*@*7#-'.'/0=34&:>/' '89:;"<'//////41' 'D8EF)G mshtml.dll Event Testing Module - Mouse click on email send button Authorization Creation Module - Email from memory & screen - Reconstruct network output Enforcement Module - Validate outgoing emails 21 Monday, May 24, 2010
22. Event Testing Module Mouse click at x=520, y=430 Determine if coordinates map to the “Send Email” button in Outlook Express x=520 y=430 22 Monday, May 24, 2010
23. Authorization Creation Module Authorization contains email recipients, subject, and message body. 23 Monday, May 24, 2010
24. Enforcement Module Validation script compares outgoing emails to authorizations in the database. ProxSMTP Outlook Express Email Client Validate Email Script SMTP Session Email Server Authorization Database Security Virtual Machine User VM SQL DB Content Filter SMTP Proxy 24 Monday, May 24, 2010
25. Security Evaluation • Not vulnerable to current generation malware - Stopped Spammer:Win32/Cutwail.gen!B • Not vulnerable to future malware - Within the bounds of our security assumptions - Breaking into the hypervisor can defeat Gyrus • No false positives (all legit emails passed) - Assuming low OCR edit distance threshold • No false negatives (stopped all spam) 25 Monday, May 24, 2010
26. Performance Evaluation User Interaction Mean StdDev Click, OE not running 3.8 ms 0.42 ms Click, no OE compose window 23.7 ms 0.48 ms Click in compose edit area 28.0 ms 0.00 ms Click in compose tool bar 109.9 ms 0.32 ms Click on send button 2067.6 ms 73.17 ms • Most delays are not perceptible to users • “Click on send button” is too slow... 26 Monday, May 24, 2010
27. Towards Improving Performance Baseline A B C D 0 1000 2000 3000 4000 5000 Time ￿milliseconds￿ Baseline Memory Snapshot Screen Capture A: xm save -c <domid> B: XenAccess dump_memory.c C: XenAccess save to memory buffer D: Save only necessary pages Copy-on-write snapshots still a potential option 27 Monday, May 24, 2010
28. Web Browser Support In Gyrus 28 Monday, May 24, 2010
29. Goal: Stop Clickbots Key Insight... Clickbots cannot press a key on your keyboard (or use your mouse)! 29 Monday, May 24, 2010
30. Internet Explorer Module Design Squid Internet Explorer Web Browser Greasyspoon (ICAP Server) Gyrus ICAP Request Mod Gyrus ICAP Response Mod Link Extraction Script HTTP Request Web Server Authorization Database HTTP Response Security Virtual Machine User VM SQL DB ICAP Web Proxy Server-side Browser EnvJS Event Testing Module - ENTER after typing URL in browser Authorization Creation Module - User clicks in browser - Parse HTTP response packets Enforcement Module - Validate HTTP requests 30 Monday, May 24, 2010
31. Event Testing Module Initial authorization occurs when ENTER key is pressed inside the location bar. 31 Monday, May 24, 2010
32. Authorization Creation and Enforcement Modules Receive Network Data Authorization Database Reject Send Data To Network Dynamic Authorization Creation Enforcement Receive Hardware Event ENTER Key Pressed? Click Within Webpage? Extract URLs From Data Keyboard Mouse HTTP Response Extract URL From Location Bar Yes Yes Insert URL as Auto Link Increment Token Counter Insert URLs and URL Types URL an Auto Link? HTTP Request Yes URL a Token Link? No Available Tokens? Yes Yes No DB Query DB Query DB Query Decrement Token Counter Automatically create authorizations based on user actions and HTTP Response packets 32 Monday, May 24, 2010
33. Dynamic Content <script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> var pageTracker = _gat._getTracker("UA-4479582-1"); pageTracker._initData(); pageTracker._trackPageview(); </script> http://www.google-analytics.com/ga.js http://www.google-analytics.com/__utm.gif? utmwv=4.6.5&utmn=708793014&utmhn=tutorial.getwindmill.com&ut mcs=utf-8&utmsr=1024x768&utmsc=24-bit&utmul=en- us&utmje=1&utmfl=10.0%20r45&utmcn=1&utmdt=FC2%20-%20Free %20Website%20Access%20Analysis%20Blog%20Rental%20Server %20SEO%20Countermeasures%20etc.%20-&utmhid=2118468594&utmr=- &utmp=%2F&utmac=UA-7509326-1&utmcc=__utma %3D80279954.585291908.1270848936.1270848936.1270848936.1%3B %2B__utmz%3D80279954.1270848936.1.1.utmcsr%3D(direct) %7Cutmccn%3D(direct)%7Cutmcmd%3D(none)%3B 33 Monday, May 24, 2010
34. Dynamic Content • JavaScript Challenges - Dynamic DOM manipulation - OnClick handlers - Timers - Inline functions - Imported JS files • Many other dynamic web technologies - Java, Flash, ActiveX, VBScript, Shockwave, etc 34 Monday, May 24, 2010
35. Security Evaluation • Not vulnerable to current generation malware - Stopped AdClicker-AD, DR/Click.HSP.A.2, and AdClicker-BY • Not vulnerable to future malware - Within the bounds of our security assumptions - Breaking into the hypervisor can defeat Gyrus • Lots of false positives - Rejected 81.9% of legit HTTP Requests when loading the Alexa top 1000 web sites • No false negatives (stopped all clickbots) 35 Monday, May 24, 2010
36. Performance Evaluation User Interaction Mean StdDev ENTER while IE not running 2.0 ms 0.00 ms ENTER while focus on search bar 2.5 ms 0.53 ms ENTER while focus on location bar 456.7 ms 10.37 ms Click while IE not running 3.8 ms 0.42 ms Click in IE window (not web page) 28.1 ms 0.32 ms Click on web page 35.3 0.95 ms • Most delays are not perceptible to users • ENTER while focus on location bar doesnʼt feel too slow because user is waiting for web page to load at same time 36 Monday, May 24, 2010
37. Towards Better Web Coverage Option 1: Complete Web Browser Option 2: Improve Env-JS • Black box approach • Simulate user behavior • Good coverage • Security concerns • Performance concerns • Walk DOM tree • Reflection to exec code • Reasonable performance • Limited coverage • Immature code base 37 Monday, May 24, 2010
38. Conclusions 38 Monday, May 24, 2010
39. • Turret Monitoring Framework - XenAccess VMI Library - Secure Active Monitoring - Design principles for external monitoring - Integrated monitoring framework • Memory Analysis - Use of HMMs to model data structures - Integrating machine learning into memory analysis • Gyrus Security Application Framework - Framework and supporting APIs - User interaction based security for email and web Major Contributions 39 Monday, May 24, 2010
40. Future Work • Semantic gap problem • Simplify VMI programming • Better suited hypervisor • Audit-aware software • Enhance and generalize VMI libraries • Simplify Gyrus application support 40 Monday, May 24, 2010
41. Improving Host-Based Computer Security Using Secure Active Monitoring and Memory Analysis Thesis Defense Bryan D. Payne School of Computer Science Georgia Institute of Technology 41 Monday, May 24, 2010