Improving Host-Based Computer Security Using Secure Active Monitoring and Memory Analysis Thesis Defense Bryan D. Payne School of Computer Science Georgia Institute of Technology 1 Monday, May 24, 2010 

Hardware / Firmware Operating System Users / Processes 2 Monday, May 24, 2010 

Hardware / Firmware Operating System Users / Processes 3 Monday, May 24, 2010 

Hardware / Firmware Operating System Users / Processes 4 Monday, May 24, 2010 

Hardware / Firmware Operating System Users / Processes 5 Monday, May 24, 2010 

Virtual Machine Introspection (VMI) Security VM User VM Hypervisor Hardware 6 Monday, May 24, 2010 

Thesis Statement This thesis investigates a practical approach to enabling simple, flexible, and comprehensive active monitoring and memory analysis techniques for security software in a virtualized environment. 7 Monday, May 24, 2010 

Problem Decomposition Secure Access - Passive - Active Memory Analysis - Semantic gap Applications - Feasibility - Performance Hypervisor User VM Security VM Hooks User Processes ... Memory Protector Virtual Machine Introspection Network Traffic Trampoline Hardware Events Hook Events Security Application Memory Analysis Mouse / Keyboard Network Disk Architecture enables secure active monitoring of virtual machines. 8 Monday, May 24, 2010 

Problem Decomposition Secure Access - Passive - Active Memory Analysis - Semantic gap Applications - Feasibility - Performance Memory analysis techniques to locate data structures across software versions. 9 Monday, May 24, 2010 

Problem Decomposition Secure Access - Passive - Active Memory Analysis - Semantic gap Applications - Feasibility - Performance Gyrus is the primary focus of todayʼs presentation. Anti-virus Linking user intent to security policy 10 Monday, May 24, 2010 

The Gyrus Framework 11 Monday, May 24, 2010 

Using Hardware Events Hi 㱺 e content (Hi) = f (content (e)) [1] [2] User Application h1 = Click h2 = Key(A) h3 = Key(D) h4 = Click h5 = Key(S) H Hi = {h2, h3, h5} content(Hi) = 'ADS' e1 = HTTP GET e2 = EMAIL 'ADS' e3 = HTTP GET content(e2) = 'ADS' Security Monitor E 12 Monday, May 24, 2010 

Gyrus Framework Hypervisor Security Virtual Machine User VM Network-Based User Application User Kernel User Kernel Transparent Network Redirection Mouse / Keyboard Network Disk Transparent Proxy Enforcement Module Authorization Database User VM Device Model H/W Event 1 2 3 4 5 6 7 Authorization Definition Event Testing — Authorization Creation — Enforcement 1,2 3,4 5,6,7 13 Monday, May 24, 2010 

Hardware Event Interposition Memory Access Screen Scraping Supporting API For App Modules 14 Monday, May 24, 2010 

15 Monday, May 24, 2010 

16 Monday, May 24, 2010 

17 Monday, May 24, 2010 

Application Support Email Client Web Browser 18 Monday, May 24, 2010 

Email Support In Gyrus 19 Monday, May 24, 2010 

Goal: Stop Spambots Key Insight... Spambots cannot press a key on your keyboard (or use your mouse)! 20 Monday, May 24, 2010 

Outlook Express Module Design User VM Outlook Express Email Client User Kernel comctl32.dll win32k !"#$%&#'()$**+,"-'.'/012*34& !56"&,77*&#*5-' '89:;"<'////1:=/ !"#$%&#'()$**+,"-'.'/0=34:>=2 !"#$%&#'(?9*@*7#-'.'/0=3A5B&/' '89:;"<'////1/C1' 'D8EF)G !"#$%&#'()$**+,"-'.'/0=/:H53& !56"&,77*&#*5-' '89:;"<'//////:> !"#$%&#'()$**+,"-'.'/0=3A*>5/ !"#$%&#'(?9*@*7#-'.'/0=3A=H2/' '89:;"<'/////&H1' 'DIEJKG !"#$%&#'()$**+,"-'.'/0=/:H5A2 !56"&,77*&#*5-' '89:;"<'/////4:2 !"#$%&#'()$**+,"-'.'/0=/:H5C> !56"&,77*&#*5-' '89:;"<'/////43> !"#$%&#'()$**+,"-'.'/0=3A*>*/ !"#$%&#'(?9*@*7#-'.'/0=3A=H2/' '89:;"<'//////4=' 'DLIEJKG !"#$%&#'()$**+,"-'.'/0=34:452 !"#$%&#'(?9*@*7#-'.'/0=34&3H/' '89:;"<'/////=H=' 'DLJMNG !"#$%&#'()$**+,"-'.'/0=34:C1/ !"#$%&#'(?9*@*7#-'.'/0=34&::/' '89:;"<'//////4=' 'DLJMNG !"#$%&#'()$**+,"-'.'/0=34:H&/ !"#$%&#'(?9*@*7#-'.'/0=34&32/' '89:;"<'/////=H=' 'DL8EF)G !"#$%&#'()$**+,"-'.'/0=34:C// !"#$%&#'(?9*@*7#-'.'/0=34&::/' '89:;"<'//////41' 'DJMNG !"#$%&#'()$**+,"-'.'/0=34:C2/ !"#$%&#'(?9*@*7#-'.'/0=34&:>/' '89:;"<'//////C=' 'DL8EF)G !"#$%&#'()$**+,"-'.'/0=3A*/1/ !"#$%&#'(?9*@*7#-'.'/0=3A=45/' '89:;"<'/////24=' 'DLO)PQG !"#$%&#'()$**+,"-'.'/0=3=&B:2 !"#$%&#'(?9*@*7#-'.'/05=4=2' '89:;"<'//////4=' 'DLG !"#$%&#'()$**+,"-'.'/0=34:4&2 !"#$%&#'(?9*@*7#-'.'/0=34&3H/' '89:;"<'//////H1' 'DJMNG !"#$%&#'()$**+,"-'.'/0=3A*/// !"#$%&#'(?9*@*7#-'.'/0=3A=45/' '89:;"<'/////=H1' 'DO)PQG !"#$%&#'()$**+,"-'.'/0=3=&B32 !"#$%&#'(?9*@*7#-'.'/05=4=2' '89:;"<'//////H1' 'DG !"#$%&#'()$**+,"-'.'/0=3A*>C/ !"#$%&#'(?9*@*7#-'.'/0=3A=4>/' '89:;"<'/////24=' 'DLO?RJG !"#$%&#'()$**+,"-'.'/0=3A*HS2 !"#$%&#'(?9*@*7#-'.'/0=3A1A/2' '89:;"<'/////2C=' 'DLT)KQ?G !"#$%&#'()$**+,"-'.'/0=34:>A2 !"#$%&#'(?9*@*7#-'.'/0=3A5B&/' '89:;"<'//////H=' 'DL8EF)G !"#$%&#'()$**+,"-'.'/0=3A*>// !"#$%&#'(?9*@*7#-'.'/0=3A===/' '89:;"<'/////=4=' 'DLP?)RG !"#$%&#'()$**+,"-'.'/0=3A*AS/ !"#$%&#'(?9*@*7#-'.'/0=3A===/' '89:;"<'//////C1' 'DP?)RG !"#$%&#'()$**+,"-'.'/0=3A*H*2 !"#$%&#'(?9*@*7#-'.'/0=3A1A/2' '89:;"<'/////4C1' 'DT)KQ?G !"#$%&#'()$**+,"-'.'/0=3A*>32 !"#$%&#'(?9*@*7#-'.'/0=3A=HH/' '89:;"<'/////=H1' 'D)M)Q?G !"#$%&#'()$**+,"-'.'/0=3A*>4/ !"#$%&#'(?9*@*7#-'.'/0=3A=4>/' '89:;"<'//////C1' 'DO?RJG !"#$%&#'()$**+,"-'.'/0=3A*>:2 !"#$%&#'(?9*@*7#-'.'/0=3A=HH/' '89:;"<'/////>C=' 'DL)M)Q?G !"#$%&#'()$**+,"-'.'/0=34:>5/ !"#$%&#'(?9*@*7#-'.'/0=3A542/' '89:;"<'//////41' 'DJMNG !"#$%&#'()$**+,"-'.'/0=/:H5>& !56"&,77*&#*5-' '89:;"<'//////B2 !"#$%&#'()$**+,"-'.'/0=34:>*/ !"#$%&#'(?9*@*7#-'.'/0=3A542/' '89:;"<'//////4=' 'DLJMNG !"#$%&#'()$**+,"-'.'/0=34:HB/ !"#$%&#'(?9*@*7#-'.'/0=34&32/' '89:;"<'//////C1' 'D8EF)G !"#$%&#'()$**+,"-'.'/0=34:CC/ !"#$%&#'(?9*@*7#-'.'/0=34&:>/' '89:;"<'//////41' 'D8EF)G mshtml.dll Event Testing Module - Mouse click on email send button Authorization Creation Module - Email from memory & screen - Reconstruct network output Enforcement Module - Validate outgoing emails 21 Monday, May 24, 2010 

Event Testing Module Mouse click at x=520, y=430 Determine if coordinates map to the “Send Email” button in Outlook Express x=520 y=430 22 Monday, May 24, 2010 

Authorization Creation Module Authorization contains email recipients, subject, and message body. 23 Monday, May 24, 2010 

Enforcement Module Validation script compares outgoing emails to authorizations in the database. ProxSMTP Outlook Express Email Client Validate Email Script SMTP Session Email Server Authorization Database Security Virtual Machine User VM SQL DB Content Filter SMTP Proxy 24 Monday, May 24, 2010 

Security Evaluation • Not vulnerable to current generation malware - Stopped Spammer:Win32/Cutwail.gen!B • Not vulnerable to future malware - Within the bounds of our security assumptions - Breaking into the hypervisor can defeat Gyrus • No false positives (all legit emails passed) - Assuming low OCR edit distance threshold • No false negatives (stopped all spam) 25 Monday, May 24, 2010 

Performance Evaluation User Interaction Mean StdDev Click, OE not running 3.8 ms 0.42 ms Click, no OE compose window 23.7 ms 0.48 ms Click in compose edit area 28.0 ms 0.00 ms Click in compose tool bar 109.9 ms 0.32 ms Click on send button 2067.6 ms 73.17 ms • Most delays are not perceptible to users • “Click on send button” is too slow... 26 Monday, May 24, 2010 

Towards Improving Performance Baseline A B C D 0 1000 2000 3000 4000 5000 Time ￿milliseconds￿ Baseline Memory Snapshot Screen Capture A: xm save -c B: XenAccess dump_memory.c C: XenAccess save to memory buffer D: Save only necessary pages Copy-on-write snapshots still a potential option 27 Monday, May 24, 2010 

Web Browser Support In Gyrus 28 Monday, May 24, 2010 

Goal: Stop Clickbots Key Insight... Clickbots cannot press a key on your keyboard (or use your mouse)! 29 Monday, May 24, 2010 

Internet Explorer Module Design Squid Internet Explorer Web Browser Greasyspoon (ICAP Server) Gyrus ICAP Request Mod Gyrus ICAP Response Mod Link Extraction Script HTTP Request Web Server Authorization Database HTTP Response Security Virtual Machine User VM SQL DB ICAP Web Proxy Server-side Browser EnvJS Event Testing Module - ENTER after typing URL in browser Authorization Creation Module - User clicks in browser - Parse HTTP response packets Enforcement Module - Validate HTTP requests 30 Monday, May 24, 2010 

Event Testing Module Initial authorization occurs when ENTER key is pressed inside the location bar. 31 Monday, May 24, 2010 

Authorization Creation and Enforcement Modules Receive Network Data Authorization Database Reject Send Data To Network Dynamic Authorization Creation Enforcement Receive Hardware Event ENTER Key Pressed? Click Within Webpage? Extract URLs From Data Keyboard Mouse HTTP Response Extract URL From Location Bar Yes Yes Insert URL as Auto Link Increment Token Counter Insert URLs and URL Types URL an Auto Link? HTTP Request Yes URL a Token Link? No Available Tokens? Yes Yes No DB Query DB Query DB Query Decrement Token Counter Automatically create authorizations based on user actions and HTTP Response packets 32 Monday, May 24, 2010 

Dynamic Content var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ ga.js' type='text/javascript'%3E%3C/script%3E")); var pageTracker = _gat._getTracker("UA-4479582-1"); pageTracker._initData(); pageTracker._trackPageview(); http://www.google-analytics.com/ga.js http://www.google-analytics.com/__utm.gif? utmwv=4.6.5&utmn=708793014&utmhn=tutorial.getwindmill.com&ut mcs=utf-8&utmsr=1024x768&utmsc=24-bit&utmul=en- us&utmje=1&utmfl=10.0%20r45&utmcn=1&utmdt=FC2%20-%20Free %20Website%20Access%20Analysis%20Blog%20Rental%20Server %20SEO%20Countermeasures%20etc.%20-&utmhid=2118468594&utmr=- &utmp=%2F&utmac=UA-7509326-1&utmcc=__utma %3D80279954.585291908.1270848936.1270848936.1270848936.1%3B %2B__utmz%3D80279954.1270848936.1.1.utmcsr%3D(direct) %7Cutmccn%3D(direct)%7Cutmcmd%3D(none)%3B 33 Monday, May 24, 2010 

Dynamic Content • JavaScript Challenges - Dynamic DOM manipulation - OnClick handlers - Timers - Inline functions - Imported JS files • Many other dynamic web technologies - Java, Flash, ActiveX, VBScript, Shockwave, etc 34 Monday, May 24, 2010 

Security Evaluation • Not vulnerable to current generation malware - Stopped AdClicker-AD, DR/Click.HSP.A.2, and AdClicker-BY • Not vulnerable to future malware - Within the bounds of our security assumptions - Breaking into the hypervisor can defeat Gyrus • Lots of false positives - Rejected 81.9% of legit HTTP Requests when loading the Alexa top 1000 web sites • No false negatives (stopped all clickbots) 35 Monday, May 24, 2010 

Performance Evaluation User Interaction Mean StdDev ENTER while IE not running 2.0 ms 0.00 ms ENTER while focus on search bar 2.5 ms 0.53 ms ENTER while focus on location bar 456.7 ms 10.37 ms Click while IE not running 3.8 ms 0.42 ms Click in IE window (not web page) 28.1 ms 0.32 ms Click on web page 35.3 0.95 ms • Most delays are not perceptible to users • ENTER while focus on location bar doesnʼt feel too slow because user is waiting for web page to load at same time 36 Monday, May 24, 2010 

Towards Better Web Coverage Option 1: Complete Web Browser Option 2: Improve Env-JS • Black box approach • Simulate user behavior • Good coverage • Security concerns • Performance concerns • Walk DOM tree • Reflection to exec code • Reasonable performance • Limited coverage • Immature code base 37 Monday, May 24, 2010 

Conclusions 38 Monday, May 24, 2010 

• Turret Monitoring Framework - XenAccess VMI Library - Secure Active Monitoring - Design principles for external monitoring - Integrated monitoring framework • Memory Analysis - Use of HMMs to model data structures - Integrating machine learning into memory analysis • Gyrus Security Application Framework - Framework and supporting APIs - User interaction based security for email and web Major Contributions 39 Monday, May 24, 2010 

Future Work • Semantic gap problem • Simplify VMI programming • Better suited hypervisor • Audit-aware software • Enhance and generalize VMI libraries • Simplify Gyrus application support 40 Monday, May 24, 2010 

Improving Host-Based Computer Security Using Secure Active Monitoring and Memory Analysis Thesis Defense Bryan D. Payne School of Computer Science Georgia Institute of Technology 41 Monday, May 24, 2010