Fix What Matters Michael Roytman SIRAcon October 21, 2013 

Why You Should(n’t) Listen • Naive Grad Student Not Too Long Ago • Still Plays With Legos • Barely Passed Regression Analysis • MS Operations Research, Georgia Tech Michael Roytman • Data Scientist, Risk I/O • Fraud Detection, Large Bank 

Roadmap • The Struggle • What’s Good? • Data Driven Insights • Framework • Decision-Making • What’s Bad? 

Starting From Scratch “It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.” -Sir Arthur Conan Doyle, 1887 

Starting From Scratch 

Starting From Scratch Academia! • GScholar! • JSTOR! • IEEE! • ProQuest! InfoSec Blogs! • CSIOs! • Pen Testers! • Threat Reports! • SOTI/DBIR! ! Twitter! • Thought Leaders (you know who you are)! • BlackHats! • Vuln Researchers! Primary Sources! • MITRE! • OSVDB! • NIST CVSS Committee(s)! • Internal Message Boards for ^! Text CISOs 

Data Fundamentalism Don’t Ignore What a Vulnerability Is: Creation Bias ! (http://blog.risk.io/2013/04/data-fundamentalism/) ! Jerico/Sushidude @ BlackHat ! (https://www.blackhat.com/us-13/briefings.html#Martin)! Luca Allodi - CVSS DDOS ! (http://disi.unitn.it/~allodi/allodi-12-badgers.pdf):! 

Data Fundamentalism - What’s The Big Deal? ! ”Since 2006 Vulnerabilities have declined by 26 percent.” ! (http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf)! ! ! “The total number of vulnerabilities in 2013 is up 16 percent so far when compared to what we saw in the same time period in 2012. ”! (http://www.symantec.com/content/en/us/enterprise/other_resources/b-intelligence_report_06-2013.en-us.pdf)! ! ! 

What’s Good? Bad For Vulnerability Statistics:! ! NVD, OSVDB, ExploitDB, CVSS, Patches, Microsoft Reports, etc, et al, and so on. ! Good For Vulnerability Statistics:! ! Vulnerabilities. ! 

Data Is Everything And Everything Is Data. 

What’s Good? 

What’s Good? 

What’s Good? 

What’s Good? 

What’s Good? 

What’s Good? 

Counterterrorism Known Groups Surveillance Threat Intel, Analysts Targets, Layouts Past Incidents, Close Calls 

What’s Good? 

Uh, Sports? Opposing Teams, Specific Players Gameplay Scouting Reports, Gametape Roster, Player Skills Learning from Losing 

InfoSec? 

Defend Like You’ve Done It Before Groups, Motivations Exploits Vulnerability Definitions Asset Topology, Actual Vulns on System Learning from Breaches 

Work With What You’ve Got: Akamai, Safenet ExploitDB, Metasploit NVD, MITRE 

Add Some Spice 

Show Me The Money 23,000,000 Vulnerabilities! Across 1,000,000 Assets! Representing 9,500 Companies! Using 22 Unique Scanners! 

Whatchu Know About Dat?(a) ! Duplication Vulnerability Density Remediation 

Duplication 0 225,000 450,000 675,000 900,000 1,125,000 1,350,000 1,575,000 1,800,000 2,025,000 2,250,000 2 or more scanners 3 or more 4 or more 5 or more 6 or more 

Duplication We Have: F(Number of Scanners) => Number of Duplicate Vulnerabilities We Want: F(Number of Scanners) => Vulnerability Coverage Make Decisions At The Margins! <---------Good Luck! 0.0 25.0 50.0 75.0 100.0 0 1 2 3 4 5 6 

Density Type of Asset ~Count Hostname 20,000 Netbios 1000 IP Address 200,000 File 10,000 Url 5,000 Hostname Netbios IP File Url 0.0 22.5 45.0 67.5 90.0 

CVSS And Remediation Metrics 0.0 350.0 700.0 1050.0 1400.0 1 2 3 4 5 6 7 8 9 10 Average Time To Close By Severity Oldest Vulnerability By Severity 

CVSS And Remediation - Lessons From A CISO Remediation/Lack Thereof, by CVSS 1 2 3 4 5 6 7 8 9 10 NVD Distribution by CVSS 

The Kicker - Live Breach Data 1,500,000 ! Vulnerabilities Related to Live Breaches Recorded! June, July 2013 ! 

CVSS And Remediation - Nope 0.0 1750.0 3500.0 5250.0 7000.0 1 2 3 4 5 6 7 8 9 10 Oldest Breached Vulnerability By Severity 

CVSS - A VERY General Guide For Remediation - Yep 0.0 40000.0 80000.0 120000.0 160000.0 1 2 3 4 5 6 7 8 9 10 Open Vulns With Breaches Occuring By Severity 

The One Billion Dollar Question Probability(You Will Be Breached On A Particular Open Vulnerability)? 1.98% =(Open Vulnerabilities | Breaches Occurred On Their CVE)/(Total Open Vulnerabilities) 

I Love It When You Call Me Big Data Probability A Vulnerability Having Property X Has Observed Breaches RANDOM VULN CVSS 10 CVSS 9 CVSS 8 CVSS 6 CVSS 7 CVSS 5 CVSS 4 Has Patch 0.00000 0.01000 0.02000 0.03000 0.04000 

What’s the Alternative? 

I Love It When You Call Me Big Data Probability A Vulnerability Having Property X Has Observed Breaches Random Vuln CVSS 10 Exploit DB Metasploit MSP+EDB 0.0 0.1 0.2 0.2 0.3 

Data Is Everything And Everything Is Data. 

Be Better Than The Gap 

I Love It When You Call Me Big Data Spray and Pray => 2% ! CVSS 10 => 4% ! Metasploit + ExploitDB => 30% ! A Good Model That’s Not Built By One Kid Without Hadoop => ???! 

Thank You Don’t Be A Stranger Blog: http://blog.risk.io Twitter: @mroytman