Under the Surface of Optum's Security Big Data Lake

Optum’s Cyber Defense organization utilizes Elastic within its Security Big Data Lake to search and pivot between cyber threats. The Hadoop and Elastic architecture of the data lake allows correlation and enrichment of logs prior to Elastic ingestion, accelerating investigation timelines. The SBDL can replace and improve on many cyber products offered by third parties at significantly lower cost and risk.

Transcript

1. Michael Levin, Director, Cyber Defense Security Big Data Lake Optum October 13, 2016
2. Presenter Introduction •  Michael Levin, JD, CISSP, EnCE, GLEG, GSLC §  Director of Cyber Defense for Optum §  Former Director of Security Design and Innovation with U.S. Dept. Health and Human Services, Senior Associate with Deloitte, and Investigative Counsel with U.S. Office of Special Counsel •  Optum provides Cyber Security Services to UnitedHealth Group, Fortune 6 business, 41st largest organization world wide. •  Twitter:@Mischa_Levin •  Linkedin: Linkedin.com/in/michaellevin 2
3. Agenda •  Issues in Cybersecurity (the problem) •  Need and Purpose of a Data Lake •  History of Data Lake Projects •  The Security Big Data Lake Design •  Working in the SBDL •  Future of the SBDL 3
4. The Problem of Cybersecurity •  2015 was a record year: §  500 million plus personal records were reported stolen* §  Spear-Phishing campaigns increased by 55%* §  Crypto-Ransomware saw a 35% increase* §  Total breaches increased 25% since 2013* §  54 Zero-day vulnerabilities were identified, a 125% increase since 2013* •  2016 will be a record year: §  Mobile malware saw a 1.7 increase from Q1 to Q2** §  Magnitude of new & unpatchable IP enabled devices §  Healthcare focused Ransomware due to Hollywood Presbyterian Medical Center’s ransom payment •  2017 and every year forever after will be a record year: 4 ¯\_(ツ)_/¯ *Symantec Internet Security Threat report, Volume 21, April 2016 **IT Threat Evolu=on in Q2 2016, Kaspersky Labs
5. Our Problem of Cybersecurity •  Challenges of Monitoring: §  140 million customers §  Operating in 130 countries §  Over 250,000 endpoints §  Over 8 TB of raw logs daily §  110 different log device categories. §  3 billion network, security and endpoint events each day §  Overlapping regulatory requirements regarding data collection §  Breach notification requirements within hours of discovery §  Poorly implemented application logging resulting in PII leakage into the SIEM §  Security tools being used as trouble shooting tools for network devices 5
6. Choking a Security Information & Event Manager (SIEM) •  Commercial SIEMs becomes unusable when correlating data older than 24 hours •  SIEM searches become non-exhaustive: §  Performance is exchanged for thoroughness, data is left un-reviewed §  Complexity focused searches do not produce results §  False negatives are the greatest risk §  Analysts cannot complete research within acceptable timeframe and events are over escalated 6
7. A new broom sweeps in a new way – Russian Proverb •  Security data exists in silos – even in aggregators and loggers data can only be queried in silos •  Records need to be enriched by other sources, internal, external, and derived •  Custom development of utilities, sources of truth, and non-rules-based reporting •  Reduces reliance on third party vendors, who offer data lake-like enrichment on a smaller scale •  Legacy systems take minutes and hours for simple investigative searches that take seconds in a data lake •  Freedom! – Taking ownership of organizations data 7
8. Optum Security Data is Big Data 8 •  8+ TB of transac=onal, referen=al, and enriching logs daily •  Myriad collectors, sensors, aggregators, parsers, filters •  Inves=ga=ve and analy=c needs for 1+ year of live data Volume Velocity Veracity •  Streaming, near-real-=me, some daily and weekly batches •  Increased velocity reduces =me-to-detect •  Ability to quickly call back cold storage data for priority inves=ga=ons •  Analyst-derived knowledge •  Network and endpoint collectors, agents, and appliances •  Organiza=onal enriching and referen=al data •  Combina=ons of unstructured, semi-structured, and structured data •  Difficult to validate completeness from many sources •  Volumes vary considerably by =me-of-day and day-of-week Variety
9. The many lives of the Data Lake •  The development and implementation of the Data Lake went through three very different project teams, each led to valuable lessons learned that eventually brought the current successful iteration of the project. 9
10. Enterprise Data Lake •  January - June 2015 •  Key players: Enterprise Big Data Platform as a Service, Cyber Defense engineers, Information Security Operations engineers •  Hardware: Enterprise Data Lake (internal Hadoop-purposed cloud) •  Ingestion: SIEM CEF •  Applications: MapR, MapReduce (BI/Search capable, not implemented) •  Lessons learned: §  Big Data is hard and takes time! §  Enterprise support for Big Data is hard! §  Enterprise Data Lake offerings must align to client needs or client must align implementation to data lake offerings §  Enterprise support must align to client needs §  Establish ALL requirement as early in project as possible 10
11. Collective Intelligence Framework •  June-November 2015 •  Key Players: Cyber Defense engineers, project managers, outside contractor •  Hardware: POC 5 virtual machines, 160GB/RAM, 2.5TB/disk •  Ingestion: Security Logs, Threat feeds •  Applications: MapR, Elastic, Hive, Wildbrain ingest, R studio, D3 visuals •  Lessons learned: §  Big data is still hard and takes time! §  Platform is a blank slate and must be shaped to meet the organization’s needs §  Requirements for success should be well-defined §  Presentation of results and communication of progress are critical to success §  Measurable results required to meet goals 11
12. Security Big Data Lake •  December 2015 – May 2016 (launch) •  Key Players: Cyber Defense engineers and data scientists, project managers, outside contractors, Big Data Platform as a Service •  Hardware: 550 virtual nodes, 4.5PB/disk, 73.5TB/ram •  Ingestion: SIEM CEF, native sources •  Applications: Hortonworks, Elastic, Zeppelin, Kibana, X-Pack, Oozie, Spark •  Lessons learned: §  Show progress in iterations §  Enterprise support works best with client-dedicated resources §  Choose your contractors, partners, and team wisely! §  Know at least enough of the technology to define your specific requirements and shape design decisions along the way §  Persistence is key 12
13. SBDL Design and Analyst Interface 13 SBDL ingests 3 billion events per day; 100k events per second at daily peak, 500k EPS if necessary
14. SBDL Security Features 14 •  Role-based access control using HDFS permissions •  Group-based access to source data streams •  Project-based access to analy=c projects and aggregated data Hortonworks HDFS •  Document-level or index-level security for each user and role •  User-level query and access logs to track PII and PHI •  Ac=ve directory and LDAP integra=on Elas=csearch •  nginx proxy for ac=ve directory integra=on of applica=ons without na=ve AD capability •  iptables restrict SBDL access to hosts and IPs predetermined to be allowed access •  Encryp=on zones for each team and data source, all user-writable areas are encrypted •  Logging all user access All SBDL Access
15. SBDL Workstreams 15 •  High-level network awareness •  Topic-specific awareness •  Infrastructure monitoring Situa=onal awareness dashboards •  IOC search and pivot •  Pivot on outliers from situa=onal awareness dashboards or indicators from SIEM •  Allows more thorough inves=ga=on Inves=ga=on •  Baselining behavior to highlight anomalies •  Put real-=me logs in historical context •  Machine learning for predic=ve analy=cs Analy=cs •  Put security events in organiza=onal context •  Enrich with data from external sources •  Enhance partnership with IRM teams Inges=on
16. Dashboards for Situational Awareness 16 Aler=ng can be set on any volume threshold for a search/visualiza=on
17. Security Big Data Lake: Asking Bigger Questions 17 Ques,on Answer Which user profiles have accessed files at strange hours that are exclusively accessed by another business unit? Cluster file access by file and user, then determine movement between clusters by a file or user How do I detect port enumera=on if it’s done slowly and inconspicuously? Query the unique number of des=na=on ports by UHG source IP/host over 48-72 hours Which users have downloaded executables or zipped files and have subsequently shown unusual traffic pacerns? Query web/proxy logs for file extension types and for all IPs with hits, compare to new user agent strings, Kansa processes, Sinkhole, A/V, and Damballa alerts Are uncommon user agent strings indica=ve of malicious ac=vity? Gather data from all inves=gated and rebuilt machines and compare incidence of uncommon user agent strings in those machines compared to others Is this system vulnerable and is the vulnerability being ac=vely exploited? Tying vulnerability scan data to actual security events within the enterprise.
18. Data Sets on SBDL Ingestion Pipeline 18 •  All SIEM data •  Firewall •  Webproxy •  Database Access •  Ac=ve Directory •  Endpoint sensors •  Vulnerability scans •  Security =cke=ng •  Kansa scans Transac=onal Enriching •  IP Reputa=on •  Threat feeds •  External vulnerabili=es •  External geoloca=on •  Contextual transac=on data •  Analyst input and feedback •  PeopleSoe •  CMDB •  OASIS •  AE references •  ASK •  Geoloca=on •  SNOW DB Referen=al 3 billion events per day from 110+ sources Reac=ve •  Forensic Data Collec=on •  Forensic Data Analysis •  Vulnerability Scan Data Correla=on
19. SBDL Maturation and Development Roadmap •  Increasing Capacity §  Data volumes are increasing 10-20% annually requiring hardware expansion for the same levels of retention •  Expanding Scope §  Flexibility of platform allows IRM to assist other operations and technology groups with urgent data services needs •  Evolving with the Hadoop and Elasticsearch Platforms §  IRM and Optum ownership of the platform allows us to adapt and evolve with open-source development of cutting edge Big Data technology •  Expansion of native source data collection §  SIEM CEF strategy jump started SBDL adoption but some source systems have higher fidelity data than can be parsed into CEF 19
20. Future SBDL Use Cases •  Automated Forensic Data §  Memory dump is immediately acquired before malicious activity is pushed out of this highly volatile data. §  Custom Scripts initiate acquisition of basic forensic data from an endpoint. §  Automated Timelining of Forensic activity. 20 *Image acquired from EH-Net Online Magazine (ethicalhacker.net)
21. Future SBDL Use Cases – Con’t 21 •  Vulnerability Scan Data §  Vulnerability Scan results are fed to SBDL and correlated to security events
22. Things We Wish We Knew •  Plans for Staffing •  Hardware Support Plan •  These clusters are large and WILL fail •  Functionality Prioritization •  Don’t drink the BD Koolaid •  Plan for long term benefit •  Not a magic bullet •  Plan for training 22
23. Recognition -  Johanna Favole -  Will Casey -  Aparna Nadimpalli -  Susan Bullwinkel -  Naushad Kasu -  Mark Archer -  Nathan Necaise -  Mir Siadaty -  John Jones -  Oliver Chan 23 *Image source: Anthony Clark (nedroid.com)
24. Except where otherwise noted, this work is licensed under hcp://crea=vecommons.org/licenses/by-nd/4.0/ Crea=ve Commons and the double C in a circle are registered trademarks of Crea=ve Commons in the United States and other countries. Third party marks and brands are the property of their respec=ve holders.