Building Security Data Lake

vBrownBag podcast
Building Security Data Lake

https://youtu.be/6qQ7_asdI4I?si=CSsn0jz2vo00Y02Q

Transcript

1. Building Security Data Lake Richard Fan March 29, 2023
2. EXPRESSVPN Richard Fan Security Engineer from ExpressVPN A Builder and Tech advocate AWS Community Builder • https://dev.to/richardfan1126 • https://medium.com/@richardfan1126 • https://github.com/richardfan1126 Who am I?
3. Security Data
4. EXPRESSVPN What is security data? Security Data
5. EXPRESSVPN Why do we need Security logs? • Detect threat • Incident response • Compliance • Vulnerability management Security Data
6. EXPRESSVPN Security Data Storing (or Not storing) locally • No correlation • Difficult to track • Time consuming during incident response Send to SIEM • Centralized • Analytics / threat detection • Strong query capability Capturing security logs
7. EXPRESSVPN The growing amount/complexity of security logs • Shift-left • Adoption of cloud • 2 common approaches ◦ Drop less-important events ◦ Scale-up SIEM and send all events to it Security Data
8. EXPRESSVPN SIEM is not an ultimate solution • Too expensive • Short retention period • Difficult to integrate with other data processor Security Data
9. Data Lake
10. EXPRESSVPN Data Lake comes in Data Lake • Store data in large scale • Centralize data repository • Turn raw data into useful data • NOT a data archive • NOT a database (Security) Data Lake Security Data Lake • Threat detection • Event context • Real-time alert
11. EXPRESSVPN How to start? • Identify all your data sources • Identify ingestion methods • Evaluate your situation ◦ Engineering ◦ Threat hunting ◦ SIEM options • Decide where SIEM fits in Data Lake
12. EXPRESSVPN Connector split Source split Data Lake SIEM in Security data lake
13. EXPRESSVPN Data lake to SIEM SIEM to Data lake Data Lake SIEM in Security data lake
14. Threat hunting
15. EXPRESSVPN Threat hunting life cycle in data lake Threat hunting
16. EXPRESSVPN Detection as Code • Better documentation • Code repository / Code Review (GitOps) • Common language • Vendor agnostic Threat hunting
17. EXPRESSVPN Detection as Code - Sigma Threat hunting
18. EXPRESSVPN Detection as Code - Sigma Threat hunting Splunk Elasticsearch
19. Our Story
20. EXPRESSVPN How do we build security data lake? Technology • Ingestion • Storage ◦ S3 • Analytics • Detection-as-code ◦ Sigma Our story
21. EXPRESSVPN How do we build security data lake? Company • SOC team • IT team • Security Engineering team • Cross-team collaboration • Security knowledge Our story
22. EXPRESSVPN Takeaway • Evaluate your current state • Start small • Estimate cost • Embrace IaC / DaC • Don’t forget about people Our story
23. Thank you