Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Security Data Lake

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on.
Avatar for Richard Fan Richard Fan
December 13, 2023

Building Security Data Lake

vBrownBag podcast
Building Security Data Lake

https://youtu.be/6qQ7_asdI4I?si=CSsn0jz2vo00Y02Q

Avatar for Richard Fan

Richard Fan

December 13, 2023
Tweet

More Decks by Richard Fan

Other Decks in Technology

Transcript

  1. EXPRESSVPN Richard Fan Security Engineer from ExpressVPN A Builder and

    Tech advocate AWS Community Builder • https://dev.to/richardfan1126 • https://medium.com/@richardfan1126 • https://github.com/richardfan1126 Who am I?
  2. EXPRESSVPN Why do we need Security logs? • Detect threat

    • Incident response • Compliance • Vulnerability management Security Data
  3. EXPRESSVPN Security Data Storing (or Not storing) locally • No

    correlation • Difficult to track • Time consuming during incident response Send to SIEM • Centralized • Analytics / threat detection • Strong query capability Capturing security logs
  4. EXPRESSVPN The growing amount/complexity of security logs • Shift-left •

    Adoption of cloud • 2 common approaches ◦ Drop less-important events ◦ Scale-up SIEM and send all events to it Security Data
  5. EXPRESSVPN SIEM is not an ultimate solution • Too expensive

    • Short retention period • Difficult to integrate with other data processor Security Data
  6. EXPRESSVPN Data Lake comes in Data Lake • Store data

    in large scale • Centralize data repository • Turn raw data into useful data • NOT a data archive • NOT a database (Security) Data Lake Security Data Lake • Threat detection • Event context • Real-time alert
  7. EXPRESSVPN How to start? • Identify all your data sources

    • Identify ingestion methods • Evaluate your situation ◦ Engineering ◦ Threat hunting ◦ SIEM options • Decide where SIEM fits in Data Lake
  8. EXPRESSVPN Data lake to SIEM SIEM to Data lake Data

    Lake SIEM in Security data lake
  9. EXPRESSVPN Detection as Code • Better documentation • Code repository

    / Code Review (GitOps) • Common language • Vendor agnostic Threat hunting
  10. EXPRESSVPN How do we build security data lake? Technology •

    Ingestion • Storage ◦ S3 • Analytics • Detection-as-code ◦ Sigma Our story
  11. EXPRESSVPN How do we build security data lake? Company •

    SOC team • IT team • Security Engineering team • Cross-team collaboration • Security knowledge Our story
  12. EXPRESSVPN Takeaway • Evaluate your current state • Start small

    • Estimate cost • Embrace IaC / DaC • Don’t forget about people Our story