Upgrade to Pro — share decks privately, control downloads, hide ads and more …

"Large enterprise SIEM: get ready for oversize", Svetlana (Mona) Arkhipova

"Large enterprise SIEM: get ready for oversize", Svetlana (Mona) Arkhipova

OWASP Russia Meetup #2

47a3212bc9721c62f1135ead56569f17?s=128

OWASP Moscow

December 04, 2017
Tweet

Transcript

  1. Large enterprise SIEM: get ready for oversize Svetlana/Mona Arkhipova Qiwi

    OWASP Meetup, Moscow, 28 Feb 2015
  2. What are we talking about? • Log collecting != Security

    Information and Event Management • Systems monitoring is not enough • Logs as a ‘Big Data’ •
  3. WTF is qRadar? Hello IBM! • Log management • Network

    activity/anomaly detection • SIEM • Nice API
  4. WTF is qRadar? Administrator’s nightmare: • Frontend: Java+Tomcat • Backend:

    Java daemons • DB: Ariel for collected+ indexed data, PostgreSQL for ‘static’ data • Painful performance metrics and load balancing
  5. Architecture

  6. To log or not to log Guides/best practices • https://www.owasp.org/index.php/Logging_Cheat_

    Sheet • http://www.syslog.org/logged/logging-and-syslog- best-practices/ • http://sniperforensicstoolkit.squarespace.com/stora ge/logging/Windows%20Logging%20Cheat%20Shee t%20v1.1.pdf • https://zeltser.com/media/docs/security-incident- log-review-checklist.pdf • …
  7. To log or not to log Huston, we got a

    problem: • Standard syslog message size (RFC 5424) • Windows security logs permissions on W7/2008+ • Database audit – what to log? • Log files on FS (IIS and so on) • In-house developed apps
  8. To log or not to log Standard sources: Windows •

    Event collectors vs. agents • Extended system audit • Non-English logs:
  9. To log or not to log Standard sources: *nix, network

    devices • Syslog as a standard • TCP syslog+network issues=pain (google: “TCP is not reliable”) • UDP syslog message size • Auditd – what to log?
  10. To log or not to log Standard sources: Databases •

    Is login history enough? • Syslog vs DB connection
  11. To log or not to log Non-Standard sources: • Exotic

    network devices • In-house developed apps • 1C (OMG…) and other specific apps • Integration with other security systems (NGFW, DBFW, AV, Security scanners…)
  12. To log or not to log When syslog is powerless:

    WAF CEF log file
  13. Normalizing/indexing Event at a glance • Standard properties: timestamp, src

    IP, dst IP, log source identifier and so on • Custom event properties – KISS principle • No search – no property. Indexing • Standard properties – index, index, index! • Custom event properties indexing: with great power comes great responsibility… • BTW, watch your index size.
  14. Over(sizing) Current Qiwi SIEM metrics: • 1800 log sources •

    10 000 - 24 000 RAW events per second (EPS) • ~11 600 network flows per second (FPS), ~700 000 flows per minute(FPM) SIEM system: 39 virtual servers, 2 hardware servers with Napatech 2x10G cards, 1 archive server
  15. Over(sizing) Expectations (sizing) Reality vCPU 140 160 vRAM 272 Gb

    521 Gb vHDD 15 TB 61 TB Once upon a time in a far far galaxy we decided to build our own SIEM…
  16. Online/offline storage Daily stats: • 67-145 Gb raw event logs

    per day • 37-53 Gb network communication events per day • Online storage – fast access (realtime + some previoius data) • Offline – archive storage
  17. What if... …EPS or FPM x2 ?

  18. Internal security scanners “Normal paranormal” activity inside and outside. •

    Butthurt :( • Log or drop events? • Custom rules set for nodes • Keep an eye on credentials! • Balancers NAT/SNAThttps://f5.com/resources/white- papers/load-balancing-101-nuts-and-bolts
  19. Autopilot: ON • Simple rules • Chained rules:

  20. Autopilot: ON

  21. Questions? Svetlana/Mona Arkhipova Lead information security expert QIWI infrastructure security

    team mona@qiwi.com mona.sax m0na_sax