"Large enterprise SIEM: get ready for oversize", Svetlana (Mona) Arkhipova

Transcript

1. Large enterprise SIEM: get ready for oversize Svetlana/Mona Arkhipova Qiwi

   OWASP Meetup, Moscow, 28 Feb 2015

2. What are we talking about? • Log collecting != Security

   Information and Event Management • Systems monitoring is not enough • Logs as a ‘Big Data’ •

3. WTF is qRadar? Hello IBM! • Log management • Network

   activity/anomaly detection • SIEM • Nice API

4. WTF is qRadar? Administrator’s nightmare: • Frontend: Java+Tomcat • Backend:

   Java daemons • DB: Ariel for collected+ indexed data, PostgreSQL for ‘static’ data • Painful performance metrics and load balancing

5. Architecture

6. To log or not to log Guides/best practices • https://www.owasp.org/index.php/Logging_Cheat_

   Sheet • http://www.syslog.org/logged/logging-and-syslog- best-practices/ • http://sniperforensicstoolkit.squarespace.com/stora ge/logging/Windows%20Logging%20Cheat%20Shee t%20v1.1.pdf • https://zeltser.com/media/docs/security-incident- log-review-checklist.pdf • …

7. To log or not to log Huston, we got a

   problem: • Standard syslog message size (RFC 5424) • Windows security logs permissions on W7/2008+ • Database audit – what to log? • Log files on FS (IIS and so on) • In-house developed apps

8. To log or not to log Standard sources: Windows •

   Event collectors vs. agents • Extended system audit • Non-English logs:

9. To log or not to log Standard sources: *nix, network

   devices • Syslog as a standard • TCP syslog+network issues=pain (google: “TCP is not reliable”) • UDP syslog message size • Auditd – what to log?

10. To log or not to log Standard sources: Databases •

    Is login history enough? • Syslog vs DB connection

11. To log or not to log Non-Standard sources: • Exotic

    network devices • In-house developed apps • 1C (OMG…) and other specific apps • Integration with other security systems (NGFW, DBFW, AV, Security scanners…)

12. To log or not to log When syslog is powerless:

    WAF CEF log file

13. Normalizing/indexing Event at a glance • Standard properties: timestamp, src

    IP, dst IP, log source identifier and so on • Custom event properties – KISS principle • No search – no property. Indexing • Standard properties – index, index, index! • Custom event properties indexing: with great power comes great responsibility… • BTW, watch your index size.

14. Over(sizing) Current Qiwi SIEM metrics: • 1800 log sources •

    10 000 - 24 000 RAW events per second (EPS) • ~11 600 network flows per second (FPS), ~700 000 flows per minute(FPM) SIEM system: 39 virtual servers, 2 hardware servers with Napatech 2x10G cards, 1 archive server

15. Over(sizing) Expectations (sizing) Reality vCPU 140 160 vRAM 272 Gb

    521 Gb vHDD 15 TB 61 TB Once upon a time in a far far galaxy we decided to build our own SIEM…

16. Online/offline storage Daily stats: • 67-145 Gb raw event logs

    per day • 37-53 Gb network communication events per day • Online storage – fast access (realtime + some previoius data) • Offline – archive storage

17. What if... …EPS or FPM x2 ?

18. Internal security scanners “Normal paranormal” activity inside and outside. •

    Butthurt :( • Log or drop events? • Custom rules set for nodes • Keep an eye on credentials! • Balancers NAT/SNAThttps://f5.com/resources/white- papers/load-balancing-101-nuts-and-bolts

19. Autopilot: ON • Simple rules • Chained rules:

20. Autopilot: ON

21. Questions? Svetlana/Mona Arkhipova Lead information security expert QIWI infrastructure security

    team mona@qiwi.com mona.sax m0na_sax