"Large enterprise SIEM: get ready for oversize", Svetlana (Mona) Arkhipova

OWASP Russia Meetup #2

Transcript

1. Large enterprise SIEM: get ready for oversize Svetlana/Mona Arkhipova Qiwi OWASP Meetup, Moscow, 28 Feb 2015
2. What are we talking about? • Log collecting != Security Information and Event Management • Systems monitoring is not enough • Logs as a ‘Big Data’ •
3. WTF is qRadar? Hello IBM! • Log management • Network activity/anomaly detection • SIEM • Nice API
4. WTF is qRadar? Administrator’s nightmare: • Frontend: Java+Tomcat • Backend: Java daemons • DB: Ariel for collected+ indexed data, PostgreSQL for ‘static’ data • Painful performance metrics and load balancing
5. Architecture
6. To log or not to log Guides/best practices • https://www.owasp.org/index.php/Logging_Cheat_ Sheet • http://www.syslog.org/logged/logging-and-syslog- best-practices/ • http://sniperforensicstoolkit.squarespace.com/stora ge/logging/Windows%20Logging%20Cheat%20Shee t%20v1.1.pdf • https://zeltser.com/media/docs/security-incident- log-review-checklist.pdf • …
7. To log or not to log Huston, we got a problem: • Standard syslog message size (RFC 5424) • Windows security logs permissions on W7/2008+ • Database audit – what to log? • Log files on FS (IIS and so on) • In-house developed apps
8. To log or not to log Standard sources: Windows • Event collectors vs. agents • Extended system audit • Non-English logs:
9. To log or not to log Standard sources: *nix, network devices • Syslog as a standard • TCP syslog+network issues=pain (google: “TCP is not reliable”) • UDP syslog message size • Auditd – what to log?
10. To log or not to log Standard sources: Databases • Is login history enough? • Syslog vs DB connection
11. To log or not to log Non-Standard sources: • Exotic network devices • In-house developed apps • 1C (OMG…) and other specific apps • Integration with other security systems (NGFW, DBFW, AV, Security scanners…)
12. To log or not to log When syslog is powerless: WAF CEF log file
13. Normalizing/indexing Event at a glance • Standard properties: timestamp, src IP, dst IP, log source identifier and so on • Custom event properties – KISS principle • No search – no property. Indexing • Standard properties – index, index, index! • Custom event properties indexing: with great power comes great responsibility… • BTW, watch your index size.
14. Over(sizing) Current Qiwi SIEM metrics: • 1800 log sources • 10 000 - 24 000 RAW events per second (EPS) • ~11 600 network flows per second (FPS), ~700 000 flows per minute(FPM) SIEM system: 39 virtual servers, 2 hardware servers with Napatech 2x10G cards, 1 archive server
15. Over(sizing) Expectations (sizing) Reality vCPU 140 160 vRAM 272 Gb 521 Gb vHDD 15 TB 61 TB Once upon a time in a far far galaxy we decided to build our own SIEM…
16. Online/offline storage Daily stats: • 67-145 Gb raw event logs per day • 37-53 Gb network communication events per day • Online storage – fast access (realtime + some previoius data) • Offline – archive storage
17. What if... …EPS or FPM x2 ?
18. Internal security scanners “Normal paranormal” activity inside and outside. • Butthurt :( • Log or drop events? • Custom rules set for nodes • Keep an eye on credentials! • Balancers NAT/SNAThttps://f5.com/resources/white- papers/load-balancing-101-nuts-and-bolts
19. Autopilot: ON • Simple rules • Chained rules:
20. Autopilot: ON
21. Questions? Svetlana/Mona Arkhipova Lead information security expert QIWI infrastructure security team mona@qiwi.com mona.sax m0na_sax