Logs hunting

Talk given at sfLive 2015 Paris

Transcript

1. LOGS HUNTING 1
2. WHO AM I? Olivier Dolbeau @odolbeau Work at BlaBlaCar 2
3. THIS IS AN ELK 3
7. Inputs Filters Outputs 41 inputs • syslog • udp • varnishlog • gelf • … 50 filters • date • geoip • i18n • urldecode • … 55 outputs • elasticsearch • redis • email • graphite • … And there are also some codecs 7
8. Kibana 8
11. Which logs are we
 talking about? 11
12. 12 Access Logs Population: High Difficulty: Easy Weapon
13. 13 Application logs Population: Medium / Low Difficulty: Medium Weapon Monolog <3
14. syslog 14 Syslog Population: Medium Difficulty: Easy Weapon RSYSLOG
15. *.* @127.0.0.1:514;RSYSLOG_ForwardFormat 15
16. input { udp { port => 514 type => syslog } } Logstash - Input 16
17. filter { if [type] == "syslog" { grok { match => [ "message", "<%{POSINT:syslog_pri}>% {TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} % {DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: % {GREEDYDATA:syslog_message}" ] add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] add_tag => [ "rsyslog" ] } } } Logstash - Filter 17
18. output { elasticsearch_http { host => “my_es.blablacar.com” port => 9200 index => "logstashv1-%{+YYYY.MM.dd}" manage_template => false } } Logstash - Output 18
20. syslog 20
22. @odolbeau 22 On recrute ! https://speakerdeck.com/odolbeau/logs-hunting