Logs hunting

Logs hunting

Talk given at sfLive 2015 Paris

418997665c4a3368515ecf9c3d746b95?s=128

Olivier Dolbeau

April 09, 2015
Tweet

Transcript

  1. LOGS HUNTING 1

  2. WHO AM I? Olivier Dolbeau @odolbeau Work at BlaBlaCar 2

  3. THIS IS AN ELK 3

  4. 4

  5. 5

  6. 6

  7. Inputs Filters Outputs 41 inputs • syslog • udp •

    varnishlog • gelf • … 50 filters • date • geoip • i18n • urldecode • … 55 outputs • elasticsearch • redis • email • graphite • … And there are also some codecs 7
  8. Kibana 8

  9. 9

  10. 10

  11. Which logs are we
 talking about? 11

  12. 12 Access Logs Population: High Difficulty: Easy Weapon

  13. 13 Application logs Population: Medium / Low Difficulty: Medium Weapon

    Monolog <3
  14. syslog 14 Syslog Population: Medium Difficulty: Easy Weapon RSYSLOG

  15. *.* @127.0.0.1:514;RSYSLOG_ForwardFormat 15

  16. input { udp { port => 514 type => syslog

    } } Logstash - Input 16
  17. filter { if [type] == "syslog" { grok { match

    => [ "message", "<%{POSINT:syslog_pri}>% {TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} % {DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: % {GREEDYDATA:syslog_message}" ] add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] add_tag => [ "rsyslog" ] } } } Logstash - Filter 17
  18. output { elasticsearch_http { host => “my_es.blablacar.com” port => 9200

    index => "logstashv1-%{+YYYY.MM.dd}" manage_template => false } } Logstash - Output 18
  19. 19

  20. syslog 20

  21. 21

  22. @odolbeau 22 On recrute ! https://speakerdeck.com/odolbeau/logs-hunting