Don't let your log go away

Don't let your log go away

Talk given at Paris Tech Talk MeetUp

418997665c4a3368515ecf9c3d746b95?s=128

Olivier Dolbeau

March 24, 2015
Tweet

Transcript

  1. DON’T LET YOUR LOG GO AWAY @odolbeau 1

  2. WHO AM I? Olivier Dolbeau @odolbeau Work at BlaBlaCar 2

  3. Log 3

  4. Which logs are we
 talking about? 4

  5. access logs 5

  6. syslog syslog 6

  7. application logs 7

  8. Access 8

  9. SSH 9

  10. Analyze 10

  11. tail grep cat 11

  12. This is specific to its access logs My roommate uses

    this to colorise his access logs… 12
  13. 13

  14. 14

  15. 15

  16. 16

  17. 17

  18. Inputs Filters Outputs 41 inputs • syslog • udp •

    varnishlog • gelf • … 50 filters • date • geoip • i18n • urldecode • … 55 outputs • elasticsearch • redis • email • graphite • … And there are also some codecs 18
  19. Kibana 19

  20. 20

  21. ELK 21

  22. 22

  23. syslog syslog 23

  24. 24

  25. *.* @127.0.0.1:514;RSYSLOG_ForwardFormat 25

  26. input { udp { port => 514 type => syslog

    } } Logstash - Input 26
  27. filter { if [type] == "syslog" { grok { match

    => [ "message", "<%{POSINT:syslog_pri}>% {TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} % {DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: % {GREEDYDATA:syslog_message}" ] add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] add_tag => [ "rsyslog" ] } } } Logstash - Filter 27
  28. output { elasticsearch_http { host => “my_es.blablacar.com” port => 9200

    index => "logstashv1-%{+YYYY.MM.dd}" manage_template => false } } Logstash - Output 28
  29. 29

  30. syslog 30

  31. 31

  32. 32

  33. None