Don't let your log go away

Talk given at Paris Tech Talk MeetUp

Transcript

1. DON’T LET YOUR LOG GO AWAY @odolbeau 1
2. WHO AM I? Olivier Dolbeau @odolbeau Work at BlaBlaCar 2
3. Log 3
4. Which logs are we
 talking about? 4
5. access logs 5
6. syslog syslog 6
7. application logs 7
8. Access 8
9. SSH 9
10. Analyze 10
11. tail grep cat 11
12. This is specific to its access logs My roommate uses this to colorise his access logs… 12
13. 13
14. 14
15. 15
16. 16
17. 17
18. Inputs Filters Outputs 41 inputs • syslog • udp • varnishlog • gelf • … 50 filters • date • geoip • i18n • urldecode • … 55 outputs • elasticsearch • redis • email • graphite • … And there are also some codecs 18
19. Kibana 19
20. 20
21. ELK 21
22. 22
23. syslog syslog 23
24. 24
25. *.* @127.0.0.1:514;RSYSLOG_ForwardFormat 25
26. input { udp { port => 514 type => syslog } } Logstash - Input 26
27. filter { if [type] == "syslog" { grok { match => [ "message", "<%{POSINT:syslog_pri}>% {TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} % {DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: % {GREEDYDATA:syslog_message}" ] add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] add_tag => [ "rsyslog" ] } } } Logstash - Filter 27
28. output { elasticsearch_http { host => “my_es.blablacar.com” port => 9200 index => "logstashv1-%{+YYYY.MM.dd}" manage_template => false } } Logstash - Output 28
29. 29
30. syslog 30
31. 31
32. 32