Don't let your log go away

Transcript

1. 1.

   DON’T LET YOUR LOG GO AWAY @odolbeau 1

2. 2.

   WHO AM I? Olivier Dolbeau @odolbeau Work at BlaBlaCar 2

3. 3.

   Log 3

4. 4.

   Which logs are we
 talking about? 4

5. 5.

   access logs 5

6. 6.

   syslog syslog 6

7. 7.

   application logs 7

8. 8.

   Access 8

9. 9.

   SSH 9

10. 10.

    Analyze 10

11. 11.

    tail grep cat 11

12. 12.

    This is specific to its access logs My roommate uses

    this to colorise his access logs… 12
13. 13.

    13

14. 14.

    14

15. 15.

    15

16. 16.

    16

17. 17.

    17

18. 18.

    Inputs Filters Outputs 41 inputs • syslog • udp •

    varnishlog • gelf • … 50 filters • date • geoip • i18n • urldecode • … 55 outputs • elasticsearch • redis • email • graphite • … And there are also some codecs 18
19. 19.

    Kibana 19

20. 20.

    20

21. 21.

    ELK 21

22. 22.

    22

23. 23.

    syslog syslog 23

24. 24.

    24

25. 25.

    *.* @127.0.0.1:514;RSYSLOG_ForwardFormat 25

26. 26.

    input { udp { port => 514 type => syslog

    } } Logstash - Input 26
27. 27.

    filter { if [type] == "syslog" { grok { match

    => [ "message", "<%{POSINT:syslog_pri}>% {TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} % {DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: % {GREEDYDATA:syslog_message}" ] add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] add_tag => [ "rsyslog" ] } } } Logstash - Filter 27
28. 28.

    output { elasticsearch_http { host => “my_es.blablacar.com” port => 9200

    index => "logstashv1-%{+YYYY.MM.dd}" manage_template => false } } Logstash - Output 28
29. 29.

    29

30. 30.

    syslog 30

31. 31.

    31

32. 32.

    32

33. 33.

    None