Don't let your log go away

Talk given at Paris Tech Talk MeetUp

Transcript

1. DON’T LET YOUR
   LOG GO AWAY
   @odolbeau
   1
   

   View Slide

2. WHO AM I?
   Olivier Dolbeau
   @odolbeau
   Work at BlaBlaCar
   2
   

   View Slide

3. Log
   3
   

   View Slide

4. Which logs
   are we

   talking about?
   4
   

   View Slide

5. access logs
   5
   

   View Slide

6. syslog
   syslog
   6
   

   View Slide

7. application logs
   7
   

   View Slide

8. Access
   8
   

   View Slide

9. SSH
   9
   

   View Slide

10. Analyze
    10
    

    View Slide

11. tail
    grep
    cat
    11
    

    View Slide

12. This is specific to its access logs
    My roommate uses this to colorise his access logs…
    12
    

    View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. View Slide

18. Inputs Filters Outputs
    41 inputs
    • syslog
    • udp
    • varnishlog
    • gelf
    • …
    50 filters
    • date
    • geoip
    • i18n
    • urldecode
    • …
    55 outputs
    • elasticsearch
    • redis
    • email
    • graphite
    • …
    And there are also some codecs
    18
    

    View Slide

19. Kibana
    19
    

    View Slide

20. View Slide

21. ELK
    21
    

    View Slide

22. View Slide

23. syslog
    syslog
    23
    

    View Slide

24. View Slide

25. *.* @127.0.0.1:514;RSYSLOG_ForwardFormat
    25
    

    View Slide

26. input {
    udp {
    port => 514
    type => syslog
    }
    }
    Logstash - Input
    26
    

    View Slide

27. filter {
    if [type] == "syslog" {
    grok {
    match => [ "message", "%
    {TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %
    {DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %
    {GREEDYDATA:syslog_message}" ]
    add_field => [ "received_at", "%{@timestamp}" ]
    add_field => [ "received_from", "%{host}" ]
    add_tag => [ "rsyslog" ]
    }
    }
    }
    Logstash - Filter
    27
    

    View Slide

28. output {
    elasticsearch_http {
    host => “my_es.blablacar.com”
    port => 9200
    index => "logstashv1-%{+YYYY.MM.dd}"
    manage_template => false
    }
    }
    Logstash - Output
    28
    

    View Slide

29. View Slide

30. syslog
    30
    

    View Slide

31. View Slide

32. View Slide

33. View Slide