Laisse pas trainer ton log !

Given at #forumphp Paris 2014.
Slides are in english, talk was given in french.

Joind.in : https://joind.in/talk/view/11942
Vidéo (in french): https://www.youtube.com/watch?v=1r1SOeaDqH4&list=PL9zDdgiGjkIeeVlrsz9A8o3HtZhvERHT-&index=7

Transcript

1. LAISSE PAS TRAINER
   TON LOG
   @odolbeau
   1
   

   View Slide

2. WHO AM I?
   Olivier Dolbeau
   @odolbeau
   Work at BlaBlaCar
   2
   

   View Slide

3. View Slide

4. Logfile
   4
   

   View Slide

5. –Wikipedia
   “In computing, a logfile (or simply log) is a file that
   records either the events which happen while an
   operating system or other software runs, […].”
   5
   

   View Slide

6. –Wikipedia
   “In computing, a logfile (or simply log) is a file that
   records either the events which happen while an
   operating system or other software runs, […].”
   6
   

   View Slide

7. Which logs
   are we

   talking about?
   7
   

   View Slide

8. access logs
   8
   

   View Slide

9. syslog
   syslog
   9
   

   View Slide

10. application logs
    10
    

    View Slide

11. Access
    11
    

    View Slide

12. SSH
    12
    

    View Slide

13. Analyze
    13
    

    View Slide

14. tail
    grep
    cat
    14
    

    View Slide

15. View Slide

16. My roommate uses this to colorise his access logs…
    15
    

    View Slide

17. This is specific to its access logs
    My roommate uses this to colorise his access logs…
    15
    

    View Slide

18. View Slide

19. View Slide

20. View Slide

21. View Slide

22. View Slide

23. Inputs Filters Outputs
    41 inputs
    • syslog
    • udp
    • varnishlog
    • gelf
    • …
    50 filters
    • date
    • geoip
    • i18n
    • urldecode
    • …
    55 outputs
    • elasticsearch
    • redis
    • email
    • graphite
    • …
    21
    

    View Slide

24. Inputs Filters Outputs
    41 inputs
    • syslog
    • udp
    • varnishlog
    • gelf
    • …
    50 filters
    • date
    • geoip
    • i18n
    • urldecode
    • …
    55 outputs
    • elasticsearch
    • redis
    • email
    • graphite
    • …
    And there are also some codecs
    21
    

    View Slide

25. Kibana
    22
    

    View Slide

26. View Slide

27. ELK
    24
    

    View Slide

28. View Slide

29. syslog
    syslog
    26
    

    View Slide

30. View Slide

31. *.* @127.0.0.1:514;RSYSLOG_ForwardFormat
    28
    

    View Slide

32. input {
    udp {
    port => 514
    type => syslog
    }
    }
    Logstash - Input
    29
    

    View Slide

33. filter {
    if [type] == "syslog" {
    grok {
    match => [ "message", "%
    {TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %
    {DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %
    {GREEDYDATA:syslog_message}" ]
    add_field => [ "received_at", "%{@timestamp}" ]
    add_field => [ "received_from", "%{host}" ]
    add_tag => [ "rsyslog" ]
    }
    }
    }
    Logstash - Filter
    30
    

    View Slide

34. output {
    elasticsearch_http {
    host => “my_es.blablacar.com”
    port => 9200
    index => "logstashv1-%{+YYYY.MM.dd}"
    manage_template => false
    }
    }
    Logstash - Output
    31
    

    View Slide

35. application logs
    32
    

    View Slide

36. View Slide

37. • StreamHandler
    • ErrorLogHandler
    • SwiftMailerHandler
    • SyslogUdpHandler
    • FirePHPHandler
    • FingersCrossedHandler
    • NullHandler
    • …
    More than 36 handlers!
    It’s just some outputs!
    34
    

    View Slide

38. http://symfony.com/doc/current/cookbook/logging/channels_handlers.html
    35
    

    View Slide

39. View Slide

40. View Slide

41. View Slide

42. WHAT CAN
    I DO WITH
    THAT
    39
    

    View Slide

43. I want to have
    all
    my logs
    displayed on the
    console.
    40
    

    View Slide

44. View Slide

45. handlers
    41
    

    View Slide

46. handlers
    channels
    41
    

    View Slide

47. I want to add
    web context
    informations
    to my logs.
    42
    

    View Slide

48. View Slide

49. I want to
    see
    all
    my logs
    in a PRETTY interface!
    44
    

    View Slide

50. View Slide

51. input {
    gelf {
    port => 12201
    type => gelf
    }
    }
    Logstash - Input
    46
    

    View Slide

52. Logstash - Filter
    47
    

    View Slide

53. Logstash - Filter
    47
    This space has intentionally been left blank.
    

    View Slide

54. Logstash - Filter
    47
    This space has intentionally been left blank.
    We don’t need any filter
    

    View Slide

55. Logstash - Filter
    47
    This space has intentionally been left blank.
    We don’t need any filter
    Because logstash works well!
    

    View Slide

56. Logstash - Filter
    47
    This space has intentionally been left blank.
    We don’t need any filter
    Because logstash works well!
    With Heka you need to write a lot of Lua
    

    View Slide

57. This troll was dedicated to @lyrixx
    48
    

    View Slide

58. output {
    elasticsearch_http {
    host => “my_es.blablacar.com”
    port => 9200
    index => "logstashv1-%{+YYYY.MM.dd}"
    manage_template => false
    }
    }
    Logstash - Output
    49
    

    View Slide

59. output {
    elasticsearch_http {
    host => “my_es.blablacar.com”
    port => 9200
    index => "logstashv1-%{+YYYY.MM.dd}"
    manage_template => false
    }
    }
    Logstash - Output
    49
    It’s a duplicate slide!
    

    View Slide

60. syslog
    50
    

    View Slide

61. View Slide

62. View Slide

63. Channels! \o/
    53
    

    View Slide

64. View Slide

65. DEMO
    55
    

    View Slide

66. “C’est pas beautiful ça ?”
    56
    

    View Slide

67. @odolbeau
    https://joind.in/11942
    57
    On recrute !
    

    View Slide