$30 off During Our Annual Pro Sale. View Details »

Laisse pas trainer ton log !

Laisse pas trainer ton log !

Given at #forumphp Paris 2014.
Slides are in english, talk was given in french.

Joind.in : https://joind.in/talk/view/11942
Vidéo (in french): https://www.youtube.com/watch?v=1r1SOeaDqH4&list=PL9zDdgiGjkIeeVlrsz9A8o3HtZhvERHT-&index=7

Olivier Dolbeau

October 23, 2014
Tweet

More Decks by Olivier Dolbeau

Other Decks in Programming

Transcript

  1. LAISSE PAS TRAINER
    TON LOG
    @odolbeau
    1

    View Slide

  2. WHO AM I?
    Olivier Dolbeau
    @odolbeau
    Work at BlaBlaCar
    2

    View Slide

  3. 3

    View Slide

  4. Logfile
    4

    View Slide

  5. –Wikipedia
    “In computing, a logfile (or simply log) is a file that
    records either the events which happen while an
    operating system or other software runs, […].”
    5

    View Slide

  6. –Wikipedia
    “In computing, a logfile (or simply log) is a file that
    records either the events which happen while an
    operating system or other software runs, […].”
    6

    View Slide

  7. Which logs
    are we

    talking about?
    7

    View Slide

  8. access logs
    8

    View Slide

  9. syslog
    syslog
    9

    View Slide

  10. application logs
    10

    View Slide

  11. Access
    11

    View Slide

  12. SSH
    12

    View Slide

  13. Analyze
    13

    View Slide

  14. tail
    grep
    cat
    14

    View Slide

  15. 15

    View Slide

  16. My roommate uses this to colorise his access logs…
    15

    View Slide

  17. This is specific to its access logs
    My roommate uses this to colorise his access logs…
    15

    View Slide

  18. 16

    View Slide

  19. 17

    View Slide

  20. 18

    View Slide

  21. 19

    View Slide

  22. 20

    View Slide

  23. Inputs Filters Outputs
    41 inputs
    • syslog
    • udp
    • varnishlog
    • gelf
    • …
    50 filters
    • date
    • geoip
    • i18n
    • urldecode
    • …
    55 outputs
    • elasticsearch
    • redis
    • email
    • graphite
    • …
    21

    View Slide

  24. Inputs Filters Outputs
    41 inputs
    • syslog
    • udp
    • varnishlog
    • gelf
    • …
    50 filters
    • date
    • geoip
    • i18n
    • urldecode
    • …
    55 outputs
    • elasticsearch
    • redis
    • email
    • graphite
    • …
    And there are also some codecs
    21

    View Slide

  25. Kibana
    22

    View Slide

  26. 23

    View Slide

  27. ELK
    24

    View Slide

  28. 25

    View Slide

  29. syslog
    syslog
    26

    View Slide

  30. 27

    View Slide

  31. *.* @127.0.0.1:514;RSYSLOG_ForwardFormat
    28

    View Slide

  32. input {
    udp {
    port => 514
    type => syslog
    }
    }
    Logstash - Input
    29

    View Slide

  33. filter {
    if [type] == "syslog" {
    grok {
    match => [ "message", "<%{POSINT:syslog_pri}>%
    {TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %
    {DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %
    {GREEDYDATA:syslog_message}" ]
    add_field => [ "received_at", "%{@timestamp}" ]
    add_field => [ "received_from", "%{host}" ]
    add_tag => [ "rsyslog" ]
    }
    }
    }
    Logstash - Filter
    30

    View Slide

  34. output {
    elasticsearch_http {
    host => “my_es.blablacar.com”
    port => 9200
    index => "logstashv1-%{+YYYY.MM.dd}"
    manage_template => false
    }
    }
    Logstash - Output
    31

    View Slide

  35. application logs
    32

    View Slide

  36. 33

    View Slide

  37. • StreamHandler
    • ErrorLogHandler
    • SwiftMailerHandler
    • SyslogUdpHandler
    • FirePHPHandler
    • FingersCrossedHandler
    • NullHandler
    • …
    More than 36 handlers!
    It’s just some outputs!
    34

    View Slide

  38. http://symfony.com/doc/current/cookbook/logging/channels_handlers.html
    35

    View Slide

  39. 36

    View Slide

  40. 37

    View Slide

  41. 38

    View Slide

  42. WHAT CAN
    I DO WITH
    THAT
    39

    View Slide

  43. I want to have
    all
    my logs
    displayed on the
    console.
    40

    View Slide

  44. 41

    View Slide

  45. handlers
    41

    View Slide

  46. handlers
    channels
    41

    View Slide

  47. I want to add
    web context
    informations
    to my logs.
    42

    View Slide

  48. 43

    View Slide

  49. I want to
    see
    all
    my logs
    in a PRETTY interface!
    44

    View Slide

  50. 45

    View Slide

  51. input {
    gelf {
    port => 12201
    type => gelf
    }
    }
    Logstash - Input
    46

    View Slide

  52. Logstash - Filter
    47

    View Slide

  53. Logstash - Filter
    47
    This space has intentionally been left blank.

    View Slide

  54. Logstash - Filter
    47
    This space has intentionally been left blank.
    We don’t need any filter

    View Slide

  55. Logstash - Filter
    47
    This space has intentionally been left blank.
    We don’t need any filter
    Because logstash works well!

    View Slide

  56. Logstash - Filter
    47
    This space has intentionally been left blank.
    We don’t need any filter
    Because logstash works well!
    With Heka you need to write a lot of Lua

    View Slide

  57. This troll was dedicated to @lyrixx
    48

    View Slide

  58. output {
    elasticsearch_http {
    host => “my_es.blablacar.com”
    port => 9200
    index => "logstashv1-%{+YYYY.MM.dd}"
    manage_template => false
    }
    }
    Logstash - Output
    49

    View Slide

  59. output {
    elasticsearch_http {
    host => “my_es.blablacar.com”
    port => 9200
    index => "logstashv1-%{+YYYY.MM.dd}"
    manage_template => false
    }
    }
    Logstash - Output
    49
    It’s a duplicate slide!

    View Slide

  60. syslog
    50

    View Slide

  61. 51

    View Slide

  62. 52

    View Slide

  63. Channels! \o/
    53

    View Slide

  64. 54

    View Slide

  65. DEMO
    55

    View Slide

  66. “C’est pas beautiful ça ?”
    56

    View Slide

  67. @odolbeau
    https://joind.in/11942
    57
    On recrute !

    View Slide