$30 off During Our Annual Pro Sale. View Details »

Logs hunting

Logs hunting

Talk given at sfLive 2015 Paris

Olivier Dolbeau

April 09, 2015
Tweet

More Decks by Olivier Dolbeau

Other Decks in Programming

Transcript

  1. LOGS
    HUNTING
    1

    View Slide

  2. WHO AM I?
    Olivier Dolbeau
    @odolbeau
    Work at BlaBlaCar
    2

    View Slide

  3. THIS IS AN
    ELK
    3

    View Slide

  4. 4

    View Slide

  5. 5

    View Slide

  6. 6

    View Slide

  7. Inputs Filters Outputs
    41 inputs
    • syslog
    • udp
    • varnishlog
    • gelf
    • …
    50 filters
    • date
    • geoip
    • i18n
    • urldecode
    • …
    55 outputs
    • elasticsearch
    • redis
    • email
    • graphite
    • …
    And there are also some codecs
    7

    View Slide

  8. Kibana
    8

    View Slide

  9. 9

    View Slide

  10. 10

    View Slide

  11. Which logs
    are we

    talking about?
    11

    View Slide

  12. 12
    Access Logs
    Population: High
    Difficulty: Easy
    Weapon

    View Slide

  13. 13
    Application logs
    Population: Medium / Low
    Difficulty: Medium
    Weapon
    Monolog <3

    View Slide

  14. syslog
    14
    Syslog
    Population: Medium
    Difficulty: Easy
    Weapon
    RSYSLOG

    View Slide

  15. *.* @127.0.0.1:514;RSYSLOG_ForwardFormat
    15

    View Slide

  16. input {
    udp {
    port => 514
    type => syslog
    }
    }
    Logstash - Input
    16

    View Slide

  17. filter {
    if [type] == "syslog" {
    grok {
    match => [ "message", "<%{POSINT:syslog_pri}>%
    {TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %
    {DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %
    {GREEDYDATA:syslog_message}" ]
    add_field => [ "received_at", "%{@timestamp}" ]
    add_field => [ "received_from", "%{host}" ]
    add_tag => [ "rsyslog" ]
    }
    }
    }
    Logstash - Filter
    17

    View Slide

  18. output {
    elasticsearch_http {
    host => “my_es.blablacar.com”
    port => 9200
    index => "logstashv1-%{+YYYY.MM.dd}"
    manage_template => false
    }
    }
    Logstash - Output
    18

    View Slide

  19. 19

    View Slide

  20. syslog
    20

    View Slide

  21. 21

    View Slide

  22. @odolbeau
    22
    On recrute !
    https://speakerdeck.com/odolbeau/logs-hunting

    View Slide