Talk given at sfLive 2015 Paris
LOGSHUNTING1
View Slide
WHO AM I?Olivier Dolbeau@odolbeauWork at BlaBlaCar2
THIS IS ANELK3
4
5
6
Inputs Filters Outputs41 inputs• syslog• udp• varnishlog• gelf• …50 filters• date• geoip• i18n• urldecode• …55 outputs• elasticsearch• redis• email• graphite• …And there are also some codecs7
Kibana8
9
10
Which logsare we talking about?11
12Access LogsPopulation: HighDifficulty: EasyWeapon
13Application logsPopulation: Medium / LowDifficulty: MediumWeaponMonolog <3
syslog14SyslogPopulation: MediumDifficulty: EasyWeaponRSYSLOG
*.* @127.0.0.1:514;RSYSLOG_ForwardFormat15
input {udp {port => 514type => syslog}}Logstash - Input16
filter {if [type] == "syslog" {grok {match => [ "message", "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ]add_field => [ "received_at", "%{@timestamp}" ]add_field => [ "received_from", "%{host}" ]add_tag => [ "rsyslog" ]}}}Logstash - Filter17
output {elasticsearch_http {host => “my_es.blablacar.com”port => 9200index => "logstashv1-%{+YYYY.MM.dd}"manage_template => false}}Logstash - Output18
19
syslog20
21
@odolbeau22On recrute !https://speakerdeck.com/odolbeau/logs-hunting