Take care of your logs with ELK

by Matthieu Moquet

Slide 1

Slide 1 text

ELK Take care of your logs with @MattKetmo – IPC Berlin 2015

Slide 2

Slide 2 text

Matthieu Moquet @MattKetmo web engineer at

Slide 3

Slide 3 text

Logs

Slide 4

Slide 4 text

Why logs? Debug Find bugs Understand code Execution workflow Metrics Monitoring

Slide 5

Slide 5 text

Which logs?

Slide 6

Slide 6 text

Syslog

Slide 7

Slide 7 text

Access Logs

Slide 8

Slide 8 text

Application Logs

Slide 9

Slide 9 text

How to view logs?

Slide 10

Slide 10 text

ssh example.org tail -‐f /var/log/some.log

Slide 11

Slide 11 text

multitail colortail rainbow grc …

Slide 12

Slide 12 text

You should stop doing this!

Slide 13

Slide 13 text

✦ Not easily readable ✦ No aggregations ✦ Bad performances on concurrent processes

Slide 14

Slide 14 text

Log messages > Dataviz

Slide 15

Slide 15 text

Send your logs AWAY

Slide 16

Slide 16 text

Open Source

Slide 17

Slide 17 text

ELK

Slide 18

Slide 18 text

Elasticsearch Logstash Kibana

Slide 19

Slide 19 text

3 independent softwares Flexible Performant Scalable

Slide 20

Slide 20 text

« Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full- text search engine with a RESTful web interface and schema-free JSON documents » — Wikipedia elasticsearch

Slide 21

Slide 21 text

elasticsearch Aggregations Real-Time Data Distributed RESTful API Schema-Free High Availability

Slide 22

Slide 22 text

elasticsearch Download at https://www.elastic.co/downloads/elasticsearch bin/elasticsearch

Slide 23

Slide 23 text

elasticsearch See it running… curl -‐X GET http://localhost:9200/

Slide 24

Slide 24 text

elasticsearch Index a document curl -‐XPUT http://localhost:9200/users/user/mattketmo -‐d '  {   "name": "Matthieu Moquet",  "age": 26  }'

Slide 25

Slide 25 text

elasticsearch Get a document curl -‐XGET http://localhost:9200/users/user/mattketmo

Slide 26

Slide 26 text

elasticsearch Use aggregations curl -‐XPOST "http://localhost:9200/users/user/_search" -‐d' { "size": 0, "aggregations": { "age_avg": { "avg": { "field": "age" } } } }'

Slide 27

Slide 27 text

Logstash Log Processing / ETL Input Output Filters

Slide 28

Slide 28 text

echo "[ALERT] Some error message"   | sed -‐e "s/^\[$.*$\] $.*$/\1,\2/"   > output.csv ! cat output.csv ALERT,Some error message Logstash

Slide 29

Slide 29 text

Logstash gelf syslog nginx logs varnish logs udp … date dns geoip grok urldecode … elasticsearch redis graphite nagios zabbix … Inputs Filters Outputs

Slide 30

Slide 30 text

Logstash Download at https://www.elastic.co/downloads/logstash bin/logstash -‐e ' input { stdin {} } output { stdout {} }'

Slide 31

Slide 31 text

Logstash Download at https://www.elastic.co/downloads/logstash bin/logstash -‐f /path/to/logstash.conf

Slide 32

Slide 32 text

Logstash input { ... } filter { ... } output { elasticsearch_http { host => "elasticsearch.tld" port => 9200 index => "logstash-‐%{+YYYY.MM.dd}" } }

Slide 33

Slide 33 text

Curator https://github.com/elastic/curator Allow you to remove old indexes (via a CRON) curator -‐-‐host 10.0.0.2 delete indices \ -‐-‐older-‐than 30 -‐-‐time-‐unit days \ -‐-‐timestring '%Y.%m.%d'

Slide 34

Slide 34 text

Heka « Heka is an open source stream processing software system developed by Mozilla. Heka is a “Swiss Army Knife” type tool for data processing. » http://hekad.readthedocs.org

Slide 35

Slide 35 text

« Fluentd is an open source data collector for unified logging layer »

Slide 36

Slide 36 text

Kibana ✦ Data visualization web app ✦ Many graphes (histogram, pie chart, geo map, …) ✦ Built in HTML / CSS / Javascript

Slide 37

Slide 37 text

Kibana Download at https://www.elastic.co/downloads/kibana bin/kibana

Slide 38

Slide 38 text

upstream es_backend { server 127.0.0.1:9200; keepalive 64; } ! server { listen 80; server_name kibana.tld; root /var/www/kibana; try_files $uri $uri/ index.html @elasticsearch; ! location @elasticsearch { proxy_pass http://es_backend; proxy_read_timeout 90; proxy_redirect off; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header X-‐Real-‐IP $remote_addr; proxy_set_header X-‐Forwarded-‐For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_pass_header Access-‐Control-‐Allow-‐Origin; proxy_pass_header Access-‐Control-‐Allow-‐Methods; proxy_hide_header Access-‐Control-‐Allow-‐Headers; add_header Access-‐Control-‐Allow-‐Headers 'X-‐Requested-‐With, Content-‐Type'; add_header Access-‐Control-‐Allow-‐Credentials true; } } Kibana 3 & nginx

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

Basic pipeline Indexer

Slide 41

Slide 41 text

Our current pipeline Shipper + Indexer Broker

Slide 42

Slide 42 text

Kafka pipeline 1M+ messages per second Logstash >= 1.5.0

Slide 43

Slide 43 text

How to forward logs?

Slide 44

Slide 44 text

Forwarding Syslog

Slide 45

Slide 45 text

RSYSLOG The rocket-fast system for log processing *.* @127.0.0.1:514;RYSLOG_ForwardFormat

Slide 46

Slide 46 text

Logstash input input { udp { port => 514 type => syslog } }

Slide 47

Slide 47 text

Logstash filter filter { if [type] == "syslog" { grok { match => ["message", "<%{POSINT:syslog_pri}>% {TIMESTAMP_ISO8601:syslog_timestamp} % {SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\ [%{POSINT:syslog_pid}\])?: % {GREENDYDATA:syslog_message}"] add_field => ["received_at", "%{@timestamp}"] add_field => ["received_from", "%{host}"] add_tag => ["rsyslog"] } } }

Slide 48

Slide 48 text

Forwarding Access Log

Slide 49

Slide 49 text

logstash-forwarder A tool to collect logs locally in preparation for processing elsewhere https://github.com/elastic/logstash-forwarder logstash-‐forwarder -‐config conf.json

Slide 50

Slide 50 text

logstash-forwarder ... { "paths": [ "/var/log/nginx/access.log" ], "fields": { "type": "nginx-‐access" } } ...

Slide 51

Slide 51 text

logtail Small Perl script to read HTTP access logs and send it directly to Redis ! https://github.com/shtouff/logtail

Slide 52

Slide 52 text

Forwarding Application Logs

Slide 53

Slide 53 text

Monolog https://github.com/Seldaek/monolog

Slide 54

Slide 54 text

PSR-3 php-fig.org/psr/psr-3 github.com/php-fig/log

Slide 55

Slide 55 text

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Usage: PHP pushHandler($handler); ! // Add records to the log $log-‐>warning('Foo'); $log-‐>error('Bar');

Slide 58

Slide 58 text

Usage: Symfony monolog: handlers: main: type: fingers_crossed action_level: warning handler: file file: type: stream level: debug path: /var/log/symfony.log syslog: type: syslog level: error

Slide 59

Slide 59 text

Usage: Symfony Controller get('logger')-‐>info('Executing foo bar'); ! return $this-‐>render('@AppBundle/Foo/bar.html.twig'); } }

Slide 60

Slide 60 text

Usage: Symfony Service use Psr\Log\LoggerInterface; use Psr\Log\NullLogger; ! class AcmeService { protected $foo; protected $logger; ! public function __construct(Foo $foo, LoggerInterface $logger = null) { $this-‐>foo = $foo; $this-‐>logger = $logger ?: new NullLogger(); } } Avoid if (null !== $logger)

Slide 61

Slide 61 text

Usage: Symfony Service use Psr\Log\LoggerInterface; use Psr\Log\NullLogger; ! class AcmeService { protected $foo; protected $logger; ! public function __construct(Foo $foo) { $this-‐>foo = $foo; $this-‐>logger = new NullLogger(); } ! public function setLogger(LoggerInterface $logger) { $this-‐>logger = $logger; } }

Slide 62

Slide 62 text

40+ Handlers AmqpHandler ErrorLogHandler FirePHPHandler GelfHandler MailHandler NewRelicHandler NullHandler RedisHandler StreamHandler SyslogUdpHandler ...

Slide 63

Slide 63 text

SyslogHandler monolog: handlers: syslog: type: syslog level: warning Ops problem now!

Slide 64

Slide 64 text

GelfHandler monolog: handlers: main: type: fingers_crossed action_level: warning handler: file gelf: type: gelf level: notice publisher: hostname: %logstash_host%

Slide 65

Slide 65 text

Logstash input input { gelf { port => 12201 type => gelf } }

Slide 66

Slide 66 text

Logstash filter filter {}

Slide 67

Slide 67 text

Logstash output output { elasticsearch { host => ["127.0.0.1"] port => 9200 index => "logstash-‐%{+YYYY.MM.dd}" } }

Slide 68

Slide 68 text

Log { "message": "Lorem ipsum", "level": "200", "level_name": "info", "@timestamp": "1432825193000" }

Slide 69

Slide 69 text

No content

Slide 70

Slide 70 text

Wait! There is more…

Slide 71

Slide 71 text

Channels Useful to group logs by category Make it easy to filter Use different rules / handlers

Slide 72

Slide 72 text

Channels - assetic - doctrine - event - php - (php_error) - profiler - request - … Symfony is shipped with many default channels

Slide 73

Slide 73 text

Usage: Symfony monolog: channels: -‐ api -‐ business_domain -‐ super_feature -‐ rabbitmq Create as many as you want

Slide 74

Slide 74 text

Usage: Symfony monolog: handlers: main: type: stream path: /var/log/symfony.log channels: !doctrine doctrine: type: stream path: /var/log/doctrine.log channels: doctrine

Slide 75

Slide 75 text

Usage: Symfony monolog: handlers: default_notice: type: gelf level: NOTICE channels: [request, security] default_info: type: gelf level: INFO channels: [!request, !security]

Slide 76

Slide 76 text

Usage: Symfony services: my_service: class: Acme\Class\Name arguments: ["@logger"] tags: -‐ name: monolog.logger channel: acme

Slide 77

Slide 77 text

Channels Visualize proportions Easy filtering

Slide 78

Slide 78 text

Context Metadata for your logs

Slide 79

Slide 79 text

Context interface LoggerInterface { public function log( $level, $message, array $context = array() ); } Use it! Abuse it!

Slide 80

Slide 80 text

Context $logger-‐>info('User #42 has logout'); $logger-‐>info('User logout', ['user_id' => 42]) Instead of building dynamic log messages Use static strings and add data into the context

Slide 81

Slide 81 text

{ "template": "logstash-‐*", "settings" : "...", "mappings": { "syslog": "...", "http": "...", "gelf": { "dynamic_templates": [{ "ctxt": { "match": "ctxt_*", "match_mapping_type": "string", "mapping": { "type": "string", "index": "not_analyzed" } } }], "properties" : { "@timestamp" : { "type" : "date", "index" : "not_analyzed" }, "message" : { "type" : "string", "index" : "analyzed" }, "facility" : { "type" : "string", "index" : "not_analyzed" }, "type" : { "type" : "string", "index" : "not_analyzed" }, "ctxt_user_id" : { "type" : "string", "index" : "not_analyzed" } } } } }

Slide 82

Slide 82 text

Processor Add metadata for ALL your logs

Slide 83

Slide 83 text

Processors GitProcessor IntrospectionProcessor MemoryPeakUsageProcessor MemoryProcessor MemoryUsageProcessor ProcessIdProcessor PsrLogMessageProcessor TagProcessor UidProcessor WebProcessor

Slide 84

Slide 84 text

Processor Current user ID User-Agent Locale Country Code IP Current route Application name / type Environment (dev / staging / prod) Request UUID API client …

Slide 85

Slide 85 text

public function __invoke(array $record) { $env = $this-‐>container-‐>get('kernel')-‐>getEnvironment(); $context = $this-‐>container-‐>get('context'); ! $record['extra']['env'] = $env; $record['extra']['locale'] = $context-‐>getLocale(); $record['extra']['media'] = $context-‐>getMedia(); $record['extra']['cli'] = $context-‐>isCli() ? 1 : 0; ! if ($this-‐>container-‐>isScopeActive('request')) { $request = $this-‐>container-‐>get('request_stack')-‐>getMasterRequest(); ! if ($request-‐>headers-‐>has('X-‐Request-‐Id')) { $requestId = $request-‐>headers-‐>get('X-‐Request-‐Id'); $record['extra']['request_id'] = $requestId; } ! $record['extra']['route'] = $request-‐>get('_route'); $record['extra']['client_ip'] = $request-‐>getClientIp(); } ! if (null !== $user = $userManager-‐>getCurrentUser()) { $record['extra']['user_id'] = $user-‐>getId(); } ! // ... ! return $record; }

Slide 86

Slide 86 text

No content

Slide 87

Slide 87 text

Context + Processor $project = new Project(42, 'foobar'); ! $logger-‐>info('Project created', [ 'project' => $project ]); Use processor to pretty format the context

Slide 88

Slide 88 text

Context + Processor class ProjectProcessor { public function __invoke(array $records) { if (!isset($records['context']['project'])) { return; } ! $project = $records['context']['project']; if (!$project instanceof Project) { return; } ! $records['context']['project'] = [ 'id' => $project-‐>getId(), 'name' => $project-‐>getName(), ] ! return $records; } }

Slide 89

Slide 89 text

ELK is very powerful & flexible

Slide 90

Slide 90 text

Other usages

Slide 91

Slide 91 text

Business Metrics

Slide 92

Slide 92 text

Business Metrics http://david.pilato.fr/blog/2015/04/28/exploring-capitaine-train-dataset/

Slide 93

Slide 93 text

PacketBeat: Performance Management

Slide 94

Slide 94 text

Suricate: Security https://home.regit.org/tag/suricata/

Slide 95

Slide 95 text

Graphana: Monitoring

Slide 96

Slide 96 text

Thank You Slides available at moquet.net/talks/ipc-2015-elk Leave feedbacks at @MattKetmo