Take care of your logs with ELK

ELK Take care of your logs with @MattKetmo – IPC
Berlin 2015

Matthieu Moquet @MattKetmo web engineer at

Why logs? Debug Find bugs Understand code Execution workflow Metrics
Monitoring

Which logs?

Syslog

Access Logs

Application Logs

How to view logs?

ssh example.org tail -‐f /var/log/some.log

multitail colortail rainbow grc …

You should stop doing this!

✦ Not easily readable ✦ No aggregations ✦ Bad performances
on concurrent processes

Log messages > Dataviz

Send your logs AWAY

Open Source

Elasticsearch Logstash Kibana

3 independent softwares Flexible Performant Scalable

« Elasticsearch is a search server based on Lucene. It
provides a distributed, multitenant-capable full- text search engine with a RESTful web interface and schema-free JSON documents » — Wikipedia elasticsearch

elasticsearch Aggregations Real-Time Data Distributed RESTful API Schema-Free High Availability

elasticsearch Download at https://www.elastic.co/downloads/elasticsearch bin/elasticsearch

elasticsearch See it running… curl -‐X GET http://localhost:9200/

elasticsearch Index a document curl -‐XPUT http://localhost:9200/users/user/mattketmo -‐d '  {
  "name": "Matthieu Moquet",  "age": 26  }'

elasticsearch Get a document curl -‐XGET http://localhost:9200/users/user/mattketmo

elasticsearch Use aggregations curl -‐XPOST "http://localhost:9200/users/user/_search" -‐d' {
"size": 0, "aggregations": { "age_avg": { "avg": { "field": "age" } } } }'

Logstash Log Processing / ETL Input Output Filters

echo "[ALERT] Some error message"   | sed -‐e
"s/^\[$.*$\] $.*$/\1,\2/"   > output.csv ! cat output.csv ALERT,Some error message Logstash

Logstash gelf syslog nginx logs varnish logs udp … date
dns geoip grok urldecode … elasticsearch redis graphite nagios zabbix … Inputs Filters Outputs

Logstash Download at https://www.elastic.co/downloads/logstash bin/logstash -‐e ' input {
stdin {} } output { stdout {} }'

Logstash Download at https://www.elastic.co/downloads/logstash bin/logstash -‐f /path/to/logstash.conf

Logstash input { ... }
filter { ... } output { elasticsearch_http { host => "elasticsearch.tld" port => 9200 index => "logstash-‐%{+YYYY.MM.dd}" } }

Curator https://github.com/elastic/curator Allow you to remove old indexes (via a
CRON) curator -‐-‐host 10.0.0.2 delete indices \ -‐-‐older-‐than 30 -‐-‐time-‐unit days \ -‐-‐timestring '%Y.%m.%d'

Heka « Heka is an open source stream processing software
system developed by Mozilla. Heka is a “Swiss Army Knife” type tool for data processing. » http://hekad.readthedocs.org

« Fluentd is an open source data collector for unified
logging layer »

Kibana ✦ Data visualization web app ✦ Many graphes (histogram,
pie chart, geo map, …) ✦ Built in HTML / CSS / Javascript

Kibana Download at https://www.elastic.co/downloads/kibana bin/kibana

upstream es_backend { server 127.0.0.1:9200;
keepalive 64; } ! server { listen 80; server_name kibana.tld; root /var/www/kibana; try_files $uri $uri/ index.html @elasticsearch; ! location @elasticsearch { proxy_pass http://es_backend; proxy_read_timeout 90; proxy_redirect off; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header X-‐Real-‐IP $remote_addr; proxy_set_header X-‐Forwarded-‐For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_pass_header Access-‐Control-‐Allow-‐Origin; proxy_pass_header Access-‐Control-‐Allow-‐Methods; proxy_hide_header Access-‐Control-‐Allow-‐Headers; add_header Access-‐Control-‐Allow-‐Headers 'X-‐Requested-‐With, Content-‐Type'; add_header Access-‐Control-‐Allow-‐Credentials true; } } Kibana 3 & nginx

Basic pipeline Indexer

Our current pipeline Shipper + Indexer Broker

Kafka pipeline 1M+ messages per second Logstash >= 1.5.0

How to forward logs?

Forwarding Syslog

RSYSLOG The rocket-fast system for log processing *.* @127.0.0.1:514;RYSLOG_ForwardFormat

Logstash input input { udp {
port => 514 type => syslog } }

Logstash filter filter { if [type] == "syslog"
{ grok { match => ["message", "<%{POSINT:syslog_pri}>% {TIMESTAMP_ISO8601:syslog_timestamp} % {SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\ [%{POSINT:syslog_pid}\])?: % {GREENDYDATA:syslog_message}"] add_field => ["received_at", "%{@timestamp}"] add_field => ["received_from", "%{host}"] add_tag => ["rsyslog"] } } }

Forwarding Access Log

logstash-forwarder A tool to collect logs locally in preparation for
processing elsewhere https://github.com/elastic/logstash-forwarder logstash-‐forwarder -‐config conf.json

logstash-forwarder ... { "paths": [
"/var/log/nginx/access.log" ], "fields": { "type": "nginx-‐access" } } ...

logtail Small Perl script to read HTTP access logs and
send it directly to Redis ! https://github.com/shtouff/logtail

Forwarding Application Logs

Monolog https://github.com/Seldaek/monolog

PSR-3 php-fig.org/psr/psr-3 github.com/php-fig/log

<?php ! namespace Psr\Log; ! interface LoggerInterface
{ public function log($level, $message, array $context = array()); ! // Shortcuts public function emergency($message, array $context = array()); public function alert($message, array $context = array()); public function critical($message, array $context = array()); public function error($message, array $context = array()); public function warning($message, array $context = array()); public function notice($message, array $context = array()); public function info($message, array $context = array()); public function debug($message, array $context = array()); }

<?php ! namespace Psr\Log; ! class LogLevel
{ const EMERGENCY = 'emergency'; const ALERT = 'alert'; const CRITICAL = 'critical'; const ERROR = 'error'; const WARNING = 'warning'; const NOTICE = 'notice'; const INFO = 'info'; const DEBUG = 'debug'; }

Usage: PHP <?php ! use Monolog\Logger; use Monolog\Handler\StreamHandler;
! // Create a log channel $handler = new StreamHandler('path/to/your.log', Logger::WARNING); $log = new Logger('name'); $log-‐>pushHandler($handler); ! // Add records to the log $log-‐>warning('Foo'); $log-‐>error('Bar');

Usage: Symfony monolog: handlers:
main: type: fingers_crossed action_level: warning handler: file file: type: stream level: debug path: /var/log/symfony.log syslog: type: syslog level: error

Usage: Symfony Controller <?php ! namespace Acme\Bundle\AppBundle\Controller; !
use Symfony\Bundle\FrameworkBundle\Controller\Controller; ! class FooController extends Controller { public function barAction() { $this-‐>get('logger')-‐>info('Executing foo bar'); ! return $this-‐>render('@AppBundle/Foo/bar.html.twig'); } }

Usage: Symfony Service use Psr\Log\LoggerInterface; use Psr\Log\NullLogger; !
class AcmeService { protected $foo; protected $logger; ! public function __construct(Foo $foo, LoggerInterface $logger = null) { $this-‐>foo = $foo; $this-‐>logger = $logger ?: new NullLogger(); } } Avoid if (null !== $logger)

Usage: Symfony Service use Psr\Log\LoggerInterface; use Psr\Log\NullLogger; !
class AcmeService { protected $foo; protected $logger; ! public function __construct(Foo $foo) { $this-‐>foo = $foo; $this-‐>logger = new NullLogger(); } ! public function setLogger(LoggerInterface $logger) { $this-‐>logger = $logger; } }

40+ Handlers AmqpHandler ErrorLogHandler FirePHPHandler GelfHandler MailHandler NewRelicHandler NullHandler RedisHandler
StreamHandler SyslogUdpHandler ...

SyslogHandler monolog: handlers:
syslog: type: syslog level: warning Ops problem now!

GelfHandler monolog: handlers:
main: type: fingers_crossed action_level: warning handler: file gelf: type: gelf level: notice publisher: hostname: %logstash_host%

Logstash input input { gelf {
port => 12201 type => gelf } }

Logstash filter filter {}

Logstash output output { elasticsearch {
host => ["127.0.0.1"] port => 9200 index => "logstash-‐%{+YYYY.MM.dd}" } }

Log { "message": "Lorem ipsum", "level":
"200", "level_name": "info", "@timestamp": "1432825193000" }

Wait! There is more…

Channels Useful to group logs by category Make it easy
to filter Use different rules / handlers

Channels - assetic - doctrine - event - php -
(php_error) - profiler - request - … Symfony is shipped with many default channels

Usage: Symfony monolog: channels:
-‐ api -‐ business_domain -‐ super_feature -‐ rabbitmq Create as many as you want

main: type: stream path: /var/log/symfony.log channels: !doctrine doctrine: type: stream path: /var/log/doctrine.log channels: doctrine

default_notice: type: gelf level: NOTICE channels: [request, security] default_info: type: gelf level: INFO channels: [!request, !security]

Usage: Symfony services: my_service:
class: Acme\Class\Name arguments: ["@logger"] tags: -‐ name: monolog.logger channel: acme

Channels Visualize proportions Easy filtering

Context Metadata for your logs

Context interface LoggerInterface { public
function log( $level, $message, array $context = array() ); } Use it! Abuse it!

Context $logger-‐>info('User #42 has logout'); $logger-‐>info('User logout', ['user_id' =>
42]) Instead of building dynamic log messages Use static strings and add data into the context

{ "template": "logstash-‐*",
"settings" : "...", "mappings": { "syslog": "...", "http": "...", "gelf": { "dynamic_templates": [{ "ctxt": { "match": "ctxt_*", "match_mapping_type": "string", "mapping": { "type": "string", "index": "not_analyzed" } } }], "properties" : { "@timestamp" : { "type" : "date", "index" : "not_analyzed" }, "message" : { "type" : "string", "index" : "analyzed" }, "facility" : { "type" : "string", "index" : "not_analyzed" }, "type" : { "type" : "string", "index" : "not_analyzed" }, "ctxt_user_id" : { "type" : "string", "index" : "not_analyzed" } } } } }

Processor Add metadata for ALL your logs

Processors GitProcessor IntrospectionProcessor MemoryPeakUsageProcessor MemoryProcessor MemoryUsageProcessor ProcessIdProcessor PsrLogMessageProcessor TagProcessor UidProcessor
WebProcessor

Processor Current user ID User-Agent Locale Country Code IP Current
route Application name / type Environment (dev / staging / prod) Request UUID API client …

public function __invoke(array $record) {
$env = $this-‐>container-‐>get('kernel')-‐>getEnvironment(); $context = $this-‐>container-‐>get('context'); ! $record['extra']['env'] = $env; $record['extra']['locale'] = $context-‐>getLocale(); $record['extra']['media'] = $context-‐>getMedia(); $record['extra']['cli'] = $context-‐>isCli() ? 1 : 0; ! if ($this-‐>container-‐>isScopeActive('request')) { $request = $this-‐>container-‐>get('request_stack')-‐>getMasterRequest(); ! if ($request-‐>headers-‐>has('X-‐Request-‐Id')) { $requestId = $request-‐>headers-‐>get('X-‐Request-‐Id'); $record['extra']['request_id'] = $requestId; } ! $record['extra']['route'] = $request-‐>get('_route'); $record['extra']['client_ip'] = $request-‐>getClientIp(); } ! if (null !== $user = $userManager-‐>getCurrentUser()) { $record['extra']['user_id'] = $user-‐>getId(); } ! // ... ! return $record; }

Context + Processor $project = new Project(42, 'foobar'); !
$logger-‐>info('Project created', [ 'project' => $project ]); Use processor to pretty format the context

Context + Processor class ProjectProcessor {
public function __invoke(array $records) { if (!isset($records['context']['project'])) { return; } ! $project = $records['context']['project']; if (!$project instanceof Project) { return; } ! $records['context']['project'] = [ 'id' => $project-‐>getId(), 'name' => $project-‐>getName(), ] ! return $records; } }

ELK is very powerful & flexible

Other usages

Business Metrics

Business Metrics http://david.pilato.fr/blog/2015/04/28/exploring-capitaine-train-dataset/

PacketBeat: Performance Management

Suricate: Security https://home.regit.org/tag/suricata/

Graphana: Monitoring

Thank You Slides available at moquet.net/talks/ipc-2015-elk Leave feedbacks at @MattKetmo

Take care of your logs with ELK

Take care of your logs with ELK

More Decks by Matthieu Moquet

Other Decks in Programming

Featured

Transcript