Take care of your logs with ELK

IPC Spring Berlin 2015

Transcript

1. ELK
   Take care of your logs with
   @MattKetmo – IPC Berlin 2015
   

   View Slide

2. Matthieu Moquet
   @MattKetmo
   web engineer at
   

   View Slide

3. Logs
   

   View Slide

4. Why logs?
   Debug
   Find bugs
   Understand code
   Execution workflow
   Metrics
   Monitoring
   

   View Slide

5. Which logs?
   

   View Slide

6. Syslog
   

   View Slide

7. Access Logs
   

   View Slide

8. Application Logs
   

   View Slide

9. How to view logs?
   

   View Slide

10. ssh  example.org  
    tail  -­‐f  /var/log/some.log
    

    View Slide

11. multitail
    colortail
    rainbow
    grc
    …
    

    View Slide

12. You should stop doing this!
    

    View Slide

13. ✦ Not easily readable
    ✦ No aggregations
    ✦ Bad performances on concurrent processes
    

    View Slide

14. Log messages > Dataviz
    

    View Slide

15. Send your logs
    AWAY
    

    View Slide

16. Open Source
    

    View Slide

17. ELK
    

    View Slide

18. Elasticsearch
    Logstash
    Kibana
    

    View Slide

19. 3 independent softwares
    Flexible
    Performant
    Scalable
    

    View Slide

20. « Elasticsearch is a search server based on Lucene.
    It provides a distributed, multitenant-capable full-
    text search engine with a RESTful web interface
    and schema-free JSON documents »
    — Wikipedia
    elasticsearch
    

    View Slide

21. elasticsearch
    Aggregations
    Real-Time Data
    Distributed
    RESTful API
    Schema-Free
    High Availability
    

    View Slide

22. elasticsearch
    Download at https://www.elastic.co/downloads/elasticsearch
    bin/elasticsearch
    

    View Slide

23. elasticsearch
    See it running…
    curl  -­‐X  GET  http://localhost:9200/
    

    View Slide

24. elasticsearch
    Index a document
    curl  -­‐XPUT  http://localhost:9200/users/user/mattketmo  -­‐d  '

    {  

       "name":  "Matthieu  Moquet",

       "age":  26

    }'
    

    View Slide

25. elasticsearch
    Get a document
    curl  -­‐XGET  http://localhost:9200/users/user/mattketmo
    

    View Slide

26. elasticsearch
    Use aggregations
    curl  -­‐XPOST  "http://localhost:9200/users/user/_search"  -­‐d'  
    {  
         "size":  0,    
         "aggregations":  {  
                 "age_avg":  {  
                         "avg":  {  
                                   "field":  "age"  
                           }  
                   }  
           }  
    }'
    

    View Slide

27. Logstash
    Log Processing / ETL
    Input
    Output
    Filters
    

    View Slide

28. echo  "[ALERT]  Some  error  message"  

       |  sed  -­‐e  "s/^\[\(.*\)\]  \(.*\)/\1,\2/"  

       >  output.csv  
    !
    cat  output.csv  
    ALERT,Some  error  message
    Logstash
    

    View Slide

29. Logstash
    gelf
    syslog
    nginx logs
    varnish logs
    udp
    …
    date
    dns
    geoip
    grok
    urldecode
    …
    elasticsearch
    redis
    graphite
    nagios
    zabbix
    …
    Inputs Filters Outputs
    

    View Slide

30. Logstash
    Download at https://www.elastic.co/downloads/logstash
    bin/logstash  -­‐e  '  
    input  {    
       stdin  {}    
    }    
    output  {    
       stdout  {}    
    }'
    

    View Slide

31. Logstash
    Download at https://www.elastic.co/downloads/logstash
    bin/logstash  -­‐f  /path/to/logstash.conf
    

    View Slide

32. Logstash
    input  {    
       ...    
    }  
    filter  {  
       ...  
    }  
    output  {  
       elasticsearch_http  {  
           host  =>  "elasticsearch.tld"  
           port  =>  9200  
           index  =>  "logstash-­‐%{+YYYY.MM.dd}"  
       }  
    }
    

    View Slide

33. Curator
    https://github.com/elastic/curator
    Allow you to remove old indexes (via a CRON)
    curator  -­‐-­‐host  10.0.0.2  delete  indices  \  
         -­‐-­‐older-­‐than  30  -­‐-­‐time-­‐unit  days  \  
         -­‐-­‐timestring  '%Y.%m.%d'
    

    View Slide

34. Heka
    « Heka is an open source stream processing software
    system developed by Mozilla. Heka is a “Swiss Army Knife”
    type tool for data processing. »
    http://hekad.readthedocs.org
    

    View Slide

35. « Fluentd is an open source data collector for unified logging layer »
    

    View Slide

36. Kibana
    ✦ Data visualization web app
    ✦ Many graphes (histogram, pie chart, geo map, …)
    ✦ Built in HTML / CSS / Javascript
    

    View Slide

37. Kibana
    Download at https://www.elastic.co/downloads/kibana
    bin/kibana
    

    View Slide

38. upstream  es_backend  {  
           server  127.0.0.1:9200;  
           keepalive  64;  
    }  
    !
    server  {  
       listen  80;  
       server_name    kibana.tld;  
       root  /var/www/kibana;  
       try_files  $uri  $uri/  index.html  @elasticsearch;  
    !
       location  @elasticsearch  {  
           proxy_pass  http://es_backend;  
           proxy_read_timeout  90;  
           proxy_redirect  off;  
           proxy_http_version  1.1;  
           proxy_set_header  Connection  "";  
           proxy_set_header    X-­‐Real-­‐IP    $remote_addr;  
           proxy_set_header    X-­‐Forwarded-­‐For  $proxy_add_x_forwarded_for;  
           proxy_set_header    Host  $http_host;  
           proxy_pass_header  Access-­‐Control-­‐Allow-­‐Origin;  
           proxy_pass_header  Access-­‐Control-­‐Allow-­‐Methods;  
           proxy_hide_header  Access-­‐Control-­‐Allow-­‐Headers;  
           add_header  Access-­‐Control-­‐Allow-­‐Headers  'X-­‐Requested-­‐With,  Content-­‐Type';  
           add_header  Access-­‐Control-­‐Allow-­‐Credentials  true;  
       }  
    }
    Kibana 3 & nginx
    

    View Slide

39. View Slide

40. Basic pipeline
    Indexer
    

    View Slide

41. Our current pipeline
    Shipper + Indexer
    Broker
    

    View Slide

42. Kafka pipeline
    1M+ messages
    per second
    Logstash >= 1.5.0
    

    View Slide

43. How to forward logs?
    

    View Slide

44. Forwarding
    Syslog
    

    View Slide

45. RSYSLOG
    The rocket-fast system for log processing
    *.*  @127.0.0.1:514;RYSLOG_ForwardFormat
    

    View Slide

46. Logstash input
    input  {    
       udp  {  
           port  =>  514  
           type  =>  syslog  
       }    
    }  
    

    View Slide

47. Logstash filter
    filter  {  
       if  [type]  ==  "syslog"  {  
           grok  {  
               match        =>  ["message",  "<%{POSINT:syslog_pri}>%
    {TIMESTAMP_ISO8601:syslog_timestamp}  %
    {SYSLOGHOST:syslog_hostname}  %{DATA:syslog_program}(?:\
    [%{POSINT:syslog_pid}\])?:  %
    {GREENDYDATA:syslog_message}"]  
               add_field  =>  ["received_at",  "%{@timestamp}"]  
               add_field  =>  ["received_from",  "%{host}"]  
               add_tag      =>  ["rsyslog"]  
           }  
       }  
    }
    

    View Slide

48. Forwarding
    Access Log
    

    View Slide

49. logstash-forwarder
    A tool to collect logs locally in preparation for processing elsewhere
    https://github.com/elastic/logstash-forwarder
    logstash-­‐forwarder  -­‐config  conf.json
    

    View Slide

50. logstash-forwarder
    ...  
    {  
       "paths":  [  
             "/var/log/nginx/access.log"  
       ],  
       "fields":  {  "type":  "nginx-­‐access"  }  
    }  
    ...
    

    View Slide

51. logtail
    Small Perl script to read HTTP access logs
    and send it directly to Redis
    !
    https://github.com/shtouff/logtail
    

    View Slide

52. Forwarding
    Application Logs
    

    View Slide

53. Monolog
    https://github.com/Seldaek/monolog
    

    View Slide

54. PSR-3
    php-fig.org/psr/psr-3
    github.com/php-fig/log
    

    View Slide

55. !
    namespace  Psr\Log;  
    !
    interface  LoggerInterface  
    {  
           public  function  log($level,  $message,  array  $context  =  array());  
    !
           //  Shortcuts  
           public  function  emergency($message,  array  $context  =  array());  
           public  function  alert($message,  array  $context  =  array());  
           public  function  critical($message,  array  $context  =  array());  
           public  function  error($message,  array  $context  =  array());  
           public  function  warning($message,  array  $context  =  array());  
           public  function  notice($message,  array  $context  =  array());  
           public  function  info($message,  array  $context  =  array());  
           public  function  debug($message,  array  $context  =  array());  
    }
    

    View Slide

56. !
    namespace  Psr\Log;  
    !
    class  LogLevel  
    {  
           const  EMERGENCY  =  'emergency';  
           const  ALERT  =  'alert';  
           const  CRITICAL  =  'critical';  
           const  ERROR  =  'error';  
           const  WARNING  =  'warning';  
           const  NOTICE  =  'notice';  
           const  INFO  =  'info';  
           const  DEBUG  =  'debug';  
    }
    

    View Slide

57. Usage: PHP
    !
    use  Monolog\Logger;  
    use  Monolog\Handler\StreamHandler;  
    !
    //  Create  a  log  channel  
    $handler  =  new  StreamHandler('path/to/your.log',  Logger::WARNING);  
    $log  =  new  Logger('name');  
    $log-­‐>pushHandler($handler);  
    !
    //  Add  records  to  the  log  
    $log-­‐>warning('Foo');  
    $log-­‐>error('Bar');
    

    View Slide

58. Usage: Symfony
    monolog:  
           handlers:  
                   main:  
                           type:  fingers_crossed  
                           action_level:  warning  
                           handler:  file  
                   file:  
                           type:  stream  
                           level:  debug  
                           path:  /var/log/symfony.log  
                   syslog:  
                           type:  syslog  
                           level:  error
    

    View Slide

59. Usage: Symfony Controller
    !
    namespace  Acme\Bundle\AppBundle\Controller;  
    !
    use  Symfony\Bundle\FrameworkBundle\Controller\Controller;  
    !
    class  FooController  extends  Controller  
    {  
           public  function  barAction()  
           {  
                   $this-­‐>get('logger')-­‐>info('Executing  foo  bar');  
    !
                   return  $this-­‐>render('@AppBundle/Foo/bar.html.twig');  
           }  
    }
    

    View Slide

60. Usage: Symfony Service
    use  Psr\Log\LoggerInterface;  
    use  Psr\Log\NullLogger;  
    !
    class  AcmeService  
    {  
           protected  $foo;  
           protected  $logger;  
    !
           public  function  __construct(Foo  $foo,  LoggerInterface  $logger  =  null)  
           {  
                   $this-­‐>foo  =  $foo;  
                   $this-­‐>logger  =  $logger  ?:  new  NullLogger();  
           }  
    }
    Avoid if  (null  !==  $logger)
    

    View Slide

61. Usage: Symfony Service
    use  Psr\Log\LoggerInterface;  
    use  Psr\Log\NullLogger;  
    !
    class  AcmeService  
    {  
           protected  $foo;  
           protected  $logger;  
    !
           public  function  __construct(Foo  $foo)  
           {  
                   $this-­‐>foo  =  $foo;  
                   $this-­‐>logger  =  new  NullLogger();  
           }  
    !
           public  function  setLogger(LoggerInterface  $logger)  
           {  
                   $this-­‐>logger  =  $logger;  
           }  
    }
    

    View Slide

62. 40+ Handlers
    AmqpHandler
    ErrorLogHandler
    FirePHPHandler
    GelfHandler
    MailHandler
    NewRelicHandler
    NullHandler
    RedisHandler
    StreamHandler
    SyslogUdpHandler
    ...
    

    View Slide

63. SyslogHandler
    monolog:  
       handlers:  
             syslog:  
                   type:  syslog  
                   level:  warning
    Ops problem now!
    

    View Slide

64. GelfHandler
    monolog:  
           handlers:  
                   main:  
                           type:  fingers_crossed  
                           action_level:  warning  
                           handler:  file  
                   gelf:  
                           type:  gelf  
                           level:  notice  
                           publisher:  
                                   hostname:  %logstash_host%
    

    View Slide

65. Logstash input
    input  {  
       gelf  {  
           port  =>  12201  
           type  =>  gelf  
       }  
    }
    

    View Slide

66. Logstash filter
    filter  {}
    

    View Slide

67. Logstash output
    output  {  
       elasticsearch  {  
           host  =>  ["127.0.0.1"]  
           port    =>  9200  
           index  =>  "logstash-­‐%{+YYYY.MM.dd}"  
       }  
    }
    

    View Slide

68. Log
    {  
       "message":  "Lorem  ipsum",  
       "level":  "200",  
       "level_name":  "info",  
       "@timestamp":  "1432825193000"  
    }
    

    View Slide

69. View Slide

70. Wait! There is more…
    

    View Slide

71. Channels
    Useful to group logs by category
    Make it easy to filter
    Use different rules / handlers
    

    View Slide

72. Channels
    - assetic
    - doctrine
    - event
    - php
    - (php_error)
    - profiler
    - request
    - …
    Symfony is shipped with many default channels
    

    View Slide

73. Usage: Symfony
    monolog:  
           channels:  
                   -­‐  api  
                   -­‐  business_domain  
                   -­‐  super_feature  
                   -­‐  rabbitmq
    Create as many as you want
    

    View Slide

74. Usage: Symfony
    monolog:  
           handlers:  
                   main:  
                           type:  stream  
                           path:  /var/log/symfony.log  
                           channels:  !doctrine  
                   doctrine:  
                           type:  stream  
                           path:  /var/log/doctrine.log  
                           channels:  doctrine
    

    View Slide

75. Usage: Symfony
    monolog:  
           handlers:  
               default_notice:  
                       type:  gelf  
                       level:  NOTICE  
                       channels:  [request,  security]  
               default_info:  
                       type:  gelf  
                       level:  INFO  
                       channels:  [!request,  !security]
    

    View Slide

76. Usage: Symfony
    services:  
           my_service:  
                   class:  Acme\Class\Name  
                   arguments:  ["@logger"]  
                   tags:  
                       -­‐  name:  monolog.logger  
                           channel:  acme
    

    View Slide

77. Channels
    Visualize proportions
    Easy filtering
    

    View Slide

78. Context
    Metadata for your logs
    

    View Slide

79. Context
    interface  LoggerInterface  
    {  
           public  function  log(  
                   $level,    
                   $message,    
                   array  $context  =  array()  
           );  
    }
    Use it! Abuse it!
    

    View Slide

80. Context
    $logger-­‐>info('User  #42  has  logout');
    $logger-­‐>info('User    logout',  ['user_id'  =>  42])
    Instead of building dynamic log messages
    Use static strings and add data into the context
    

    View Slide

81. {  
           "template":  "logstash-­‐*",  
           "settings"  :  "...",  
           "mappings":  {  
                   "syslog":  "...",  
                   "http":  "...",  
                   "gelf":  {  
                           "dynamic_templates":  [{  
                                   "ctxt":  {  
                                           "match":  "ctxt_*",  
                                           "match_mapping_type":  "string",  
                                           "mapping":  {  
                                                   "type":  "string",  
                                                   "index":  "not_analyzed"  
                                           }  
                                   }  
                           }],  
                           "properties"  :  {  
                                   "@timestamp"  :  {  "type"  :  "date",  "index"  :  "not_analyzed"  },  
                                   "message"  :  {  "type"  :  "string",  "index"  :  "analyzed"  },  
                                   "facility"  :  {  "type"  :  "string",  "index"  :  "not_analyzed"  },  
                                   "type"  :  {  "type"  :  "string",  "index"  :  "not_analyzed"  },  
                                   "ctxt_user_id"  :  {  "type"  :  "string",  "index"  :  "not_analyzed"  }  
                           }  
                   }  
           }  
    }
    

    View Slide

82. Processor
    Add metadata for ALL your logs
    

    View Slide

83. Processors
    GitProcessor
    IntrospectionProcessor
    MemoryPeakUsageProcessor
    MemoryProcessor
    MemoryUsageProcessor
    ProcessIdProcessor
    PsrLogMessageProcessor
    TagProcessor
    UidProcessor
    WebProcessor
    

    View Slide

84. Processor
    Current user ID
    User-Agent
    Locale
    Country Code
    IP
    Current route
    Application name / type
    Environment (dev / staging / prod)
    Request UUID
    API client
    …
    

    View Slide

85. public  function  __invoke(array  $record)  
    {  
           $env          =  $this-­‐>container-­‐>get('kernel')-­‐>getEnvironment();  
           $context  =  $this-­‐>container-­‐>get('context');  
    !
           $record['extra']['env']          =  $env;  
           $record['extra']['locale']    =  $context-­‐>getLocale();  
           $record['extra']['media']      =  $context-­‐>getMedia();  
           $record['extra']['cli']          =  $context-­‐>isCli()  ?  1  :  0;  
    !
           if  ($this-­‐>container-­‐>isScopeActive('request'))  {  
                   $request  =  $this-­‐>container-­‐>get('request_stack')-­‐>getMasterRequest();  
    !
                   if  ($request-­‐>headers-­‐>has('X-­‐Request-­‐Id'))  {  
                           $requestId  =  $request-­‐>headers-­‐>get('X-­‐Request-­‐Id');  
                           $record['extra']['request_id']  =  $requestId;  
                   }  
    !
                   $record['extra']['route']  =  $request-­‐>get('_route');  
                   $record['extra']['client_ip']  =  $request-­‐>getClientIp();  
           }  
    !
           if  (null  !==  $user  =  $userManager-­‐>getCurrentUser())  {  
                   $record['extra']['user_id']  =  $user-­‐>getId();  
           }  
    !
           //  ...  
    !
           return  $record;  
    }
    

    View Slide

86. View Slide

87. Context + Processor
    $project  =  new  Project(42,  'foobar');  
    !
    $logger-­‐>info('Project  created',  [  
           'project'  =>  $project  
    ]);  
    Use processor to pretty format the context
    

    View Slide

88. Context + Processor
    class  ProjectProcessor    
    {  
           public  function  __invoke(array  $records)  
           {  
                   if  (!isset($records['context']['project']))  {  
                           return;  
                   }  
    !
                   $project  =  $records['context']['project'];  
                   if  (!$project  instanceof  Project)  {  
                           return;  
                   }  
    !
                   $records['context']['project']  =  [  
                           'id'      =>  $project-­‐>getId(),  
                           'name'  =>  $project-­‐>getName(),  
                   ]  
    !
                   return  $records;  
           }  
    }
    

    View Slide

89. ELK is very powerful & flexible
    

    View Slide

90. Other usages
    

    View Slide

91. Business Metrics
    

    View Slide

92. Business Metrics
    http://david.pilato.fr/blog/2015/04/28/exploring-capitaine-train-dataset/
    

    View Slide

93. PacketBeat: Performance Management
    

    View Slide

94. Suricate: Security
    https://home.regit.org/tag/suricata/
    

    View Slide

95. Graphana: Monitoring
    

    View Slide

96. Thank You
    Slides available at
    moquet.net/talks/ipc-2015-elk
    Leave feedbacks at @MattKetmo
    

    View Slide