La Suite ELK au service des développeurs et de la production

La Suite ELK au service des développeurs et de la production

Comment la Suite ELK facilite la centralisation et la corrélation d'indicateurs hétérogènes, pour une meilleure prise de décision.

7d1caf9df777b3b2cf474ff743494335?s=128

Jérémy Lecour

October 14, 2016
Tweet

Transcript

  1. 4.
  2. 7.
  3. 8.

    metrics • utilisateurs connectés • nombre de commandes • panier

    moyen • état du stock • temps de chargement • taux d’échec • réponse de la BDD • disponibilité des services métier système
  4. 16.

    xeonys-fr-01 HAProxy Apache/FPM A1 Apache/FPM A2 BDD-A Queues Varnish Apache/FPM

    B1 Apache/FPM B2 BDD-B Queues Apache/FPM B3 xeonys-uk HAProxy Varnish Apache/FPM 1 Apache/FPM 2 BDD Queues xeonys-de HAProxy Varnish Apache/FPM 1 Apache/FPM 2 BDD Queues Apache/FPM 3
  5. 27.

    ELK

  6. 31.
  7. 34.

    Logstash Extract Transform Load inputs: beats: port: 5044 gelf: port:

    12201 use_udp: true filters: grok: pattern: httpd date: match: dd-MMM-YYYY HH:mm:ss mutate: convert: [line: integer] geoip: src: remote_ip target: geo
  8. 35.

    Logstash Extract Transform Load inputs: beats: port: 5044 gelf: port:

    12201 use_udp: true filters: grok: pattern: httpd date: match: dd-MMM-YYYY HH:mm:ss mutate: convert: [line: integer] geoip: src: remote_ip target: geo outputs: elasticsearch: host: 127.0.0.1:9200 index: logs-%{type}-%{+YYYY.MM.dd} stdout: codec: rubydebug
  9. 61.

    administration facile paquets à installer très peu de config de

    départ cluster trivial sauvegardes par snapshot
  10. 62.
  11. 63.

    $ curl -XGET 'http://localhost:9200/_search' -d '{ "query": { "bool" :

    { "must" : { "query_string" : { "query" : "some query string here" } }, "filter" : { "term" : { "user" : "kimchy" } } } } } ' documents
  12. 64.

    $ curl -XGET 'http://localhost:9200/_search' -d '{ "query": { "bool" :

    { "must" : { "query_string" : { "query" : "some query string here" } }, "filter" : { "term" : { "user" : "kimchy" } } } } } ' documents { "_shards":{ "total" : 5, "successful" : 5, "failed" : 0 }, "hits":{ "total" : 1, "hits" : [ { "_index" : "twitter", "_type" : "tweet", "_id" : "1", "_source" : { "user" : "kimchy", "postDate" : "2009-11-15T14:12:12", "message" : "hello Elasticsearch" } } ] } }
  13. 66.

    agrégations GET /cars/transactions/_search { "size" : 0, "aggs": { "colors":

    { "terms": { "field": "color" }, "aggs": { "avg_price": { "avg": { "field": "price" } } } } } }
  14. 67.

    agrégations GET /cars/transactions/_search { "size" : 0, "aggs": { "colors":

    { "terms": { "field": "color" }, "aggs": { "avg_price": { "avg": { "field": "price" } } } } } } { ... "aggregations": { "colors": { "buckets": [ { "key": "red", "doc_count": 4, "avg_price": { "value": 32500 } }, { "key": "blue", "doc_count": 2, "avg_price": { "value": 20000 } }, { "key": "green", "doc_count": 2, "avg_price": { "value": 21000 } } ] } } ... }
  15. 73.
  16. 78.
  17. 79.
  18. 80.
  19. 87.

    éléments de corrélation Queues PHP HTTP Proxy date request method

    referrer user-agent host stack date file line class function message severity app_name app_env host stack
  20. 88.

    éléments de corrélation Queues PHP HTTP Proxy date request method

    referrer user-agent host stack date file line class function message severity app_name app_env host stack date request method frontend backend host stack
  21. 89.

    éléments de corrélation Queues PHP HTTP Proxy date request method

    referrer user-agent host stack date file line class function message severity app_name app_env host stack date request method frontend backend host stack date message severity app_name app_env host stack
  22. 94.

    collecte inputs: beats: port: 5044 gelf: port: 12201 use_udp: true

    Logstash + lib/gelf.php Icons by Chameleon Design from the Noun Project
  23. 95.

    collecte inputs: beats: port: 5044 gelf: port: 12201 use_udp: true

    Logstash + lib/gelf.php Icons by Chameleon Design from the Noun Project
  24. 96.

    collecte inputs: beats: port: 5044 gelf: port: 12201 use_udp: true

    Logstash + lib/gelf.php prospectors: paths: /var/log/*.log input_type: log output: logstash: hosts: logstash:5044 Filebeat Icons by Chameleon Design from the Noun Project
  25. 97.

    collecte inputs: beats: port: 5044 gelf: port: 12201 use_udp: true

    Logstash + lib/gelf.php prospectors: paths: /var/log/*.log input_type: log output: logstash: hosts: logstash:5044 Filebeat Icons by Chameleon Design from the Noun Project
  26. 98.
  27. 121.

    modèle économique Libre, gratuit • Elasticsearch • Logstash • Kibana

    • Beat • Curator • … Services • formation • assistance • consulting • X-Pack
  28. 122.

    modèle économique Libre, gratuit • Elasticsearch • Logstash • Kibana

    • Beat • Curator • … Services • formation • assistance • consulting • X-Pack Cloud • As A Service • Hébergé
  29. 125.

    forge.evolix.org www.evolix.fr @evolix – @jlecour info@evolix.fr @ wiki.evolix.org Git logo

    from git-scm.org Chain by Bohdan Burmich from the Noun Project Paw Print by Mattijs Dekkers from the Noun Project Twitter logo from twitter.com