Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Logstash: l’arme secrète pour vaincre le coté obscur de vos logs

Logstash: l’arme secrète pour vaincre le coté obscur de vos logs

Talk given at Devoxx France 2015

http://cfp.devoxx.fr/2015/talk/LMO-2774/Logstash:_l%E2%80%99arme_secrete_pour_vaincre_le_cote_obscur_de_vos_logs

Les logs générés par vos applications, services, systèmes d’exploitations et équipements réseaux (soit toutes les composantes nécessaires au fonctionnement des systèmes d’information) sont typiquement une masse obscure de données non-structurée et décentralisée. La nature non structurée de cette information la rend chaotique et pratiquement incompréhensible.

Grâce à sa flexibilité et sa simplicité, Logstash vous permet de transformer cette information non-structurée en données structurées, indexables et analysables.

Venez découvrir comment l’outil Open Source Logstash peut vous aider à collecter et transformer les streams de données hétérogènes, décentralisées et non-structurées tels que les logs, pour passer du mode réactif de la recherche de la cause à la visualisation et l’analyse pro-active, en temps-réel, de vos données opérationnelles.

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

April 09, 2015
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. @colinsurprenant #logstash Logstash: l’arme secrète pour vaincre le coté obscur

    de vos logs @colinsurprenant Elastic, http://elastic.co/ Développeur
  2. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash Logstash?

  3. @colinsurprenant #logstash Logstash! log lôɡ,läɡ/ noun 1. a part of

    the trunk or a large branch of a tree that has fallen or been cut off. stash as in mustache mus·tache ˈməәsˌtaSH,məәˈstaSH/Submit noun 1. a strip of hair left to grow above the upper lip.
  4. @colinsurprenant #logstash logstash! 66.249.73.185 - - [16/Feb/2014:09:47:54 -0500] "GET /

    HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http:// www.google.com/bot.html)" ELASTICSEARCH REST / JSON log lôɡ,läɡ/ noun 1. event that occur in an operating system or software stash staSH 1. a store of something.
  5. @colinsurprenant #logstash logstash Collect, parse and store logs

  6. @colinsurprenant #logstash logstash ALL  THE  DATA  +  timestamp  =  log

    Pas  de  timestamp?     Pas  de  problème!    Nous  en   ajouterons  un  pour  vous.   Qu’est-ce qu’un log?
  7. @colinsurprenant #logstash logstash Transport & processing
 of
 streaming / continuous

    data
  8. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash logs? quel problèmes?

  9. @colinsurprenant #logstash logs? quel problèmes? 66.249.73.185 - - [16/Feb/2014:09:47:54 -0500]

    "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" Exception in thread "main" Fooz$FancyPantsException at Fooz.bar(Fooz.java:14) at Fooz.foo(Fooz.java:10) at Fooz.main(Fooz.java:6)
  10. 89.96.171.210 - - [13/Sep/2012:02:32:49 -0400] "GET /files/logstash/logstash-1.1.0-monolithic.jar HTTP/1.1" 200 40923996

    "-" "Chef Client/0.10.10 (rub 37.57.128.238 - - [13/Sep/2012:02:37:24 -0400] "GET / HTTP/1.1" 200 41687 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent" 199.21.99.109 - - [13/Sep/2012:02:38:12 -0400] "GET /blog/tags/packaging HTTP/1.1" 200 15152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yand 180.76.6.232 - - [13/Sep/2012:02:38:23 -0400] "GET /blog/tags/wrt54gl HTTP/1.1" 200 8867 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.ba 217.227.233.68 - - [13/Sep/2012:02:38:25 -0400] "GET /articles/ssh-security/ HTTP/1.1" 200 16543 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/2 217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 ( 217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/articles/ssh-security/" "Mo 217.227.233.68 - - [13/Sep/2012:02:38:31 -0400] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla 184.73.137.50 - - [13/Sep/2012:02:38:28 -0400] "GET /files/logstash/logstash-1.1.1-monolithic.jar HTTP/1.1" 200 53813805 "-" "Chef Client/0.10.8 (ruby 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/ HTTP/1.1" 200 4483 "http://news.ycombinator.com/item?id=4417660" "Mozilla/5.0 (Windows NT 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/bootstrap/css/bootstrap-responsive.min.css HTTP/1.1" 200 7680 "http://semicomplete.com/kiba 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/css/style.css HTTP/1.1" 200 2715 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/css/jquery.ui.datepicker.css HTTP/1.1" 200 33035 "http://semicomplete.com/kibana/" "Mozilla 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/excanvas.min.js HTTP/1.1" 200 19415 "http://semicomplete.com/kibana/" "Mozilla/5.0 ( 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/bootstrap/css/bootstrap.min.css HTTP/1.1" 200 71463 "http://semicomplete.com/kibana/" "Mozi 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.history.js HTTP/1.1" 200 6466 "http://semicomplete.com/kibana/" "Mozilla/5.0 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/css/jquery-ui-1.8.16.custom.css HTTP/1.1" 200 50829 "http://semicomplete.com/kibana/" "Mozi 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.flot.min.js HTTP/1.1" 200 37554 "http://semicomplete.com/kibana/" "Mozilla/5. 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.flot.selection.min.js HTTP/1.1" 200 3532 "http://semicomplete.com/kibana/" "M 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.smartresize.js HTTP/1.1" 200 1123 "http://semicomplete.com/kibana/" "Mozilla/ 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/iso8601.min.js HTTP/1.1" 200 486 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Win 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/safebase64.js HTTP/1.1" 200 3264 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Win 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.min.js HTTP/1.1" 200 93868 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Wi 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/js/lib/json2.js HTTP/1.1" 200 17541 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/js/ajax.js HTTP/1.1" 200 37584 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery-ui-timepicker-addon.js HTTP/1.1" 200 50902 "http://semicomplete.com/kibana/" 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.ui.datepicker.js HTTP/1.1" 200 76332 "http://semicomplete.com/kibana/" "Mozil 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/bootstrap/js/bootstrap-dropdown.js HTTP/1.1" 200 2558 "http://semicomplete.com/kibana/" "Mo 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/bootstrap/js/bootstrap.js HTTP/1.1" 200 47395 "http://semicomplete.com/kibana/" "Mozilla/5. 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/js/lib/jquery.ui.accordion.js HTTP/1.1" 200 16265 "http://semicomplete.com/kibana/" "Mozill 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery-ui-1.8.16.custom.min.js HTTP/1.1" 200 196695 "http://semicomplete.com/kibana/ 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/loader2.php?page=eyJzZWFyY2giOiIiLCJmaWVsZHMiOlsiIl0sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNDggaG 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/images/logo.png HTTP/1.1" 200 1051 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/css/images/ui-icons_222222_256x240.png HTTP/1.1" 200 4369 "http://semicomplete.com/kibana/" 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/images/kibana_banner.png HTTP/1.1" 200 16930 "http://semicomplete.com/kibana/" "Mozilla/5.0 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/images/feed.png HTTP/1.1" 200 689 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows N 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/images/csv.gif HTTP/1.1" 200 154 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/images/stream.png HTTP/1.1" 200 569 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/images/ajax-loader.gif HTTP/1.1" 200 723 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Wi 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/favicon.ico HTTP/1.1" 200 4286 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 ( 24.24.235.59 - - [13/Sep/2012:02:38:55 -0400] "GET /kibana/loader2.php?page=eyJzZWFyY2giOiIiLCJmaWVsZHMiOlsiIl0sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiYWxsIi 24.24.235.59 - - [13/Sep/2012:02:38:56 -0400] "GET /kibana/loader2.php?page=eyJzZWFyY2giOiIiLCJmaWVsZHMiOlsiIl0sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNDggaG 50.19.56.78 - - [13/Sep/2012:02:38:54 -0400] "GET /files/logstash/logstash-1.1.1-monolithic.jar HTTP/1.1" 200 53813805 "-" "Chef Client/0.10.8 (ruby-1 178.1.226.1 - - [13/Sep/2012:02:39:23 -0400] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 108.166.15.188 - - [13/Sep/2012:02:40:07 -0400] "GET /files/logstash/logstash-1.1.0-monolithic.jar HTTP/1.1" 200 40923996 "-" "Chef Client/0.10.8 (rub 74.125.183.29 - - [13/Sep/2012:02:40:15 -0400] "GET /?flav=rss20 HTTP/1.1" 200 32996 "-" "FeedBurner/1.0 (http://www.FeedBurner.com)" 178.77.103.6 - - [13/Sep/2012:02:40:16 -0400] "GET /blog HTTP/1.1" 200 41691 "-" "Tiny Tiny RSS/1.5.5 (http://tt-rss.org/)" 174.37.213.34 - - [13/Sep/2012:02:40:48 -0400] "GET /blog HTTP/1.1" 200 41691 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:S
  11. @colinsurprenant #logstash logs? quel problèmes? xkcd https://xkcd.com/208/

  12. @colinsurprenant #logstash /regex/ (?<a0>(?<a1>(?<a2>\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?| Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b) +(?<a3>(?:(?:0[1-9])|(?:[12] [0-9])|(?:3[01])|[1-9])) (?<a4>(?!<[0-9])(?<a5>(?:2[0123]|[01][0-9])):(?<a6>(?:[0-5][0-9]))(?::(?<a7>(?:(?:[0-5] [0-9]|60)(?:[.,][0-9]+)?)))(?![0-9]))) (?<a8>(?:(?<a9>\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z]

    [0-9A-Za-z-]{0,62}))*(\.?|\b))|(?<a10>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]| [0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])))) (? <a11>(?<a12>(?:[\w._/%-]+))(?:\[(?<a13>\b(?:[1-9][0-9]*)\b)\])?): (?<a14>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]| [0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]| 2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])):(?<a15>(?:[+-]?(?:[0-9]+))) \[(?<a16>(?<a17>(?:(?:0[1-9])|(?:[12][0-9])| (?:3[01])|[1-9]))/(?<a18>\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?| Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b)/(?<a19>[0-9]+):(?<a20>(?! <[0-9])(?<a21>(?:2[0123]|[01][0-9])):(?<a22>(?:[0-5][0-9]))(?::(?<a23>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?! [0-9])).(?<a24>(?:[+-]?(?:[0-9]+))))\] (?<a25>\S+) (?<a26>\S+)/(?<a27>\S+) (?<a28>(?:[+-]?(?:[0-9]+)))/(? <a29>(?:[+-]?(?:[0-9]+)))/(?<a30>(?:[+-]?(?:[0-9]+)))/(?<a31>(?:[+-]?(?:[0-9]+)))/(?<a32>\S+) (?<a33>(?:[+-]? (?:[0-9]+))) (?<a34>\S+) (?<a35>.*?) (?<a36>.*?) (?<a37>\S+) (?<a38>(?:[+-]?(?:[0-9]+)))/(?<a39>(?:[+-]?(?: [0-9]+)))/(?<a40>(?:[+-]?(?:[0-9]+)))/(?<a41>(?:[+-]?(?:[0-9]+)))/(?<a42>\S+) (?<a43>(?:[+-]?(?:[0-9]+)))/(? <a44>(?:[+-]?(?:[0-9]+))) \{(?<a45>(?<a46>.*?))\} \{(?<a47>(?<a48>.*?))\} "(?<a49>\b\w+\b) (?<a50>(? <a51>(?:/[A-Za-z0-9$.+!*'(){},~:;=#%_-]*)+)(?:(?<a52>\?[A-Za-z0-9$.+!*'(){},~#%&/=:;_-]*))?) HTTP/(? <a53>(?:(?<a54>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))")
  13. @colinsurprenant #logstash devenez un clavier humain http://www.designbuzz.com/keyboard-jacket-a-weird-fashion-statement/

  14. @colinsurprenant #logstash timestamp 66.249.73.185 - - [16/Feb/2014:09:47:54 -0500] "GET /

    HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" Application Logs tim estam p
  15. @colinsurprenant #logstash timestamp Metrics cpu_usage 0.74 1395532585 tim estam p

  16. @colinsurprenant #logstash quelle heure est-il? @4000000037c219bf2ef02e94 110429.071055,118 020805 13:51:24 Oct

    11 20:21:47 Fri, 21 Nov 2014 09:55:06 -0600 29/Apr/2014:07:05:26 +0000 1304060505
  17. @colinsurprenant #logstash problèmes • Formats et protocoles hétérogènes • Textuel,

    non structuré • Décentralisé
  18. @colinsurprenant #logstash C’est de la folie!

  19. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash Logstash

  20. @colinsurprenant / @dadoonet #elk

  21. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash Logstash - Open Source

  22. @colinsurprenant #logstash open source github.com/elastic/logstash github.com/logstash-plugins 180 plugins repositories

  23. @colinsurprenant #logstash

  24. @colinsurprenant #logstash JRuby? • Librairies : ~50k librairies dans Maven

    • Polyglot: Java, Closure, Scala, Groovy, Scala, … • GC • Ruby == beacoup d’objets • Plusieurs type de GC • One GC to rule them all • One standard memory model • Ruby thread == JVM thread == native thread == parallelism
  25. @colinsurprenant #logstash JRuby? • Profiling • VisualVM, YourKit, NetBeans, JXInsight

    • jruby [--profile | --profile.graph] • JVM command-line profilers • Monitoring • Java Management Extensions (JMX) • jconsole and VisualVM
  26. @colinsurprenant #logstash JRuby? Garbage Collection Native JIT Profiled Optimizations Native

    Threading Tooling Cross Platform JVM
  27. @colinsurprenant #logstash JRuby? • Contribuer des plugins • Simple et

    accessible en Ruby • Utilisation des librairies et languages sur la JVM • Optimisation en Java au besoin
  28. @colinsurprenant #logstash Performance • Ruby est un language interprété •

    JRuby 1.7.x est efficace • génère bytecode si possible et selon threshold d’invocations • profite du JIT de la JVM • JRuby 9k est très prometteur • nouveau backend IR (Intermediate Representation) • génération de bytecode plus efficace • Truffle + Graal
  29. @colinsurprenant #logstash Performance

  30. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash Logstash - définition

  31. @colinsurprenant #logstash Définitions • Event • Pipeline • Plugin (input,

    filter, output)
  32. @colinsurprenant #logstash Event • Objet interne à Logstash • Données

    externes transformées en Event • Les plugins manipulent et enrichissent les objets Event
  33. @colinsurprenant #logstash Logstash Pipeline

  34. @colinsurprenant #logstash Logstash Pipeline pipeline à 3 étapes source des

    logs storage
  35. @colinsurprenant #logstash Architecture de plugins • ~180 plugins github.com/logstash-plugins •

    Infrastructure de tests
 • Input plugins: captures external data+format & transform it to logstash events • Filter plugins: process/transform events • Output plugins: send events to external destination & format
  36. @colinsurprenant #logstash Architecture de plugins • Codecs • Codecs are

    plugins • Optional part of input and output plugins • encode and decode raw data that enter or exit the pipeline • Character encoding/transcoding into UTF-8
  37. @colinsurprenant #logstash Inputs

  38. @colinsurprenant #logstash Filters advisor, alter, anonymize, checksum, cidr, cipher, clone,

    collate, csv, date, dns, drop, elapsed, elasticsearch, environment, extractnumbers, fingerprint,gelfify, geoip, grep, grok, grokdiscovery, i18n, json, json_encode, kv, metaevent, metrics, multiline, mutate, noop, prune, punch, railsparallelrequest, range, ruby, sleep, split, sumnumbers, syslog_pri, throttle, translate, unique, urldecode, useragent, uuid, wms, wmts, xml, zeromq
  39. @colinsurprenant #logstash Grok Filter Qui aime écrire des /regex/ ?

    (?<a0>(?<a1>(?<a2>\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?| Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b) +(?<a3>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) (?<a4>(?!<[0-9])(?<a5>(?:2[0123]|[01][0-9])):(?<a6>(?:[0-5][0-9]))(?::(?<a7>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?! [0-9]))) (?<a8>(?:(?<a9>\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b))|(? <a10>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4] [0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])))) (?<a11>(?<a12>(?:[\w._/%-]+))(?:\[(?<a13> \b(?:[1-9][0-9]*)\b)\])?): (?<a14>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9] {1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])):(?<a15>(?:[+-]?(?: [0-9]+))) \[(?<a16>(?<a17>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))/(?<a18>\b(?:Jan(?:uary)?|Feb(?:ruary)?| Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?| Dec(?:ember)?)\b)/(?<a19>[0-9]+):(?<a20>(?!<[0-9])(?<a21>(?:2[0123]|[01][0-9])):(?<a22>(?:[0-5][0-9]))(?::(? <a23>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9])).(?<a24>(?:[+-]?(?:[0-9]+))))\] (?<a25>\S+) (?<a26>\S+)/(?<a27> \S+) (?<a28>(?:[+-]?(?:[0-9]+)))/(?<a29>(?:[+-]?(?:[0-9]+)))/(?<a30>(?:[+-]?(?:[0-9]+)))/(?<a31>(?:[+-]?(?:[0-9]+)))/ (?<a32>\S+) (?<a33>(?:[+-]?(?:[0-9]+))) (?<a34>\S+) (?<a35>.*?) (?<a36>.*?) (?<a37>\S+) (?<a38>(?:[+-]?(?: [0-9]+)))/(?<a39>(?:[+-]?(?:[0-9]+)))/(?<a40>(?:[+-]?(?:[0-9]+)))/(?<a41>(?:[+-]?(?:[0-9]+)))/(?<a42>\S+) (?<a43>(?: [+-]?(?:[0-9]+)))/(?<a44>(?:[+-]?(?:[0-9]+))) \{(?<a45>(?<a46>.*?))\} \{(?<a47>(?<a48>.*?))\} "(?<a49>\b\w+\b) (? <a50>(?<a51>(?:/[A-Za-z0-9$.+!*'(){},~:;=#%_-]*)+)(?:(?<a52>\?[A-Za-z0-9$.+!*'(){},~#%&/=:;_-]*))?) HTTP/(? <a53>(?:(?<a54>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))")
  40. @colinsurprenant #logstash Grok Filter • Regex réutilisables • Librairie •

    ~100 patterns • patterns de base et meta-patterns • Extensible: ajoutez vos propres patterns
  41. @colinsurprenant #logstash Grok %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[% {HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request}

    HTTP/% {NUMBER:httpversion}" %{NUMBER:response:int} (?:-|% {NUMBER:bytes:int}) %{QS:referrer} %{QS:agent} %{COMBINEDAPACHELOG} • Format: %{SYNTAX:fieldname} %{SYNTAX:fieldname:type}
  42. @colinsurprenant #logstash Grok Patterns

  43. @colinsurprenant #logstash Grok

  44. @colinsurprenant #logstash Outputs

  45. @colinsurprenant #logstash input { stdin { codec => line }

    } filter { grok { match => [ "message", "%{COMBINEDAPACHELOG}" ] } date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] } } output { stdout { codec => json_lines } } Example: config
  46. @colinsurprenant #logstash echo '66.249.73.185 - - [16/Feb/2014:09:47:54 -0500] "GET /

    HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"' | bin/ logstash -f simple.conf Example: execution
  47. @colinsurprenant #logstash { "message": "66.249.73.185 - - [16/Feb/2014:09:47:54 -0500] \"GET

    / HTTP/1.1\" 200 37932 \"-\" \"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/ bot.html)\"", "@version": "1", "@timestamp": "2014-02-16T14:47:54.000Z", "host": "colin-mbp13r-2.local", "clientip": "66.249.73.185", "ident": "-", "auth": "-", "timestamp": "16/Feb/2014:09:47:54 -0500", "verb": "GET", "request": "/", "httpversion": "1.1", "response": "200", "bytes": "37932", "referrer": "\"-\"", "agent": "\"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/ bot.html)\"" } Example: output
  48. @colinsurprenant #logstash output { if [type] == "apache" { if

    [response] =~ /^5\d\d/ { nagios {…} } else if [response] =~ /^4\d\d/ { elasticsearch {…} } statsd { increment => "apache.%{status}" } } } Alternatives (conditionals)
  49. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash Scaling Logstash

  50. @colinsurprenant #logstash Pipeline Scaling • La Pipeline de Logstash est

    multi threads • Chaque étape offre des options pour régler la concurrence • Maximise l’utilisation des multiples cores pour le parrallelisme
  51. @colinsurprenant #logstash Étape Input

  52. @colinsurprenant #logstash Étape Filter

  53. @colinsurprenant #logstash Étape Filter $ bin/logstash -w 2 …

  54. @colinsurprenant #logstash Étape Output

  55. @colinsurprenant #logstash Pipeline - Queues Internes

  56. @colinsurprenant #logstash Pipeline - Backpressure

  57. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash Logstash - Le futur

  58. @colinsurprenant #logstash Aujourd’hui: Queues volatiles Volatile. Taille fixe. Crash =

    Perte d’Events.
  59. @colinsurprenant #logstash Bientôt: Persistence Persistante. Taille fixe. Aucune perte d’Events.

  60. @colinsurprenant #logstash Futur: Qtée variable Persistante. Taille fixe Variable.

  61. @colinsurprenant #logstash Aujourd’hui: Architecture elasticsearch Payments   Server Database Web

      Server
  62. @colinsurprenant #logstash Futur: Architecture simplifiée elasticsearch Payments   Server Database

    Web   Server … … … … …
  63. @colinsurprenant #logstash Aujourd’hui: Erreurs Filters Outputs … … … …

    … ❌
  64. @colinsurprenant #logstash … … … … … Dead Letter Queue

    Futur: Erreurs …to dead letter input Filters Outputs … … … … … ❌
  65. @colinsurprenant #logstash Futur: Logstash cluster Payments   Server Database Web

      Server
  66. @colinsurprenant #logstash Futur • API • Pas de redémarrage pour

    les changements de config • Instance simple ou cluster • Metriques internes • Integration plus simple a Java et aux autres languages JVM • Java plugin API
  67. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash Q & A Nous recrutons! elastic.co/about/careers

    Support elastic.co/subscriptions