Logstash: l’arme secrète pour vaincre le coté obscur de vos logs

Talk given at Devoxx France 2015

http://cfp.devoxx.fr/2015/talk/LMO-2774/Logstash:_l%E2%80%99arme_secrete_pour_vaincre_le_cote_obscur_de_vos_logs

Les logs générés par vos applications, services, systèmes d’exploitations et équipements réseaux (soit toutes les composantes nécessaires au fonctionnement des systèmes d’information) sont typiquement une masse obscure de données non-structurée et décentralisée. La nature non structurée de cette information la rend chaotique et pratiquement incompréhensible.

Grâce à sa flexibilité et sa simplicité, Logstash vous permet de transformer cette information non-structurée en données structurées, indexables et analysables.

Venez découvrir comment l’outil Open Source Logstash peut vous aider à collecter et transformer les streams de données hétérogènes, décentralisées et non-structurées tels que les logs, pour passer du mode réactif de la recherche de la cause à la visualisation et l’analyse pro-active, en temps-réel, de vos données opérationnelles.

Transcript

1. @colinsurprenant #logstash Logstash: l’arme secrète pour vaincre le coté obscur de vos logs @colinsurprenant Elastic, http://elastic.co/ Développeur
2. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash Logstash?
3. @colinsurprenant #logstash Logstash! log lôɡ,läɡ/ noun 1. a part of the trunk or a large branch of a tree that has fallen or been cut off. stash as in mustache mus·tache ˈməәsˌtaSH,məәˈstaSH/Submit noun 1. a strip of hair left to grow above the upper lip.
4. @colinsurprenant #logstash logstash! 66.249.73.185 - - [16/Feb/2014:09:47:54 -0500] "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http:// www.google.com/bot.html)" ELASTICSEARCH REST / JSON log lôɡ,läɡ/ noun 1. event that occur in an operating system or software stash staSH 1. a store of something.
5. @colinsurprenant #logstash logstash Collect, parse and store logs
6. @colinsurprenant #logstash logstash ALL  THE  DATA  +  timestamp  =  log Pas  de  timestamp?     Pas  de  problème!    Nous  en   ajouterons  un  pour  vous.   Qu’est-ce qu’un log?
7. @colinsurprenant #logstash logstash Transport & processing
 of
 streaming / continuous data
8. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash logs? quel problèmes?
9. @colinsurprenant #logstash logs? quel problèmes? 66.249.73.185 - - [16/Feb/2014:09:47:54 -0500] "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" Exception in thread "main" Fooz$FancyPantsException at Fooz.bar(Fooz.java:14) at Fooz.foo(Fooz.java:10) at Fooz.main(Fooz.java:6)
10. 89.96.171.210 - - [13/Sep/2012:02:32:49 -0400] "GET /files/logstash/logstash-1.1.0-monolithic.jar HTTP/1.1" 200 40923996 "-" "Chef Client/0.10.10 (rub 37.57.128.238 - - [13/Sep/2012:02:37:24 -0400] "GET / HTTP/1.1" 200 41687 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent" 199.21.99.109 - - [13/Sep/2012:02:38:12 -0400] "GET /blog/tags/packaging HTTP/1.1" 200 15152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yand 180.76.6.232 - - [13/Sep/2012:02:38:23 -0400] "GET /blog/tags/wrt54gl HTTP/1.1" 200 8867 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.ba 217.227.233.68 - - [13/Sep/2012:02:38:25 -0400] "GET /articles/ssh-security/ HTTP/1.1" 200 16543 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/2 217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 ( 217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/articles/ssh-security/" "Mo 217.227.233.68 - - [13/Sep/2012:02:38:31 -0400] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla 184.73.137.50 - - [13/Sep/2012:02:38:28 -0400] "GET /files/logstash/logstash-1.1.1-monolithic.jar HTTP/1.1" 200 53813805 "-" "Chef Client/0.10.8 (ruby 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/ HTTP/1.1" 200 4483 "http://news.ycombinator.com/item?id=4417660" "Mozilla/5.0 (Windows NT 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/bootstrap/css/bootstrap-responsive.min.css HTTP/1.1" 200 7680 "http://semicomplete.com/kiba 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/css/style.css HTTP/1.1" 200 2715 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/css/jquery.ui.datepicker.css HTTP/1.1" 200 33035 "http://semicomplete.com/kibana/" "Mozilla 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/excanvas.min.js HTTP/1.1" 200 19415 "http://semicomplete.com/kibana/" "Mozilla/5.0 ( 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/bootstrap/css/bootstrap.min.css HTTP/1.1" 200 71463 "http://semicomplete.com/kibana/" "Mozi 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.history.js HTTP/1.1" 200 6466 "http://semicomplete.com/kibana/" "Mozilla/5.0 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/css/jquery-ui-1.8.16.custom.css HTTP/1.1" 200 50829 "http://semicomplete.com/kibana/" "Mozi 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.flot.min.js HTTP/1.1" 200 37554 "http://semicomplete.com/kibana/" "Mozilla/5. 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.flot.selection.min.js HTTP/1.1" 200 3532 "http://semicomplete.com/kibana/" "M 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.smartresize.js HTTP/1.1" 200 1123 "http://semicomplete.com/kibana/" "Mozilla/ 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/iso8601.min.js HTTP/1.1" 200 486 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Win 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/safebase64.js HTTP/1.1" 200 3264 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Win 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.min.js HTTP/1.1" 200 93868 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Wi 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/js/lib/json2.js HTTP/1.1" 200 17541 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/js/ajax.js HTTP/1.1" 200 37584 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery-ui-timepicker-addon.js HTTP/1.1" 200 50902 "http://semicomplete.com/kibana/" 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.ui.datepicker.js HTTP/1.1" 200 76332 "http://semicomplete.com/kibana/" "Mozil 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/bootstrap/js/bootstrap-dropdown.js HTTP/1.1" 200 2558 "http://semicomplete.com/kibana/" "Mo 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/bootstrap/js/bootstrap.js HTTP/1.1" 200 47395 "http://semicomplete.com/kibana/" "Mozilla/5. 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/js/lib/jquery.ui.accordion.js HTTP/1.1" 200 16265 "http://semicomplete.com/kibana/" "Mozill 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery-ui-1.8.16.custom.min.js HTTP/1.1" 200 196695 "http://semicomplete.com/kibana/ 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/loader2.php?page=eyJzZWFyY2giOiIiLCJmaWVsZHMiOlsiIl0sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNDggaG 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/images/logo.png HTTP/1.1" 200 1051 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/css/images/ui-icons_222222_256x240.png HTTP/1.1" 200 4369 "http://semicomplete.com/kibana/" 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/images/kibana_banner.png HTTP/1.1" 200 16930 "http://semicomplete.com/kibana/" "Mozilla/5.0 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/images/feed.png HTTP/1.1" 200 689 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows N 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/images/csv.gif HTTP/1.1" 200 154 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/images/stream.png HTTP/1.1" 200 569 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/images/ajax-loader.gif HTTP/1.1" 200 723 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Wi 24.24.235.59 - - [13/Sep/2012:02:38:47 -0400] "GET /kibana/favicon.ico HTTP/1.1" 200 4286 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 ( 24.24.235.59 - - [13/Sep/2012:02:38:55 -0400] "GET /kibana/loader2.php?page=eyJzZWFyY2giOiIiLCJmaWVsZHMiOlsiIl0sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiYWxsIi 24.24.235.59 - - [13/Sep/2012:02:38:56 -0400] "GET /kibana/loader2.php?page=eyJzZWFyY2giOiIiLCJmaWVsZHMiOlsiIl0sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNDggaG 50.19.56.78 - - [13/Sep/2012:02:38:54 -0400] "GET /files/logstash/logstash-1.1.1-monolithic.jar HTTP/1.1" 200 53813805 "-" "Chef Client/0.10.8 (ruby-1 178.1.226.1 - - [13/Sep/2012:02:39:23 -0400] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 108.166.15.188 - - [13/Sep/2012:02:40:07 -0400] "GET /files/logstash/logstash-1.1.0-monolithic.jar HTTP/1.1" 200 40923996 "-" "Chef Client/0.10.8 (rub 74.125.183.29 - - [13/Sep/2012:02:40:15 -0400] "GET /?flav=rss20 HTTP/1.1" 200 32996 "-" "FeedBurner/1.0 (http://www.FeedBurner.com)" 178.77.103.6 - - [13/Sep/2012:02:40:16 -0400] "GET /blog HTTP/1.1" 200 41691 "-" "Tiny Tiny RSS/1.5.5 (http://tt-rss.org/)" 174.37.213.34 - - [13/Sep/2012:02:40:48 -0400] "GET /blog HTTP/1.1" 200 41691 "-" "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:S
11. @colinsurprenant #logstash logs? quel problèmes? xkcd https://xkcd.com/208/
12. @colinsurprenant #logstash /regex/ (?<a0>(?<a1>(?<a2>\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?| Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b) +(?<a3>(?:(?:0[1-9])|(?:[12] [0-9])|(?:3[01])|[1-9])) (?<a4>(?!<[0-9])(?<a5>(?:2[0123]|[01][0-9])):(?<a6>(?:[0-5][0-9]))(?::(?<a7>(?:(?:[0-5] [0-9]|60)(?:[.,][0-9]+)?)))(?![0-9]))) (?<a8>(?:(?<a9>\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z] [0-9A-Za-z-]{0,62}))*(\.?|\b))|(?<a10>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]| [0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])))) (? <a11>(?<a12>(?:[\w._/%-]+))(?:\[(?<a13>\b(?:[1-9][0-9]*)\b)\])?): (?<a14>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]| [0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]| 2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])):(?<a15>(?:[+-]?(?:[0-9]+))) \[(?<a16>(?<a17>(?:(?:0[1-9])|(?:[12][0-9])| (?:3[01])|[1-9]))/(?<a18>\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?| Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b)/(?<a19>[0-9]+):(?<a20>(?! <[0-9])(?<a21>(?:2[0123]|[01][0-9])):(?<a22>(?:[0-5][0-9]))(?::(?<a23>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?! [0-9])).(?<a24>(?:[+-]?(?:[0-9]+))))\] (?<a25>\S+) (?<a26>\S+)/(?<a27>\S+) (?<a28>(?:[+-]?(?:[0-9]+)))/(? <a29>(?:[+-]?(?:[0-9]+)))/(?<a30>(?:[+-]?(?:[0-9]+)))/(?<a31>(?:[+-]?(?:[0-9]+)))/(?<a32>\S+) (?<a33>(?:[+-]? (?:[0-9]+))) (?<a34>\S+) (?<a35>.*?) (?<a36>.*?) (?<a37>\S+) (?<a38>(?:[+-]?(?:[0-9]+)))/(?<a39>(?:[+-]?(?: [0-9]+)))/(?<a40>(?:[+-]?(?:[0-9]+)))/(?<a41>(?:[+-]?(?:[0-9]+)))/(?<a42>\S+) (?<a43>(?:[+-]?(?:[0-9]+)))/(? <a44>(?:[+-]?(?:[0-9]+))) \{(?<a45>(?<a46>.*?))\} \{(?<a47>(?<a48>.*?))\} "(?<a49>\b\w+\b) (?<a50>(? <a51>(?:/[A-Za-z0-9$.+!*'(){},~:;=#%_-]*)+)(?:(?<a52>\?[A-Za-z0-9$.+!*'(){},~#%&/=:;_-]*))?) HTTP/(? <a53>(?:(?<a54>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))")
13. @colinsurprenant #logstash devenez un clavier humain http://www.designbuzz.com/keyboard-jacket-a-weird-fashion-statement/
14. @colinsurprenant #logstash timestamp 66.249.73.185 - - [16/Feb/2014:09:47:54 -0500] "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" Application Logs tim estam p
15. @colinsurprenant #logstash timestamp Metrics cpu_usage 0.74 1395532585 tim estam p
16. @colinsurprenant #logstash quelle heure est-il? @4000000037c219bf2ef02e94 110429.071055,118 020805 13:51:24 Oct 11 20:21:47 Fri, 21 Nov 2014 09:55:06 -0600 29/Apr/2014:07:05:26 +0000 1304060505
17. @colinsurprenant #logstash problèmes • Formats et protocoles hétérogènes • Textuel, non structuré • Décentralisé
18. @colinsurprenant #logstash C’est de la folie!
19. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash Logstash
20. @colinsurprenant / @dadoonet #elk
21. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash Logstash - Open Source
22. @colinsurprenant #logstash open source github.com/elastic/logstash github.com/logstash-plugins 180 plugins repositories
23. @colinsurprenant #logstash
24. @colinsurprenant #logstash JRuby? • Librairies : ~50k librairies dans Maven • Polyglot: Java, Closure, Scala, Groovy, Scala, … • GC • Ruby == beacoup d’objets • Plusieurs type de GC • One GC to rule them all • One standard memory model • Ruby thread == JVM thread == native thread == parallelism
25. @colinsurprenant #logstash JRuby? • Profiling • VisualVM, YourKit, NetBeans, JXInsight • jruby [--profile | --profile.graph] • JVM command-line profilers • Monitoring • Java Management Extensions (JMX) • jconsole and VisualVM
26. @colinsurprenant #logstash JRuby? Garbage Collection Native JIT Profiled Optimizations Native Threading Tooling Cross Platform JVM
27. @colinsurprenant #logstash JRuby? • Contribuer des plugins • Simple et accessible en Ruby • Utilisation des librairies et languages sur la JVM • Optimisation en Java au besoin
28. @colinsurprenant #logstash Performance • Ruby est un language interprété • JRuby 1.7.x est efficace • génère bytecode si possible et selon threshold d’invocations • profite du JIT de la JVM • JRuby 9k est très prometteur • nouveau backend IR (Intermediate Representation) • génération de bytecode plus efficace • Truffle + Graal
29. @colinsurprenant #logstash Performance
30. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash Logstash - définition
31. @colinsurprenant #logstash Définitions • Event • Pipeline • Plugin (input, filter, output)
32. @colinsurprenant #logstash Event • Objet interne à Logstash • Données externes transformées en Event • Les plugins manipulent et enrichissent les objets Event
33. @colinsurprenant #logstash Logstash Pipeline
34. @colinsurprenant #logstash Logstash Pipeline pipeline à 3 étapes source des logs storage
35. @colinsurprenant #logstash Architecture de plugins • ~180 plugins github.com/logstash-plugins • Infrastructure de tests
 • Input plugins: captures external data+format & transform it to logstash events • Filter plugins: process/transform events • Output plugins: send events to external destination & format
36. @colinsurprenant #logstash Architecture de plugins • Codecs • Codecs are plugins • Optional part of input and output plugins • encode and decode raw data that enter or exit the pipeline • Character encoding/transcoding into UTF-8
37. @colinsurprenant #logstash Inputs
38. @colinsurprenant #logstash Filters advisor, alter, anonymize, checksum, cidr, cipher, clone, collate, csv, date, dns, drop, elapsed, elasticsearch, environment, extractnumbers, fingerprint,gelfify, geoip, grep, grok, grokdiscovery, i18n, json, json_encode, kv, metaevent, metrics, multiline, mutate, noop, prune, punch, railsparallelrequest, range, ruby, sleep, split, sumnumbers, syslog_pri, throttle, translate, unique, urldecode, useragent, uuid, wms, wmts, xml, zeromq
39. @colinsurprenant #logstash Grok Filter Qui aime écrire des /regex/ ? (?<a0>(?<a1>(?<a2>\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?| Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b) +(?<a3>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) (?<a4>(?!<[0-9])(?<a5>(?:2[0123]|[01][0-9])):(?<a6>(?:[0-5][0-9]))(?::(?<a7>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?! [0-9]))) (?<a8>(?:(?<a9>\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b))|(? <a10>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4] [0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])))) (?<a11>(?<a12>(?:[\w._/%-]+))(?:\[(?<a13> \b(?:[1-9][0-9]*)\b)\])?): (?<a14>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9] {1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])):(?<a15>(?:[+-]?(?: [0-9]+))) \[(?<a16>(?<a17>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))/(?<a18>\b(?:Jan(?:uary)?|Feb(?:ruary)?| Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?| Dec(?:ember)?)\b)/(?<a19>[0-9]+):(?<a20>(?!<[0-9])(?<a21>(?:2[0123]|[01][0-9])):(?<a22>(?:[0-5][0-9]))(?::(? <a23>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9])).(?<a24>(?:[+-]?(?:[0-9]+))))\] (?<a25>\S+) (?<a26>\S+)/(?<a27> \S+) (?<a28>(?:[+-]?(?:[0-9]+)))/(?<a29>(?:[+-]?(?:[0-9]+)))/(?<a30>(?:[+-]?(?:[0-9]+)))/(?<a31>(?:[+-]?(?:[0-9]+)))/ (?<a32>\S+) (?<a33>(?:[+-]?(?:[0-9]+))) (?<a34>\S+) (?<a35>.*?) (?<a36>.*?) (?<a37>\S+) (?<a38>(?:[+-]?(?: [0-9]+)))/(?<a39>(?:[+-]?(?:[0-9]+)))/(?<a40>(?:[+-]?(?:[0-9]+)))/(?<a41>(?:[+-]?(?:[0-9]+)))/(?<a42>\S+) (?<a43>(?: [+-]?(?:[0-9]+)))/(?<a44>(?:[+-]?(?:[0-9]+))) \{(?<a45>(?<a46>.*?))\} \{(?<a47>(?<a48>.*?))\} "(?<a49>\b\w+\b) (? <a50>(?<a51>(?:/[A-Za-z0-9$.+!*'(){},~:;=#%_-]*)+)(?:(?<a52>\?[A-Za-z0-9$.+!*'(){},~#%&/=:;_-]*))?) HTTP/(? <a53>(?:(?<a54>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))")
40. @colinsurprenant #logstash Grok Filter • Regex réutilisables • Librairie • ~100 patterns • patterns de base et meta-patterns • Extensible: ajoutez vos propres patterns
41. @colinsurprenant #logstash Grok %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[% {HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/% {NUMBER:httpversion}" %{NUMBER:response:int} (?:-|% {NUMBER:bytes:int}) %{QS:referrer} %{QS:agent} %{COMBINEDAPACHELOG} • Format: %{SYNTAX:fieldname} %{SYNTAX:fieldname:type}
42. @colinsurprenant #logstash Grok Patterns
43. @colinsurprenant #logstash Grok
44. @colinsurprenant #logstash Outputs
45. @colinsurprenant #logstash input { stdin { codec => line } } filter { grok { match => [ "message", "%{COMBINEDAPACHELOG}" ] } date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] } } output { stdout { codec => json_lines } } Example: config
46. @colinsurprenant #logstash echo '66.249.73.185 - - [16/Feb/2014:09:47:54 -0500] "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"' | bin/ logstash -f simple.conf Example: execution
47. @colinsurprenant #logstash { "message": "66.249.73.185 - - [16/Feb/2014:09:47:54 -0500] \"GET / HTTP/1.1\" 200 37932 \"-\" \"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/ bot.html)\"", "@version": "1", "@timestamp": "2014-02-16T14:47:54.000Z", "host": "colin-mbp13r-2.local", "clientip": "66.249.73.185", "ident": "-", "auth": "-", "timestamp": "16/Feb/2014:09:47:54 -0500", "verb": "GET", "request": "/", "httpversion": "1.1", "response": "200", "bytes": "37932", "referrer": "\"-\"", "agent": "\"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/ bot.html)\"" } Example: output
48. @colinsurprenant #logstash output { if [type] == "apache" { if [response] =~ /^5\d\d/ { nagios {…} } else if [response] =~ /^4\d\d/ { elasticsearch {…} } statsd { increment => "apache.%{status}" } } } Alternatives (conditionals)
49. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash Scaling Logstash
50. @colinsurprenant #logstash Pipeline Scaling • La Pipeline de Logstash est multi threads • Chaque étape offre des options pour régler la concurrence • Maximise l’utilisation des multiples cores pour le parrallelisme
51. @colinsurprenant #logstash Étape Input
52. @colinsurprenant #logstash Étape Filter
53. @colinsurprenant #logstash Étape Filter $ bin/logstash -w 2 …
54. @colinsurprenant #logstash Étape Output
55. @colinsurprenant #logstash Pipeline - Queues Internes
56. @colinsurprenant #logstash Pipeline - Backpressure
57. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash Logstash - Le futur
58. @colinsurprenant #logstash Aujourd’hui: Queues volatiles Volatile. Taille fixe. Crash = Perte d’Events.
59. @colinsurprenant #logstash Bientôt: Persistence Persistante. Taille fixe. Aucune perte d’Events.
60. @colinsurprenant #logstash Futur: Qtée variable Persistante. Taille fixe Variable.
61. @colinsurprenant #logstash Aujourd’hui: Architecture elasticsearch Payments   Server Database Web   Server
62. @colinsurprenant #logstash Futur: Architecture simplifiée elasticsearch Payments   Server Database Web   Server … … … … …
63. @colinsurprenant #logstash Aujourd’hui: Erreurs Filters Outputs … … … … … ❌
64. @colinsurprenant #logstash … … … … … Dead Letter Queue Futur: Erreurs …to dead letter input Filters Outputs … … … … … ❌
65. @colinsurprenant #logstash Futur: Logstash cluster Payments   Server Database Web   Server
66. @colinsurprenant #logstash Futur • API • Pas de redémarrage pour les changements de config • Instance simple ou cluster • Metriques internes • Integration plus simple a Java et aux autres languages JVM • Java plugin API
67. @YourTwitterHandle @YourTwitterHandle @colinsurprenant #logstash Q & A Nous recrutons! elastic.co/about/careers Support elastic.co/subscriptions