Devoxx FR : Ingest Node - (re)indexing and enriching your documents within Elasticsearch

Lorsque vous injectez des données dans elasticsearch, vous pouvez avoir besoin de réaliser des opérations de transformation assez simples. Jusqu'à présent, ces opérations devaient s'effectuer en dehors d'elasticsearch, avant l'indexation proprement dite.

Souhaitez la bienvenue à Ingest node ! Un nouveau type de noeud qui vous permet justement de faire cela.

Ce talk explique le concept de Ingest Node, comment l'intégrer avec le reste de la suite logicielle Elastic et comment développer son propre plugin Ingest par la pratique en montrant comment j'ai développé le plugin ingest-bano pour enrichir des adresses postales et/ou des coordonnées géographiques françaises (pour l'instant).

Ce talk parlera également de l'API de réindexation qui peut également bénéficier du pipeline d'ingestion pour modifier vos données à la volée lors de la réindexation. Notamment, comment relire vos données existantes et les enrichir grace à ingest-bano.

Talk donné à l'occasion de Devoxx France 2017

Transcript

1. #DevoxxFR #elasticsearch (re)indexer et enrichir ses documents avec Elasticsearch David Pilato Elasticsearch Ingest Node in action
3. infom ercial
4. infom ercial Scannez votre badge pour tenter de gagner
5. #DevoxxFR #elasticsearch 5 Pourquoi Ingest Node ?
6. ‹#› I just want to tail a file.
7. #DevoxxFR #elasticsearch Logstash common setup 7 127.0.0.1 - - [19/Apr/2016:12:00:00 +0200] "GET /robots.txt HTTP/1.1" 200 68 127.0.0.1 - - [19/Apr/2016:12:00:01 +0200] "GET /cgi-bin/try/ HTTP/1.1" 200 3395 127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] "GET /not_found/ HTTP/1.1" 404 7218 127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:15 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:18 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:00 +0200] "GET /robots.txt HTTP/1.1" 200 68 127.0.0.1 - - [19/Apr/2016:12:00:01 +0200] "GET /cgi-bin/try/ HTTP/1.1" 200 3395 127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] "GET /not_found/ HTTP/1.1" 404 7218 127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:15 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:18 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:00 +0200] "GET /robots.txt HTTP/1.1" 200 68 127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] "GET /not_found/ HTTP/1.1" 404 7218 127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:15 +0200] "GET / HTTP/1.1" 200 24 message
8. #DevoxxFR #elasticsearch Ingest node setup 8 127.0.0.1 - - [19/Apr/2016:12:00:00 +0200] "GET /robots.txt HTTP/1.1" 200 68 127.0.0.1 - - [19/Apr/2016:12:00:01 +0200] "GET /cgi-bin/try/ HTTP/1.1" 200 3395 127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] "GET /not_found/ HTTP/1.1" 404 7218 127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:15 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:18 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:00 +0200] "GET /robots.txt HTTP/1.1" 200 68 127.0.0.1 - - [19/Apr/2016:12:00:01 +0200] "GET /cgi-bin/try/ HTTP/1.1" 200 3395 127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] "GET /not_found/ HTTP/1.1" 404 7218 127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:15 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:18 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:00 +0200] "GET /robots.txt HTTP/1.1" 200 68 127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] "GET /not_found/ HTTP/1.1" 404 7218 127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:15 +0200] "GET / HTTP/1.1" 200 24
9. #DevoxxFR #elasticsearch Filebeat: collect and ship 9 127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] "GET /not_found/ HTTP/1.1" 404 7218 127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 { "message" : "127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] \"GET / HTTP/1.1\" 200 24" } { "message" : "127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] \"GET /not_found/ HTTP/1.1\" 404 7218" } { "message" : "127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] \"GET /favicon.ico HTTP/1.1\" 200 3638" }
10. #DevoxxFR #elasticsearch Elasticsearch: enrich and index 10 { "message" : "127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] \"GET / HTTP/1.1\" 200 24" } { "request" : "/", "auth" : "-", "ident" : "-", "verb" : "GET", "@timestamp" : "2016-04-19T10:00:04.000Z", "response" : 200, "bytes" : 24, "clientip" : "127.0.0.1", "httpversion" : "1.1" }
11. #DevoxxFR #elasticsearch From logstash filters… 11 grok date mutate input output Filters The file Elasticsearch
12. #DevoxxFR #elasticsearch …to an Ingest pipeline 12 grok date remove doc doc Pipeline: a set of processors
13. #DevoxxFR #elasticsearch Utilisation des pipelines 13 PUT /_ingest/pipeline/apache-log { ... } PUT /my_index/my_type/my_id?pipeline=apache-log { "message" : "..." }
14. #DevoxxFR #elasticsearch 14 ingest-bano plugin From postal address to geo_point From geo_point to postal address
15. #DevoxxFR #elasticsearch BANO = Base d'Adresses Nationale 15 http://openstreetmap.fr/bano 
 http://bano.openstreetmap.fr/data/ 976030950H-26,26,RUE DISMA,97660,Bandrélé,CAD,-12.891701,45.202652
 976030950H-28,28,RUE DISMA,97660,Bandrélé,CAD,-12.891900,45.202700
 976030950H-30,30,RUE DISMA,97660,Bandrélé,CAD,-12.891781,45.202535
 976030950H-32,32,RUE DISMA,97660,Bandrélé,CAD,-12.892005,45.202564
 976030950H-3,3,RUE DISMA,97660,Bandrélé,CAD,-12.892444,45.202135
 976030950H-34,34,RUE DISMA,97660,Bandrélé,CAD,-12.892068,45.202450
 976030950H-4,4,RUE DISMA,97660,Bandrélé,CAD,-12.892446,45.202367
 976030950H-5,5,RUE DISMA,97660,Bandrélé,CAD,-12.892461,45.202248
 976030950H-6,6,RUE DISMA,97660,Bandrélé,CAD,-12.892383,45.202456
 976030950H-8,8,RUE DISMA,97660,Bandrélé,CAD,-12.892300,45.202555
 976030950H-9,9,RUE DISMA,97660,Bandrélé,CAD,-12.892355,45.202387 976030951J-103,103,RTE NATIONALE 3,97660,Bandrélé,CAD,-12.893639,45.201696 \_ ID | \_ Street Name | \ \_ Source \_ Geo point | | \ |_ Street Number |_ Zipcode \_ City Name
16. #DevoxxFR #elasticsearch in action Elasticsearch Ingest Node 16
17. #DevoxxFR #elasticsearch Merci @ Devoxx FR Team A suivre sur : https://github.com/dadoonet et http://david.pilato.fr/ 17 2012 2013 2014 2015 2016 2017