Indexing and Parsing Documents into Elasticsearch with Ingest

Transcript

1. ‹#› (re)indexing and parsing documents within Elasticsearch @talevy

2. www.elastic.co Copyright Elasticsearch BV 2016 Copying, publishing and/or distributing without

   written permission is strictly prohibited 2 The Elastic Stack

3. Really Quick Intro: Elasticsearch 3 POST /myindex/_search { "query": {

   "bool": { "must": { "match": { "title": "elasticsearch" } }, "filter": { "range": { "price": { "lt": 40 } } } } } } POST /myindex/books/1 {"title": "Elasticsearch in Action", "price": 35.69} POST /myindex/books/2 {"title": "Elasticsearch: The Definitive Guide", "price": 39.90} POST /myindex/books/3 {"title": "Relevant Search", "price": 39.61} index query

4. Really Quick Intro: Logstash 4 input { kafka { topics

   => [“my-topic”] } } filter { mutate { remove => [ “field-a” ] } } output { elasticsearch { index => “my-kafka-index” } } Pipeline

5. Really Quick Intro: Beats 5

6. Really Quick Intro: Kibana 6

7. ‹#› Why ingest node? Parsing Indexed Documents

8. ‹#› I just want to tail a file.

9. Logstash: collect, enrich & transport 9 grok date mutate input

   output Filters The file Elasticsearch

10. Logstash common setup 10 127.0.0.1 - - [19/Apr/2016:12:00:00 +0200] "GET

    /robots.txt HTTP/1.1" 200 68 127.0.0.1 - - [19/Apr/2016:12:00:01 +0200] "GET /cgi-bin/try/ HTTP/1.1" 200 3395 127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] "GET /not_found/ HTTP/1.1" 404 7218 127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:15 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:18 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:00 +0200] "GET /robots.txt HTTP/1.1" 200 68 127.0.0.1 - - [19/Apr/2016:12:00:01 +0200] "GET /cgi-bin/try/ HTTP/1.1" 200 3395 127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] "GET /not_found/ HTTP/1.1" 404 7218 127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:15 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:18 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:00 +0200] "GET /robots.txt HTTP/1.1" 200 68 127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] "GET /not_found/ HTTP/1.1" 404 7218 127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:15 +0200] "GET / HTTP/1.1" 200 24 message

11. Ingest node setup 11 127.0.0.1 - - [19/Apr/2016:12:00:00 +0200] "GET

    /robots.txt HTTP/1.1" 200 68 127.0.0.1 - - [19/Apr/2016:12:00:01 +0200] "GET /cgi-bin/try/ HTTP/1.1" 200 3395 127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] "GET /not_found/ HTTP/1.1" 404 7218 127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:15 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:18 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:00 +0200] "GET /robots.txt HTTP/1.1" 200 68 127.0.0.1 - - [19/Apr/2016:12:00:01 +0200] "GET /cgi-bin/try/ HTTP/1.1" 200 3395 127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] "GET /not_found/ HTTP/1.1" 404 7218 127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:15 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:18 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:00 +0200] "GET /robots.txt HTTP/1.1" 200 68 127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] "GET /not_found/ HTTP/1.1" 404 7218 127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 127.0.0.1 - - [19/Apr/2016:12:00:15 +0200] "GET / HTTP/1.1" 200 24

12. Filebeat: collect and ship 12 127.0.0.1 - - [19/Apr/2016:12:00:04 +0200]

    "GET / HTTP/1.1" 200 24 127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] "GET /not_found/ HTTP/1.1" 404 7218 127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] "GET /favicon.ico HTTP/1.1" 200 3638 { "message" : "127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] \"GET / HTTP/1.1\" 200 24" } { "message" : "127.0.0.1 - - [19/Apr/2016:12:00:07 +0200] \"GET /not_found/ HTTP/1.1\" 404 7218" } { "message" : "127.0.0.1 - - [19/Apr/2016:12:00:09 +2000] \"GET /favicon.ico HTTP/1.1\" 200 3638" }

13. Elasticsearch: parse and index 13 { "message" : "127.0.0.1 -

    - [19/Apr/2016:12:00:04 +0200] \"GET / HTTP/1.1\" 200 24" } { "request" : "/", "auth" : "-", "ident" : "-", "verb" : "GET", "@timestamp" : "2016-04-19T10:00:04.000Z", "response" : "200", "bytes" : "24", "clientip" : "127.0.0.1", "httpversion" : "1.1", "rawrequest" : null, "timestamp" : "19/Apr/2016:12:00:04 +0200" }

14. ‹#› How does ingest node work?

15. Ingest pipeline 15 Pipeline: a set of processors grok date

    remove document enriched document

16. Define a pipeline PUT /_ingest/pipeline/apache-log { "processors" : [ {

    "grok" : { "field": "message", "pattern": "%{COMMONAPACHELOG}" } }, { "date" : { "match_field" : "timestamp", "match_formats" : ["dd/MMM/YYYY:HH:mm:ss Z"] } }, { "remove" : { "field" : "message" } } ] } 16

17. Index a document Provide the id of the pipeline to

    execute PUT /logs/apache/1?pipeline=apache-log { "message" : "127.0.0.1 - - [19/Apr/2016:12:00:04 +0200] \"GET / HTTP/1.1\" 200 24" } 17

18. GET /logs/apache/1 { "request" : "/", "auth" : "-", "ident"

    : "-", "verb" : "GET", "@timestamp" : "2016-04-19T10:00:04.000Z", "response" : "200", "bytes" : "24", "clientip" : "127.0.0.1", "httpversion" : "1.1", "rawrequest" : null, "timestamp" : "19/Apr/2016:12:00:04 +0200" } What has actually been indexed 18

19. PUT /_ingest/pipeline/apache-log { … } GET /_ingest/pipeline/apache-log GET /_ingest/pipeline/* DELETE

    /_ingest/pipeline/apache-log Pipeline management Create, Read, Update & Delete 19

20. 20 grok remove attachment convert uppercase foreach trim append gsub

    set split fail geoip join lowercase rename date

21. Extracts structured fields out of a single text field 21

    Grok processor { "grok": { "field": "message", "pattern": "%{DATE:date}" } }

22. set, remove, rename, convert, gsub, split, join, lowercase, uppercase, trim,

    append 22 Mutate processors { "remove": { "field": "message" } }

23. Parses a date from a string 23 Date processor {

    "date": { "field": "timestamp", "match_formats": ["YYYY"] } }

24. Adds information about the geographical location of IP addresses 24

    Geoip processor { "geoip": { "field": "ip" } }

25. parses structured documents like PDF, doc, etc. 25 Attachment Processor

    { "attachment": { "field": “my-doc-field“ } }

26. entity recognition and stores the output in the JSON before

    it is being stored. 26 Open NLP Processor { "opennlp": { "field": “my-field" } } { "my_field" : "Kobe Bryant was one of the best basketball players of all times. Not even Michael Jordan has ever scored 81 points in one game. Munich is really an awesome city, but New York is as well. Yesterday has been the hottest day of the year.", "entities" : { "locations" : [ "Munich", "New York" ], "dates" : [ "Yesterday" ], "names" : [ "Kobe Bryant", "Michael Jordan" ] } }

27. Introducing new processors is as easy as writing a plugin

    https://www.elastic.co/blog/ writing-your-own-ingest- processor-for-elasticsearch 27 Plugins { "your_plugin": { … } }

28. ‹#› Error handling

29. 29 grok date remove { "message" : "127.0.0.1 - -

    [19/Apr/2016:12:00:00 +040] \"GET / HTTP/1.1\" 200 24" }

30. 30 grok date remove { "message" : "127.0.0.1 - -

    [19/Apr/2016:12:00:00 +040] \"GET / HTTP/1.1\" 200 24" } 400 Bad Request unable to parse date [19/Apr/2016:12:00:00 +040]

31. 31 { "message" : "127.0.0.1 - - [19/Apr/2016:12:00:00 +040] \"GET

    / HTTP/1.1\" 200 24" } grok date remove set on failure processors at the pipeline level

32. 32 { "message" : "127.0.0.1 - - [19/Apr/2016:12:00:00 +040] \"GET

    / HTTP/1.1\" 200 24" } remove 200 OK grok date set on failure processors at the pipeline level

33. 33 { "message" : "127.0.0.1 - - [19/Apr/2016:12:00:00 +040] \"GET

    / HTTP/1.1\" 200 24" } grok date remove set on failure processors at the processor level remove

34. 34 { "message" : "127.0.0.1 - - [19/Apr/2016:12:00:00 +040] \"GET

    / HTTP/1.1\" 200 24" } grok date remove set remove 200 OK on failure processors at the processor level

35. ‹#› Ingest node internals

36. cluster Default scenario 36 Client node1 logs 2P logs 3R

    CS node2 logs 3P logs 1R CS node3 logs 1P logs 2R CS Cluster State logs index: 3 primary shards, 1 replica each All nodes are equal: - node.data: true - node.master: true - node.ingest: true

37. cluster Default scenario 37 Client node1 logs 2P logs 3R

    CS node2 logs 3P logs 1R CS node3 logs 1P logs 2R CS Pre-processing on the coordinating node All nodes are equal: - node.data: true - node.master: true - node.ingest: true index request for shard 3

38. cluster Default scenario 38 Client node1 logs 2P logs 3R

    CS node2 logs 3P logs 1R CS node3 logs 1P logs 2R CS Indexing on the primary shard All nodes are equal: - node.data: true - node.master: true - node.ingest: true index request for shard 3

39. cluster Default scenario 39 Client node1 logs 2P logs 3R

    CS node2 logs 3P logs 1R CS node3 logs 1P logs 2R CS Indexing on the replica shard All nodes are equal: - node.data: true - node.master: true - node.ingest: true index request for shard 3

40. cluster Ingest dedicated nodes 40 Client node1 logs 2P logs

    3R CS node2 logs 3P logs 1R CS node3 logs 1P logs 2R CS node4 CS node5 CS node.data: false node.master: false node.ingest: true node.data: true node.master: true node.ingest: false

41. cluster Ingest dedicated nodes 41 Client node1 logs 2P logs

    3R CS node2 logs 3P logs 1R CS node3 logs 1P logs 2R CS node4 CS node5 CS index request for shard 3 Forward request to an ingest node

42. cluster Ingest dedicated nodes 42 Client node1 logs 2P logs

    3R CS node2 logs 3P logs 1R CS node3 logs 1P logs 2R CS node4 CS node5 CS index request for shard 3 Pre-processing on the ingest node

43. cluster Ingest dedicated nodes 43 Client node1 logs 2P logs

    3R CS node2 logs 3P logs 1R CS node3 logs 1P logs 2R CS node4 CS node5 CS index request for shard 3 Indexing on the primary shard

44. cluster Ingest dedicated nodes 44 Client node1 logs 2P logs

    3R CS node2 logs 3P logs 1R CS node3 logs 1P logs 2R CS node4 CS node5 CS index request for shard 3 Indexing on the replica shard

45. ‹#› Where can ingest pipelines be used?

46. 46 Beats output.elasticsearch.pipeline: ‘%{[fields.example]}’ output.elasticsearch.pipelines: - pipeline: 'ok-pipeline' when.range: http.code:

    [200, 299] - pipeline: 'verybad-pipeline' when.range: http.code: [500, 999] - pipeline: 'default-pipeline'

47. 47 Index API PUT /logs/apache/1?pipeline=apache-log { "message" : "…" }

48. 48 Bulk api PUT /logs/_bulk { "index": { "_type": "apache",

    "_id": "1", "pipeline": "apache-log" } }\n { "message" : "…" }\n { "index": {"_type": "mysql", "_id": "1", "pipeline": "mysql-log" } }\n { "message" : "…" }\n

49. Scan/scroll & bulk indexing made easy 49 Reindex API POST

    /_reindex { "source": { "index": "logs", "type": "apache" }, "dest": { "index": "apache-logs", "pipeline" : "apache-log" } }

50. Great way to upgrade if you have the hardware for

    two 50 Reindex From Remote POST /_reindex { "source": { "index": "logs", "type": "apache", "remote": { "host": "http://localhost:9200" } }, "dest": { "index": "apache-logs", "pipeline" : "apache-log" } }

51. ‹#› https://www.elastic.co/products https://writequit.org/org/es/presentations/ whats-new-elasticsearch-5.0.html https://www.elastic.co/blog/elastic-stack-5-0-0- released Go Get Elastic Stack

    v5!

52. ‹#› Thank you