Elastic Stack - What's New in 5.x?

This is the slide set from 8 Feb 2017 Elastic Meetup in Istanbul that briefly goes over some of the new features of Elastic Stack version 5.x.

Transcript

1. 1Selim Seynur - www.seynur.com Elastic 5.x What’s new in Elastic Stack & X-Pack
2. 2 Elastic Cloud Security Monitoring Alerting Graph X-Pack Kibana User Interface Elasticsearch Store, Index, & Analyze Ingest Logstash Beats + Elastic Stack Product Portfolio Reporting
3. 3 Elastic Makes it Easy Data Complex/Diverse Meets Developer Requirements Use Cases Many users/use cases Real-Time Availability Rapid Query Execution Flexible Data Model Schemaless Horizontal Scale Sophisticated Query Language Value/Impact Short/mid/long term Location Machine/Log Files User-Activity Documents Social Revenue Growth Launch new applications, Monetize services; Personalize user experiences Cost Savings/Risk Mgmt Application Search Embedded Search Logging Security Analytics Operational Analytics More … Next generation architecture; Retool existing systems; manage risk and compliance
4. 4 IT Operations Application Management Security Analytics Marketing Insights Business Development Customer Sentiment Website/App Search Internal/Intranet Search URL Search Internal Systems/Applications External Systems/Applications Developers IT/Ops Business Users Solving Many Use Cases Log Analysis Analytics Search Security
5. 5 What’s new in 5.x?
6. 6 elasticsearch
7. 7 Search and Query DSL Changes • Deprecated search_type and queries removed ‒ search_type=count ‒ search_type=scan ‒ filtered (use bool instead) ‒ and (use must instead) ‒ or (use should instead) ‒ missing (use negated exists instead) ‒ …
8. 8 Mapping Changes String Mappings string text analyzed keyword not-analyzed
9. 9 Mapping Changes • Preventing mapping explosion (field mapping limits) ‒ The maximum number of fields in an index is limited to 1000. ‒ The maximum depth for a field is limited to 20. ‒ The maximum number of nested fields in an index is limited to 50. • _ttl and _timestamp cannot be created ‒ You can no longer create indexes with _ttl or _timestamp enabled. ‒ Indexes with them enabled created before 5.0 will continue to work.
10. 10 Reindex API POST _reindex { "source": { "index": "oldindex" }, "dest": { "index": "newindex" }, "script": { "lang": "painless", "inline": "ctx._source.timestamp = ctx._timestamp; ctx._timestamp = null" } }
11. 11 Painless Scripting Language {"name”:”Selim”,”goals":[9,27,15],"assists":[0,0,0]} GET hockey/_search { "query": { "function_score": { "script_score": { "script": { "lang": "painless", "inline": "int total = 0; for (int i = 0; i < doc['goals']. length; ++i) { total += doc['goals'][i]; } return total;” } } } } }
12. 12 Ingest Node • pre-process documents before the actual indexing takes place • intercepts bulk and index requests, applies the transformations, and then passes the documents back to the index or bulk APIs • Ingest is enabled by default on all nodes • To pre-process documents before indexing, you define a pipeline that specifies a series of processors. • The description is a special field to store a helpful description of what the pipeline does. • The processors parameter defines a list of processors to be executed in order.
13. 13 Ingest API Example { "description": "mysql pipeline", "processors": [ { "grok" : { "field" : "message", } "pattern" : "..." }, { "remove" : { "field" : "message" } } ] }
14. 14 kibana
15. 15 UI
16. 16 Console
17. 17 Timelion
18. 18 Easily create bar charts, line and scatter plots, histograms, pie charts, and maps Visualize and Explore
19. 19 logstash Ingest
20. 20 Ingest • Data collection and enrichment; 200+ plugins • Next generation data pipeline; micro-batches, process groups of events ES-Hadoop • Platform to build lightweight, data shippers • Forward host-based metrics and any data to Elasticsearch • Two-way connector to integrate with HDFS, Spark, MapReduce, etc. • Enable real-time search queries on Hadoop data
21. 21 Collect, Enrich, and Transport
22. 22 Lightweight Data Shippers Library for forwarding host- based metrics to Elasticsearch Libbeat Packetbeat Metricbeat Real-time network packet analytics for web, database, and any network protocols Gather resource utilization data such as CPU, memory, and other pre-process/system data Next-generation Logstash forwarder to collect, pre-process, and forward log files. Filebeat Winlogbeat {Future}beats System, application, and security information from Window event logs Growing list of beats from the community including http, Redis, Nginx, Docker, Twitter, etc
23. 23 x-pack
24. 24 X-Pack: A Single Extension Security (Shield) Monitor (Marvel) Alerting (Watcher) Graph X-pack Report
26. 26 Prelert
27. 27 Try It https://github.com/xeraa/vagrant-elastic-stack/
28. 28 Website: www.elastic.co @elastic Meetup: https://www.meetup.com/Turkey-Elastic-Fantastics/ Credits: Philipp Krenn @xeraa Selim Seynur @selimseynur Thank You