Gearing Up for Elastic Stack, X-Pack 5.0 Releases

Introductions for new features and breaking changes of Elasticsearch, Logstash, Kibana and Beats version 5.0 releases. Information based on 5.0.0-alpha4 releases and subject to change before the general availability.

Transcript

1. ‹#› Kosho Owa, Solutions Architect, Elastic July 14th, 2016 Gearing Up for 5.0 Elastic Stack, X-Pack 5.0.0-alpha4 releases
2. 2 Elastic Cloud Security X-Pack Kibana User Interface Elasticsearch Store, Index,
 & Analyze Ingest Logstash Beats + Elastic Stack Elastic Stack, X-Pack, and Cloud Alerting Monitoring Reporting Graph
3. What’s New in Elasticsearch 5.0 • Lucene 6 • Ingest node • Painless scripting • Text/keyword to replace strings • Completion suggester v2 • Settings validation • Safety in production • Percolate query • Deleted Index Tombstones 3 • Indexed scripts/templates • Dots in Field Names • Cluster Allocation Explain API • Rollover/Shrink Index API • Delete by Query and more!
4. Ingest Node • The data goes through a pipeline before being indexed • Popular Logstash filters (grok, split, convert and date) are implemented • Filebeat to forward a log file to Elasticsearch directly 4 PUT _ingest/pipeline/my_pipeline_id { "description" : "...", "processors": [ { "grok": { "field": "message", "patterns": ["%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} % {NUMBER:duration}"] }}]} PUT my-index/my-type/my-id?pipeline=my_pipeline_id { "message": "55.3.244.1 GET /index.html 15824 0.043" }
5. Text/Keyword to Replace Strings • The string field datatype has been replaced by the text field for full text analyzed content, and the keyword field for not-analyzed exact string values • The string field to be deprecated in 6.0 • Default string mappings: 5 { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }
6. Safety in Production • Bootstrap Checks Must be Passed in Production Mode - Heap size check - File descriptor check - Memory lock check - Minimum master nodes check - Maximum number of threads check - Maximum size virtual memory check - Maximum map count check - Server JVM check • Development vs Production Mode - No external interface bound - development (default), otherwise production 6 • Docs > Elasticsearch Reference > Setup Elasticsearch > Bootstrap Checks
7. Dots in Field Names • Treat dots in field names as though they were an object • The following two documents are treated as thought they were the same: 7 PUT my_index/my_type/1 { "aaa.bbb.ccc": "some_val", "aaa.ddd": "other_val" } PUT my_index/my_type/2 { "aaa": { "bbb": { "ccc": "some_val" }, "ddd": "other_val" } }
8. Rollover Index API • Rolls an alias over to a new index when the existing index is considered to be too large or too old • Settings, mappings, and aliases can be specified other than using index templates 8 PUT /logs-0001 { "aliases": { "logs_write": {} }} POST logs_write/_rollover { "conditions": { "max_age": "7d", "max_docs": 1000 }, "settings": { "index.number_of_shards": 2 }}
9. Shrink Index API • Shrinks an existing index into a new index with fewer primary shards • The requested number of primary shards in the target index must be a factor of the number of shards in the source index (e.g. 8 primary shards into 4, 2 or 1; 15 primary shards into 5, 3 or 1) 9 POST my_source_index/_shrink/my_target_index { "settings": { "index.number_of_replicas": 1, "index.number_of_shards": 1, "index.codec": "best_compression" }, "aliases": { "my_search_indices": {} } }
10. Delete by Query • Delete-By-Query plugin has been removed and replaced by the Delete By Query API • Performs a deletion on every document that match a query 10 POST twitter/_delete_by_query { "query": { "match": { "message": "some message" } } }
11. Field Data is Disabled in Analyzed Fields • Field data cache is no longer loaded to memory by default • Aggregation does not work in analyzed fields • “fielddata” must be set “true” explicitly when using 11 PUT my_index { "mappings": { "_default_" :{ "properties": { “my_text_field” : { "type": "text", "index": "analyzed", "fielddata": true } } } } }
12. • Features: - Cluster Checkup - Reindex Helper - Deprecation Logging • Works on Elasticsearch 2.3.x only Migration Helper 12 • github > Elasticsearch Migration Helper • Docs Elasticsearch Reference > Breaking changes > Breaking changes in 5.0
13. What’s New in Kibana • Flat plug-in structure • Upload CSV • Sense in Console • X-Pack Integration 13
14. Upload CSV • Import a CSV file into Elasticsearch directory through Kibana up to 1 GB • Tips - Date/time field must look like “2016-07-12T03:47:19.744Z” or “2016-07-12” - Single title and at least 10 data lines are required - No extra spaces between columns - Save in UTF-8 14
15. What’s New in X-Pack • Kibana in Monitoring • Reporting • Security - Users/Roles management - Change password 15
16. Kibana in Monitoring • Keep track of requests, response times, memory usage, and more 16
17. Reporting • Manually export dashboards in PDF • Automatically generate reports by Alerting 17
18. Security • Users, Roles can be managed through Kibana • The user can change own password 18
19. What’s New in Logstash 5.0 • Monitoring API • Kafka 0.9, 0.10 support • Improved Release packages • Settings file • Java event API • Plugin generator • JSON logging • Thread safe Elasticsearch output 19
20. Monitoring API • Provides the following monitoring APIs to retrieve runtime metrics - Plugins API - Node Stats API - Pipeline Stats API - Hot Threads API 20 • Docs > Logstash Reference > Monitoring APIs curl "localhost:9600/?pretty" { "host" : “logstash-host.local", "version" : "5.0.0-alpha4", "http_address" : "127.0.0.1:9600", "build_date" : "2016-06-28T22:22:41+02:00", "build_sha" : "f77ec6c845d864a28862e590dff621c68f0823e4", "build_snapshot" : false }
21. What’s New in Beats • JSON support in Filebeat • Integration with Ingest Node • Packetbeat IP/TCP flows • Kafka output • Winlogbeat improvements • Improved Redis output • Kibana dashboards per Beat • New directory layout • Metricbeat replaces Topbeat 21 • Support for gzip compression in Elasticsearch output • Ignore Symlinks log files
22. Metricbeat • Contains Apache, MongoDB, MySQL, Nginx, Redis, System and Zookeeper modules • Developer Guide available for your own metrics 22 metricbeat.modules: #---------------------------- Apache Status Module --------------------------- - module: apache metricsets: ["status"] enabled: true period: 1s filters: hosts: ["http://127.0.0.1/"] #---------------------------- MySQL Status Module ---------------------------- - module: mysql metricsets: ["status"] enabled: true period: 2s hosts: ["root@tcp(127.0.0.1:3306)/"]
23. Be a Pioneer! www.elastic.co/v5 23