5.x new features - Elastic Kansas City meetup

Join us for our Fall meetup at TGS Midwest Technology Center in Lenexa. We'll have a talk from Elastic's Adam Quan on new features in the v5.4 and v5.5 releases.

Some of the 5.4 and 5.5 new features include:

- Machine Learning

- Elastic Stack Monitoring Service

- Elasticsearch Windows installer

- Kibana Timer Series Visual Builder

- Kibana Cross Cluster Search

- Logstash Persistent Queues

We will also cover some of the new features in the upcoming 6.0 release, like the new Logstash Pipeline Viewer and the Alert UI etc.

Doors will open at 4pm –– food & beverages will be provided.

Adam Quan is a Solutions Architect at Elastic, where he works with customers on architecting distributed solutions using the Elastic Stack. Adam has 20 years of experience with Java application development and application integration technologies.

https://www.meetup.com/Elastic-Kansas-City-Meetup/events/242546268/

Transcript

1. 1 Adam Quan 5.4/5.5/5.6 New Features
2. 2 Machine learning is now GA!
3. 3 ML Job Time Estimator
4. 4 ML create watch for real-time job
5. 5 ML Jobs Monitoring
6. 6 Elastic Stack Monitoring Service
7. 7 Elastic Stack Monitoring Service • The Elastic support team will create and manage a dedicated cluster to host the customer’s X-Pack monitoring data (retain for 7 days) • Gold/Platinum self-hosted customers are eligible Official announcement on July 6th, 2017
8. 8 Benefit to Elastic - Save time - Customer insights - Help with Sales and renewals - Elastic Cloud awareness Benefit to Customer - Available at no charge - Streamlined support - Simplified workflow and management
14. 14 Type removal • Encouraged for any projects just getting started now • Blog explaining this/the future: https://www.elastic.co/blog/index-type-parent- child-join-now-future-in-elasticsearch • Phase 2 coming with 6.0 Phase 1
15. 15 Other features • ip_range field (like date_range or float_range, etc) • icu_collation_keyword field • index.blocks.read_only_allow_delete: delete a read-only index • Several notable deprecations, preparing for 6.0: file scripts (use stored/inline), native scripts (use ScriptEngine), custom users/groups via rpm/deb (use zip/tar/config management), and others: https://www.elastic.co/guide/en/elasticsearch/reference/5.5/release-notes-5.5.0.html
16. 16 Highlights in Elasticsearch 5.6 • New High-Level Java REST client • Deprecations API • New join data type • Improvements to search scalability
17. 17 High-Level RESTful Java Client • Improve the Elasticsearch developer experience, migrate away from Transport client • Low level REST Java client was difficult to use for developers • Built on top of low level client, abstracting away some of the low level code needed – E.g. Low level REST calls, GET /index-name/_search vs .query() – Request/response objects vs JSON blobs • APIs for: info, get, index, delete, update, bulk, search, search-scroll, and clear-scroll • Other requests can still be handled by the low level REST client
18. 18 High-Level RESTful Java Client Response response = restClient.performRequest("GET", "/posts/_search", params, null, consumerFactory); searchSourceBuilder.query(QueryBuilders.matchAllQuery()); Low level REST client match all query: High-level REST client match all query:
19. 19 Deprecations API • Improve confidence in upgrades! • X-pack feature to find deprecated settings before an upgrade • Results include “none”, “info”, “warning”, “critical” levels (and links to documentation) • API used by the upgrade assistant to Kibana 5.6 GET /_xpack/migration/deprecations { … { "level" : "info", "message" : "Coercion of boolean fields", "url" : "https://www.elastic.co/guide/en/elasticsearch/reference/6.0/breaking_60_mappings_changes.html#_coercion_of_bool ean_fields", "details" : "<anchor id="type: doc" xreflabel="field: spins]"/>" } }
20. 20 • Create parent/child relationships between documents within a single index (without using types) • Ease the transition to one type per index • Added in 5.6 to ease the transition to 6.0 New join Datatype "mappings": { "doc": { "properties": { "my_join_field": { "type": "join", "relations": { "question" : "answer" ...
22. 22 • 21 Kibana ER’s closed with 5.5 • 8 New filtering capabilities – Dropdowns for filters, support for OR • 7 Event context viewer – With support for filtering • 3 Cross cluster search – Support in Kibana • 2 Region maps – Custom shape vectors • 1 Usability enhancements – Auto-complete for selecting fields Go Kibana!
23. 23 Time Series Visual Builder (5.4) A curated UI - just for time series data with features such as... Chart multiple indices Pipeline aggregations Complex calculations Conditional formatting Annotations Series offset
24. 24 Watcher UI (5.4) Phase 1 - View, manage, and create Watches
25. 25 Elastic Maps Service 18 Zoom Levels Hosted geoJSON shape files Easier to comprehend
26. 26 CONFIDENTIAL / TENTATIVE - DO NOT SHARE Visualizations Region map & Coordinate map update
27. 27 CONFIDENTIAL / TENTATIVE - DO NOT SHARE Visualizations Gauge, Goal & enhanced Metrics
28. 28 CONFIDENTIAL / TENTATIVE - DO NOT SHARE Usability Enhancements Dashboard cloning, field combo box and more!
29. 29 CONFIDENTIAL / TENTATIVE - DO NOT SHARE Discover A UI made just for filters
30. 30 CONFIDENTIAL / TENTATIVE - DO NOT SHARE Kibana Cross Cluster Search Configuration Example cluster with .kibana index cluster 1 cluster settings API to configure any node for cross cluster cluster 2
31. 31 CONFIDENTIAL / TENTATIVE - DO NOT SHARE Platform Support for Cross Cluster Search!
32. 32 CONFIDENTIAL / TENTATIVE - DO NOT SHARE Dev Tools Grok Debugger
34. 34 Persistent Queues Disk-Based Queuing - GA in 5.4 Resiliency • Durability across node failures • At-least-once delivery guarantees Adaptive buffering • External queuing layer no longer required to absorb throughput Features Control max disk usage Limited impact on performance Monitoring UI integration • Queue type and queue lag • Future: disk usage and disk IO Opt-in feature
35. 35 Feature Overview (5.5) • Resiliency from data loss of events that can’t be processed • Local disk-based queue • Inspect and reprocess dead events with a DLQ input plugin • Off by default Dead Letter Queues Dead Event Content • Original event • Event metadata ◦ Entry timestamp ◦ Plugin ID ◦ Plugin type ◦ Reason - error info Logstash Inputs Filters Outputs DLQ
36. 36 Dead Letter Queue Processing
37. 37 5.6 Logstash Modules are here...
38. 38 38 Logstash Modules Accelerated Time to Insight with the Elastic Stack • Data to dashboard in one command • Start exploring in minutes • Modules include: Automatic data processing and enrichment Suite of Kibana dashboards • Logstash 5.6 introduces modules for ArcSight and Netflow
39. 39 The Magic Underneath Logstash Modules Internals
40. 40 40 Logstash ArcSight Module X-Pack Basic ArcSight Certified
41. 41 41 Network Data • Dashboards: Network Overview, Network Suspicious Activity • Data Types: Network firewalls, intrusion systems, VPN devices Endpoint Data • Dashboards: Endpoint Overview, Endpoint OS Activity • Data Types: Operating systems, applications, host intrusion systems DNS Data • Dashboards: Microsoft DNS Overview • Data Types: Microsoft DNS devices Network & Firewall Device Dashboards Logstash ArcSight Module
42. 42 42 Logstash Netflow Module Open Source
43. 43 43 Logstash Netflow Module What is Netflow? • Record of connections traversing network devices • Who is talking to who? • How much traffic? • Over which protocols? • Using which services? • At what quality of service? • AND MORE...
44. 44 44 Logstash Netflow Module Instantly analyze Netflow data with the Elastic Stack Supports Netflow v5 and v9 datasets Complex data processing and enrichment is automatic Comprehensive suite of Kibana dashboards to help you get started • UDP input + Netflow codec (default port 2055) • Common user-friendly service names • ASN and GeoIP enrichment • TCP Flag decoding
45. 45 45 Logstash Netflow Module Netflow Module Dashboards • Overview • Conversation Partners • Traffic Analysis • Top-N • Geo Location • Autonomous Systems • Flow Exporters • Raw Flow Records
47. 47 ML shrink-wrapped configurations in Beats module • Installation includes 5 ML Jobs and 2 new dashboards ‒ 5.6 ‒ 6.0 • ML Jobs are installed, but are not started. • Think of them as templates for reference implementations that can be further edited or cloned Filebeat - nginx module ./filebeat -setup -modules=nginx ./filebeat setup --modules nginx
48. 48 Installed ML Jobs & Linked Dashboards 1. Changes in Website Visitor Counts - Single-metric jobs a. Low Count of Website Visitors, (low count, no field) b. Count of remote IPs, (distinct count, remote_IP) 2. Changes in Website Behaviour - HTTP status codes - Multi-metric job - response code by rate, influenced by remote IP 3. Unusual Clients - Advanced jobs for population analysis a. Detect Unusual Remote IPs - HIgh request rates b. Detect Unusual Remote IPs by distinct URL Drill into canned Dashboards 1. Remote IP Explorer 2. Remote IP URL Explorer
49. 49 Questions?