What's new on the Elastic Stack 6.7 - 7.4

Elastic Stack overview & major release highlights between versions 6.7 to 7.4.
Focuses mostly on OSS & Elastic Basic License features.

The original slides contain animations, which cannot be preserved here.

Transcript

1. 1 Luca Belluccini Support Tech Lead, Elastic 24 Oct 2019, Elastic Meetup Marseille What's new on the Elastic Stack 6.7 - 7.4
2. 2 @lucabelluccini Software Engineer Elastic Support Engineer
3. 3 What is the Elastic Stack?
4. 4 4 Elastic Stack Store, Search, & Analyze Elasticsearch Visualize & Manage Kibana Beats Ingest Logstash
5. 5 Elasticsearch You know… For search Scale: Distributed by design Speed: Find matches in milliseconds Relevance: Get highly relevant results Storage, search & analytics layer Built on top of Lucene REST API, Query DSL
6. 6 Logstash ETL for Elasticsearch Ingest data of all shapes, sizes, and sources Parse and dynamically transform data Transport data to any output Secure and encrypt data inputs Build your own pipelines Lots of plugins
7. 7 Beats Lightweight data shippers Ship data from the source Ship and centralize in Elasticsearch Ship to Logstash for transformation and parsing Ship to Elastic Cloud Libbeat: API framework to build custom beats 70+ community Beats
8. 8 Beats All the modules Elastic Stack
9. 9 Kibana Window into the Elastic Stack Navigate and explore data Manage the Stack and Configuration Manage Index and Snapshots Lifecycle Maps, Infra, APM, Logs Spaces to separate scopes Canvas for pixel perfect visualizations
10. 10 10 Store, Search, & Analyze Elasticsearch Visualize & Manage Kibana Beats Ingest Logstash Metrics Logging APM Site Search Application Search Business Analytics Enterprise Search Security Analytics Future Solutions Elastic Stack
11. 11 11 Store, Search, & Analyze Elasticsearch Visualize & Manage Kibana Beats Ingest Logstash Metrics Logging APM Site Search Application Search Business Analytics Enterprise Search Security Analytics Future Solutions SaaS Elastic Cloud Self Managed Elastic Cloud Enterprise Standalone Deployment Elastic Stack Elastic Cloud on Kubernetes (beta) All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement.
12. 12 • Performant search and analytics • Scalable, resilient and highly available • Configurable node types Elasticsearch Kibana Logstash Beats • Lightweight data shippers • Files, metrics, packets, audit events • Pre-built modules for parsing and visualization • Explore and search • Visual interaction • Development and management tools • Normalize, filter and enrich • Centralized configuration management • Persistent queues “Typical” data flow
13. 13 ES-Hadoop Data store Web APIs Social Sensors Kafka Redis Messaging Queue Logstash Workers (2+) LDAP Authentication AD Notification SSO Kibana Custom UI Elasticsearch Clients Elasticsearch Master (3) Ingest (X) Data – Hot (X) Data – Warm (X) Machine Learning (2+) Coordinating (X) Alerting (X) HEARTBEAT Beats FILEBEAT METRICBEAT PACKETBEAT WINGLOGBEAT AUDITBEAT SAML All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement. “Full picture”
14. 14 Release Highlights 6.7 to 7.4
15. 15 Frozen Indices • Freeze Index API ‒ Read only indices ‒ Use less memory when idle ‒ Slower when being searched ‒ Not intended for real-time • Tips ‒ Requires ignore_throttled=false in the search request ‒ Enable pre_filter_shard_size=1 ‒ On Kibana, you have to enable includeFrozen in Advanced Settings 6.7 1 2 3 Search query opens frozen index Search executes against frozen index Frozen index is closed after search completes
16. 16 Elasticsearch SQL APIs & CLI • Query in SQL ‒ API call can mix SQL & Elasticsearch DSL • Get results in different formats • Learn DSL using the Translate API • SQL in Canvas as data source 6.7
17. 17 Index Lifecycle Management 6.7 Hot Phase Actions: Rollover H W C D Warm Phase Actions: Read-Only Allocate Shrink Force Merge Cold Phase Actions: Allocate Freeze Delete Phase Actions: Delete
18. 18 Functionbeat • Serverless shipper from ‒ AWS CloudWatch ‒ AWS SQS ‒ AWS Kinesis • Support for more vendors to be added in the future • The “Functionbeat” binary deploys a remote function 6.7
19. 19 Canvas • Canvas expression language (with TinyMath support) • “Presentations” with live data • Interactive filters / dropdowns • Workpads ‒ Pages ‒ Elements 6.7
20. 20 Security (Basic license) • Encrypted communications with TLS • Role-based access control • File and native authentication • Kibana Spaces • Kibana feature controls (7.2) ‒ Selective access to Kibana features at Role & Space level • API keys management (7.4) ‒ Users can create API keys and use them to provide long-term credentials while interacting with Elasticsearch 6.8 & 7.1
21. 21 New cluster coordination in Elasticsearch • Relies on formal methods to validate its designs (TLA+ model) • Automatic migration to the new cluster coordination module • No more discovery.zen.minimum_master_nodes to be set • Master elections are quicker • New concept of cluster bootstrapping (cluster.initial_master_nodes) • discovery.zen.ping.unicast.hosts is now discovery.seed_hosts • Introduction of “Voting-only master-eligible” role (7.3) ‒ Interesting for HA when we have only 2 AZ 7.0
22. 22 Index refresh changes • Notion of "search idle" shards ‒ index.search.idle.after (30s) • Skipping the refresh is cancelled on the next search and that search operation will block until the next refresh happens automatically (1s) • Only applies to indices where index.refresh_interval is not set explicitly 7.0
23. 23 Real memory circuit breakers • Best-effort mechanism based on estimates • Tries to avoid OutOfMemoryError rejecting tasks • The parent circuit breaker can use real memory usage instead of relying on memory usage reported by child circuit breakers • Allows for more aggressive limits (95% vs 70% of JVM Heap) 7.0 The image has been provided by Kiran Raja Bahadur SRK under the CC BY-NC-ND 2.0 license (original source).
24. 24 Adaptive Replica Selection • Elasticsearch ranks each copy of the shard to determine which is likeliest to be the "best" copy to send the request to • Choice based on ‒ Outstanding search requests to a node ‒ Number of data nodes in the system ‒ EWMA of the response time (as seen from the node coordinating the search) in ms ‒ EWMA of the number of events in the search thread pool queue ‒ EWMA of service time of search events on the data node in milliseconds • Enabled by default (available since 6.1) 7.0
25. 25 Removal of types • Phase “Elasticsearch 7.x” ‒ Specifying types in requests is deprecated ‒ Indexing doesn’t require a type anymore (_doc) ‒ include_type_name in index mappings defaults to false ‒ No need to provide it ‒ The _default_ mapping type is removed 7.0
26. 26 Automatic Query cancellation • Auto-terminate queries sent through the _search endpoint when the initiating connection is closed 7.4
27. 27 Replicated Closed Indices • Capitalize the work done on Frozen indices • Closed indices are replicated ‒ Important in case of disk failure 7.2
28. 28 AdoptOpenJDK 13 bundled with Elasticsearch • AdoptOpenJDK 13 bundled with Elasticsearch • Docker images always use the most up to date JDK available • Less “variance” in JDK configurations • Notarized AdoptOpenJDK package ‒ Apple macOS Catalina 7.4
29. 29 Snapshot Lifecycle Management • When and how often automatic snapshots of an Elasticsearch cluster are taken • Dedicated Snapshot Management Kibana UI (7.2) • Enhancements soon... 7.4
30. 30 Data Frames / Transform 7.3 (beta) POST _data_frame/transforms /_preview { "source": { "index": "kibana_sample_data_ecommerce" }, "dest" : { "index" : "sample_ecommerce_orders_by_customer" }, "pivot": { "group_by": { "user": { "terms": { "field": "user" }}, "customer_id": { "terms": { "field": "customer_id" }} }, "aggregations": { "order_count": { "value_count": { "field": "order_id" }}, "total_order_amt": { "sum": { "field": "taxful_total_price" }}, "avg_amt_per_order": { "avg": { "field": "taxful_total_price" }}, "avg_unique_products_per_order": { "avg": { "field": "total_unique_products" }}, "total_unique_products": { "cardinality": { "field": "products.product_id" }} } } }
31. 31 Flattened datatype • The flattened type mapping will parse out its leaf values and index them into one field as keywords • Avoids field explosion • Trade Offs ‒ All values as keyword type ‒ Not all search types are allowed ‒ Flattened object fields currently cannot be stored 7.4
32. 32 Rare terms aggregations • Conceptually like a terms aggregation that is sorted by _count ascending ‒ But with a bounded error (customizable precision) • Based on CuckooFilters ‒ Fan, Bin, et al. "Cuckoo filter: Practically better than bloom." Proceedings of the 10th ACM International on Conference on emerging Networking Experiments and Technologies. ACM, 2014. 7.4
33. 33 Kibana authentication support • PKI (7.4 Platinum) • Kerberos (7.3 Platinum) • OpenID Connect (7.2 Platinum) • SAML (6.2 Platinum)
34. 34 Kibana enhancements • Copy Object to space • Every panel in a dashboard can have a custom time picker • Better Canvas expression editor aka “Monaco” • Index Templates management from Kibana • Feature Controls in (7.2) 7.x
35. 35 Filebeat • Kafka Input (7.4) • AWS S3 Input (7.4, beta) - Requires SQS • NetFlow Input (7.4) • Google Pub/Sub input (7.2) • IBM MQ Error log module (7.4, beta) • Oracle, Amazon RDS, CockroachDB, MS SQL Server modules (7.3) • Script Processor (7.2, experimental) • Elastic Common Schema “compliance” (6.7 and 6.8 w/ migration, 7.0)
36. 36 Metricbeat • Elastic Common Schema “compliance” (6.7 and 6.8 w/ migration, 7.0) • Metricbeat Ceph, Aerospike, Couchbase, Prometheus stats (7.4) • Metricbeat Kubernetes monitoring (7.3) ‒ kube-controller-manager ‒ kube-proxy ‒ kube-scheduler
37. 37 Logstash • Java 11 Support (6.7) • Java pipeline execution (7.0) • Pipeline to Pipeline become GA (7.4) • Logstash JMS input plugin (7.3)
38. 38 Much more... APM • Angular for RUM JS Agent • .NET framework in the .NET agent • JAVA ECS Logging library Elasticsearch • Reloadable search_analyzers • circle ingestion processor & shape_field datatype • cardinality aggregation 7.4 Kibana • Maps GA (7.3) • Show common path between 2 points
39. Thank you!