Elastic v6.0 - complete presentation about this new release

complete presentation about this new release v6.0. Elastic Stack (Elasticsearch, Kibana & logshtash, Watcher, Beats and more)

Transcript

1. 1 Camilo Sierra What is new in Elastic 6.0, @CamiloSierrax 6.0 is coming 6.0
2. 2 ALONE WE CAN DO SO LITTLE; TOGETHER WE CAN DO SO MUCH. Helen Keller ‘‘
3. 3 Q & A
4. 4 Elasticsearch
5. 5 Nouvelle expérience de mise à jour Mise à niveau progressive et plus simple Upgrading to 2.x Upgrading to 5.x Upgrading to 6.x
6. 6 Brand new upgrade experience • New Upgrade Assistant (UI & API) • Zero downtime upgrades ‒Rolling restarts from latest 5.x to 6.x ‒Cross-cluster search across major version Upgrades just got oh so simpler X-Pack feature (Basic, free)
7. 7 Space-saving columnar store • Better for storing sparse fields • Save on disk space & file system cache Tapping into Lucene 7 goodness (sparse doc value) user first middle last age phone johns Alex Smith jrice Jill Amy Rice 508.567.121 1 mt123 Jeff Twain 56 sadams Sue Adams adoe Amy Doe 31 lp12 Liz Potter
8. 8 age 56 31 5.X 32 bit 32 bit 32 bit 32 bit 32 bit 32 bit age 56 31 6.0 32 bit 32 bit Space-saving columnar store Large improvements to how sparse data is stored • More efficient data structure in Elasticsearch 6.0 for storing sparse fields • In real life, calculate space on disk saved may not be that simple in most cases.
9. 9 Much speedier sorted queries Tapping into Lucene 7 goodness (index sorting) Player 1 Score: 600 5.x Query for top 3 player scores Player 2 Score: 0 Player 3 Score: 200 Player 4 Score: 700 Player 5 Score: 300 Player 1907 Score: 800 ... Query for top 3 player scores ... Player 1907 Score: 800 Player 4 Score: 700 Player 1 Score: 600 Player 5 Score: 300 Player 3 Score: 200 Player 2 Score: 0 6.x Sort at index time vs. query time Optimize on-disk format for some use cases Improve query performance at the cost of index performance
10. 10 Game Points Joust 1000 Joust 990 Joust 980 Dragon’s Lair 1000 Dragon’s Lair 950 Dragon’s Lair 940 Index Sorting: Define a Sorted Index
11. 11 Index Sorting: Early termination of Queries Sort at index time vs. query time
12. 12 Large Improvements to Replication • Limit syncs to only changed documents (instead of file-based recovery) • Fast replica recovery after temporary unavailability (network issues, etc.) • Re-sync on primary failure • Laying foundation for future big league features ‒Cross-datacenter replication ‒Changes API (tbd) New operation-based approach to recovery (sequence numbers)
13. 13 Large Improvements to Replication 5.x - We have to do a file-based recovery
14. 14 Large Improvements to Replication 6.x - New operation-based approach to recovery (sequence numbers)
15. 15 Simpler data models with type removal • Breaking change • Gradual migration path ‒ 6.0 indices can be created with only one type ‒ Existing 5.x indices using _type will continue to function • Introducing new APIs for type-less operations Say goodbye to _type confusion
16. 16 What happens in 6.x with Parent - Child ?
17. 17 Join datatype !!! Only one join field mapping is allowed per index. PUT my_index { "mappings": { "doc": { "properties": { "my_join_field": { "type": "join", "relations": { "question": "answer" } } } } } } PUT my_index/doc/1 { "text": "a question", "my_join_field": { "name": "question" } } PUT my_index/doc/3?routing=1 { "text": "an answer", "my_join_field": { "name": "answer", "parent": "1" } }
18. 18 Distributed watch execution • Watches are no longer executed on only the master node • They are executed on nodes which hold shards of the .watches index • Configure all or specific nodes dedicated to watch execution • To improve scalability, you need to add more replicas to .watches index X-Pack feature (Gold)
19. 19 Logstash
20. 20 • Run multiple, distinct workloads on a single Logstash JVM • Simplify dataflow logic by managing per data source logic independently • Monitor each pipeline separately with the new Pipeline Viewer Multiple Pipelines, One Logstash Logstash JDBC Pipeline Netflow Pipeline Apache Pipeline
21. 21 • Visualize pipeline topologies as graphs
 • Reveal bottlenecks at the plugin level
 • Optimize dataflow with better metrics
 • Integrated with Monitoring UI Zoom in on your Pipelines Pipeline Viewer X-Pack feature (Basic, free)
22. 22 Runtime Stats for Your Plugins • Filters and Outputs display : • CPU usage • Performance • Throughput
 • Highlights areas of high CPU usage and execution time • Provide an easy way to diagnose bottlenecks Pipeline Viewer X-Pack feature (Basic, free)
23. 23 • Manage multiple pipelines from multiple nodes in a single UI
 • Logstash nodes can poll and dynamically reload pipelines on configuration change
 • Secure access to configuration management with X-Pack Centrally Manage Logstash Pipelines Configuration Management X-Pack feature (Gold) Elasticsearch Kibana Logstash Apache Logstash Logstash Config Mgmt UI DevOps / Admins Auto-Update Pipelines JDBC Netflow
24. 24 Centrally Manage Logstash Pipelines X-Pack feature (Gold)
25. 25 Kibana
26. 26 Export saved searches to CSV with a single click Highly requested feature Trigger export via Watcher X-Pack feature (Basic, free)
27. 27 Export saved searches to CSV Export complex objects into CSV X-Pack feature (Basic, free)
28. 28 Lock down edits with Dashboard Only mode Share dashboards without worrying about accidental changes X-Pack feature (Gold)
29. 29 Maximize screen space with Full Screen mode Optimized viewing for your NOCs & SOCs
31. 31 6.0 starts Kibana on the accessibility path • High contrast color scheme • Keyboard accessibility • Screen reader support • More improvements on the way Accessibility improvements
32. 32 6.0 starts Kibana on the accessibility path Accessibility improvements
33. 33 Kibana now supports multiple query languages • Lucene Query Language (default) • Kuery (off by default, experimental in 6.0) • ... perhaps others in the future We want your feedback! • Enable Kuery from Advanced Settings More ways to query with Kuery Consistent syntax and simple to get started
34. 34 Kuery syntax • Function("field", value) We want your feedback! • Kuery: is("response", 200) • Lucene: response:200 • Kuery: not(is("response", 404)) • Lucene: !response:404 • Kuery: range("bytes", gt=1000, lt=8000) • Lucene: bytes:[1000 to 8000] • Kuery: geoPolygon("geo.coordinates", "40.97, -127.26", "24.20, -84.375", "40.44, -66.09") • Lucene: not supported More ways to query with Kuery Consistent syntax and simple to get started
35. 35 Get e-mail alerts on Cluster Alerts • Cluster Alerts are built-in Watches for cluster issues • Get e-mails when Cluster Alerts get triggered and resolved • Add admin e-mail in Kibana Advanced Settings
 X-Pack feature (Gold)
36. 36 Easily create simple threshold alerts New form based UI for threshold alerts X-Pack feature (Gold)
37. 37 Actions in threshold alerts UI / API Actions in UI for threshold alerts / Actions supported by API watcher Watcher API supports the following types of actions: • email, • webhook, • index, • logging, • hipchat, • Slack, • and pagerduty.
38. 38 Beats
39. 39 • New Kubernetes module in Metricbeat ‒ CPU, memory, bytes on network and more. • New processor to add_docker_metadata ‒ Container ID, name, image, labels • New processor to add_kubernetes_metadata ‒ Pod name, pod namespace, container name, pod labels Beats <3 containerization Monitor your Docker and Kubernetes deployments with ease
40. 40 More modules for more data sources New Filebeat modules New Metricbeat Modules
41. 41 • Improved dashboards for Metricbeat system module • Filebeat NGINX module ships with Machine Learning jobs ‒ We want your feedback Better Modules
42. 42 • Skip the hassle of parsing auditd logs ‒ Auditbeat subscribes to the kernel directly • Reuse auditd rule formats (no need to learn new rule formats) • Plus, file integrity checks on Linux, macOS, and Windows ‒ Watch files or directories (non-recursively) for changes ‒ Report file metadata and MD5, SHA1, SHA256 hashes on changes Auditbeat - a simpler way to track audit logs An alternative to auditd on Linux
43. 43 Elasticsearch • Rolling upgrades (upgrade with no downtime), • Top queries are super-fast (sort at index time), • Save disk space (save a lot of both disk space and file-system cache), • Say goodbye to multiple _types within index (parent/child become Join datatype) • Fast replica recovery (Sequence numbers improve replicas synchronization, recovery if primary shard fails... much more to come)
44. 44 Logstash • Run multiple, distinct workloads on a single Logstash (Simplify dataflow logic by managing per data source logic independently) • Runtime Stats for Your Plugins (indicators on the components spend extra time, easy way to diagnose bottlenecks) • Manage multiple pipelines from multiple nodes in a single UI (change pipelines configuration in running services)
45. 45 Kibana • CSV Export (Highly requested. Most commented issue in the Kibana repo), • Dashboard only mode (Hides all other applications, Dashboard's presented are in read only mode), • Full screen mode (Perfect for operations use case and "command centers"), • Compliance & Accessibility (High contrast colors for the color blind, Keyboard accessible, Improved support for screen readers), • Watcher UI (UI for creating and editing alerts based on thresholds)
46. 46 Become an Elastic Pioneer Download 6.0 preview release Provide feedback via GitHub or Discuss forum Get limited edition Pioneer swag 1 2 3
47. 47 Elastic Pioneer Program Download 6.0 preview release (Beta, RC, etc) Provide feedback via GitHub or Discuss forum Get limited edition Pioneer swag 1 2 3 We want your feedback!