6.0 is coming - Elastic San Francisco meetup

6.0 is coming

To help our awesome community prepare we will be doing a run through of the major changes that will land with the Elastic Stack in 6.0.

We have things like rolling-major version upgrades, even more Lucene data structure optimizations, sequence IDs, saying farewell to _types, dashboard modes, automated cluster alerts, Logstash pipelines and a pipeline viewer, Beats integration with kubernetes and a whole raft of new Beats modules.

Our Pioneer Program also makes a return, so if you are already using the beta releases of the stack and are submitting issues, you're a winner!

Christoph Wurm is a Solutions Architect at Elastic living in San Francisco. He spends most of his time working with users and customers of the Elastic Stack while alternating between the different coffee roasters the city has to offer (but green tea is good as well).

https://www.meetup.com/Elasticsearch-San-Francisco/events/244064018/

Transcript

1. 1 Christoph Wurm Principal Solutions Architect cw@elastic.co 6.0 is coming
2. 2 Beats
3. 3 Packetbeat Nov '15 Elasticsearch Filebeat Nov '15 Winlogbeat Jan '16 Metricbeat Oct '16 Heartbeat Jan '17 Auditbeat Fall 2017 Logstash Kafka Redis Beats
4. 4 Metricbeat Aerospike Apache Ceph Couchbase Docker Dropwizard Elasticsearch Golang graphite HAProxz HTTP Jolokia Kafka Kibana Kubernetes Memcached MongoDB MySQL Nginx PHP_FPM PostgreSQL Prometheus RabbitMQ Redis System vSphere Windows ZooKeeper Elasticsearch Kibana 28 modules
6. 6 Auditbeat Auditbeat File changes Syscalls Linux macOS Windows Linux Elasticsearch Kibana
8. 8 • New Kubernetes module in Metricbeat ‒ CPU, memory, bytes on network and more. • New processor to add_docker_metadata ‒ Container ID, name, image, labels • New processor to add_kubernetes_metadata ‒ Pod name, pod namespace, container name, pod labels Beats <3 containerization Monitor your Docker and Kubernetes deployments with ease
9. 9 Logstash
10. 10 • Run multiple, distinct workloads on a single Logstash JVM • Simplify dataflow logic by managing per data source logic independently • Monitor each pipeline separately with the new Pipeline Viewer Multiple Pipelines, One Logstash Logstash JDBC Pipeline Netflow Pipeline Apache Pipeline
11. 11 Queues in Logstash Logstash Beats Elasticsearch PQ Filters DLQ
12. 12 • Visualize pipeline topologies as graphs • Reveal bottlenecks at the plugin level • Optimize dataflow with better metrics • Integrated with Monitoring UI Zoom in on your Pipelines Pipeline Viewer X-Pack feature (Basic, free)
13. 13 Elasticsearch
14. 14 Upgrade (2015) Upgrading to 2.x
15. 15 Upgrade (2016) Upgrading to 5.x
16. 16 Upgrade (2017) •New Upgrade Assistant (UI & API) • Zero downtime upgrades ‒Rolling restarts from latest 5.x to 6.x ‒Cross-cluster search across major version
17. 17 Space-saving columnar store • Better for storing sparse fields • Save on disk space & file system cache Tapping into Lucene 7 goodness (sparse doc value) user first middle last age phone johns Alex Smith jrice Jill Amy Rice 508.567.1211 mt123 Jeff Twain 56 sadams Sue Adams adoe Amy Doe 31 lp12 Liz Potter
18. 18 Much speedier sorted queries Tapping into Lucene 7 goodness (index sorting) Player 1 Score: 600 5.x Query for top 3 player scores Player 2 Score: 0 Player 3 Score: 200 Player 4 Score: 700 Player 5 Score: 300 Player 1907 Score: 800 ... Query for top 3 player scores ... Player 1907 Score: 800 Player 4 Score: 700 Player 1 Score: 600 Player 5 Score: 300 Player 3 Score: 200 Player 2 Score: 0 6.x Sort at index time vs. query time Optimize on-disk format for some use cases Improve query performance at the cost of index performance
19. 19 Sequence Numbers
20. 20 Sequence Numbers
21. 21 Sequence Numbers
22. 22 Sequence Numbers
23. 23 Sequence Numbers
24. 24 Sequence Numbers
25. 25 Large Improvements to Replication • Limit syncs to only changed documents (instead of file-based recovery) • Fast replica recovery after temporary unavailability (network issues, etc.) • Re-sync on primary failure • Laying foundation for future big league features ‒Cross-datacenter replication ‒Changes API (tbd) New operation-based approach to recovery (sequence numbers)
26. 26 Simpler data models with type removal • Breaking change • Gradual migration path ‒ 6.0 indices can be created with only one type ‒ Existing 5.x indices using _type will continue to function • Introducing new APIs for type-less operations Say goodbye to _type confusion
27. 27 X-Pack
28. 28 New in X-Pack 6.0 • Logstash ‒ Central configuration management in Elasticsearch/Kibana • Elasticsearch ‒ No default passwords • Kibana ‒ Dashboard Only mode ‒ Alerting UI & Cluster Alerts
29. 29 Kibana
30. 30 Export saved searches to CSV with a single click I guess people still like Excel? X-Pack feature (Basic, free)
31. 31 Maximize screen space with Full Screen mode Optimized viewing for your NOCs & SOCs
32. 32 32
33. 33 6.0 starts Kibana on the accessibility path • High contrast color scheme • Keyboard accessibility • Screen reader support • More improvements on the way Accessibility improvements
34. 34 6.0 starts Kibana on the accessibility path Accessibility improvements
35. 35 Kibana now supports multiple query languages • Lucene Query Language (default) • Kuery (off by default, experimental in 6.0) • ... perhaps others in the future We want your feedback! • Enable Kuery from Advanced Settings More ways to query with Kuery Consistent syntax and simple to get started geoBoundingBox("coordinates", topLeft="40.73, -74.1", bottomRight="40.01, -71.12")
36. 36 • Visualize pipeline topologies as graphs • Reveal bottlenecks at the plugin level • Optimize dataflow with better metrics • Integrated with Monitoring UI Zoom in on your Pipelines Pipeline Viewer X-Pack feature (Basic, free)
37. 37 6.0-rc2 Available in Elastic Cloud today cloud.elastic.co
38. 38 6.0 Coming very soon Available in Elastic Cloud on Day 1 cloud.elastic.co
39. 39 Elastic{ON} Feb 27 – Mar 1, 2018 https://www.elastic.co/elasticon
40. 40 You know, for Search... Questions?