Elastic Stack 5 update

Transcript

1. 5.0 Updates Tobias Karlsson 1
2. 5.0 is here. All new versions. All aligned.
3. APIs Plugins Visualization ELK Stack
4. Along Came Beats ELKB
5. 5 Elastic Stack Kibana Elasticsearch Beats Logstash Kibana Elasticsearch Beats Logstash
6. 6 Jun 9, 2015 1.6 Jul 16, 2015 1.7 Feb 19, 2015 4.0 Jun 10, 2015 4.1 May 14th, 2015 1.5 May 27th, 2015 1.0 Beta 1 July 13th, 2015 1.0 Beta 2 Sept 4 th, 2015 1.0 Beta 3 May 23, 2015 1.5 Nov 5, 2014 1.4 It’s complicated es kibana ls beats
7. 7 Working beautifully together es kibana ls beats 6.0 7.0 x-pack 5.0 5.0 5.0 5.0 5.0
8. 8 Kibana Elasticsearch Beats Logstash Security Alerting Monitoring Reporting X-Pack Graph
9. 9 Kibana Elasticsearch Beats Logstash X-Pack Elastic Cloud Security Alerting Monitoring Reporting Graph Kibana Elasticsearch Beats Logstash
10. Elasticsearch 5.0: What You Need to Know
11. The Heart of the Elastic Stack
12. Core Tenants of Elasticsearch 12 Developer Friendly Speed Scalability
13. 5.0 What You Need to Know
14. Better Support for Numbers • BKD trees • Lower heap usage • IPv6 support • Scaled / Half float 14 Faster & reduced memory/disk for many use cases
15. Improved Indexing Time Performance 15
16. Fast, Safe Scripting Language 16 • Secure and production-safe • Significantly faster than Groovy • Familiar syntax • Can be used in various places: • ingest node pipeline, function scoring, scripted result filtering, watch conditions, and more Say “Heya” to Painless
17. 17 • Rollover APIs Logs-0001 Logs-0002 Logs-0003 1000 docs 800 docs 0 docs Logs (alias) Simplified Architecture Automatic time-based index management
18. 18 • Shrink APIs Simplified Architecture Automatic time-based index management Shard 1 Compressed Shard 2 /_shrink API High-volume Writes Hot nodes Lower-resource warm nodes Compressed Shard 1 Shard 2 Shard 3 Shard 4
19. Simplified Architecture 19 • Simplified experience for interactive pages • Wait-for-Refresh • Simplified getting started experience • Ingest Node: More to come on this today
20. Resiliency and Safety Improvements 20 • Bootstrap checks • Safeguards • Keyword type • Circuit breakers • 2 phase cluster state commit • Safe primary relocations
21. Faster, more normalized DSL 21 • Completion Suggester v2 • Percolation is now a normal query • Profile API expansion to include aggregations and not just queries
22. Java HTTP Client 22 • Decouple server/client • Minimize dependencies • Similar to other clients
23. Kibana 5.0: It’s More Than Charts and Graphs
24. Data Visualization Management Kibana Evolution
25. Discover Visualize Dashboard Data Visualization Management Kibana Evolution: 4.x
26. Discover Dashboard Monitoring Data Visualization Management Kibana Evolution: 4.x Visualize
27. Discover Dashboard Graph Data Visualization Management Monitoring Kibana Evolution: 4.x Visualize
28. Kibana Evolution: 4.x Discover Dashboard Graph Timelion Sense Data Visualization Management Monitoring Visualize
29. Kibana Evolution: 5.0 Discover Dashboard Graph DevTools Data Visualization Management Timelion Console Monitoring Visualize
30. Kibana Evolution: 5.0 Discover Dashboard Graph Settings Users DevTools Data Visualization Management Timelion Monitoring Visualize Console
31. New look and more space-optimized 31
32. Improved time-series analytics 32 Timelion is now a default Kibana app
33. Window into the Elastic Stack 33 Console (formerly Sense) is a default app
34. Window into the Elastic Stack 34 Monitoring app now includes Kibana monitoring * requires X-Pack
35. Window into the Elastic Stack 35 New UI to manage users and roles * requires X-Pack
36. Share the Kibana <3 36 Create reports of your visualizations and dashboards * requires X-Pack
37. Ingesting in 5.0: Logstash and Beats
38. Ingest data from any source, in any format 38 Beats Logstash
40. Say Heya to Ingest Node 40 Process incoming data directly in Elasticsearch
41. Beats: 5.0 Updates • Filter out data on the edge • Beats processors • Count and bytes on the TCP/IP layer not application layer • Packetbeat • No more double Logstash • Kafka output for Beats 41
42. New in 5.0: Metricbeat • Collect metrics from systems and services • One beat, many modules • System stats • Services • Container-ready 42
43. • Node Info • Node Stats • Plugins • Hot Threads 43 Monitoring API Logstash: Goodbye Black Box!
44. Logstash: Goodbye Black Box! • Debug active pipelines • Component level granularity Log4j2 Internal Logging
45. Logstash: Performance++ • New Java Event • 20%+ increase in overall pipeline performance • Rewrite of Beats input • 50% performance boost ingesting from Beats
46. Logstash: Performance++ • New Plugins • Kinesis input, Protobuf codec, Dissect filter, IPv6 Support with GeoIP2 • Improved Kafka support • Kafka 0.10 Support + Basic Auth & SSL/TLS • Plugin Generator • Developers can generate new plugins in seconds
47. 47 Elasticsearch-Hadoop 5.0 Spark 2.0 & Better Streaming Support Ingest Node Pipeline Integration Elasticsearch 5.0 Parallel Reader
48. X-Pack 5.0: Extending the Elastic Stack
49. 49 Kibana Elasticsearch Beats Logstash Security Alerting Monitoring Reporting X-Pack Graph X-Pack: One Pack. Many Features.
50. 50 We loved extensions
51. Simplified Getting Started Experience 51 $ bin/elasticsearch-plugin install x-pack $ bin/kibana-plugin install x-pack
52. Deeper Integration with Kibana 52 New UI to manage users and roles
53. Deeper Integration with Kibana 53 Monitoring app now includes Kibana monitoring
54. Share the Kibana <3 54 Create reports of your visualizations and dashboards
55. New and improved Graph app 55 New way to explore relationships in your data Now includes the ability to save workspaces, and drill down to supporting data
56. Alerting Improvements • Chained Inputs • Run multiple inputs serially • Condition per Action • E.g. Slack message if outage for 5 minutes. SMS messages if outage for 30 minutes { "input" : { "chain" : { "inputs" : [ { "first" : { "simple" : { "path" : "/_search" } } }, { "second" : { "http" : { "request" : { "host" : "localhost", "port" : 9200, "path" : "{{ctx.payload.first.path}}" } } } } ] } } ... }
57. Elastic Stack 5.2
58. Elastic Stack 5.2 • Tribe Node ✦ Broke in Kibana 5.0 ✦ 5.2.0 introduces the separation of a "data" cluster from an "admin" cluster ✦ Still a couple of shortcomings we’re working on • Start of internationalization • Tile service enhancements ✦ From 10 to 12 zoom levels
59. Heatmaps
60. Container monitoring
61. Resources
62. Upgrading to 5.0 v1.7 v2.4 v5.x 1.x Lucene 4 1.x Lucene 4 2.x Lucene 5 2.x Lucene 5 5.x Lucene 6 read/write read read/write read read/write Full cluster restart Full cluster restart reindex from remote reindex in place Data (segments) Software Upgrading Elasticsearch
 major version
63. Resources: Upgrading to 5.0 • Webinar - Upgrade your Elastic Stack to 5.0 (Nov 29) • Documentation - cross-stack upgrade guide • Elastic Support & Services - contact us for upgrade guidance
64. Coming Soon …
65. 65 Behavioral analytics and unsupervised machine learning 65 Welcome Prelert
66. 66 Elastic Cloud as a Product In ANY cloud … In YOUR cloud … Many clusters / use cases Single use case, as a service Available in AWS today
67. 67 Provisioning, orchestration, and management of multiple Elastic Stacks Expected GA Q1 2017 Same technical foundation as the Elastic Cloud service
68. www.elastic.co