Opening Keynote: From ELK to Elastic Stack

The Elastic Stack is a new name and vision for our open source projects: Elasticsearch, Kibana, Logstash, and Beats. But what does that really mean, and where are we headed? This session will set the stage (literally) for the day.

Transcript

1. OPENING KEYNOTE From ELK to the Elastic Stack Shay Banon, Co-Founder & CTO, Creator of Elasticsearch 1 London | November 15 2016
2. 2 2010 2012 2013 2014 2015 2016 First version of Elasticsearch
 released in February
3. 3 2010 2012 2013 2014 2015 2016 Elasticsearch founded as a company Total cumulative downloads 2M
4. 2010 Kibana and Logstash open source projects join Elasticsearch Total cumulative downloads 5M 2012 2013 2014 2015 2016 4
5. 2010 1.0 GA Elasticsearch Release of Marvel (monitoring) Total cumulative downloads 18M 2012 2013 2014 2015 2016 5
6. 2010 1st Elastic{ON} user conference Company name changed to “Elastic” Found acquired (now Elastic Cloud) Packetbeat team joins Elastic (now Beats) Total cumulative downloads 45M 2012 2013 2014 2015 2016 6
7. 2010 2nd Elastic{ON} user conference ELK “Elastic Stack” Prelert acquisition Total cumulative downloads … 2012 2013 2014 2015 2016 7
8. 75,000,000 DOWNLOADS
9. APIs Plugins Visualization ELK Stack
10. Along Came Beats ELKB
11. 11 Elastic Stack Kibana Elasticsearch Beats Logstash
12. 12 Kibana Elasticsearch Beats Logstash Security Alerting Monitoring Reporting X-Pack Graph
13. 13 Kibana Elasticsearch Beats Logstash X-Pack Elastic Cloud Security Alerting Monitoring Reporting Graph
14. 14 Jun 9, 2015 1.6 Jul 16, 2015 1.7 Feb 19, 2015 4.0 Jun 10, 2015 4.1 May 14th, 2015 1.5 May 27th, 2015 1.0 Beta 1 July 13th, 2015 1.0 Beta 2 Sept 4 th, 2015 1.0 Beta 3 May 23, 2015 1.5 Nov 5, 2014 1.4 It’s complicated es kibana ls beats
15. Elasticsearch Beats Logstash Kibana 5.0 is here. All new versions. All aligned.
16. 16 Working beautifully together es kibana ls beats 6.0 7.0 x-pack 5.0 5.0 5.0 5.0 5.0
17. Search and analytics, it all started here More than 60% of our customers have a search or analytics use case
20. Logs Logs Logs, 
 many devices,
 many systems More than 40% of our
 customers use our products
 for operational log analysis
21. We collect more than 1.2 TB logs every day from our infrastructure, web servers, and applications. 21
22. 22 We handle more than 3 Billion daily events while meeting our all of our data security requirements.
23. Sniff sniff sniff,
 find the bad actors
 in your data 200% YoY growth in security use cases with our products
24. We analyze piles of data: 13B AMP queries/day 600B emails/day 16B web requests/day 24
25. We are on track to achieve our goal to handle more than 20 PB of data to serve over 100 technical and business teams at scale across the globe. 25
26. The Elastic Stack: 
 A foundation to solve many use cases 75% of our customers use our products for more than one use case SEARCH SECURITY CUSTOM APPS METRICS OPERATIONAL
 ANALYTICS LOG ANALYSIS
27. Operational analytics Flight telemetry analysis Anomaly resolution Internal search engine 27
28. 28 Enterprise search Intranet search Real-time log analytics Legal contract repository Trade tracking application HR recruiting application
29. Cloud is 4real, let us host and manage the stack 250% growth in our cloud business since March 2015
30. 30 elastic cloud Hosted Elasticsearch & Kibana From the Source Log into Elastic Cloud Login Password adrian.jozwik@elastic.co Forgot your password? We’ll help. Don't have an account? Sign up. by
31. 31 Elastic Cloud as a Product In ANY cloud … In YOUR cloud … Many clusters / use cases Single use case, as a service Available in AWS today
32. 32 Provisioning, orchestration, and management of multiple Elastic Stacks Expected GA Q1 2017 Same technical foundation as the Elastic Cloud service
33. 33 Behavioral analytics and unsupervised machine learning 33 Welcome Prelert
34. 34 Thank you sponsors Thank you speakers
35. 35 March 7-9, 2017 • Pier 48 • San Francisco, CA • 2,500 attendees 3rd Annual Elastic User Conference Elastic Cause Awards • Recognizing 3 projects serving the greater good • 2 comp tickets (conf + hotel) for the project team • Deadline: December 1st
36. Elasticsearch 5.0: What You Need to Know Clint Gormley
37. The Heart of the Elastic Stack
38. Core Tenets of Elasticsearch 38 Developer Friendly Speed Scalability
39. 5.0 What You Need to Know
40. Better support for Numb3rs • BKD Trees • Lower heap usage • IPv6 Support 40 Faster & reduced memory/disk for many use cases
41. 0 10000 20000 30000 40000 50000 60000 70000 80000 float half float scaled float (factor = 4000) scaled float (factor = 100) On Disk Usage in kb Points disk usage (kb) docs_values disk usage (kb) Better support for Numb3rs Scaled / Half float 41 Faster & reduced memory/disk for many use cases
42. 42 Improved Indexing Time Performance
43. Fast, Safe Scripting Language 43 • Secure and production-safe • Significantly faster than Groovy • Familiar syntax • Can be used in various places: • Ingest node pipeline, function scoring,
 scripted result filtering, watch conditions,
 and more Say “Heya” to Painless
44. 44 • Automatic time-series index management • Rollover APIs Logs-0001 Logs-0002 Logs-0003 1000 docs 800 docs 0 docs Logs (alias) Simplified Architecture
45. Simplified Architecture 45 • Automatic time-series index management • Shrink APIs Shard 1 Compressed Shard 2 /_shrink API High-volume Writes Hot nodes Lower-resource warm nodes Compressed Shard 1 Shard 2 Shard 3 Shard 4
46. Simplified Architecture 46 • Simplified experience for interactive pages • Wait-for-Refresh • Simplified getting started experience • Ingest Node: More to come on this today
47. Resiliency and Safety Improvements 47 • We saw some common problems when getting started or new users on a multi-tenant environment • Bootstrap checks • Circuit breakers • Safeguards
48. Resiliency and Safety Improvements 48 • Understanding and preventing a terrible Friday afternoon • 2 phase cluster state commit • safe primary relocations
49. Faster, more normalized DSL 49 • Completion Suggester v2 • Percolation is now a normal query • Profile API expansion to include aggregations and not just queries
50. Beyond 5.0 50 • Higher timestamp resolution (great for logging use cases) • More improvements on resiliency • Build on BKD: range fields, geo • Increased performance for append-only time series use cases • Native RESTful Java client
51. Kibana 5.0: The Window into the Elastic Stack Uri Boness
52. Data Visualization Management Kibana Evolution
53. Discover Visualize Dashboard Data Visualization Management Kibana Evolution: 4.x
54. Discover Dashboard Monitoring Data Visualization Management Kibana Evolution: 4.x Visualize
55. Discover Dashboard Graph Data Visualization Management Monitoring Kibana Evolution: 4.x Visualize
56. Kibana Evolution: 4.x Discover Dashboard Graph Timelion Sense Data Visualization Management Monitoring Visualize
57. Kibana Evolution: 5.0 Discover Dashboard Graph DevTools Data Visualization Management Timelion Console Monitoring Visualize
58. Kibana Evolution: 5.0 Discover Dashboard Graph Management Users DevTools Data Visualization Management Timelion Monitoring Visualize Console
59. Demo: Kibana 5.0
60. Beyond 5.0 60 • Kibana is the Window into the Elastic Stack — management and visualization • Embrace more diversity: New user interfaces, visualizations, and dev management tools • Kibana for everyone — developers, technical, non-technical business users • “Unexpected apps”
61. Coffee Break Sponsor
62. Ingest: Beats & Logstash 5.0 Shay Banon
63. Ingest data from any source, in any format 63 Beats Logstash
64. X-pack X-pack Nodes (X) Logstash Messaging Queue Kafka Redis Elasticsearch Master Nodes (3) Data Nodes - Warm (X) Instances (X) Kibana Custom UI Datastore Web APIs Social Sensors Log Files Beats Metrics Wire Data your{beat} Hadoop Ecosystem ES-Hadoop Ingest Nodes (X) Data Nodes - Hot (X) Authentication Notification LDAP AD SSO
65. Say Heya to Ingest Node 65 Process incoming data directly in Elasticsearch I N G E S T
66. New in 5.0 66 Streamline network & storage Count and bytes on the TCP/IP layer not application layer No more double Logstash Beats Processors Packetbeat Kafka output for Beats
67. Metricbeat is here Topbeat New in 5.0
69. Logstash: Goodbye Black Box! 69 logstash:9600/ _node Node Info
 Node Stats
 Plugins
 Hot Threads Monitoring API Debug active pipelines with new logging API Component level logging granularity Log4j2 Internal Logging
70. Logstash: Performance++ 70 20%+ increase in overall pipeline performance 50% performance boost ingesting from Beats New Java Event Beats Input Java Rewrite
71. Logstash: Plugin Features 71 Developers can generate new plugins in seconds Kafka 0.10 Support Basic Auth & SSL/TLS Plugin Generator Kafka Support++ Kinesis Input
 Protobuf Codec
 Dissect Filter IPv6 Support with GeoIP2 New Plugins
72. 72 Elasticsearch Kibana ES-Hadoop Backup Elasticsearch with HDFS Efficiently move data between Elasticsearch & Hadoop Elasticsearch-Hadoop 5.0 Spark 2.0 & Better Streaming Support Ingest Node Pipeline Integration Elasticsearch 5.0 Parallel Reader
73. Beyond 5.0 (Beats) 73 • Moar modules in Metricbeat • Moar Beats • Even easier getting started experience • Centralized configuration & monitoring
74. Beyond 5.0 (Logstash) 74 • Logstash persistence (disk-based queuing) • Monitoring UI & centralized configuration • Multiple pipelines, one JVM • Error event routing
75. X-Pack 5.0: Extending the Elastic Stack Uri Cohen
76. 76 We loved extensions
77. X-Pack: One Pack. Many Features. 77 Kibana Elasticsearch Beats Logstash Security Alerting Monitoring Reporting X-Pack Graph 77
78. Simplified Getting Started Experience 78 $ bin/elasticsearch-plugin install x-pack $ bin/kibana-plugin install x-pack
79. Demo: X-Pack
80. Beyond 5.0 80 • Security • Kerberos & SAML realms • Monitoring • UI for monitoring Logstash & Beats • Automatic identification of issues • Alerting • Distributed watch execution • UI • Graph • Improvements to U/X • Enhanced user experience • Reporting • CSV Export • New Output Types (.png)
81. Prelert: Behavioral Analytics for the Elastic Stack Steve Dodson
82. 82 • How do I know my systems are behaving normally? • Where to set thresholds for good alerting?
 • How to find the root cause of problems when I don’t know what to look for? IT Operations
83. 83 • Do I have systems that are compromised with malware?
 • Which users could be an insider threat? IT Security
84. 84 • Is my factory working normally?
 • What do I do with thousands of time-series data?
 • Which traffic incidents are causing the most delay? Other
85. 85 Search Aggregations Visualization Machine Learning Extracting useful, valuable information is hard
86. Example: Detecting anomalies in data 86 Notify when current behavior deviates significantly from the predictive model Unsupervised machine learning automatically models behaviors in data
87. Demo: Prelert
88. Coming Soon 88 • Beta available for download now • Working on tighter integration into the Elastic Stack • GA targeted in first half of 2017
89. It’s all in the {find} y’know @ BBC BBC Mike Satterthwaite
90. Coffee Break Sponsor
91. {elastic}searching for Insights: Fighting Financial Crime at HSBC HSBC Adam Tun
92. www.elastic.co
93. Thank You