Elastic{ON}16 Opening Keynote

Elastic Founder & CEO Steven Schuurman, Elastic Founder & CTO Shay Banon, and other members of the Elastic engineering team kick off Elastic{ON}16 with company and product highlights from the past year as well as what to expect with the upcoming 5.0 release. You’ll hear about updates coming to the Elastic Stack, learn about what the heck a Pack is, and see some awesome live demos of the new Kibana interface and Graph API. Jason McGee, VP and CTO of the IBM Cloud Platform, also shares how IBM leverages the Elastic Stack within its organization.

Transcript

1. #elasticon
2. STEVEN
 SCHUURMAN CEO & Co-Founder, Elastic
7. Last year…
8. Attendees 1800+
9. 51,000+ Members
10. Meetups in 2015 500
11. Total Cumulative Downloads Across Elastic Products Millions of downloads 10 20 30 40 50 2012 2013 2014 2015 Feb'16
12. Downloads 50,000,000
13. “Improving patient care with real-time clinical decision making.”
14. “Combating our global human trafficking problem.”
15. “Mining 3-4 billion events per day to ensure security intelligence.”
16. “Care free deployments using Hosted Elasticsearch.”
17. “Many use cases from trade optimization to compliance to HR recruiting.”
18. The future
20. More than Search
21. Beyond Developers
22. Centralising your deployment
23. CTO & Co-Founder, Elastic Creator of Elasticsearch
24. ABRVTNS We love naming
25. Elastic{ON}
26. EON Do
27. ESON EON Do Ray
28. EGON ESON EON Ray Do
29. ELK Stack
30. ELK Stack
31. APIs Plugins Visualization ELK Stack
32. Along Came Beats ELKB
34. The Elastic Stack
35. elasticsearch
36. elasticsearch Columnar Store Not Yo’ Mama’s Inverted Index Date Customer Store Product Price
37. elasticsearch SECURITY VALUABLE RESOURCES CODE JVM CLASS LOADER One does not simply fork a process Java Security Manager
38. elasticsearch 1. { 2. “nodes”: 3. [ 4. [”10,42.1.120”,9200], 5. [”10,42.0.121”,9200], 6. [”10,42.0.123”,9200], 7. [”10,42.0.124”,9200], 8. [”10,42.0.125”,9200] 9. ] 10. } 1. { 2. “nodes”: 3. [ 4. [”10,42.0.121”,9200], 5. [”10,42.0.122”,9200], 6. [”10,42.0.123”,9200], 7. [”10,42.0.124”,9200], 8. [”10,42.0.125”,9200] 9. ] 10. } Cluster State Diffs
39. elasticsearch Y U SO SLOW Profile API
40. elasticsearch Location Location
 Location GEO
41. Pipeline Aggregations Thu 31 Smooth Average Data Value Upper Control Limit August Aug 03 Tue 05 Thu 07 Sat 09 Mon 11 Wed 13 Fri 15 Aug 17 Tue 19 10 20 30 40 50 60 70 10 20 30 40 50 60 70 Numb3rs elasticsearch
42. kibana
43. kibana Eye-meltingly colourific Colour picker
44. kibana Naming
 matters Custom Legends
45. kibana Not only for the 1% Field formatters
46. kibana Back in Black Black theme
47. kibana Predator Vision Heat map
48. kibana Ice Ice Baby Pluggable Tile Servers
49. kibana Which 3pm? Global timezone
50. logstash
51. logstash Faster, more reliable pipeline New pipeline Architecture In-Memory Synchronous Queue InputThread1 Worker Thread 1 <Event> <Event> [<Event>, <Event>, ...] [<Event>, <Event>, ...] Batch Stage Filter Stage Output Stage RedisInput InputThread1 Worker Thread 1 <Event> <Event> [<Event>, <Event>, ...] [<Event>, <Event>, ...] Batch Stage Filter Stage Output Stage BeatsInput
52. logstash Responsive design Config Reload
53. logstash Kafka, HDFS, Salesforce, HTTP, Oh my! Plugins Analysis Alerting Monitoring Archiving Elasticsearch + ANY data store Watcher + ANY notifications tools Marvel + ANY monitoring tools Hadoop + ANY cloud storage platform Log and metrics data Web and social data Sensor and device data Data stores and streams
54. logstash JAVA is the assembly of (J)Ruby Java Event
55. logstash Events 0 1000000 2000000 3000000 4000000 5000000 6000000 7000000 8000000 Logstash Releases 1.4.4 1.5.0 2.0.0 2.1.0 2.2.0 (Pipeline) 2.3.0 (Java Event) Apache Parsing Complex Syslog Now you can grok faster Performance
56. beats
57. Capture the Packet Packetbeat
58. Capture the Packet Packetbeat
59. Unleash the Beats libbeat Beat 1 libbeat Beat 2 Beat 3 +
60. It was only supposed to be a demo topbeat
61. It was only supposed to be a demo topbeat
62. To tail a File filebeat + logstash
63. To tail a File filebeat + logstash
64. Welcome to 1998 winlogbeat
65. Now winlogbeat
66. MySQL metricbeat Redis Apache + Connecting Numb3rs metricbeat
67. Versions
68. Jun 9, 2015 1.6 Jul 16, 2015 1.7 Feb 19, 2015 4.0 Jun 10, 2015 4.1 May 14th, 2015 1.5 May 27th, 2015 1.0 Beta 1 July 13th, 2015 1.0 Beta 2 Sept 4 th, 2015 1.0 Beta 3 May 23, 2015 1.5 Nov 5, 2014 1.4 It’s complicated es kibana ls beats
69. es kibana ls beats Oct 28th Nov 21st Feb 2nd 2.0 4.2 2.0 2.1 4.3 2.1 1.0 2.2 4.4 2.2 1.1 Release Bonanza
70. v 5. 0
71. “I just want to tail a file.” Ingest
72. Grok Geo
73. Simple things should be simple Not like this Like this
74. I N G E S T
75. kibana
76. Kibana 4 Discover Visualize Dashboard
77. Marvel 2.0
78. Timelion
79. Tag Cloud
80. A Window 
 to our Stack
81. Rashid Khan Creator of Kibana Demo
82. Extensions
83. We love extensions
84. Packs
85. NO OPEN SOURCE ENTERPRISE EDITION
86. Security: Shield Authentication Authorization Encryption IP Filtering Audit Logging
87. Security: Shield Field and Document Level Security
88. Alerting: Watcher • Alerts • Notifications to email, Slack, JIRA, Hipchat, PagerDuty, and more • Analyze Watch history
89. Security in Kibana • Session Management • Login/ & Logout Capabilities
90. Bundled set of features: Security, Alerting, Monitoring, and more … x-pack
91. Uri Boness Co-Founder, Elastic X-Pack Engineering Lead Demo
92. Gephi
93. Mark Harwood Software Engineer Graph Demo
94. Elasticsearch + Kibana as a Service Latest release of the Elastic Stack and X-Pack
95. Cute but hard to find Did I mention naming?
96. cloud
97. Cloud as a Product * Not actual packaging *
98. It’s all about choice We want to install it
99. Even more choice Many clusters / use cases, exposed within the organization Single use case, as a service, exposed within the organization You want to install it
100. PRIVATE BETA cloud Elastic Cloud Deploy Elastic Cloud Enterprise Download
101. Michael Basnight Software Engineer Njal Karevoll Software Engineer Demo
103. 103 Jason McGee @jrmcgee IBM Fellow, VP and CTO, IBM Cloud Platform
104. ©2015 IBM Corporation IBM 104 Reaching new Horizons with Elastic Jason McGee @jrmcgee IBM Fellow, VP & CTO, IBM Cloud Platform February 18, 2016
105. ©2015 IBM Corporation IBM 105 IBM 2 ©2015 IBM Corporation How IBM is using Elastic
106. ©2015 IBM Corporation IBM 106 Kibana Logstash Cluster App
 logs Object Storage Application server Logstash agent Elasticsearch Cluster Kafka Cluster Zookeeper Cluster IBM 3 Watson Developer Cloud 500m events/day 50+ apps 300-500GB data/day Bare-metal nodes w/24 TB storage Env tags added, filtered by “topic” Can someone get me a screen shot as I do not have access to Kibana.
107. ©2015 IBM Corporation IBM 107 IBM 4 Twitter Insights for Bluemix Elasticsearch Cluster Data compliancy (deleted/ protected/ unprotected tweets current) ~100m events/day 400-700 tweets/sec ES Cluster: 44 Bare-metal nodes, each w/8 TB storage, 128GB RAM ~2 years of searchable Twitter data (~25B tweets, ~50TB of original data) Kafka Cluster Tweets queue Compliance queue Zookeeper Cluster Spark HA Cluster Tweets Receiver Compliance Events Recv’r Ingestion / Enrichment Compliance processing WebSphere Cluster REST Search API Twitter Insights Bluemix Service
108. ©2015 IBM Corporation IBM 108 IBM 5 ©2015 IBM Corporation Bluemix monitoring & logging service
109. ©2015 IBM Corporation IBM 109 Openstack Control Plane Virtual Machines on bare metal Elasticsearch Cluster Logstash Cluster Kafka Cluster App
 logs Cloud-based services Metrics topic Logging topic Graphite Cluster Zookeeper Cluster Carbon Cache Cluster Containers Grafana Kibana Lumberjack Cluster Load balancing IBM 6 Bluemix monitoring & logging service Metrics ingestion Log ingestion Replace with Kibana dashboard and include new KI product icon Can someone get me a screen shot as I do not have access to Kibana.
110. ©2015 IBM Corporation IBM 110 7 Up to 7GB & 7 days of logs per Bluemix space 5 production deployments on 2 continents & growing 5 1000+ tenants growing to millions Several TB data/day & growing 10s of nodes per production deployment 25k+ shards per production deployment Expanding Quickly… IBM 7 ©2015 IBM Corporation
111. ©2015 IBM Corporation IBM 111 Lessons we’ve learned IBM 8 ©2015 IBM Corporation
112. ©2015 IBM Corporation IBM 112 Upgrading, scaling, rebalancing w/o impacting users is an ongoing challenge In a 
 cloud service, data flow never stops IBM 9 ©2015 IBM Corporation
113. ©2015 IBM Corporation IBM 113 Learning to anticipate load & dynamically allocate shards is critical to dealing with diverse conditions on the cloud IBM 10 ©2015 IBM Corporation
114. ©2015 IBM Corporation IBM 114 Looking at the future: where we’re going IBM 11 ©2015 IBM Corporation
115. ©2015 IBM Corporation IBM 115 IBM 12 Container Log Crawler Host Multi-tenant Logstash forwarder Logstash Cluster Kafka Cluster Zookeeper Cluster Kibana Elasticsearch Cluster Multi-tenant proxy Built in multi-tenancy IBM 12 Not sure what to do here? Maybe update image with new KI dashboard and use the KI icon?
116. ©2015 IBM Corporation IBM 116 IBM 13 ©2015 IBM Corporation Combined Logs and Metrics I don’t think we can replace this graphic as it has data from their clusters. Perhaps, clip out the Kibana at the top and put the new KI icon somewhere?
117. ©2015 IBM Corporation IBM 117 Internationalization & Embedability IBM 14 ©2015 IBM Corporation
118. ©2015 IBM Corporation IBM 118 Design Thinking Containers Extreme Agile Mobile IoT APIs Microservices Cognitive Build
 something 
 cool! Watson IBM Containers ElasticSearch by Compose Twilio Slack IBM 15 ©2015 IBM Corporation Watson IBM Containers ElasticSearch by Compose Twilio Slack Build Something Cool!
119. ©2015 IBM Corporation IBM 119
120. ©2015 IBM Corporation IBM 120 Lorem Ipsum dolor sit, to amet consectetur irare a adispicing elit done et ectals tempus. Quote “ “ Author, Secondary Information September 16, 2015 Presentation Title Thank You! ©2015 IBM Corporation 17 Jason McGee @jrmcgee
121. Thank you!