Elastic{ON}16 Opening Keynote

by Elastic Co

Slide 1

Slide 1 text

#elasticon

Slide 2

Slide 2 text

STEVEN  SCHUURMAN CEO & Co-Founder, Elastic

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Last year…

Slide 8

Slide 8 text

Attendees 1800+

Slide 9

Slide 9 text

51,000+ Members

Slide 10

Slide 10 text

Meetups in 2015 500

Slide 11

Slide 11 text

Total Cumulative Downloads Across Elastic Products Millions of downloads 10 20 30 40 50 2012 2013 2014 2015 Feb'16

Slide 12

Slide 12 text

Downloads 50,000,000

Slide 13

Slide 13 text

“Improving patient care with real-time clinical decision making.”

Slide 14

Slide 14 text

“Combating our global human traﬃcking problem.”

Slide 15

Slide 15 text

“Mining 3-4 billion events per day to ensure security intelligence.”

Slide 16

Slide 16 text

“Care free deployments using Hosted Elasticsearch.”

Slide 17

Slide 17 text

“Many use cases from trade optimization to compliance to HR recruiting.”

Slide 18

Slide 18 text

The future

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

More than Search

Slide 21

Slide 21 text

Beyond Developers

Slide 22

Slide 22 text

Centralising your deployment

Slide 23

Slide 23 text

CTO & Co-Founder, Elastic Creator of Elasticsearch

Slide 24

Slide 24 text

ABRVTNS We love naming

Slide 25

Slide 25 text

Elastic{ON}

Slide 26

Slide 26 text

EON Do

Slide 27

Slide 27 text

ESON EON Do Ray

Slide 28

Slide 28 text

EGON ESON EON Ray Do

Slide 29

Slide 29 text

ELK Stack

Slide 30

Slide 30 text

ELK Stack

Slide 31

Slide 31 text

APIs Plugins Visualization ELK Stack

Slide 32

Slide 32 text

Along Came Beats ELKB

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

The Elastic Stack

Slide 35

Slide 35 text

elasticsearch

Slide 36

Slide 36 text

elasticsearch Columnar Store Not Yo’ Mama’s Inverted Index Date Customer Store Product Price

Slide 37

Slide 37 text

elasticsearch SECURITY VALUABLE RESOURCES CODE JVM CLASS LOADER One does not simply fork a process Java Security Manager

Slide 38

Slide 38 text

elasticsearch 1. { 2. “nodes”: 3. [ 4. [”10,42.1.120”,9200], 5. [”10,42.0.121”,9200], 6. [”10,42.0.123”,9200], 7. [”10,42.0.124”,9200], 8. [”10,42.0.125”,9200] 9. ] 10. } 1. { 2. “nodes”: 3. [ 4. [”10,42.0.121”,9200], 5. [”10,42.0.122”,9200], 6. [”10,42.0.123”,9200], 7. [”10,42.0.124”,9200], 8. [”10,42.0.125”,9200] 9. ] 10. } Cluster State Diﬀs

Slide 39

Slide 39 text

elasticsearch Y U SO SLOW Proﬁle API

Slide 40

Slide 40 text

elasticsearch Location Location  Location GEO

Slide 41

Slide 41 text

Pipeline Aggregations Thu 31 Smooth Average Data Value Upper Control Limit August Aug 03 Tue 05 Thu 07 Sat 09 Mon 11 Wed 13 Fri 15 Aug 17 Tue 19 10 20 30 40 50 60 70 10 20 30 40 50 60 70 Numb3rs elasticsearch

Slide 42

Slide 42 text

kibana

Slide 43

Slide 43 text

kibana Eye-meltingly colouriﬁc Colour picker

Slide 44

Slide 44 text

kibana Naming  matters Custom Legends

Slide 45

Slide 45 text

kibana Not only for the 1% Field formatters

Slide 46

Slide 46 text

kibana Back in Black Black theme

Slide 47

Slide 47 text

kibana Predator Vision Heat map

Slide 48

Slide 48 text

kibana Ice Ice Baby Pluggable Tile Servers

Slide 49

Slide 49 text

kibana Which 3pm? Global timezone

Slide 50

Slide 50 text

logstash

Slide 51

Slide 51 text

logstash Faster, more reliable pipeline New pipeline Architecture In-Memory Synchronous Queue InputThread1 Worker Thread 1 [, , ...] [, , ...] Batch Stage Filter Stage Output Stage RedisInput InputThread1 Worker Thread 1 [, , ...] [, , ...] Batch Stage Filter Stage Output Stage BeatsInput

Slide 52

Slide 52 text

logstash Responsive design Conﬁg Reload

Slide 53

Slide 53 text

logstash Kafka, HDFS, Salesforce, HTTP, Oh my! Plugins Analysis Alerting Monitoring Archiving Elasticsearch + ANY data store Watcher + ANY notiﬁcations tools Marvel + ANY monitoring tools Hadoop + ANY cloud storage platform Log and metrics data Web and social data Sensor and device data Data stores and streams

Slide 54

Slide 54 text

logstash JAVA is the assembly of (J)Ruby Java Event

Slide 55

Slide 55 text

logstash Events 0 1000000 2000000 3000000 4000000 5000000 6000000 7000000 8000000 Logstash Releases 1.4.4 1.5.0 2.0.0 2.1.0 2.2.0 (Pipeline) 2.3.0 (Java Event) Apache Parsing Complex Syslog Now you can grok faster Performance

Slide 56

Slide 56 text

beats

Slide 57

Slide 57 text

Capture the Packet Packetbeat

Slide 58

Slide 58 text

Capture the Packet Packetbeat

Slide 59

Slide 59 text

Unleash the Beats libbeat Beat 1 libbeat Beat 2 Beat 3 +

Slide 60

Slide 60 text

It was only supposed to be a demo topbeat

Slide 61

Slide 61 text

It was only supposed to be a demo topbeat

Slide 62

Slide 62 text

To tail a File ﬁlebeat + logstash

Slide 63

Slide 63 text

To tail a File ﬁlebeat + logstash

Slide 64

Slide 64 text

Welcome to 1998 winlogbeat

Slide 65

Slide 65 text

Now winlogbeat

Slide 66

Slide 66 text

MySQL metricbeat Redis Apache + Connecting Numb3rs metricbeat

Slide 67

Slide 67 text

Versions

Slide 68

Slide 68 text

Jun 9, 2015 1.6 Jul 16, 2015 1.7 Feb 19, 2015 4.0 Jun 10, 2015 4.1 May 14th, 2015 1.5 May 27th, 2015 1.0 Beta 1 July 13th, 2015 1.0 Beta 2 Sept 4 th, 2015 1.0 Beta 3 May 23, 2015 1.5 Nov 5, 2014 1.4 It’s complicated es kibana ls beats

Slide 69

Slide 69 text

es kibana ls beats Oct 28th Nov 21st Feb 2nd 2.0 4.2 2.0 2.1 4.3 2.1 1.0 2.2 4.4 2.2 1.1 Release Bonanza

Slide 70

Slide 70 text

v 5. 0

Slide 71

Slide 71 text

“I just want to tail a ﬁle.” Ingest

Slide 72

Slide 72 text

Grok Geo

Slide 73

Slide 73 text

Simple things should be simple Not like this Like this

Slide 74

Slide 74 text

I N G E S T

Slide 75

Slide 75 text

kibana

Slide 76

Slide 76 text

Kibana 4 Discover Visualize Dashboard

Slide 77

Slide 77 text

Marvel 2.0

Slide 78

Slide 78 text

Timelion

Slide 79

Slide 79 text

Slide 80

Slide 80 text

A Window   to our Stack

Slide 81

Slide 81 text

Rashid Khan Creator of Kibana Demo

Slide 82

Slide 82 text

Extensions

Slide 83

Slide 83 text

We love extensions

Slide 84

Slide 84 text

Packs

Slide 85

Slide 85 text

NO OPEN SOURCE ENTERPRISE EDITION

Slide 86

Slide 86 text

Security: Shield Authentication Authorization Encryption IP Filtering Audit Logging

Slide 87

Slide 87 text

Security: Shield Field and Document Level Security

Slide 88

Slide 88 text

Alerting: Watcher • Alerts • Notiﬁcations to email, Slack, JIRA, Hipchat, PagerDuty, and more • Analyze Watch history

Slide 89

Slide 89 text

Security in Kibana • Session Management • Login/ & Logout Capabilities

Slide 90

Slide 90 text

Bundled set of features: Security, Alerting, Monitoring, and more … x-pack

Slide 91

Slide 91 text

Uri Boness Co-Founder, Elastic X-Pack Engineering Lead Demo

Slide 92

Slide 92 text

Gephi

Slide 93

Slide 93 text

Mark Harwood Software Engineer Graph Demo

Slide 94

Slide 94 text

Elasticsearch + Kibana as a Service Latest release of the Elastic Stack and X-Pack

Slide 95

Slide 95 text

Cute but hard to ﬁnd Did I mention naming?

Slide 96

Slide 96 text

cloud

Slide 97

Slide 97 text

Cloud as a Product * Not actual packaging *

Slide 98

Slide 98 text

It’s all about choice We want to install it

Slide 99

Slide 99 text

Even more choice Many clusters / use cases, exposed within the organization Single use case, as a service, exposed within the organization You want to install it

Slide 100

Slide 100 text

PRIVATE BETA cloud Elastic Cloud Deploy Elastic Cloud Enterprise Download

Slide 101

Slide 101 text

Michael Basnight Software Engineer Njal Karevoll Software Engineer Demo

Slide 102

Slide 102 text

No content

Slide 103

Slide 103 text

103 Jason McGee @jrmcgee IBM Fellow, VP and CTO, IBM Cloud Platform

Slide 104

Slide 104 text

Slide 105

Slide 105 text

Slide 106

Slide 106 text

©2015 IBM Corporation IBM 106 Kibana Logstash Cluster App  logs Object Storage Application server Logstash agent Elasticsearch Cluster Kafka Cluster Zookeeper Cluster IBM 3 Watson Developer Cloud 500m events/day 50+ apps 300-500GB data/day Bare-metal nodes w/24 TB storage Env tags added, filtered by “topic” Can someone get me a screen shot as I do not have access to Kibana.

Slide 107

Slide 107 text

©2015 IBM Corporation IBM 107 IBM 4 Twitter Insights for Bluemix Elasticsearch Cluster Data compliancy (deleted/ protected/ unprotected tweets current) ~100m events/day 400-700 tweets/sec ES Cluster: 44 Bare-metal nodes, each w/8 TB storage, 128GB RAM ~2 years of searchable Twitter data (~25B tweets, ~50TB of original data) Kafka Cluster Tweets queue Compliance queue Zookeeper Cluster Spark HA Cluster Tweets Receiver Compliance Events Recv’r Ingestion / Enrichment Compliance processing WebSphere Cluster REST Search API Twitter Insights Bluemix Service

Slide 108

Slide 108 text

Slide 109

Slide 109 text

©2015 IBM Corporation IBM 109 Openstack Control Plane Virtual Machines on bare metal Elasticsearch Cluster Logstash Cluster Kafka Cluster App  logs Cloud-based services Metrics topic Logging topic Graphite Cluster Zookeeper Cluster Carbon Cache Cluster Containers Grafana Kibana Lumberjack Cluster Load balancing IBM 6 Bluemix monitoring & logging service Metrics ingestion Log ingestion Replace with Kibana dashboard and include new KI product icon Can someone get me a screen shot as I do not have access to Kibana.

Slide 110

Slide 110 text

©2015 IBM Corporation IBM 110 7 Up to 7GB & 7 days of logs per Bluemix space 5 production deployments on 2 continents & growing 5 1000+ tenants growing to millions Several TB data/day & growing 10s of nodes per production deployment 25k+ shards per production deployment Expanding Quickly… IBM 7 ©2015 IBM Corporation

Slide 111

Slide 111 text

Slide 112

Slide 112 text

Slide 113

Slide 113 text

Slide 114

Slide 114 text

Slide 115

Slide 115 text

©2015 IBM Corporation IBM 115 IBM 12 Container Log Crawler Host Multi-tenant Logstash forwarder Logstash Cluster Kafka Cluster Zookeeper Cluster Kibana Elasticsearch Cluster Multi-tenant proxy Built in multi-tenancy IBM 12 Not sure what to do here? Maybe update image with new KI dashboard and use the KI icon?

Slide 116

Slide 116 text

©2015 IBM Corporation IBM 116 IBM 13 ©2015 IBM Corporation Combined Logs and Metrics I don’t think we can replace this graphic as it has data from their clusters. Perhaps, clip out the Kibana at the top and put the new KI icon somewhere?

Slide 117

Slide 117 text

Slide 118

Slide 118 text

©2015 IBM Corporation IBM 118 Design Thinking Containers Extreme Agile Mobile IoT APIs Microservices Cognitive Build  something   cool! Watson IBM Containers ElasticSearch by Compose Twilio Slack IBM 15 ©2015 IBM Corporation Watson IBM Containers ElasticSearch by Compose Twilio Slack Build Something Cool!

Slide 119

Slide 119 text

Slide 120

Slide 120 text

©2015 IBM Corporation IBM 120 Lorem Ipsum dolor sit, to amet consectetur irare a adispicing elit done et ectals tempus. Quote “ “ Author, Secondary Information September 16, 2015 Presentation Title Thank You! ©2015 IBM Corporation 17 Jason McGee @jrmcgee

Slide 121

Slide 121 text

Thank you!