Slide 1

Slide 1 text

Elastic{ON} 2018 Stretching the Cloud: Flexibility in Cloud Deployments Jonathan Halterman, Cloud Software Engineer Uri Cohen, Cloud Product Manager

Slide 2

Slide 2 text

Agenda 2 1 Intro to Elastic Cloud [Enterprise] 2 Why We’re Here 4 How It Really Works 5 What’s Next 3 A Cluster’s Tale

Slide 3

Slide 3 text

3 Providers AWS | GCP Azure is coming soon 2 Regions 9 AWS + 4 GCP More in the pipe 13 Clusters For any use case you can think of 10K+ Docs And growing as we speak... 300B+ Elastic Cloud in A Nutshell

Slide 4

Slide 4 text

4 Elastic Cloud Enterprise

Slide 5

Slide 5 text

5 Customers From all industries and sizes Dozens Releases 1 major 1 minor 5 service releases 7 Clusters Managed by ECE Thousands Hosts Already running ECE in Production Hundreds Elastic Cloud Enterprise, 10 Months Post GA

Slide 6

Slide 6 text

Why We’re Here

Slide 7

Slide 7 text

In Elastic Cloud, All Nodes & Hosts Are the Same • Every node is Master, Data and Ingest, all at the same time • They all run on the same I/O optimized setup

Slide 8

Slide 8 text

In Elastic Cloud, All Nodes & Hosts Are the Same AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest

Slide 9

Slide 9 text

A Cluster’s Tale

Slide 10

Slide 10 text

10 The typical Elastic adoption pattern Start Small 1

Slide 11

Slide 11 text

11 The typical Elastic adoption pattern Start Small 1 Grow Bigger 2 More data

Slide 12

Slide 12 text

12 The typical Elastic adoption pattern Start Small 1 Even Bigger Grow Bigger 2 3 More data More Users

Slide 13

Slide 13 text

13 The typical Elastic adoption pattern Start Small 1 Even Bigger Grow Bigger 2 3 More data More Users Do More with Your Data: ML and APM! 4 More Insights

Slide 14

Slide 14 text

Demo Time!

Slide 15

Slide 15 text

15 Grow Your Cluster Further - Hot/Warm Master Kibana Hot Nodes Warm Nodes Automatic Index Curation Data Feed

Slide 16

Slide 16 text

Demo Time!

Slide 17

Slide 17 text

17 Grow Your Cluster Further - Hot/Warm

Slide 18

Slide 18 text

A Peek Under the Hood

Slide 19

Slide 19 text

What We’ve Seen Multiple Node Types In the Same Cluster 1 2 3 Place each node on different HW that’s most appropriate for it Deployment templates: a way to model and deploy an entire Elastic stack

Slide 20

Slide 20 text

Why Different Hardware Profiles? SSDs Fast data access, fast writes. But $$$$ HDDs slow data access, but cheap. Good for archiving Lots of CPU Processing at Ingestion Many queries Lots of RAM Aggregations Kibana Master nodes APM

Slide 21

Slide 21 text

AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 Host Tags and Metadata AWS i3 - High I/O AWS d2 - Dense Storage AWS c5 - High CPU AWS r4 - High Mem AWS m5 - Gen Purpose instance-type: aws.i3 ssd: true vpc: true foo: bar ---- RAM: 244GB Disk: 7600GB vCPUs: 32 instance-type: aws.d2 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 48000GB vCPUs: 36 instance-type: aws.c5 ssd: false vpc: true foo: qux ---- RAM: 144GB Disk: 1024GB vCPUs: 36 instance-type: aws.r4 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 1024 vCPUs: 32 instance-type: aws.m5 ssd: false vpc: true foo: bar ---- RAM: 192GB Disk: 1024 vCPUs: 48

Slide 22

Slide 22 text

AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 Host Tags AWS d2 - Dense Storage AWS c5 - High CPU AWS r4 - High Mem AWS m5 - Gen Purpose instance-type: aws.d2 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 48000GB vCPUs: 36 instance-type: aws.c5 ssd: false vpc: true foo: qux ---- RAM: 144GB Disk: 1024GB vCPUs: 36 instance-type: aws.r4 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 1024 vCPUs: 32 instance-type: aws.m5 ssd: false vpc: true foo: bar ---- RAM: 192GB Disk: 1024 vCPUs: 48 AWS i3 - High I/O instance-type: aws.i3 ssd: true vpc: true foo: bar ---- RAM: 244GB Disk: 7600GB vCPUs: 32

Slide 23

Slide 23 text

AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 Host Metadata AWS d2 - Dense Storage AWS c5 - High CPU AWS r4 - High Mem AWS m5 - Gen Purpose instance-type: aws.d2 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 48000GB vCPUs: 36 instance-type: aws.c5 ssd: false vpc: true foo: qux ---- RAM: 144GB Disk: 1024GB vCPUs: 36 instance-type: aws.r4 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 1024 vCPUs: 32 instance-type: aws.m5 ssd: false vpc: true foo: bar ---- RAM: 192GB Disk: 1024 vCPUs: 48 AWS i3 - High I/O instance-type: aws.i3 ssd: true vpc: true foo: bar ---- RAM: 244GB Disk: 7600GB vCPUs: 32

Slide 24

Slide 24 text

Elastic Stack “Instance Types” Elasticsearch Master Ingest ML Kibana Logstash APM Data

Slide 25

Slide 25 text

Choosing the Right Hardware Data - High I/O Requirements: Fast I/O, Ample Memory Filter: instance-type is "aws.i3" Disk:RAM ratio: 30GB disk for every 1GB RAM i3 m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4

Slide 26

Slide 26 text

Choosing the Right Hardware Requirements: Tons of storage, Ample Memory Filter: instance-type is "aws.d2" Disk:RAM ratio: 100GB disk for every 1GB RAM i3 Data - High Storage m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4

Slide 27

Slide 27 text

Choosing the Right Hardware Requirements: Ample CPU, Ample RAM Machine Learning Filter: instance-type is "aws.m5" Disk:RAM ratio: 5GB disk for every 1GB RAM i3 m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4

Slide 28

Slide 28 text

Choosing the Right Hardware Requirements: Ample RAM Kibana Filter: instance-type is "aws.r4" Disk:RAM ratio: 4GB disk for every 1GB RAM i3 m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4

Slide 29

Slide 29 text

How It Works: Instance Configurations

Slide 30

Slide 30 text

Instance Configurations Demo

Slide 31

Slide 31 text

Tying It All Together: Deployment Templates Template: hot/warm Data - Hot Default size: 240GB Storage Number of zones: 3 Filter: instance-type is "aws.i3" Disk:RAM ratio: 30 Data - Warm Default size: 1TB Storage Number of zones: 2 Filter: instance-type is "aws.d2" Disk:RAM ratio: 100 Kibana Default size: 1GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 4 Master Default size: 2GB RAM Number of zones: 3 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3 Machine Learning Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.m5" Disk:RAM ratio: 4 APM Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3

Slide 32

Slide 32 text

Tying It All Together: Deployment Templates Template: hot/warm Kibana Default size: 1GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 4 Master Default size: 2GB RAM Number of zones: 3 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3 Machine Learning Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.m5" Disk:RAM ratio: 4 APM Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3 Data - High Storage Default size: 1TB Storage Number of zones: 2 Filter: instance-type is "aws.d2" Disk:RAM ratio: 100 Data - High I/O Default size: 240GB Storage Number of zones: 3 Filter: instance-type is "aws.i3" Disk:RAM ratio: 30

Slide 33

Slide 33 text

Deployment Template Demo

Slide 34

Slide 34 text

What’s In It for You • Cost effectiveness for more use cases: ▪ Choose the right hardware for the job ▪ e.g. for logging: o RAM:Disk ratio of ~1:100, way cheaper per GB storage • Deploy common architectural patterns in a few clicks • Bigger clusters ▪ With dedicated masters • Machine Learning, APM and Logstash* on Elastic Cloud! * Future

Slide 35

Slide 35 text

Availability & Future Plans

Slide 36

Slide 36 text

• Roll out planned for Q2 this year, stay tuned! ▪ For both Elastic Cloud and Elastic Cloud Enterprise ▪ Includes I/O, storage, CPU and RAM optimized HW o Templates for hot/warm ▪ Includes Elasticsearch (with ML), Kibana, APM • Future improvement: ▪ Gradually transition data nodes to different hardware profiles o e.g. start with hot-only, move to hot/warm What’s Next

Slide 37

Slide 37 text

37 More Questions? Visit us at the AMA

Slide 38

Slide 38 text

www.elastic.co

Slide 39

Slide 39 text

Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 39 Please attribute Elastic with a link to elastic.co