Elastic{ON} 2018 Stretching the Cloud: Flexibility in Cloud Deployments Jonathan Halterman, Cloud Software Engineer Uri Cohen, Cloud Product Manager

Agenda 2 1 Intro to Elastic Cloud [Enterprise] 2 Why We’re Here 4 How It Really Works 5 What’s Next 3 A Cluster’s Tale

3 Providers AWS | GCP Azure is coming soon 2 Regions 9 AWS + 4 GCP More in the pipe 13 Clusters For any use case you can think of 10K+ Docs And growing as we speak... 300B+ Elastic Cloud in A Nutshell

4 Elastic Cloud Enterprise

5 Customers From all industries and sizes Dozens Releases 1 major 1 minor 5 service releases 7 Clusters Managed by ECE Thousands Hosts Already running ECE in Production Hundreds Elastic Cloud Enterprise, 10 Months Post GA

Why We’re Here

In Elastic Cloud, All Nodes & Hosts Are the Same • Every node is Master, Data and Ingest, all at the same time • They all run on the same I/O optimized setup

In Elastic Cloud, All Nodes & Hosts Are the Same AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest

A Cluster’s Tale

10 The typical Elastic adoption pattern Start Small 1

11 The typical Elastic adoption pattern Start Small 1 Grow Bigger 2 More data

12 The typical Elastic adoption pattern Start Small 1 Even Bigger Grow Bigger 2 3 More data More Users

13 The typical Elastic adoption pattern Start Small 1 Even Bigger Grow Bigger 2 3 More data More Users Do More with Your Data: ML and APM! 4 More Insights

Demo Time!

15 Grow Your Cluster Further - Hot/Warm Master Kibana Hot Nodes Warm Nodes Automatic Index Curation Data Feed

Demo Time!

17 Grow Your Cluster Further - Hot/Warm

A Peek Under the Hood

What We’ve Seen Multiple Node Types In the Same Cluster 1 2 3 Place each node on different HW that’s most appropriate for it Deployment templates: a way to model and deploy an entire Elastic stack

Why Different Hardware Profiles? SSDs Fast data access, fast writes. But $$$$ HDDs slow data access, but cheap. Good for archiving Lots of CPU Processing at Ingestion Many queries Lots of RAM Aggregations Kibana Master nodes APM

AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 Host Tags and Metadata AWS i3 - High I/O AWS d2 - Dense Storage AWS c5 - High CPU AWS r4 - High Mem AWS m5 - Gen Purpose instance-type: aws.i3 ssd: true vpc: true foo: bar ---- RAM: 244GB Disk: 7600GB vCPUs: 32 instance-type: aws.d2 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 48000GB vCPUs: 36 instance-type: aws.c5 ssd: false vpc: true foo: qux ---- RAM: 144GB Disk: 1024GB vCPUs: 36 instance-type: aws.r4 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 1024 vCPUs: 32 instance-type: aws.m5 ssd: false vpc: true foo: bar ---- RAM: 192GB Disk: 1024 vCPUs: 48

AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 Host Tags AWS d2 - Dense Storage AWS c5 - High CPU AWS r4 - High Mem AWS m5 - Gen Purpose instance-type: aws.d2 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 48000GB vCPUs: 36 instance-type: aws.c5 ssd: false vpc: true foo: qux ---- RAM: 144GB Disk: 1024GB vCPUs: 36 instance-type: aws.r4 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 1024 vCPUs: 32 instance-type: aws.m5 ssd: false vpc: true foo: bar ---- RAM: 192GB Disk: 1024 vCPUs: 48 AWS i3 - High I/O instance-type: aws.i3 ssd: true vpc: true foo: bar ---- RAM: 244GB Disk: 7600GB vCPUs: 32

AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 Host Metadata AWS d2 - Dense Storage AWS c5 - High CPU AWS r4 - High Mem AWS m5 - Gen Purpose instance-type: aws.d2 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 48000GB vCPUs: 36 instance-type: aws.c5 ssd: false vpc: true foo: qux ---- RAM: 144GB Disk: 1024GB vCPUs: 36 instance-type: aws.r4 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 1024 vCPUs: 32 instance-type: aws.m5 ssd: false vpc: true foo: bar ---- RAM: 192GB Disk: 1024 vCPUs: 48 AWS i3 - High I/O instance-type: aws.i3 ssd: true vpc: true foo: bar ---- RAM: 244GB Disk: 7600GB vCPUs: 32

Elastic Stack “Instance Types” Elasticsearch Master Ingest ML Kibana Logstash APM Data

Choosing the Right Hardware Data - High I/O Requirements: Fast I/O, Ample Memory Filter: instance-type is "aws.i3" Disk:RAM ratio: 30GB disk for every 1GB RAM i3 m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4

Choosing the Right Hardware Requirements: Tons of storage, Ample Memory Filter: instance-type is "aws.d2" Disk:RAM ratio: 100GB disk for every 1GB RAM i3 Data - High Storage m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4

Choosing the Right Hardware Requirements: Ample CPU, Ample RAM Machine Learning Filter: instance-type is "aws.m5" Disk:RAM ratio: 5GB disk for every 1GB RAM i3 m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4

Choosing the Right Hardware Requirements: Ample RAM Kibana Filter: instance-type is "aws.r4" Disk:RAM ratio: 4GB disk for every 1GB RAM i3 m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4

How It Works: Instance Configurations

Instance Configurations Demo

Tying It All Together: Deployment Templates Template: hot/warm Data - Hot Default size: 240GB Storage Number of zones: 3 Filter: instance-type is "aws.i3" Disk:RAM ratio: 30 Data - Warm Default size: 1TB Storage Number of zones: 2 Filter: instance-type is "aws.d2" Disk:RAM ratio: 100 Kibana Default size: 1GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 4 Master Default size: 2GB RAM Number of zones: 3 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3 Machine Learning Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.m5" Disk:RAM ratio: 4 APM Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3

Tying It All Together: Deployment Templates Template: hot/warm Kibana Default size: 1GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 4 Master Default size: 2GB RAM Number of zones: 3 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3 Machine Learning Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.m5" Disk:RAM ratio: 4 APM Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3 Data - High Storage Default size: 1TB Storage Number of zones: 2 Filter: instance-type is "aws.d2" Disk:RAM ratio: 100 Data - High I/O Default size: 240GB Storage Number of zones: 3 Filter: instance-type is "aws.i3" Disk:RAM ratio: 30

Deployment Template Demo

What’s In It for You • Cost effectiveness for more use cases: ▪ Choose the right hardware for the job ▪ e.g. for logging: o RAM:Disk ratio of ~1:100, way cheaper per GB storage • Deploy common architectural patterns in a few clicks • Bigger clusters ▪ With dedicated masters • Machine Learning, APM and Logstash* on Elastic Cloud! * Future

Availability & Future Plans

• Roll out planned for Q2 this year, stay tuned! ▪ For both Elastic Cloud and Elastic Cloud Enterprise ▪ Includes I/O, storage, CPU and RAM optimized HW o Templates for hot/warm ▪ Includes Elasticsearch (with ML), Kibana, APM • Future improvement: ▪ Gradually transition data nodes to different hardware profiles o e.g. start with hot-only, move to hot/warm What’s Next

37 More Questions? Visit us at the AMA

www.elastic.co

Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 39 Please attribute Elastic with a link to elastic.co