Elastic{ON} 2018 - Stretching the Cloud -- Flexibility in cloud deployments

New Elastic Cloud capabilities allow you to match your provisioned hardware to your use cases, making it easier to efficiently run Elasticsearch for multiple workloads such as hot/warm architectures, dedicated master nodes, and machine learning and enable Elastic Stack features like APM, Logstash, and more.

Jonathan Halterman | Cloud Engineer | Elastic
Uri Cohen | Director of Product Management | Elastic

Transcript

1. Elastic{ON} 2018 Stretching the Cloud: Flexibility in Cloud Deployments Jonathan Halterman, Cloud Software Engineer Uri Cohen, Cloud Product Manager
2. Agenda 2 1 Intro to Elastic Cloud [Enterprise] 2 Why We’re Here 4 How It Really Works 5 What’s Next 3 A Cluster’s Tale
3. 3 Providers AWS | GCP Azure is coming soon 2 Regions 9 AWS + 4 GCP More in the pipe 13 Clusters For any use case you can think of 10K+ Docs And growing as we speak... 300B+ Elastic Cloud in A Nutshell
4. 4 Elastic Cloud Enterprise
5. 5 Customers From all industries and sizes Dozens Releases 1 major 1 minor 5 service releases 7 Clusters Managed by ECE Thousands Hosts Already running ECE in Production Hundreds Elastic Cloud Enterprise, 10 Months Post GA
6. Why We’re Here
7. In Elastic Cloud, All Nodes & Hosts Are the Same • Every node is Master, Data and Ingest, all at the same time • They all run on the same I/O optimized setup
8. In Elastic Cloud, All Nodes & Hosts Are the Same AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest
9. A Cluster’s Tale
10. 10 The typical Elastic adoption pattern Start Small 1
11. 11 The typical Elastic adoption pattern Start Small 1 Grow Bigger 2 More data
12. 12 The typical Elastic adoption pattern Start Small 1 Even Bigger Grow Bigger 2 3 More data More Users
13. 13 The typical Elastic adoption pattern Start Small 1 Even Bigger Grow Bigger 2 3 More data More Users Do More with Your Data: ML and APM! 4 More Insights
14. Demo Time!
15. 15 Grow Your Cluster Further - Hot/Warm Master Kibana Hot Nodes Warm Nodes Automatic Index Curation Data Feed
16. Demo Time!
17. 17 Grow Your Cluster Further - Hot/Warm
18. A Peek Under the Hood
19. What We’ve Seen Multiple Node Types In the Same Cluster 1 2 3 Place each node on different HW that’s most appropriate for it Deployment templates: a way to model and deploy an entire Elastic stack
20. Why Different Hardware Profiles? SSDs Fast data access, fast writes. But $$$$ HDDs slow data access, but cheap. Good for archiving Lots of CPU Processing at Ingestion Many queries Lots of RAM Aggregations Kibana Master nodes APM
21. AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 Host Tags and Metadata AWS i3 - High I/O AWS d2 - Dense Storage AWS c5 - High CPU AWS r4 - High Mem AWS m5 - Gen Purpose instance-type: aws.i3 ssd: true vpc: true foo: bar ---- RAM: 244GB Disk: 7600GB vCPUs: 32 instance-type: aws.d2 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 48000GB vCPUs: 36 instance-type: aws.c5 ssd: false vpc: true foo: qux ---- RAM: 144GB Disk: 1024GB vCPUs: 36 instance-type: aws.r4 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 1024 vCPUs: 32 instance-type: aws.m5 ssd: false vpc: true foo: bar ---- RAM: 192GB Disk: 1024 vCPUs: 48
22. AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 Host Tags AWS d2 - Dense Storage AWS c5 - High CPU AWS r4 - High Mem AWS m5 - Gen Purpose instance-type: aws.d2 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 48000GB vCPUs: 36 instance-type: aws.c5 ssd: false vpc: true foo: qux ---- RAM: 144GB Disk: 1024GB vCPUs: 36 instance-type: aws.r4 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 1024 vCPUs: 32 instance-type: aws.m5 ssd: false vpc: true foo: bar ---- RAM: 192GB Disk: 1024 vCPUs: 48 AWS i3 - High I/O instance-type: aws.i3 ssd: true vpc: true foo: bar ---- RAM: 244GB Disk: 7600GB vCPUs: 32
23. AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 Host Metadata AWS d2 - Dense Storage AWS c5 - High CPU AWS r4 - High Mem AWS m5 - Gen Purpose instance-type: aws.d2 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 48000GB vCPUs: 36 instance-type: aws.c5 ssd: false vpc: true foo: qux ---- RAM: 144GB Disk: 1024GB vCPUs: 36 instance-type: aws.r4 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 1024 vCPUs: 32 instance-type: aws.m5 ssd: false vpc: true foo: bar ---- RAM: 192GB Disk: 1024 vCPUs: 48 AWS i3 - High I/O instance-type: aws.i3 ssd: true vpc: true foo: bar ---- RAM: 244GB Disk: 7600GB vCPUs: 32
24. Elastic Stack “Instance Types” Elasticsearch Master Ingest ML Kibana Logstash APM Data
25. Choosing the Right Hardware Data - High I/O Requirements: Fast I/O, Ample Memory Filter: instance-type is "aws.i3" Disk:RAM ratio: 30GB disk for every 1GB RAM i3 m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4
26. Choosing the Right Hardware Requirements: Tons of storage, Ample Memory Filter: instance-type is "aws.d2" Disk:RAM ratio: 100GB disk for every 1GB RAM i3 Data - High Storage m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4
27. Choosing the Right Hardware Requirements: Ample CPU, Ample RAM Machine Learning Filter: instance-type is "aws.m5" Disk:RAM ratio: 5GB disk for every 1GB RAM i3 m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4
28. Choosing the Right Hardware Requirements: Ample RAM Kibana Filter: instance-type is "aws.r4" Disk:RAM ratio: 4GB disk for every 1GB RAM i3 m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4
29. How It Works: Instance Configurations
30. Instance Configurations Demo
31. Tying It All Together: Deployment Templates Template: hot/warm Data - Hot Default size: 240GB Storage Number of zones: 3 Filter: instance-type is "aws.i3" Disk:RAM ratio: 30 Data - Warm Default size: 1TB Storage Number of zones: 2 Filter: instance-type is "aws.d2" Disk:RAM ratio: 100 Kibana Default size: 1GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 4 Master Default size: 2GB RAM Number of zones: 3 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3 Machine Learning Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.m5" Disk:RAM ratio: 4 APM Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3
32. Tying It All Together: Deployment Templates Template: hot/warm Kibana Default size: 1GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 4 Master Default size: 2GB RAM Number of zones: 3 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3 Machine Learning Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.m5" Disk:RAM ratio: 4 APM Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3 Data - High Storage Default size: 1TB Storage Number of zones: 2 Filter: instance-type is "aws.d2" Disk:RAM ratio: 100 Data - High I/O Default size: 240GB Storage Number of zones: 3 Filter: instance-type is "aws.i3" Disk:RAM ratio: 30
33. Deployment Template Demo
34. What’s In It for You • Cost effectiveness for more use cases: ▪ Choose the right hardware for the job ▪ e.g. for logging: o RAM:Disk ratio of ~1:100, way cheaper per GB storage • Deploy common architectural patterns in a few clicks • Bigger clusters ▪ With dedicated masters • Machine Learning, APM and Logstash* on Elastic Cloud! * Future
35. Availability & Future Plans
36. • Roll out planned for Q2 this year, stay tuned! ▪ For both Elastic Cloud and Elastic Cloud Enterprise ▪ Includes I/O, storage, CPU and RAM optimized HW o Templates for hot/warm ▪ Includes Elasticsearch (with ML), Kibana, APM • Future improvement: ▪ Gradually transition data nodes to different hardware profiles o e.g. start with hot-only, move to hot/warm What’s Next
37. 37 More Questions? Visit us at the AMA
38. www.elastic.co
39. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 39 Please attribute Elastic with a link to elastic.co