Elastic Cloud - Architecture deep dive

Igor Kupczynski, an Elastic Cloud Engineer, takes you into the Cloud architecture and cover a high level talk on the inner workings of Elastic Cloud, our hosted version of the Elastic Stack.

This talk was presented on the Elastic NL meetup http://www.meetup.com/Elastic-NL/events/230223068/

Transcript

1. 1 Elastic Cloud Architecture Morten Ingebrigtsen, Igor Kupczyński
2. 2 Agenda Expertise Architecture / Index / Shard Design Cluster Management (Tuning) Query Performance Optimization Dev to Production Migration & Upgrades Best Practices (Elastic Stack, X-Pack) • Introduction & demo • Dive into architecture • Use case: recommendations with Graph • Q&A
3. 3 Introduction
4. 4 Demo
5. 5 Architecture
6. 6 Architecture — key features • Services • AWS infrastructure • Regions and availability zones • EC2 — virtual servers (i2) • Elastic Load Balancer — front proxy / load balancer • S3 — backup and binary storage • RDS — PostgreSQL • Docker all-the-things • Zookeeper — state of the system + coordination
7. 7 Lets talk numbers • 7 AWS regions • Thousands of active elasticsearch clusters • Billions of elasticsearch requests per day • 15 team members • both SREs and devs, with some overlap in responsibilities • it started with 4
8. 8 High availability 2TKOCT[PQFG 4GRNKECPQFG 6KGDTGCMGT #XCKNCDKNKV[ \QPG \QPG \QPG #OC\QPņU 'NCUVKE.QCF $CNCPEGT 2TQZ[ 2TQZ[ 2TQZ[ )'6AUGCTEJ
9. 9 No-downtime scaling & upgrades 0QFG# XGT 2TQZ[ 2TQZ[ 2TQZ[   0QFG$ XGT   4GRNKECVKPIFCVC
10. 10 Architecture <QQMGGRGT 2QUVITG 53. 2TQZ[ %QPUVTWEVQT 1VJGTUGTXKEGU #FOKP %QPUQNG 7UGT%QPUQNG #NNQECVQT '5 0QFG
11. 11 Zookeeper <QQMGGRGT • Distributed data store for coordination and storing the global state • Filesystem tree-like structure — znodes • CAP — consistent in the event of network partitions • Fast reads, but only simple queries are possible • Writes need ACK from a quorum, so slower; data should fit in memory • Writes are linear — all clients see them in the same order • Possible to set watches • Elastic Cloud — zookeeper stores the state of the whole system
12. 12 Use Zookeeper. It’s mature, well-designed, and battle-tested. Because the consequences of its connection model and linearizability properties are subtle, you should, wherever possible, take advantage of tested recipes and client libraries like Curator, which do their best to correctly handle the complex state transitions associated with session and connection loss. https://aphyr.com/posts/291-jepsen-zookeeper Kyle Kingsbury a.k.a. Aphyr
13. 13 Allocator & elasticsearch nodes • Manages elasticsearch nodes • All allocators in a region form an allocator pool • Advertises its capacity and free resources in zookeeper • Each node is a docker container • Guarantees memory, CPU, IOPS, disk space • Isolates the nodes #NNQECVQT '5 0QFG
14. 14 Constructor • The brains behind allocator • Calculates what needs to be changed when a cluster is added or reconfigured • Monitors new requests from user & admin console • Writes to a znode • Allocator watches it and acts • Assigns new clusters to a proper allocator %QPUVTWEVQT
15. 15 Security • Stunnels between containers • znode ACLs (each container can access only part of the state important for it) • Dockerized elasticsearch nodes • Oauth & Time-based One Time Passwords in the admin console • Shield (Elastic Security) for all the clusters • Dedicated service to manage secrets in the cloud • Backups every 30 mins, retention 48 hours • Isolated S3 bucket
16. 16 Use case: Graph
17. 17 Q&A Go to https://cloud.elastic.co and start your trial today!