Elastic Cloud - Architecture deep dive

Slide 1

Slide 1 text

1 Elastic Cloud Architecture Morten Ingebrigtsen, Igor Kupczyński

Slide 2

Slide 2 text

2 Agenda Expertise Architecture / Index / Shard Design Cluster Management (Tuning) Query Performance Optimization Dev to Production Migration & Upgrades Best Practices (Elastic Stack, X-Pack) • Introduction & demo • Dive into architecture • Use case: recommendations with Graph • Q&A

Slide 3

Slide 3 text

3 Introduction

Slide 4

Slide 4 text

4 Demo

Slide 5

Slide 5 text

5 Architecture

Slide 6

Slide 6 text

6 Architecture — key features • Services • AWS infrastructure • Regions and availability zones • EC2 — virtual servers (i2) • Elastic Load Balancer — front proxy / load balancer • S3 — backup and binary storage • RDS — PostgreSQL • Docker all-the-things • Zookeeper — state of the system + coordination

Slide 7

Slide 7 text

7 Lets talk numbers • 7 AWS regions • Thousands of active elasticsearch clusters • Billions of elasticsearch requests per day • 15 team members • both SREs and devs, with some overlap in responsibilities • it started with 4

Slide 8

Slide 8 text

8 High availability 2TKOCT[PQFG 4GRNKECPQFG 6KGDTGCMGT #XCKNCDKNKV[ \QPG \QPG \QPG #OC\QPņU 'NCUVKE.QCF $CNCPEGT 2TQZ[ 2TQZ[ 2TQZ[ )'6AUGCTEJ

Slide 9

Slide 9 text

9 No-downtime scaling & upgrades 0QFG# XGT 2TQZ[ 2TQZ[ 2TQZ[ 0QFG$ XGT 4GRNKECVKPIFCVC

Slide 10

Slide 10 text

10 Architecture

Slide 11

Slide 11 text

11 Zookeeper

Slide 12

Slide 12 text

12 Use Zookeeper. It’s mature, well-designed, and battle-tested. Because the consequences of its connection model and linearizability properties are subtle, you should, wherever possible, take advantage of tested recipes and client libraries like Curator, which do their best to correctly handle the complex state transitions associated with session and connection loss. https://aphyr.com/posts/291-jepsen-zookeeper Kyle Kingsbury a.k.a. Aphyr

Slide 13

Slide 13 text

13 Allocator & elasticsearch nodes • Manages elasticsearch nodes • All allocators in a region form an allocator pool • Advertises its capacity and free resources in zookeeper • Each node is a docker container • Guarantees memory, CPU, IOPS, disk space • Isolates the nodes #NNQECVQT '5 0QFG

Slide 14

Slide 14 text

14 Constructor • The brains behind allocator • Calculates what needs to be changed when a cluster is added or reconfigured • Monitors new requests from user & admin console • Writes to a znode • Allocator watches it and acts • Assigns new clusters to a proper allocator %QPUVTWEVQT

Slide 15

Slide 15 text

15 Security • Stunnels between containers • znode ACLs (each container can access only part of the state important for it) • Dockerized elasticsearch nodes • Oauth & Time-based One Time Passwords in the admin console • Shield (Elastic Security) for all the clusters • Dedicated service to manage secrets in the cloud • Backups every 30 mins, retention 48 hours • Isolated S3 bucket