Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elastic Cloud Inside Out

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
March 08, 2017

Elastic Cloud Inside Out

Elastic Cloud runs thousands of clusters and is growing rapidly, while maintaining solid SLAs and allowing users to scale out, upgrade, and reliably monitor their clusters. Ever wonder how it works?

This session will go under the hood into how Elastic Cloud is built, how it handles complex tasks like efficient placement, zero-downtime upgrades, and host failures, as well as how the Elastic Cloud team uses it to orchestrate and manage thousands of clusters for Elastic Cloud customers.

Uri Cohen l Sr. Director, Product Management l Elastic
Ben Osborne l Site Reliability Engineer l Elastic
Christian Strzadala l Software Engineer l Elastic


Elastic Co

March 08, 2017


  1. Elastic March 8, 2017 Elastic Cloud Inside Out Ben Osborne

    @bj_osborne Christian Strzadala @cstrzadala Uri Cohen @uri1803
  2. Agenda 2 1 Elastic Cloud, 2 years in 2 What

    it looks like under the hood 3 What it takes to keep the lights on for thousands of clusters 4 Elastic Cloud as a Product!
  3. 3 Elastic Cloud Hosted Elasticsearch & Kibana Includes X-Pack features

    Starts at $45/mo
  4. Brief History Found Beta Found GA Elastic acquires Found Found

    Gold & Platinum Elasticsearch 2.0 available on Found Say Heya to Elastic Cloud Elasticsearch & Kibana 5.0 on Cloud 2012 2013 2015 2016 2017 Elastic Cloud Enterprise Public Beta
  5. We’ve Come a Long Way… Monthly Signups Growth

  6. We’ve Come a Long Way… Clusters Growth

  7. We’ve Come a Long Way… Customer Growth

  8. We’ve Come a Long Way… Monthly Bookings Growth

  9. Why Elastic Cloud? • Fully managed • Super easy scaling

    & upgrades • Runs on top notch instances • Always the latest and greatest • X-Pack! • Support right from the source • All the APIs are there
  10. Elastic Cloud Components & Architecture Under the hood

  11. • Service orientated architecture • Utilise AWS • Immutable services

    • Docker for containerisation • Services created specifically for Elastic stack component deployment Key features
  12. Demo: Let’s create a cluster

  13. None
  14. Whoa, slow down cowboy! Zookeeper, Constructor, plans, where do these

    fit in?
  15. None
  16. Zookeeper Our beloved coordination system • Distributed, strongly consistent datastore

    • CaP • Fast reads for small pieces of data • Watchers
  17. What's the story (with a plan) morning glory? { "availability_zones":

    1, "capacity": 1024, "failover": {}, "plan_id": "887a397f-abc1-4896-8eca-e3c60d4ca10d", "strategy": {}, "source": { "action": "elasticsearch.create-cluster", "user_id": 3403111101, "date": "2017-03-06T22:22:46.654Z", "facilitator": "console", }, "elasticsearch": { "version": "5.2.2", "user_settings_overrides": null, "plugins": [], "user_plugins": [], "user_bundles": [], "user_settings": {} }, "ssd": true, "instance_capacity": 1024, "instance_count": 1, "region": "us-east-1" }
  18. Cooking up clusters since 2012 The Constructor

  19. • Our cluster scheduler. • Calculates what needs to be

    changed when a cluster is added or reconfigured • Uses the concept of 'steps’ when processing a pending plan to progress the plan forward (or backward) The Constructor
  20. • Steps are the stages that need to complete for

    a cluster to become available • Steps involve: • Querying Zookeeper data • Querying the cluster • Controlling allocation configuration, such as moving data away from a node it’s about to kill. • Rollbacks (Cluster changes can fail) Constructor Steps
  21. We got to have a home for these instances Allocator

  22. • Manages Elasticsearch and Kibana nodes • We use large

    AWS instances to house our services • We have a pool of allocators in each region • Each allocator advertises its resources in zookeeper • Each node that runs on an allocator gets its own docker container The Allocator
  23. The Whale in the room

  24. • All services including Elasticsearch and Kibana are deployed as

    Docker containers • The history of the platform has moved from a shared cluster to clusters deployed with containers • Guarantee a share of resources • Malicious attacks are limited to a container Docker
  25. How we keep feeding the whale Docker container management

  26. The Runner

  27. None
  28. We love stunnels

  29. None
  30. Where did I put that cluster? The Proxy

  31. • We don't expose clusters directly, but route all requests

    through a proxy layer. • Maintains an in-memory routing table of Zookeeper data The Proxy
  32. Running Elastic Cloud

  33. 3 Elastic Cloud Admin

  34. Demo: Live Node Migration

  35. So Many Clusters

  36. >3 Billion requests per day

  37. Elastic Cloud ‘s Beats

  38. Whatcha Monitoring There ? • Zookeeper • Proxies • Load

    Balancers • Docker Containers • Host machines • Heartbeat SLA
  39. 3 ZooKeeper Stats

  40. 4 Proxy Stats

  41. Elastic Cloud as a Product Introducing Elastic Cloud Enterprise

  42. Provisioning, orchestration, and management of multiple clusters Deployed on-premise, in

    your private cloud … or wherever you want Leverages the same technology used in Elastic Cloud Automates frequent tasks such as snapshot/restore, upgrade and scale Public beta available; GA expected Q2 2017
  43. 43 More Questions? Visit us at the AMA

  44. www.elastic.co

  45. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/

    Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 45 Please attribute Elastic with a link to elastic.co