Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Supporting Dynamic Growth with the Elastic Stack - WIDS 2016

Supporting Dynamic Growth with the Elastic Stack - WIDS 2016

World Internet Developer Summit 2016 - Supporting Dynamic Growth with the Elastic Stack

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

June 07, 2016
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. ‹#› Christian Strzadala Elastic Engineer working on Elastic Cloud @cstrzadala

    Supporting Dynamic Growth with the Elastic Stack
  2. What makes up the Elastic stack 2

  3. ‹#› Show Me The Movies! 3

  4. What is the ‘Show Me The Movies’ offering? • A

    SaaS offering for media providers to enable streaming of their content • Each media provider will have video and meta data stored in the system • Analytical data on videos streamed will also be stored and offered to media providers • Our service provides a public API for media providers to use the meta and analytical data to build their solutions 4
  5. As our business grows How does this affect our system?

    5 • Increased traffic / load to our public API’s • Increased storage of media meta data • Increased storage of analytical data • Increased reliance of system stability
  6. ‹#› Why use the Elastic Stack? A developers perspective

  7. ‹#› The quick overview of Elasticsearch 7

  8. Elasticsearch Cluster 8

  9. Elasticsearch Index 9

  10. Elasticsearch Shards 10

  11. Types and Mapping Example Document 11 { "name": { "first":

    "John" } }
  12. Types and Mapping Example Mapping 12 { "person" : {

    "properties" : { "name" : { "properties" : { "first" : { "type" : "string" } } } } } }
  13. ‹#› Scaling Elasticsearch 13

  14. ‹#› How do we know how much data a shard

    can hold? Let’s talk sizing our cluster
  15. 15 1 2 3 Start with a single shard Ingest

    data we expect for production into test cluster Performance test application and evaluate metrics to find upper limit of single shard What’s the best cluster configuration? Well, it depends
  16. ‹#› We found our shard limits, now how can we

    scale for the future? Talking shard overallocation
  17. Shard Overallocation 17 PUT /my_index { "settings": { "number_of_shards": 2,

    "number_of_replicas": 0 } }
  18. ‹#› I don’t know how big this is going to

    be, and I can’t change the index size later on, so to be on the safe side, I’ll just give this index 1,000 shards… New Elasticsearch user
  19. The ‘Kagillion Shard’ Problem • A shard is a Lucene

    index under the covers, which uses file handles, memory, and CPU cycles. • Every search request needs to hit a copy of every shard in the index. • Term statistics, used to calculate relevance, are per shard. Having a small amount of data in many shards leads to poor relevance. 19 A little overallocation is good. A kagillion shards is bad.
  20. ‹#› Scaling search queries

  21. Replicas 21 PUT /my_index/_settings { "number_of_replicas": 1 }

  22. Balancing Load with Replicas 22 PUT /my_index/_settings { "number_of_replicas": 2

    }
  23. ‹#› Coping with failure

  24. Coping with failure Initial cluster 24

  25. Coping with failure Cluster with Node 1 failing 25

  26. ‹#› Scalable Architecture Designs 26

  27. ‹#› Storing User Data How we store our ‘Media Provider’

    meta data
  28. Index Per Media Provider • Easy to implement by appending

    media provider ID to index name • Data is easily separated between media providers • Indexes can have specific shards and replicas to deal with the amount of data for the media provider • Works really well while the amount of media providers is small 28
  29. Index Per Media Provider Example creating 3 indexes for 3

    Media Providers 29 PUT /media_provider_1 { "settings": { "number_of_shards": 1, "number_of_replicas": 1 } } PUT /media_provider_2 { "settings": { "number_of_shards": 1, "number_of_replicas": 1 } } PUT /media_provider_3 { "settings": { "number_of_shards": 1, "number_of_replicas": 1 } }
  30. What happens when we get too many media providers? 30

  31. 31

  32. 32

  33. Single Shared Index • Resources can be dedicated to the

    single index • Utilising filters and aliases, we can separate media providers data based on their ID • Define routing to increase search performance • Able to easily move heavy users off the shared index to their own dedicated index 33
  34. Single Shared Index New Media Providers Index 34 PUT /media_providers

    { "settings": { "number_of_shards": 5, "number_of_replicas": 1 } }
  35. Single Shared Index Add new meta data for Media Provider

    35 PUT /media_providers/media/1 { “media_provider_id”: 123, "title": "Zootopia" }
  36. Single Shared Index Alias 36 POST /_aliases { "actions": [

    {“add":{"alias": "media_provider_1", "index": “media_providers” }} ] }
  37. What happens if a Media Provider starts to cause performance

    issues on our cluster? 37
  38. Separating Out Large Media Providers • We can create a

    separate index specifically for the large media provider • Migrating data from the existing shared index can be done using a scroll query and the bulk API • We can then set an index alias to point to the new index 38
  39. Separating Out Large Media Providers 39 PUT /big_media_provider_1 { "settings":

    { "number_of_shards": 2, "number_of_replicas": 1 } } POST /_aliases { "actions": [ {"remove": { "alias": “media_provider_1”, "index": “media_providers"}}, {“add":{"alias": "media_provider_1", "index": “big_media_provider_1” }} ] }
  40. ‹#› Logs, logs, everywhere logs! 40

  41. Time based indices • A single large index would run

    out of space and resources quickly • Time based indexes are created for a specific time period, IE. Monthly • You can change the index configuration much quicker as new indexes are created more often • Use of aliases make querying and ingesting data easy • Your able to remove old data (old indexes) easily to save resources • Elastic Curator https://github.com/elastic/curator 41
  42. Time based indices Creating a time based index 42 PUT

    /logs_2016-06 { "settings": { "number_of_shards": 1, "number_of_replicas": 1 } }
  43. Time based indices Aliases 43 POST /_aliases { "actions": [

    { "add": { "alias": "logs_current", "index": "logs_2016-06" }}, { "remove": { "alias": "logs_current", "index": "logs_2016-05" }}, { "add": { "alias": "last_month", "index": "logs_2016-05" }}, { "remove": { "alias": "last_month", "index": "logs_2016-04" }} ] }
  44. ‹#› Ingesting Log Data

  45. 45 Logstash

  46. 46 Beats

  47. Logging Architecture 47

  48. Can we make the architecture simpler? 48

  49. Ingest Pipeline - coming in 5.0 • Elasticsearch will have

    an ingest node, which will do document enrichment • Great for processing simple documents with some enrichment • Logstash will should still be used for more complex document enrichment 49
  50. ‹#› Analytics 50

  51. Time based indexes for analytical data • Analytical data generally

    is associated with a timestamp • Allows us to query easily over a certain time period without overhead of querying all data • Example: How many plays of a particular media over the last day • We can easily remove old data that we no longer offer to our media providers • We only want to offer the last 3 years of analytical data 51
  52. Time based indexes for analytical data Example time based index

    for our analytical data 52 PUT /media_plays_01_06_2016 { "settings": { "number_of_shards": 1, "number_of_replicas": 1 } }
  53. Analytics Architecture 53 Proxy Media Player

  54. ‹#› Visualising all this data 54

  55. Kibana • Visualise all your data • Seamless integration into

    Elasticsearch • Build sophisticated dashboards for analytics • Visually interact with Elasticsearch 55
  56. 56

  57. ‹#› Scaling with our ever growing business 57

  58. 58 • Security (formally Shield) • Alerting (formally Watcher) •

    Monitoring (formally Marvel) • Graph • Reporting
  59. ‹#› Thank you!