Small, Medium, or Large: Evolve Your Elastic Stack to Fit

The few thousands operations per second indexing requirement has changed to few millions; the single use case has grown to multiple; the one user group has become tens of user groups.

In this talk, Sherry will take you through the Elastic Stack topologies from small to large to meet your evolving use case(s). She will examine deployment patterns for projects of various sizes, using Beats, Elasticsearch, Kibana, Logstash, X-Pack, and others.

Sherry Ger l Support Engineer l Elastic

Transcript

1. Elastic March 08, 2017 Small, Medium, or Large: Evolve Your Elastic Stack to Fit Sherry Ger, Support Engineer
2. Agenda 2 1 Overview 2 Customizing and Evolving - Beats, Logstash, and more 3 Seeing and Understanding - Kibana for all 4 Thinking Anew - Elastic Cloud Enterprise 5 Going forward
3. 3 Ingest, Store, and See Modularity, Flexibility, Simplicity Beats Elasticsearch Kibana X-Pack X-Pack Logstash • Scaling the Elastic Stack • Growing the stack as the use case grows or the number of use cases grow • Sharing our knowledge of the Elastic products and the experience of our customers • Evolving from an one man shop to multi data center logging platform
4. 4 Minimalist - Exploring and Visualizing Logs A Simple Tool for a Simple Task Beats Log Files Windows Log Files Wire Data Elasticsearch Master / Data Nodes (3) Ingest Nodes (X) Metrics X-Pack Output to Elasticsearch and data is ready to use in Kibana. The ingest node grok processor has 120 reusable patterns. There are dozens of ingest node processors to add new data fields or mutate existing ones. Adding Packetbeat and Metricbeat are a cinch.
5. 5 Input Variety and Advanced Processing Collect, Transform, Enrich Power Beats Log Files Windows Log files Wire Data Metrics Logstash Nodes (X) Datastore Web APIs Social Sensors Logstash collects Kafka, Kinesis, PubSub, tcp, udp, jdbc, log4j, s3,sqs, twitter, and other inputs. Logstash provides advanced processing like aggregate, translate, geoip lookup, dns lookup. Elasticsearch X-Pack Master Nodes (3) Ingest Nodes (X) Data Nodes (X) Logstash has a wide range of codecs and filters for advanced processing.
6. 6 Resiliency - Safety Net Included Do More with What You Have Beats Log Files Wire Data Logstash Nodes (X) Datastore Web APIs Social Sensors Elasticsearch X-Pack Master Nodes (3) Ingest Nodes (X) Data Nodes (X) Filebeat uses a back-pressure sensitive protocol. Logstash persistent queue provides data durability in events of abnormal terminations or bursts of events. Logstash dead letter queue persists poisson messages for future reprocessing. Windows Log files Metrics Both Filebeat and the up and coming release of Logstash guarantee at least once delivery
7. 7 Resiliency - High Availability and Replication Decoupling with Message Queue Beats Log Files Wire Data Logstash Nodes (X) Datastore Web APIs Social Sensors Elasticsearch X-Pack Master Nodes (3) Ingest Nodes (X) Data Nodes (X) Logstash Nodes (X) Kafka Messaging Queue Windows Log files Metrics
8. 8 Logging As a Service - Pre-Processing Distributed Configurations Beats Logstash Nodes (X) Datastore Web APIs Logstash Nodes (X) Kafka Messaging Queue Elasticsearch X-Pack Master Nodes (3) Ingest Nodes (X) Data Nodes – Hot (X) Data Nodes – Warm (X) Beats Datastore Web APIs Nodes (X) User Group B User Group A Logstash
9. 9 Logging As a Service - Post-Processing Centralized Configurations Beats Datastore Web APIs Kafka Messaging Queue Elasticsearch X-Pack Master Nodes (3) Ingest Nodes (X) Data Nodes – Hot (X) Data Nodes – Warm (X) Beats Datastore Web APIs Nodes (X) User Group A Logstash Logstash Nodes (X) Logstash Nodes (X) User Group B Nodes (X) Logstash
10. 10 Logging As a Service - Multi Data Centers Coping with Uncertainties Beats Logstash Nodes (X) Datastore Web APIs Logstash Nodes (X) Kafka Messaging Queue Elasticsearch X-Pack Master Nodes (3) Ingest Nodes (X) Data Nodes – Hot (X) Data Nodes – Warm (X) Beats Datastore Web APIs Nodes (X) Data Center B Data Center A Logstash Kafka Messaging Queue
11. 11 Visualization - High Availability Dashboarding Anytime Add coordinating nodes to the Elasticsearch cluster. Kibana X-Pack Instance Elasticsearch Coordinating Node X-Pack Host A Kibana X-Pack Instance Elasticsearch Coordinating Node X-Pack Host B Deploying at least 2 instances of Kibana and load balance between them to prevent single point failure and provide high availability. Kinana has small resource footprint.
12. 12 Visualization - Multiple User Groups Dashboarding with Access Control Run multiple instances of Kibana, one for each user group. Elasticsearch X-Pack Master Nodes (3) Ingest Nodes (X) Data Nodes (X) Coordinating Nodes – (X) Kibana X-Pack Kibana X-Pack Kibana X-Pack X-Pack security integrates nicely with AD or LDAP, allowing us to map its users to Elasticsearch security roles. Each Kibana instance is configured with its own metadata and user roles.
13. 13 Scaling Differently - Elastic Cloud Enterprise Centrally Manage Multiple Instances of the Stack Deploy Elastic stack on hardwares of your choice Automate resizing of the stacks Manage versioning, upgrading, taking snapshots, replication and failover. Each user group or each use case has its own mini stack Provide better security and better resource isolation
14. • Discussion Forums - https://discuss.elastic.co • Meetups - http://elasticsearch.meetup.com • Docs - https://elastic.co/docs • Community - https://elastic.co/community • More - https://www.elastic.co/learn Resources 14
15. Lau Tzu { } The journey of a thousand miles must begin with a single step.
16. 16 More Questions? Visit us at the AMA
17. www.elastic.c o
18. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 18 Please attribute Elastic with a link to elastic.co