Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elastic{ON} 2018 - “Here, Hold My Beer.” Customer Stories to Learn from As Told by Support

Elastic Co
March 01, 2018

Elastic{ON} 2018 - “Here, Hold My Beer.” Customer Stories to Learn from As Told by Support

Elastic Co

March 01, 2018
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. Elastic 1 March 2018 @GeorgeKobar @jpcarey @juliD “Here, Hold My

    Beer.” Customer Stories to Learn from as Told by Support George, Jared & Jaleh
  2. In this session, members of the Elastic support team discuss

    the top three common customer mistakes and misconfigurations along with best practices and recommendations so you can avoid those issues yourself.
  3. In this session, members of the Elastic support team discuss

    the top three common customer mistakes and misconfigurations along with best practices and recommendations so you can avoid those issues yourself.
  4. { Common Evolution of Using Elasticsearch } 9 1 The

    Honeymoon 2 The Work Maintaining 3 4 Realization Marriage
  5. { Common Evolution of Using Elasticsearch } 1 0 1

    The Honeymoon Proof of Concept 2 The Work Tuning & Optimization, Growth Maintaining Business Continuity, Upgrades 3 4 Realization Life in Production Marriage
  6. { Bootstrap Checks } Heap size check - JVM min

    and max heap are equal File descriptor check - file descriptors are set to at least 65,536 Memory lock check - Bootstrap.memory_lock is set Maximum number of threads check - Allow elasticsearch process to create at least 2048 threads Maximum size virtual memory check - Allow unlimited address space for elasticsearch process Maximum map count check - sysctl -w vm.max_map_count=262144 Client JVM check - Make sure server JVM is running, not the client JVM Use serial collector check - Make sure -XX:+UseSerialGC is enabled System call filter check - Make sure system call filters are installed and enabled OnError and OnOutOfMemoryError checks - Disallow these settings when syscall filters are enabled Early-access check Disallow use of openJDK early access builds G1GC check - Disallow use of G1GC
  7. { Shard Benchmarking } Rally (elasticsearch’s benchmarking tool) $ esrally

    --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:1" --report-file=~/result_1p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:2" --report-file=~/result_2p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:4" --report-file=~/result_4p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:8" --report-file=~/result_8p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:12" --report-file=~/result_12p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:18" --report-file=~/result_18p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:24" --report-file=~/result_24p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:48" --report-file=~/result_48p.txt esrally 0.9.2 Google Compute n1-standard-4 (4 vCPUs, 15 GB memory) CPU platform: Intel Haswell 100gb SSD persistent disk CentOS Linux release 7.4.1708 (Core)
  8. { The Work } 45 Elasticsearch Has Many Uses Cases

    • Application Search • Enterprise Search • Business Analytics • Metrics / Operational Log Analytics • Security Analysis • https://www.elastic.co/use-cases
  9. { The Work } 46 Elasticsearch Has Many Uses Cases

    •Search •Time Series Data • Application Search • Enterprise Search • Business Analytics • Metrics / Operational Log Analytics • Security Analysis • https://www.elastic.co/use-cases
  10. { The Work } Elasticsearch Has Many Uses Cases •

    Search How Fast Are My Search Results? • Time Series How Fast Can I Index?
  11. • Data Modeling • Filters!!! • Use Profiler for Query

    Optimization • Force Merge Static Indices Search Performance { Optimize for Search }
  12. • Increase Refresh Interval >30 sec • Disable OS Swapping

    • Increase Buffer Index Size Index Performance { Optimize for Index }
  13. • Rollover API - • Managing Time Based Series Data

    Efficiently • https://www.elastic.co/blog/managing-time-based-indices-efficiently • Aliases • Abstract or change your physical mapping without downtime • https://www.elastic.co/blog/aliases-ftw • Optimize for failure • Users commonly optimize and fine tune to what is available. • Rarely plan for node outage. N+1!!! OR N+2!!! { Other Considerations }
  14. 5 8 Beats Log Files Metrics Wire Data your(beat) Datastore

    Social Web APIs Sensors Kafka Redis Messaging Queue Nodes (X) Logstash Elasticsearch Kibana X-pack Authentication Notification X-pack LDAP AD SSO Instances (X) Master Nodes (3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) { Maintaining The Family }
  15. 59 Secure • Authentication • Secure Connections • Monitor -

    • Nodes, Kibana, and Logstash • Data Retention • Index Sizing { Keeping an Eye on Your Cluster } • High Availability • Disaster Recovery - Hot/Warm Site Monitoring Management HA and DR
  16. • Data Retention • Delete indices • Index size management.

    • Replicas (Change the number of replicas per shard for indices) • Rollover • Shrink • Rollover indices { Curator to the Rescue } 70 How can it help you
  17. 72 Elasticsearch X-pack Master Nodes (3) Ingest Nodes (X) Data

    Nodes - Hot (X) Data Nodes - Warm (X) Master Nodes (3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) Zone A Zone B { High Availability } Shard Allocation Awareness Beats Log Files Metrics Wire Data your(beat) Nodes (X) Logstash Kibana Instances (X)
  18. 73 { Disaster Recovery } Elasticsearch Kibana X-pack X-pack Instances

    (X) Master Nodes (3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) Elasticsearch Kibana X-pack X-pack Instances (X) Master Nodes (3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) Beats Log Files Metrics Wire Data your(beat) Datastore Social Web APIs Sensors Kafka Redis Messaging Queue Nodes (X) Logstash
  19. • Rolling Upgrade • Upgrade assistant • Interactive Upgrade Guide.

    { We Heard You } 77 Major upgrade without downtime
  20. { Backup Your Elasticsearch Data } Avoid relying solely on

    - Infrastructure strategies - OS Strategies Elasticsearch Snapshot API - Separate storage in your data Center - Cloud storage
  21. { How to Get Help } • Subscription Support -Consultative

    in Nature ◦ Dedicated Support Engineer ▪Use Case Driven Support ▪Understands Impact of Upgrade/Caveats For Your Environment • Baseline Taken -Metrics ▪Upgrade Planning ▪24/7 Break Fix For Production Issues
  22. • Support • https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html • Training • Courses in your

    area: https://www.elastic.co/blog/category/releases • Consulting • For on-site, time-sensitive assistance • https://www.elastic.co/services_policy#upgrade-strategy { How to Get Help } 91
  23. { How to Get Help } 92 ⬩ IRC on

    Freenode ◈ #elasticsearch, #logstash, #kibana, #beats ◈ #elastic-webinar ⬩ Slides and recording are available ⬩ Forum: https://discuss.elastic.co/ ⬩ Github: https://github.com/elastic ⬩ StackOverflow: http://stackoverflow.com/questions/tagged/e lasticsearch
  24. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/

    Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 95 Please attribute Elastic with a link to elastic.co