Elastic 1 March 2018 @GeorgeKobar @jpcarey @juliD “Here, Hold My Beer.” Customer Stories to Learn from as Told by Support George, Jared & Jaleh

Most good stories start with, “Here, Hold my beer.” Unless it’s a story about you...

Our Goal To avoid this...

4 And Provide This

In this session, members of the Elastic support team discuss the top three common customer mistakes and misconfigurations along with best practices and recommendations so you can avoid those issues yourself.

In this session, members of the Elastic support team discuss the top three common customer mistakes and misconfigurations along with best practices and recommendations so you can avoid those issues yourself.

{ Common Evolution of Using Elasticsearch } 7

{ Common Evolution of Using Elasticsearch } 8 Common Evolution of Marriage

{ Common Evolution of Using Elasticsearch } 9 1 The Honeymoon 2 The Work Maintaining 3 4 Realization Marriage

{ Common Evolution of Using Elasticsearch } 1 0 1 The Honeymoon Proof of Concept 2 The Work Tuning & Optimization, Growth Maintaining Business Continuity, Upgrades 3 4 Realization Life in Production Marriage

{ Honeymoon }

Proof of Concept : { Honeymoon }

Proof of Concept : { Honeymoon }

{ Bootstrap Checks } Heap size check - JVM min and max heap are equal File descriptor check - file descriptors are set to at least 65,536 Memory lock check - Bootstrap.memory_lock is set Maximum number of threads check - Allow elasticsearch process to create at least 2048 threads Maximum size virtual memory check - Allow unlimited address space for elasticsearch process Maximum map count check - sysctl -w vm.max_map_count=262144 Client JVM check - Make sure server JVM is running, not the client JVM Use serial collector check - Make sure -XX:+UseSerialGC is enabled System call filter check - Make sure system call filters are installed and enabled OnError and OnOutOfMemoryError checks - Disallow these settings when syscall filters are enabled Early-access check Disallow use of openJDK early access builds G1GC check - Disallow use of G1GC

Elastic Products { Honeymoon } 0.90 1.x 2.x 5.x 6.x 7+

{ Common User Path } POC Production

{ Common User Path } POC Production

{ Realization }

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

{ Shard Benchmarking } Rally (elasticsearch’s benchmarking tool) $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:1" --report-file=~/result_1p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:2" --report-file=~/result_2p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:4" --report-file=~/result_4p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:8" --report-file=~/result_8p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:12" --report-file=~/result_12p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:18" --report-file=~/result_18p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:24" --report-file=~/result_24p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:48" --report-file=~/result_48p.txt esrally 0.9.2 Google Compute n1-standard-4 (4 vCPUs, 15 GB memory) CPU platform: Intel Haswell 100gb SSD persistent disk CentOS Linux release 7.4.1708 (Core)

{Indexing}

{Terms}

{ The Work }

{ The Work } 45 Elasticsearch Has Many Uses Cases • Application Search • Enterprise Search • Business Analytics • Metrics / Operational Log Analytics • Security Analysis • https://www.elastic.co/use-cases

{ The Work } 46 Elasticsearch Has Many Uses Cases ●Search ●Time Series Data • Application Search • Enterprise Search • Business Analytics • Metrics / Operational Log Analytics • Security Analysis • https://www.elastic.co/use-cases

{ The Work } Elasticsearch Has Many Uses Cases ● Search How Fast Are My Search Results? ● Time Series How Fast Can I Index?

{ The Work } Default Search Indexing Search Performance Index Performance

Search { The Work } Faster Search Performance

Search { The Work } Indexing Slower Indexing Performance Faster Search Performance

Faster Index Performance { The Work } Slower Search Performance Search Indexing

● Data Modeling ● Filters!!! ● Use Profiler for Query Optimization ● Force Merge Static Indices Search Performance { Optimize for Search }

● Increase Refresh Interval >30 sec ● Disable OS Swapping ● Increase Buffer Index Size Index Performance { Optimize for Index }

{ Optimize for Both } Index Performance Search Performance

• Rollover API - • Managing Time Based Series Data Efficiently • https://www.elastic.co/blog/managing-time-based-indices-efficiently • Aliases • Abstract or change your physical mapping without downtime • https://www.elastic.co/blog/aliases-ftw • Optimize for failure • Users commonly optimize and fine tune to what is available. • Rarely plan for node outage. N+1!!! OR N+2!!! { Other Considerations }

5 6 {Maintenance}

{ Business Continuity } Big Words for It Needs to Keep Working

5 8 Beats Log Files Metrics Wire Data your(beat) Datastore Social Web APIs Sensors Kafka Redis Messaging Queue Nodes (X) Logstash Elasticsearch Kibana X-pack Authentication Notification X-pack LDAP AD SSO Instances (X) Master Nodes (3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) { Maintaining The Family }

59 Secure • Authentication • Secure Connections • Monitor - • Nodes, Kibana, and Logstash • Data Retention • Index Sizing { Keeping an Eye on Your Cluster } • High Availability • Disaster Recovery - Hot/Warm Site Monitoring Management HA and DR

60 This is a sample image Lock it up

{Chose your Lock}

62 This is a sample image Monitoring

{Don’t take your eyes off it }

6 4 Cluster Alerts { Cluster Alerts }

6 5 Nodes { Nodes }

6 6 Logstash { Logstash }

6 7 Logstash pipeline { Pipeline }

{Do it your way}

69 This is a sample image Index Management

• Data Retention • Delete indices • Index size management. • Replicas (Change the number of replicas per shard for indices) • Rollover • Shrink • Rollover indices { Curator to the Rescue } 70 How can it help you

71 This is a sample image High Availability Disaster Recovery

72 Elasticsearch X-pack Master Nodes (3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) Master Nodes (3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) Zone A Zone B { High Availability } Shard Allocation Awareness Beats Log Files Metrics Wire Data your(beat) Nodes (X) Logstash Kibana Instances (X)

73 { Disaster Recovery } Elasticsearch Kibana X-pack X-pack Instances (X) Master Nodes (3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) Elasticsearch Kibana X-pack X-pack Instances (X) Master Nodes (3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) Beats Log Files Metrics Wire Data your(beat) Datastore Social Web APIs Sensors Kafka Redis Messaging Queue Nodes (X) Logstash

{ Upgrades } Keep it Fresh

75 This is a sample image But It’s a Tough Climb

76 This is a sample image Sudden leaps are more risky

• Rolling Upgrade • Upgrade assistant • Interactive Upgrade Guide. { We Heard You } 77 Major upgrade without downtime

78 This is a sample image { Upgrade Assistant }

79 This is a sample image { Cluster Checkup }

80 This is a sample image { Reindex Helper }

81 This is a sample image { Toggle Deprecation Logger }

82 This is a sample image { Interactive Upgrade Guide }

83 This is a sample image { Interactive Upgrade Guide }

{ Backup } Don’t Lose Your Data

85 Not This Way

{ Backup Your Elasticsearch Data } Avoid relying solely on - Infrastructure strategies - OS Strategies Elasticsearch Snapshot API - Separate storage in your data Center - Cloud storage

{ Curator to the Rescue }

{ How to Get Help }

Timeshare Time

{ How to Get Help } ● Subscription Support -Consultative in Nature ○ Dedicated Support Engineer ■Use Case Driven Support ■Understands Impact of Upgrade/Caveats For Your Environment ● Baseline Taken -Metrics ■Upgrade Planning ■24/7 Break Fix For Production Issues

• Support • https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html • Training • Courses in your area: https://www.elastic.co/blog/category/releases • Consulting • For on-site, time-sensitive assistance • https://www.elastic.co/services_policy#upgrade-strategy { How to Get Help } 91

{ How to Get Help } 92 ⬩ IRC on Freenode ◈ #elasticsearch, #logstash, #kibana, #beats ◈ #elastic-webinar ⬩ Slides and recording are available ⬩ Forum: https://discuss.elastic.co/ ⬩ Github: https://github.com/elastic ⬩ StackOverflow: http://stackoverflow.com/questions/tagged/e lasticsearch

93 More Questions? Visit us at the AMA

www.elastic.c o

Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 95 Please attribute Elastic with a link to elastic.co