Upgrading to 5.0 1 

2 Pius Fung Support Engineer Chris Earle Monitoring Lead 

Housekeeping & Logistics • Slides and recording will be available following the webinar • Chat via IRC #elastic-webinar ‒ #elastic-webinar @ Freenode ‒ Click ”Join the Chat” link, create an IRC account It has to happen… 

Register for Elastic{ON}17. Holiday pricing of $1,195 ($1,495 full price) expires January 6. https://www.elastic.co/elasticon/conf/2017/sf/registration 

All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement. 5 Elastic Stack Version 5.0. All new versions. All aligned. 

All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement. 6 Security Alerting Monitoring Reporting X-Pack Graph 

All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement. 7 Elastic Cloud Security Alerting Monitoring Reporting X-Pack Graph 

• Full Elastic Stack upgrade order • Upgrading from 2.x • Upgrading from 1.x • Upgrading on Elastic Cloud • Best practices and caveats • Upgrade resources Topics for today: 

Full Stack Upgrade Order 9 

Full Stack Upgrade General Upgrade Order https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html [ NEW ] Upgrade Order 

Full Stack Upgrade General Upgrade Order https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html [ NEW ] Upgrade Order 

Full Stack Upgrade ES-Hadoop General Upgrade Order https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html [ NEW ] Upgrade Order 

Upgrading from 2.x Elasticsearch 13 

• Indices created prior to 2.0 are not compatible with 5.0 ▪ Elasticsearch will not start up ▪ Snapshots will not restore • Requires reindexing • Currently, upgrading between major versions requires a full cluster restart Upgrading from 2.x Breaking Changes https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes.html 

• Warns in development mode. Refuses to start in production mode. • Motivation - https://www.elastic.co/blog/bootstrap_checks_annoying_instead_of_devastating Examples: • Heap size, bootstrap.memory_lock (previously known as mlockall), OnOutOfMemoryError • File descriptor, virtual memory size, memory-mapped area, max # of threads Upgrading from 2.x Breaking Changes >> Bootstrap Checks https://www.elastic.co/guide/en/elasticsearch/reference/5.0/bootstrap-checks.html 

Examples: • filtered query ➔ bool query • search_type=count ➔ size: 0 • scroll query with search_type=scan ➔ scroll query with sort: [“_doc”] • and / or ➔ bool-must / bool-should Upgrading from 2.x Breaking Changes >> Query Changes https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_search_changes.htm 

Examples: • string field type replaced by text and keyword type • strings now mapped both as text and keyword by default (can be customized) • https://www.elastic.co/blog/strings-are-dead-long-live-strings • scoring on numeric fields requires mapping as keyword field • _timestamp and _ttl removed Upgrading from 2.x Breaking Changes >> Mapping Changes https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_mapping_changes.html 

Examples: • Most index settings must defined at index level (not in yml) • System properties changes, e.g. -Des.path.conf replaced by -Epath.conf • Define node attributes using node.attr. instead of node. • Default node.name is now first 7 characters of random UUID Upgrading from 2.x Breaking Changes >> Settings Changes https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_settings_changes.html 

Examples: • JVM options (including heap size) now in jvm.options file. • JVM terminates on OutOfMemoryError, StackOverFlowError, etc.. • Repository location for apt/yum packages is now at https://artifacts.elastic.co (previously https://packages.elastic.co) Upgrading from 2.x Breaking Changes >> Packaging https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_packaging.html 

Out of the box safeguards added to prevent simple user errors Examples: • Soft limit on the number of shards that can be queried in a single search request (1000) • Soft limit on total # of fields in an index (1000) • Soft limit on total # of nested fields in an index (50) • Soft limit on mapping depth (20) Upgrading from 2.x Breaking Changes >> Safeguards 

Upgrading from 2.x Elasticsearch Upgrading Options https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html Full Cluster Restart Upgrade • Only indices created in 2.0+ • 1.x and 2.x time-based indices • can wait for 1.x indices to be deleted Reindex to Upgrade • Indices created in 1.x • 1.x and 2.x time based indices • cannot wait for 1.x indices to be deleted • reindex 1.x indices • 2.x indices, but want new features that require reindexing (eg. BKD tree for numeric data fields) 

Cluster Checkup • Report problems on cluster, node and index settings, index mapping and segment version conflicts, installed plugins, etc.. • There are breaking changes not covered by the migration plugin. Reindex Helper • Provides 1-click local reindexing of indices that are not compatible with Elasticsearch 5.0 Deprecation Logging • Enable deprecation logging on the fly Upgrading from 2.x Demo: Elasticsearch Migration Plugin (2.x) https://github.com/elastic/elasticsearch-migration/tree/2.x 

• Elasticsearch 2.0+ • Can be enabled dynamically, or use migration plugin • _deprecation.log in Elasticsearch logs folder Upgrading from 2.x Demo: Deprecation Logging https://www.elastic.co/guide/en/elasticsearch/reference/2.4/setup-configuration.html#deprecation-logging curl -XPUT "http://localhost:9200/_cluster/settings" -d' { "transient": { "logger.deprecation":"DEBUG" } }' 

Upgrading from 2.x Kibana | X-Pack | Logstash | Beats 24 

• Binds to localhost by default • RPM / DEB installation • New repo location: https://artifacts.elastic.co • /usr/share/kibana and /etc/kibana • New plugin install command: /bin/kibana-plugin • First class support of tribe node coming in 5.x Upgrading from 2.x Kibana >> Breaking Changes https://www.elastic.co/guide/en/kibana/5.0/breaking-changes-5.0.html 

• .kibana reindexing required for Kibana 4.0/4.1 • Sense is renamed to Console • Console and Timelion now default Kibana apps • Shield/Marvel/Graph UI and Reporting now part of X-Pack • Query DSL breaking changes in saved objects Upgrading from 2.x Kibana >> Upgrading https://www.elastic.co/guide/en/kibana/5.0/upgrade.html 

● Single X-Pack plugin install ○ bin/elasticsearch-plugin install x-pack ○ bin/kibana-plugin install x-pack ● Settings renamed ○ watcher.enabled ➔ xpack.watcher.enabled ○ shield.authc.realms.ldap1.url ➔ xpack.security.authc.realms.ldap1.url ● Endpoints deprecated and renamed ○ /_shield/user ➔ /_xpack/security ● Role format changes ● Native is now the default realm ● Java keystores no longer required Upgrading from 2.x X-Pack >> Breaking Changes https://www.elastic.co/guide/en/x-pack/current/migrating-to-xpack.html 

• Changes for rpm/deb installation: ▪ New repo location: https://artifacts.elastic.co ▪ Binaries now at /usr/share/logstash • Elasticsearch output plugin ▪ workers setting replaced by pipeline.workers ▪ Auto-select logstash template for Elasticsearch version • Kafka input/output plugins ▪ Compatible with Kafka 0.10 (requires Kafka broker 0.10) • Ruby filter plugin ▪ Java event class requires refactoring o tmp = event['field'] -> tmp = event.get('field') o event['field'] = tmp -> event.set('field',tmp) • All plugins download package removed ▪ Use https://www.elastic.co/guide/en/logstash/current/offline-plugins.html Upgrading from 2.x Logstash >> Breaking Changes https://www.elastic.co/guide/en/logstash/current/breaking-changes.html 

• Migrate existing configs (scripts/migrate_beat_config_1_x_to_5_0.py) • Topbeat deprecation • Replaced by Metricbeat system module • Topbeat data is not compatible with 5.0 dashboards • rpm/deb • New repo location: https://artifacts.elastic.co • New binaries at /usr/share/ • Update beats index template using output.elasticsearch.template.overwrite Upgrading from 2.x Beats >> Breaking Changes https://www.elastic.co/guide/en/beats/libbeat/current/breaking-changes-5.0.html 

Upgrading from 1.x Elastic Stack 30 

• Option: Upgrade to 2.x first • Run migration plugin and deprecation logging • Reindex to upgrade • Option: “Start from scratch” • Reindex directly to 5.0 (Reindex API) • Lack of migration helpers • Requires separate cluster • Upgrading from 2.x > Elasticsearch section from above applies here Upgrading from 1.x Elasticsearch https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html 

• Kibana 3 not upgradable ▪ Recreate all visualizations and dashboards • Upgrading from Kibana 4.0 and 4.1 (compatible with ES 1.x) requires reindexing of .kibana • Upgrading from 2.x > Kibana section from above applies here Upgrading from 1.x Kibana https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html 

• Reindex .watches, .triggered_watches and watch history indices • Marvel indices not compatible with X-pack Monitoring • Upgrading from 2.x > x-pack section from above applies here Upgrading from 1.x X-Pack https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html 

• Elasticsearch output plugin (node or transport protocol) • Plugin that enables this feature now community-supported • Use default (since 2.x) http protocol • Upgrading from 2.x > Logstash section from above applies here Upgrading from 1.x Logstash https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html 

• Filebeat replaces Logstash forwarder (EOL) • Logstash forwarder + lumberjack input -> Filebeat + beats input • Reindex existing indices created in ES 1.x • Topbeat indices not compatible with 5.0 dashboards • Upgrading from 2.x > Beats section from above applies here Upgrading from 1.x Beats https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html 

Upgrading in Elastic Cloud 36 

1. Restore latest snapshot of indices to a test cluster running the same version Upgrading in Elastic Cloud 

2. Run the migration plugin and enable deprecation logging in the test cluster with the restored snapshot 3. Review breaking changes documentation Upgrading in Elastic Cloud 

4. Address “red” issues reported by the migration plugin 5. Upgrade (1 click!) Elasticsearch and Kibana (in that order) 6. Validate upgraded test cluster before upgrading production cluster. Upgrading in Elastic Cloud 

Best Practices & Caveats Examples from the real-world 40 

• Run compatible/supported versions of Elastic products • https://www.elastic.co/support/matrix • Backup indices before upgrading/reindexing • Review breaking changes and implement necessary remediation changes • Migration plugin and deprecation logging are only “helper” tools • Set up separate 5.0 cluster with parallel indexing • Full cluster restart inherently requires downtime/maintenance window • Local reindexing requires additional disk space. Best Practices 

• All nodes must be upgraded to the same version • Upgrade commercial and community plugins • Java transport/node clients and NEST not compatible across major versions ▪ For Java applications, check out the new Java REST Client (https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_features.html) • Upgrade to the latest version Best Practices 

• TEST! TEST! TEST! Best Practices 

Upgrading Resources 44 

• Cross stack upgrade guide • https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html • Release highlights (per product) • https://www.elastic.co/blog/category/releases • Support Matrix ( https://www.elastic.co/support/matrix ) • Upcoming End of Life (EOL) Dates ( https://www.elastic.co/support/eol ) • 1.6.x - Dec 9, 2016 • 1.7.x - Jan 16, 2017 Resources 

• Training • Courses in your area: http://training.elastic.co/classes • Consulting • For on-site, time-sensitive assistance • https://www.elastic.co/services_policy#upgrade-strategy • Support • https://www.elastic.co/subscriptions Resources 

47 { } “{support engineer} proactively contacted me as our upgrade has been delayed twice now. We are finally getting back on track on this end to the reach out was very timely and appreciated. {support engineer} provided some great advice in terms of technical specifics and a roadmap for information we will have to share during the actual implementation to make it a success!” 

48 { } “Very detailed tailored to our site upgrade instruction by {support engineer} worked flawlessly.” 

49 { } “In the length of time of using Elastic Support, you guys have saved my {fill in the blank} 2 major times. The first time was the migration from ES1.x to ES2.x … The only reason I was able to complete my maintenance without impact was due to the work of {support engineer} and team. You guys rock!! 

• Discuss • https://discuss.elastic.co/ • Github • https://github.com/elastic • Stackoverflow • http://stackoverflow.com/questions/tagged/elasticsearch • IRC • #elasticsearch, #logstash, #kibana, #beats Resources 

Questions? 51 Log into IRC to ask questions • #elastic-webinar @ Freenode • Click ”Join the Chat” link, create an IRC account 

Thank You 52