Upgrading Your Elastic Stack to 5.0

Version 5.0 introduces many new features and improvements across all components of the Elastic Stack. From a new scripting language for Elasticsearch to a new monitoring API for Logstash to a more colorific and eye-pleasing Kibana to a new Beat for gathering infrastructure metrics, there are many compelling reasons to upgrade your Elastic Stack to 5.0. In this video, Pius and Chris walk through the considerations, best practices and caveats for upgrading your Elastic Stack to 5.0.

1. Upgrading to 5.0 1
2. 2 Pius Fung Support Engineer Chris Earle Monitoring Lead
3. Housekeeping & Logistics • Slides and recording will be available following the webinar • Chat via IRC #elastic-webinar ‒ #elastic-webinar @ Freenode ‒ Click ”Join the Chat” link, create an IRC account It has to happen…
4. Register for Elastic{ON}17. Holiday pricing of $1,195 ($1,495 full price) expires January 6. https://www.elastic.co/elasticon/conf/2017/sf/registration
5. All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement. 5 Elastic Stack Version 5.0. All new versions. All aligned.
6. All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement. 6 Security Alerting Monitoring Reporting X-Pack Graph
7. All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement. 7 Elastic Cloud Security Alerting Monitoring Reporting X-Pack Graph
8. • Full Elastic Stack upgrade order • Upgrading from 2.x • Upgrading from 1.x • Upgrading on Elastic Cloud • Best practices and caveats • Upgrade resources Topics for today:
9. Full Stack Upgrade Order 9
10. Full Stack Upgrade General Upgrade Order https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html [ NEW ] Upgrade Order
11. Full Stack Upgrade General Upgrade Order https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html [ NEW ] Upgrade Order
12. Full Stack Upgrade ES-Hadoop General Upgrade Order https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html [ NEW ] Upgrade Order
13. Upgrading from 2.x Elasticsearch 13
14. • Indices created prior to 2.0 are not compatible with 5.0 ▪ Elasticsearch will not start up ▪ Snapshots will not restore • Requires reindexing • Currently, upgrading between major versions requires a full cluster restart Upgrading from 2.x Breaking Changes https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes.html
15. • Warns in development mode. Refuses to start in production mode. • Motivation - https://www.elastic.co/blog/bootstrap_checks_annoying_instead_of_devastating Examples: • Heap size, bootstrap.memory_lock (previously known as mlockall), OnOutOfMemoryError • File descriptor, virtual memory size, memory-mapped area, max # of threads Upgrading from 2.x Breaking Changes >> Bootstrap Checks https://www.elastic.co/guide/en/elasticsearch/reference/5.0/bootstrap-checks.html
16. Examples: • filtered query ➔ bool query • search_type=count ➔ size: 0 • scroll query with search_type=scan ➔ scroll query with sort: [“_doc”] • and / or ➔ bool-must / bool-should Upgrading from 2.x Breaking Changes >> Query Changes https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_search_changes.htm
17. Examples: • string field type replaced by text and keyword type • strings now mapped both as text and keyword by default (can be customized) • https://www.elastic.co/blog/strings-are-dead-long-live-strings • scoring on numeric fields requires mapping as keyword field • _timestamp and _ttl removed Upgrading from 2.x Breaking Changes >> Mapping Changes https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_mapping_changes.html
18. Examples: • Most index settings must defined at index level (not in yml) • System properties changes, e.g. -Des.path.conf replaced by -Epath.conf • Define node attributes using node.attr.<tag> instead of node.<tag> • Default node.name is now first 7 characters of random UUID Upgrading from 2.x Breaking Changes >> Settings Changes https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_settings_changes.html
19. Examples: • JVM options (including heap size) now in jvm.options file. • JVM terminates on OutOfMemoryError, StackOverFlowError, etc.. • Repository location for apt/yum packages is now at https://artifacts.elastic.co (previously https://packages.elastic.co) Upgrading from 2.x Breaking Changes >> Packaging https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_packaging.html
20. Out of the box safeguards added to prevent simple user errors Examples: • Soft limit on the number of shards that can be queried in a single search request (1000) • Soft limit on total # of fields in an index (1000) • Soft limit on total # of nested fields in an index (50) • Soft limit on mapping depth (20) Upgrading from 2.x Breaking Changes >> Safeguards
21. Upgrading from 2.x Elasticsearch Upgrading Options https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html Full Cluster Restart Upgrade • Only indices created in 2.0+ • 1.x and 2.x time-based indices • can wait for 1.x indices to be deleted Reindex to Upgrade • Indices created in 1.x • 1.x and 2.x time based indices • cannot wait for 1.x indices to be deleted • reindex 1.x indices • 2.x indices, but want new features that require reindexing (eg. BKD tree for numeric data fields)
22. Cluster Checkup • Report problems on cluster, node and index settings, index mapping and segment version conflicts, installed plugins, etc.. • There are breaking changes not covered by the migration plugin. Reindex Helper • Provides 1-click local reindexing of indices that are not compatible with Elasticsearch 5.0 Deprecation Logging • Enable deprecation logging on the fly Upgrading from 2.x Demo: Elasticsearch Migration Plugin (2.x) https://github.com/elastic/elasticsearch-migration/tree/2.x
23. • Elasticsearch 2.0+ • Can be enabled dynamically, or use migration plugin • <cluster_name>_deprecation.log in Elasticsearch logs folder Upgrading from 2.x Demo: Deprecation Logging https://www.elastic.co/guide/en/elasticsearch/reference/2.4/setup-configuration.html#deprecation-logging curl -XPUT "http://localhost:9200/_cluster/settings" -d' { "transient": { "logger.deprecation":"DEBUG" } }'
24. Upgrading from 2.x Kibana | X-Pack | Logstash | Beats 24
25. • Binds to localhost by default • RPM / DEB installation • New repo location: https://artifacts.elastic.co • /usr/share/kibana and /etc/kibana • New plugin install command: /bin/kibana-plugin • First class support of tribe node coming in 5.x Upgrading from 2.x Kibana >> Breaking Changes https://www.elastic.co/guide/en/kibana/5.0/breaking-changes-5.0.html
26. • .kibana reindexing required for Kibana 4.0/4.1 • Sense is renamed to Console • Console and Timelion now default Kibana apps • Shield/Marvel/Graph UI and Reporting now part of X-Pack • Query DSL breaking changes in saved objects Upgrading from 2.x Kibana >> Upgrading https://www.elastic.co/guide/en/kibana/5.0/upgrade.html
27. • Single X-Pack plugin install ◦ bin/elasticsearch-plugin install x-pack ◦ bin/kibana-plugin install x-pack • Settings renamed ◦ watcher.enabled ➔ xpack.watcher.enabled ◦ shield.authc.realms.ldap1.url ➔ xpack.security.authc.realms.ldap1.url • Endpoints deprecated and renamed ◦ /_shield/user ➔ /_xpack/security • Role format changes • Native is now the default realm • Java keystores no longer required Upgrading from 2.x X-Pack >> Breaking Changes https://www.elastic.co/guide/en/x-pack/current/migrating-to-xpack.html
28. • Changes for rpm/deb installation: ▪ New repo location: https://artifacts.elastic.co ▪ Binaries now at /usr/share/logstash • Elasticsearch output plugin ▪ workers setting replaced by pipeline.workers ▪ Auto-select logstash template for Elasticsearch version • Kafka input/output plugins ▪ Compatible with Kafka 0.10 (requires Kafka broker 0.10) • Ruby filter plugin ▪ Java event class requires refactoring o tmp = event['field'] -> tmp = event.get('field') o event['field'] = tmp -> event.set('field',tmp) • All plugins download package removed ▪ Use https://www.elastic.co/guide/en/logstash/current/offline-plugins.html Upgrading from 2.x Logstash >> Breaking Changes https://www.elastic.co/guide/en/logstash/current/breaking-changes.html
29. • Migrate existing configs (scripts/migrate_beat_config_1_x_to_5_0.py) • Topbeat deprecation • Replaced by Metricbeat system module • Topbeat data is not compatible with 5.0 dashboards • rpm/deb • New repo location: https://artifacts.elastic.co • New binaries at /usr/share/<beat_name> • Update beats index template using output.elasticsearch.template.overwrite Upgrading from 2.x Beats >> Breaking Changes https://www.elastic.co/guide/en/beats/libbeat/current/breaking-changes-5.0.html
30. Upgrading from 1.x Elastic Stack 30
31. • Option: Upgrade to 2.x first • Run migration plugin and deprecation logging • Reindex to upgrade • Option: “Start from scratch” • Reindex directly to 5.0 (Reindex API) • Lack of migration helpers • Requires separate cluster • Upgrading from 2.x > Elasticsearch section from above applies here Upgrading from 1.x Elasticsearch https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
32. • Kibana 3 not upgradable ▪ Recreate all visualizations and dashboards • Upgrading from Kibana 4.0 and 4.1 (compatible with ES 1.x) requires reindexing of .kibana • Upgrading from 2.x > Kibana section from above applies here Upgrading from 1.x Kibana https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
33. • Reindex .watches, .triggered_watches and watch history indices • Marvel indices not compatible with X-pack Monitoring • Upgrading from 2.x > x-pack section from above applies here Upgrading from 1.x X-Pack https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
34. • Elasticsearch output plugin (node or transport protocol) • Plugin that enables this feature now community-supported • Use default (since 2.x) http protocol • Upgrading from 2.x > Logstash section from above applies here Upgrading from 1.x Logstash https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
35. • Filebeat replaces Logstash forwarder (EOL) • Logstash forwarder + lumberjack input -> Filebeat + beats input • Reindex existing indices created in ES 1.x • Topbeat indices not compatible with 5.0 dashboards • Upgrading from 2.x > Beats section from above applies here Upgrading from 1.x Beats https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
36. Upgrading in Elastic Cloud 36
37. 1. Restore latest snapshot of indices to a test cluster running the same version Upgrading in Elastic Cloud
38. 2. Run the migration plugin and enable deprecation logging in the test cluster with the restored snapshot 3. Review breaking changes documentation Upgrading in Elastic Cloud
39. 4. Address “red” issues reported by the migration plugin 5. Upgrade (1 click!) Elasticsearch and Kibana (in that order) 6. Validate upgraded test cluster before upgrading production cluster. Upgrading in Elastic Cloud
40. Best Practices & Caveats Examples from the real-world 40
41. • Run compatible/supported versions of Elastic products • https://www.elastic.co/support/matrix • Backup indices before upgrading/reindexing • Review breaking changes and implement necessary remediation changes • Migration plugin and deprecation logging are only “helper” tools • Set up separate 5.0 cluster with parallel indexing • Full cluster restart inherently requires downtime/maintenance window • Local reindexing requires additional disk space. Best Practices
42. • All nodes must be upgraded to the same version • Upgrade commercial and community plugins • Java transport/node clients and NEST not compatible across major versions ▪ For Java applications, check out the new Java REST Client (https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_features.html) • Upgrade to the latest version Best Practices
43. • TEST! TEST! TEST! Best Practices
44. Upgrading Resources 44
45. • Cross stack upgrade guide • https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html • Release highlights (per product) • https://www.elastic.co/blog/category/releases • Support Matrix ( https://www.elastic.co/support/matrix ) • Upcoming End of Life (EOL) Dates ( https://www.elastic.co/support/eol ) • 1.6.x - Dec 9, 2016 • 1.7.x - Jan 16, 2017 Resources
46. • Training • Courses in your area: http://training.elastic.co/classes • Consulting • For on-site, time-sensitive assistance • https://www.elastic.co/services_policy#upgrade-strategy • Support • https://www.elastic.co/subscriptions Resources
47. 47 { } “{support engineer} proactively contacted me as our upgrade has been delayed twice now. We are finally getting back on track on this end to the reach out was very timely and appreciated. {support engineer} provided some great advice in terms of technical specifics and a roadmap for information we will have to share during the actual implementation to make it a success!”
48. 48 { } “Very detailed tailored to our site upgrade instruction by {support engineer} worked flawlessly.”
49. 49 { } “In the length of time of using Elastic Support, you guys have saved my {fill in the blank} 2 major times. The first time was the migration from ES1.x to ES2.x … The only reason I was able to complete my maintenance without impact was due to the work of {support engineer} and team. You guys rock!!
50. • Discuss • https://discuss.elastic.co/ • Github • https://github.com/elastic • Stackoverflow • http://stackoverflow.com/questions/tagged/elasticsearch • IRC • #elasticsearch, #logstash, #kibana, #beats Resources
51. Questions? 51 Log into IRC to ask questions • #elastic-webinar @ Freenode • Click ”Join the Chat” link, create an IRC account
52. Thank You 52