Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Upgrading Your Elastic Stack to 5.0

Elastic Co
November 29, 2016

Upgrading Your Elastic Stack to 5.0

Version 5.0 introduces many new features and improvements across all components of the Elastic Stack. From a new scripting language for Elasticsearch to a new monitoring API for Logstash to a more colorific and eye-pleasing Kibana to a new Beat for gathering infrastructure metrics, there are many compelling reasons to upgrade your Elastic Stack to 5.0. In this video, Pius and Chris walk through the considerations, best practices and caveats for upgrading your Elastic Stack to 5.0.

Elastic Co

November 29, 2016

More Decks by Elastic Co

Other Decks in Technology


  1. Housekeeping & Logistics • Slides and recording will be available

    following the webinar • Chat via IRC #elastic-webinar ‒ #elastic-webinar @ Freenode ‒ Click ”Join the Chat” link, create an IRC account It has to happen…
  2. Register for Elastic{ON}17. Holiday pricing of $1,195 ($1,495 full price)

    expires January 6. https://www.elastic.co/elasticon/conf/2017/sf/registration
  3. All product names, logos, and brands are property of their

    respective owners and are used only for identification purposes. This is not an endorsement. 5 Elastic Stack Version 5.0. All new versions. All aligned.
  4. All product names, logos, and brands are property of their

    respective owners and are used only for identification purposes. This is not an endorsement. 6 Security Alerting Monitoring Reporting X-Pack Graph
  5. All product names, logos, and brands are property of their

    respective owners and are used only for identification purposes. This is not an endorsement. 7 Elastic Cloud Security Alerting Monitoring Reporting X-Pack Graph
  6. • Full Elastic Stack upgrade order • Upgrading from 2.x

    • Upgrading from 1.x • Upgrading on Elastic Cloud • Best practices and caveats • Upgrade resources Topics for today:
  7. • Indices created prior to 2.0 are not compatible with

    5.0 ▪ Elasticsearch will not start up ▪ Snapshots will not restore • Requires reindexing • Currently, upgrading between major versions requires a full cluster restart Upgrading from 2.x Breaking Changes https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes.html
  8. • Warns in development mode. Refuses to start in production

    mode. • Motivation - https://www.elastic.co/blog/bootstrap_checks_annoying_instead_of_devastating Examples: • Heap size, bootstrap.memory_lock (previously known as mlockall), OnOutOfMemoryError • File descriptor, virtual memory size, memory-mapped area, max # of threads Upgrading from 2.x Breaking Changes >> Bootstrap Checks https://www.elastic.co/guide/en/elasticsearch/reference/5.0/bootstrap-checks.html
  9. Examples: • filtered query ➔ bool query • search_type=count ➔

    size: 0 • scroll query with search_type=scan ➔ scroll query with sort: [“_doc”] • and / or ➔ bool-must / bool-should Upgrading from 2.x Breaking Changes >> Query Changes https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_search_changes.htm
  10. Examples: • string field type replaced by text and keyword

    type • strings now mapped both as text and keyword by default (can be customized) • https://www.elastic.co/blog/strings-are-dead-long-live-strings • scoring on numeric fields requires mapping as keyword field • _timestamp and _ttl removed Upgrading from 2.x Breaking Changes >> Mapping Changes https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_mapping_changes.html
  11. Examples: • Most index settings must defined at index level

    (not in yml) • System properties changes, e.g. -Des.path.conf replaced by -Epath.conf • Define node attributes using node.attr.<tag> instead of node.<tag> • Default node.name is now first 7 characters of random UUID Upgrading from 2.x Breaking Changes >> Settings Changes https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_settings_changes.html
  12. Examples: • JVM options (including heap size) now in jvm.options

    file. • JVM terminates on OutOfMemoryError, StackOverFlowError, etc.. • Repository location for apt/yum packages is now at https://artifacts.elastic.co (previously https://packages.elastic.co) Upgrading from 2.x Breaking Changes >> Packaging https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_packaging.html
  13. Out of the box safeguards added to prevent simple user

    errors Examples: • Soft limit on the number of shards that can be queried in a single search request (1000) • Soft limit on total # of fields in an index (1000) • Soft limit on total # of nested fields in an index (50) • Soft limit on mapping depth (20) Upgrading from 2.x Breaking Changes >> Safeguards
  14. Upgrading from 2.x Elasticsearch Upgrading Options https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html Full Cluster Restart

    Upgrade • Only indices created in 2.0+ • 1.x and 2.x time-based indices • can wait for 1.x indices to be deleted Reindex to Upgrade • Indices created in 1.x • 1.x and 2.x time based indices • cannot wait for 1.x indices to be deleted • reindex 1.x indices • 2.x indices, but want new features that require reindexing (eg. BKD tree for numeric data fields)
  15. Cluster Checkup • Report problems on cluster, node and index

    settings, index mapping and segment version conflicts, installed plugins, etc.. • There are breaking changes not covered by the migration plugin. Reindex Helper • Provides 1-click local reindexing of indices that are not compatible with Elasticsearch 5.0 Deprecation Logging • Enable deprecation logging on the fly Upgrading from 2.x Demo: Elasticsearch Migration Plugin (2.x) https://github.com/elastic/elasticsearch-migration/tree/2.x
  16. • Elasticsearch 2.0+ • Can be enabled dynamically, or use

    migration plugin • <cluster_name>_deprecation.log in Elasticsearch logs folder Upgrading from 2.x Demo: Deprecation Logging https://www.elastic.co/guide/en/elasticsearch/reference/2.4/setup-configuration.html#deprecation-logging curl -XPUT "http://localhost:9200/_cluster/settings" -d' { "transient": { "logger.deprecation":"DEBUG" } }'
  17. • Binds to localhost by default • RPM / DEB

    installation • New repo location: https://artifacts.elastic.co • /usr/share/kibana and /etc/kibana • New plugin install command: /bin/kibana-plugin • First class support of tribe node coming in 5.x Upgrading from 2.x Kibana >> Breaking Changes https://www.elastic.co/guide/en/kibana/5.0/breaking-changes-5.0.html
  18. • .kibana reindexing required for Kibana 4.0/4.1 • Sense is

    renamed to Console • Console and Timelion now default Kibana apps • Shield/Marvel/Graph UI and Reporting now part of X-Pack • Query DSL breaking changes in saved objects Upgrading from 2.x Kibana >> Upgrading https://www.elastic.co/guide/en/kibana/5.0/upgrade.html
  19. • Single X-Pack plugin install ◦ bin/elasticsearch-plugin install x-pack ◦

    bin/kibana-plugin install x-pack • Settings renamed ◦ watcher.enabled ➔ xpack.watcher.enabled ◦ shield.authc.realms.ldap1.url ➔ xpack.security.authc.realms.ldap1.url • Endpoints deprecated and renamed ◦ /_shield/user ➔ /_xpack/security • Role format changes • Native is now the default realm • Java keystores no longer required Upgrading from 2.x X-Pack >> Breaking Changes https://www.elastic.co/guide/en/x-pack/current/migrating-to-xpack.html
  20. • Changes for rpm/deb installation: ▪ New repo location: https://artifacts.elastic.co

    ▪ Binaries now at /usr/share/logstash • Elasticsearch output plugin ▪ workers setting replaced by pipeline.workers ▪ Auto-select logstash template for Elasticsearch version • Kafka input/output plugins ▪ Compatible with Kafka 0.10 (requires Kafka broker 0.10) • Ruby filter plugin ▪ Java event class requires refactoring o tmp = event['field'] -> tmp = event.get('field') o event['field'] = tmp -> event.set('field',tmp) • All plugins download package removed ▪ Use https://www.elastic.co/guide/en/logstash/current/offline-plugins.html Upgrading from 2.x Logstash >> Breaking Changes https://www.elastic.co/guide/en/logstash/current/breaking-changes.html
  21. • Migrate existing configs (scripts/migrate_beat_config_1_x_to_5_0.py) • Topbeat deprecation • Replaced

    by Metricbeat system module • Topbeat data is not compatible with 5.0 dashboards • rpm/deb • New repo location: https://artifacts.elastic.co • New binaries at /usr/share/<beat_name> • Update beats index template using output.elasticsearch.template.overwrite Upgrading from 2.x Beats >> Breaking Changes https://www.elastic.co/guide/en/beats/libbeat/current/breaking-changes-5.0.html
  22. • Option: Upgrade to 2.x first • Run migration plugin

    and deprecation logging • Reindex to upgrade • Option: “Start from scratch” • Reindex directly to 5.0 (Reindex API) • Lack of migration helpers • Requires separate cluster • Upgrading from 2.x > Elasticsearch section from above applies here Upgrading from 1.x Elasticsearch https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
  23. • Kibana 3 not upgradable ▪ Recreate all visualizations and

    dashboards • Upgrading from Kibana 4.0 and 4.1 (compatible with ES 1.x) requires reindexing of .kibana • Upgrading from 2.x > Kibana section from above applies here Upgrading from 1.x Kibana https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
  24. • Reindex .watches, .triggered_watches and watch history indices • Marvel

    indices not compatible with X-pack Monitoring • Upgrading from 2.x > x-pack section from above applies here Upgrading from 1.x X-Pack https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
  25. • Elasticsearch output plugin (node or transport protocol) • Plugin

    that enables this feature now community-supported • Use default (since 2.x) http protocol • Upgrading from 2.x > Logstash section from above applies here Upgrading from 1.x Logstash https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
  26. • Filebeat replaces Logstash forwarder (EOL) • Logstash forwarder +

    lumberjack input -> Filebeat + beats input • Reindex existing indices created in ES 1.x • Topbeat indices not compatible with 5.0 dashboards • Upgrading from 2.x > Beats section from above applies here Upgrading from 1.x Beats https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
  27. 1. Restore latest snapshot of indices to a test cluster

    running the same version Upgrading in Elastic Cloud
  28. 2. Run the migration plugin and enable deprecation logging in

    the test cluster with the restored snapshot 3. Review breaking changes documentation Upgrading in Elastic Cloud
  29. 4. Address “red” issues reported by the migration plugin 5.

    Upgrade (1 click!) Elasticsearch and Kibana (in that order) 6. Validate upgraded test cluster before upgrading production cluster. Upgrading in Elastic Cloud
  30. • Run compatible/supported versions of Elastic products • https://www.elastic.co/support/matrix •

    Backup indices before upgrading/reindexing • Review breaking changes and implement necessary remediation changes • Migration plugin and deprecation logging are only “helper” tools • Set up separate 5.0 cluster with parallel indexing • Full cluster restart inherently requires downtime/maintenance window • Local reindexing requires additional disk space. Best Practices
  31. • All nodes must be upgraded to the same version

    • Upgrade commercial and community plugins • Java transport/node clients and NEST not compatible across major versions ▪ For Java applications, check out the new Java REST Client (https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_features.html) • Upgrade to the latest version Best Practices
  32. • Cross stack upgrade guide • https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html • Release highlights

    (per product) • https://www.elastic.co/blog/category/releases • Support Matrix ( https://www.elastic.co/support/matrix ) • Upcoming End of Life (EOL) Dates ( https://www.elastic.co/support/eol ) • 1.6.x - Dec 9, 2016 • 1.7.x - Jan 16, 2017 Resources
  33. • Training • Courses in your area: http://training.elastic.co/classes • Consulting

    • For on-site, time-sensitive assistance • https://www.elastic.co/services_policy#upgrade-strategy • Support • https://www.elastic.co/subscriptions Resources
  34. 47 { } “{support engineer} proactively contacted me as our

    upgrade has been delayed twice now. We are finally getting back on track on this end to the reach out was very timely and appreciated. {support engineer} provided some great advice in terms of technical specifics and a roadmap for information we will have to share during the actual implementation to make it a success!”
  35. 48 { } “Very detailed tailored to our site upgrade

    instruction by {support engineer} worked flawlessly.”
  36. 49 { } “In the length of time of using

    Elastic Support, you guys have saved my {fill in the blank} 2 major times. The first time was the migration from ES1.x to ES2.x … The only reason I was able to complete my maintenance without impact was due to the work of {support engineer} and team. You guys rock!!
  37. • Discuss • https://discuss.elastic.co/ • Github • https://github.com/elastic • Stackoverflow

    • http://stackoverflow.com/questions/tagged/elasticsearch • IRC • #elasticsearch, #logstash, #kibana, #beats Resources
  38. Questions? 51 Log into IRC to ask questions • #elastic-webinar

    @ Freenode • Click ”Join the Chat” link, create an IRC account