$30 off During Our Annual Pro Sale. View Details »

Upgrading Your Elastic Stack to 5.0

Elastic Co
November 29, 2016

Upgrading Your Elastic Stack to 5.0

Version 5.0 introduces many new features and improvements across all components of the Elastic Stack. From a new scripting language for Elasticsearch to a new monitoring API for Logstash to a more colorific and eye-pleasing Kibana to a new Beat for gathering infrastructure metrics, there are many compelling reasons to upgrade your Elastic Stack to 5.0. In this video, Pius and Chris walk through the considerations, best practices and caveats for upgrading your Elastic Stack to 5.0.

Elastic Co

November 29, 2016
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. Upgrading to 5.0
    1

    View Slide

  2. 2
    Pius Fung
    Support Engineer
    Chris Earle
    Monitoring Lead

    View Slide

  3. Housekeeping & Logistics
    • Slides and recording will be available following the webinar
    • Chat via IRC #elastic-webinar
    ‒ #elastic-webinar @ Freenode
    ‒ Click ”Join the Chat” link, create an IRC account
    It has to happen…

    View Slide

  4. Register for Elastic{ON}17.
    Holiday pricing of $1,195 ($1,495 full price) expires January 6.
    https://www.elastic.co/elasticon/conf/2017/sf/registration

    View Slide

  5. All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement. 5
    Elastic Stack
    Version 5.0.
    All new versions.
    All aligned.

    View Slide

  6. All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement. 6
    Security
    Alerting
    Monitoring
    Reporting
    X-Pack
    Graph

    View Slide

  7. All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement. 7
    Elastic
    Cloud
    Security
    Alerting
    Monitoring
    Reporting
    X-Pack
    Graph

    View Slide

  8. • Full Elastic Stack upgrade order
    • Upgrading from 2.x
    • Upgrading from 1.x
    • Upgrading on Elastic Cloud
    • Best practices and caveats
    • Upgrade resources
    Topics for today:

    View Slide

  9. Full Stack Upgrade Order
    9

    View Slide

  10. Full Stack Upgrade
    General Upgrade Order
    https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html [ NEW ]
    Upgrade Order

    View Slide

  11. Full Stack Upgrade
    General Upgrade Order
    https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html [ NEW ]
    Upgrade Order

    View Slide

  12. Full Stack Upgrade
    ES-Hadoop
    General Upgrade Order
    https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html [ NEW ]
    Upgrade Order

    View Slide

  13. Upgrading from 2.x
    Elasticsearch
    13

    View Slide

  14. • Indices created prior to 2.0 are not compatible with 5.0
    ▪ Elasticsearch will not start up
    ▪ Snapshots will not restore
    • Requires reindexing
    • Currently, upgrading between major versions requires a full cluster restart
    Upgrading from 2.x
    Breaking Changes
    https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes.html

    View Slide

  15. • Warns in development mode. Refuses to start in production mode.
    • Motivation - https://www.elastic.co/blog/bootstrap_checks_annoying_instead_of_devastating
    Examples:
    • Heap size, bootstrap.memory_lock (previously known as mlockall),
    OnOutOfMemoryError
    • File descriptor, virtual memory size, memory-mapped area, max # of threads
    Upgrading from 2.x
    Breaking Changes >> Bootstrap Checks
    https://www.elastic.co/guide/en/elasticsearch/reference/5.0/bootstrap-checks.html

    View Slide

  16. Examples:
    • filtered query ➔ bool query
    • search_type=count ➔ size: 0
    • scroll query with search_type=scan ➔ scroll query with sort: [“_doc”]
    • and / or ➔ bool-must / bool-should
    Upgrading from 2.x
    Breaking Changes >> Query Changes
    https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_search_changes.htm

    View Slide

  17. Examples:
    • string field type replaced by text and keyword type
    • strings now mapped both as text and keyword by default (can be customized)
    • https://www.elastic.co/blog/strings-are-dead-long-live-strings
    • scoring on numeric fields requires mapping as keyword field
    • _timestamp and _ttl removed
    Upgrading from 2.x
    Breaking Changes >> Mapping Changes
    https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_mapping_changes.html

    View Slide

  18. Examples:
    • Most index settings must defined at index level (not in yml)
    • System properties changes, e.g. -Des.path.conf replaced by -Epath.conf
    • Define node attributes using node.attr. instead of node.
    • Default node.name is now first 7 characters of random UUID
    Upgrading from 2.x
    Breaking Changes >> Settings Changes
    https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_settings_changes.html

    View Slide

  19. Examples:
    • JVM options (including heap size) now in jvm.options file.
    • JVM terminates on OutOfMemoryError, StackOverFlowError, etc..
    • Repository location for apt/yum packages is now at https://artifacts.elastic.co
    (previously https://packages.elastic.co)
    Upgrading from 2.x
    Breaking Changes >> Packaging
    https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_packaging.html

    View Slide

  20. Out of the box safeguards added to prevent simple user errors
    Examples:
    • Soft limit on the number of shards that can be queried in a single search request
    (1000)
    • Soft limit on total # of fields in an index (1000)
    • Soft limit on total # of nested fields in an index (50)
    • Soft limit on mapping depth (20)
    Upgrading from 2.x
    Breaking Changes >> Safeguards

    View Slide

  21. Upgrading from 2.x
    Elasticsearch Upgrading Options
    https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html
    Full Cluster Restart Upgrade
    • Only indices created in 2.0+
    • 1.x and 2.x time-based indices
    • can wait for 1.x indices to be deleted
    Reindex to Upgrade
    • Indices created in 1.x
    • 1.x and 2.x time based indices
    • cannot wait for 1.x indices to be deleted
    • reindex 1.x indices
    • 2.x indices, but want new features that
    require reindexing (eg. BKD tree for
    numeric data fields)

    View Slide

  22. Cluster Checkup
    • Report problems on cluster, node and index settings, index mapping and segment
    version conflicts, installed plugins, etc..
    • There are breaking changes not covered by the migration plugin.
    Reindex Helper
    • Provides 1-click local reindexing of indices that are not compatible with Elasticsearch
    5.0
    Deprecation Logging
    • Enable deprecation logging on the fly
    Upgrading from 2.x
    Demo: Elasticsearch Migration Plugin (2.x)
    https://github.com/elastic/elasticsearch-migration/tree/2.x

    View Slide

  23. • Elasticsearch 2.0+
    • Can be enabled dynamically, or use migration plugin
    • _deprecation.log in Elasticsearch logs folder
    Upgrading from 2.x
    Demo: Deprecation Logging
    https://www.elastic.co/guide/en/elasticsearch/reference/2.4/setup-configuration.html#deprecation-logging
    curl -XPUT "http://localhost:9200/_cluster/settings" -d'
    {
    "transient": {
    "logger.deprecation":"DEBUG"
    }
    }'

    View Slide

  24. Upgrading from 2.x
    Kibana | X-Pack | Logstash | Beats
    24

    View Slide

  25. • Binds to localhost by default
    • RPM / DEB installation
    • New repo location: https://artifacts.elastic.co
    • /usr/share/kibana and /etc/kibana
    • New plugin install command: /bin/kibana-plugin
    • First class support of tribe node coming in 5.x
    Upgrading from 2.x
    Kibana >> Breaking Changes
    https://www.elastic.co/guide/en/kibana/5.0/breaking-changes-5.0.html

    View Slide

  26. • .kibana reindexing required for Kibana 4.0/4.1
    • Sense is renamed to Console
    • Console and Timelion now default Kibana apps
    • Shield/Marvel/Graph UI and Reporting now part of X-Pack
    • Query DSL breaking changes in saved objects
    Upgrading from 2.x
    Kibana >> Upgrading
    https://www.elastic.co/guide/en/kibana/5.0/upgrade.html

    View Slide

  27. ● Single X-Pack plugin install
    ○ bin/elasticsearch-plugin install x-pack
    ○ bin/kibana-plugin install x-pack
    ● Settings renamed
    ○ watcher.enabled ➔ xpack.watcher.enabled
    ○ shield.authc.realms.ldap1.url ➔ xpack.security.authc.realms.ldap1.url
    ● Endpoints deprecated and renamed
    ○ /_shield/user ➔ /_xpack/security
    ● Role format changes
    ● Native is now the default realm
    ● Java keystores no longer required
    Upgrading from 2.x
    X-Pack >> Breaking Changes
    https://www.elastic.co/guide/en/x-pack/current/migrating-to-xpack.html

    View Slide

  28. • Changes for rpm/deb installation:
    ▪ New repo location: https://artifacts.elastic.co
    ▪ Binaries now at /usr/share/logstash
    • Elasticsearch output plugin
    ▪ workers setting replaced by pipeline.workers
    ▪ Auto-select logstash template for Elasticsearch version
    • Kafka input/output plugins
    ▪ Compatible with Kafka 0.10 (requires Kafka broker 0.10)
    • Ruby filter plugin
    ▪ Java event class requires refactoring
    o tmp = event['field'] -> tmp = event.get('field')
    o event['field'] = tmp -> event.set('field',tmp)
    • All plugins download package removed
    ▪ Use https://www.elastic.co/guide/en/logstash/current/offline-plugins.html
    Upgrading from 2.x
    Logstash >> Breaking Changes
    https://www.elastic.co/guide/en/logstash/current/breaking-changes.html

    View Slide

  29. • Migrate existing configs (scripts/migrate_beat_config_1_x_to_5_0.py)
    • Topbeat deprecation
    • Replaced by Metricbeat system module
    • Topbeat data is not compatible with 5.0 dashboards
    • rpm/deb
    • New repo location: https://artifacts.elastic.co
    • New binaries at /usr/share/
    • Update beats index template using output.elasticsearch.template.overwrite
    Upgrading from 2.x
    Beats >> Breaking Changes
    https://www.elastic.co/guide/en/beats/libbeat/current/breaking-changes-5.0.html

    View Slide

  30. Upgrading from 1.x
    Elastic Stack
    30

    View Slide

  31. • Option: Upgrade to 2.x first
    • Run migration plugin and deprecation logging
    • Reindex to upgrade
    • Option: “Start from scratch”
    • Reindex directly to 5.0 (Reindex API)
    • Lack of migration helpers
    • Requires separate cluster
    • Upgrading from 2.x > Elasticsearch section from above applies here
    Upgrading from 1.x
    Elasticsearch
    https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html

    View Slide

  32. • Kibana 3 not upgradable
    ▪ Recreate all visualizations and dashboards
    • Upgrading from Kibana 4.0 and 4.1 (compatible with ES 1.x) requires reindexing
    of .kibana
    • Upgrading from 2.x > Kibana section from above applies here
    Upgrading from 1.x
    Kibana
    https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html

    View Slide

  33. • Reindex .watches, .triggered_watches and watch history indices
    • Marvel indices not compatible with X-pack Monitoring
    • Upgrading from 2.x > x-pack section from above applies here
    Upgrading from 1.x
    X-Pack
    https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html

    View Slide

  34. • Elasticsearch output plugin (node or transport protocol)
    • Plugin that enables this feature now community-supported
    • Use default (since 2.x) http protocol
    • Upgrading from 2.x > Logstash section from above applies here
    Upgrading from 1.x
    Logstash
    https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html

    View Slide

  35. • Filebeat replaces Logstash forwarder (EOL)
    • Logstash forwarder + lumberjack input -> Filebeat + beats input
    • Reindex existing indices created in ES 1.x
    • Topbeat indices not compatible with 5.0 dashboards
    • Upgrading from 2.x > Beats section from above applies here
    Upgrading from 1.x
    Beats
    https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html

    View Slide

  36. Upgrading in Elastic Cloud
    36

    View Slide

  37. 1. Restore latest snapshot of indices to a test cluster running the same version
    Upgrading in Elastic Cloud

    View Slide

  38. 2. Run the migration plugin and enable deprecation logging in the test cluster with the
    restored snapshot
    3. Review breaking changes documentation
    Upgrading in Elastic Cloud

    View Slide

  39. 4. Address “red” issues reported by the migration plugin
    5. Upgrade (1 click!) Elasticsearch and Kibana (in that order)
    6. Validate upgraded test cluster before upgrading production cluster.
    Upgrading in Elastic Cloud

    View Slide

  40. Best Practices & Caveats
    Examples from the real-world
    40

    View Slide

  41. • Run compatible/supported versions of Elastic products
    • https://www.elastic.co/support/matrix
    • Backup indices before upgrading/reindexing
    • Review breaking changes and implement necessary remediation changes
    • Migration plugin and deprecation logging are only “helper” tools
    • Set up separate 5.0 cluster with parallel indexing
    • Full cluster restart inherently requires downtime/maintenance window
    • Local reindexing requires additional disk space.
    Best Practices

    View Slide

  42. • All nodes must be upgraded to the same version
    • Upgrade commercial and community plugins
    • Java transport/node clients and NEST not compatible across major versions
    ▪ For Java applications, check out the new Java REST Client
    (https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_features.html)
    • Upgrade to the latest version
    Best Practices

    View Slide

  43. • TEST! TEST! TEST!
    Best Practices

    View Slide

  44. Upgrading Resources
    44

    View Slide

  45. • Cross stack upgrade guide
    • https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html
    • Release highlights (per product)
    • https://www.elastic.co/blog/category/releases
    • Support Matrix ( https://www.elastic.co/support/matrix )
    • Upcoming End of Life (EOL) Dates ( https://www.elastic.co/support/eol )
    • 1.6.x - Dec 9, 2016
    • 1.7.x - Jan 16, 2017
    Resources

    View Slide

  46. • Training
    • Courses in your area: http://training.elastic.co/classes
    • Consulting
    • For on-site, time-sensitive assistance
    • https://www.elastic.co/services_policy#upgrade-strategy
    • Support
    • https://www.elastic.co/subscriptions
    Resources

    View Slide

  47. 47
    { }
    “{support engineer} proactively contacted me as our upgrade has been delayed
    twice now. We are finally getting back on track on this end to the reach out was
    very timely and appreciated. {support engineer} provided some great advice in
    terms of technical specifics and a roadmap for information we will have to share
    during the actual implementation to make it a success!”

    View Slide

  48. 48
    { }
    “Very detailed tailored to our site upgrade instruction by {support engineer}
    worked flawlessly.”

    View Slide

  49. 49
    { }
    “In the length of time of using Elastic Support, you guys have saved my {fill in the
    blank} 2 major times. The first time was the migration from ES1.x to ES2.x … The
    only reason I was able to complete my maintenance without impact was due to
    the work of {support engineer} and team. You guys rock!!

    View Slide

  50. • Discuss
    • https://discuss.elastic.co/
    • Github
    • https://github.com/elastic
    • Stackoverflow
    • http://stackoverflow.com/questions/tagged/elasticsearch
    • IRC
    • #elasticsearch, #logstash, #kibana, #beats
    Resources

    View Slide

  51. Questions?
    51
    Log into IRC to ask questions
    • #elastic-webinar @ Freenode
    • Click ”Join the Chat” link, create an IRC account

    View Slide

  52. Thank You
    52

    View Slide