Upgrading Your Elastic Stack to 5.0

Version 5.0 introduces many new features and improvements across all components of the Elastic Stack. From a new scripting language for Elasticsearch to a new monitoring API for Logstash to a more colorific and eye-pleasing Kibana to a new Beat for gathering infrastructure metrics, there are many compelling reasons to upgrade your Elastic Stack to 5.0. In this video, Pius and Chris walk through the considerations, best practices and caveats for upgrading your Elastic Stack to 5.0.

Transcript

1. Upgrading to 5.0
   1
   

   View Slide

2. 2
   Pius Fung
   Support Engineer
   Chris Earle
   Monitoring Lead
   

   View Slide

3. Housekeeping & Logistics
   • Slides and recording will be available following the webinar
   • Chat via IRC #elastic-webinar
   ‒ #elastic-webinar @ Freenode
   ‒ Click ”Join the Chat” link, create an IRC account
   It has to happen…
   

   View Slide

4. Register for Elastic{ON}17.
   Holiday pricing of $1,195 ($1,495 full price) expires January 6.
   https://www.elastic.co/elasticon/conf/2017/sf/registration
   

   View Slide

5. All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement. 5
   Elastic Stack
   Version 5.0.
   All new versions.
   All aligned.
   

   View Slide

6. All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement. 6
   Security
   Alerting
   Monitoring
   Reporting
   X-Pack
   Graph
   

   View Slide

7. All product names, logos, and brands are property of their respective owners and are used only for identification purposes. This is not an endorsement. 7
   Elastic
   Cloud
   Security
   Alerting
   Monitoring
   Reporting
   X-Pack
   Graph
   

   View Slide

8. • Full Elastic Stack upgrade order
   • Upgrading from 2.x
   • Upgrading from 1.x
   • Upgrading on Elastic Cloud
   • Best practices and caveats
   • Upgrade resources
   Topics for today:
   

   View Slide

9. Full Stack Upgrade Order
   9
   

   View Slide

10. Full Stack Upgrade
    General Upgrade Order
    https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html [ NEW ]
    Upgrade Order
    

    View Slide

11. Full Stack Upgrade
    General Upgrade Order
    https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html [ NEW ]
    Upgrade Order
    

    View Slide

12. Full Stack Upgrade
    ES-Hadoop
    General Upgrade Order
    https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html [ NEW ]
    Upgrade Order
    

    View Slide

13. Upgrading from 2.x
    Elasticsearch
    13
    

    View Slide

14. • Indices created prior to 2.0 are not compatible with 5.0
    ▪ Elasticsearch will not start up
    ▪ Snapshots will not restore
    • Requires reindexing
    • Currently, upgrading between major versions requires a full cluster restart
    Upgrading from 2.x
    Breaking Changes
    https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes.html
    

    View Slide

15. • Warns in development mode. Refuses to start in production mode.
    • Motivation - https://www.elastic.co/blog/bootstrap_checks_annoying_instead_of_devastating
    Examples:
    • Heap size, bootstrap.memory_lock (previously known as mlockall),
    OnOutOfMemoryError
    • File descriptor, virtual memory size, memory-mapped area, max # of threads
    Upgrading from 2.x
    Breaking Changes >> Bootstrap Checks
    https://www.elastic.co/guide/en/elasticsearch/reference/5.0/bootstrap-checks.html
    

    View Slide

16. Examples:
    • filtered query ➔ bool query
    • search_type=count ➔ size: 0
    • scroll query with search_type=scan ➔ scroll query with sort: [“_doc”]
    • and / or ➔ bool-must / bool-should
    Upgrading from 2.x
    Breaking Changes >> Query Changes
    https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_search_changes.htm
    

    View Slide

17. Examples:
    • string field type replaced by text and keyword type
    • strings now mapped both as text and keyword by default (can be customized)
    • https://www.elastic.co/blog/strings-are-dead-long-live-strings
    • scoring on numeric fields requires mapping as keyword field
    • _timestamp and _ttl removed
    Upgrading from 2.x
    Breaking Changes >> Mapping Changes
    https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_mapping_changes.html
    

    View Slide

18. Examples:
    • Most index settings must defined at index level (not in yml)
    • System properties changes, e.g. -Des.path.conf replaced by -Epath.conf
    • Define node attributes using node.attr. instead of node.
    • Default node.name is now first 7 characters of random UUID
    Upgrading from 2.x
    Breaking Changes >> Settings Changes
    https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_settings_changes.html
    

    View Slide

19. Examples:
    • JVM options (including heap size) now in jvm.options file.
    • JVM terminates on OutOfMemoryError, StackOverFlowError, etc..
    • Repository location for apt/yum packages is now at https://artifacts.elastic.co
    (previously https://packages.elastic.co)
    Upgrading from 2.x
    Breaking Changes >> Packaging
    https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_packaging.html
    

    View Slide

20. Out of the box safeguards added to prevent simple user errors
    Examples:
    • Soft limit on the number of shards that can be queried in a single search request
    (1000)
    • Soft limit on total # of fields in an index (1000)
    • Soft limit on total # of nested fields in an index (50)
    • Soft limit on mapping depth (20)
    Upgrading from 2.x
    Breaking Changes >> Safeguards
    

    View Slide

21. Upgrading from 2.x
    Elasticsearch Upgrading Options
    https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html
    Full Cluster Restart Upgrade
    • Only indices created in 2.0+
    • 1.x and 2.x time-based indices
    • can wait for 1.x indices to be deleted
    Reindex to Upgrade
    • Indices created in 1.x
    • 1.x and 2.x time based indices
    • cannot wait for 1.x indices to be deleted
    • reindex 1.x indices
    • 2.x indices, but want new features that
    require reindexing (eg. BKD tree for
    numeric data fields)
    

    View Slide

22. Cluster Checkup
    • Report problems on cluster, node and index settings, index mapping and segment
    version conflicts, installed plugins, etc..
    • There are breaking changes not covered by the migration plugin.
    Reindex Helper
    • Provides 1-click local reindexing of indices that are not compatible with Elasticsearch
    5.0
    Deprecation Logging
    • Enable deprecation logging on the fly
    Upgrading from 2.x
    Demo: Elasticsearch Migration Plugin (2.x)
    https://github.com/elastic/elasticsearch-migration/tree/2.x
    

    View Slide

23. • Elasticsearch 2.0+
    • Can be enabled dynamically, or use migration plugin
    • _deprecation.log in Elasticsearch logs folder
    Upgrading from 2.x
    Demo: Deprecation Logging
    https://www.elastic.co/guide/en/elasticsearch/reference/2.4/setup-configuration.html#deprecation-logging
    curl -XPUT "http://localhost:9200/_cluster/settings" -d'
    {
    "transient": {
    "logger.deprecation":"DEBUG"
    }
    }'
    

    View Slide

24. Upgrading from 2.x
    Kibana | X-Pack | Logstash | Beats
    24
    

    View Slide

25. • Binds to localhost by default
    • RPM / DEB installation
    • New repo location: https://artifacts.elastic.co
    • /usr/share/kibana and /etc/kibana
    • New plugin install command: /bin/kibana-plugin
    • First class support of tribe node coming in 5.x
    Upgrading from 2.x
    Kibana >> Breaking Changes
    https://www.elastic.co/guide/en/kibana/5.0/breaking-changes-5.0.html
    

    View Slide

26. • .kibana reindexing required for Kibana 4.0/4.1
    • Sense is renamed to Console
    • Console and Timelion now default Kibana apps
    • Shield/Marvel/Graph UI and Reporting now part of X-Pack
    • Query DSL breaking changes in saved objects
    Upgrading from 2.x
    Kibana >> Upgrading
    https://www.elastic.co/guide/en/kibana/5.0/upgrade.html
    

    View Slide

27. ● Single X-Pack plugin install
    ○ bin/elasticsearch-plugin install x-pack
    ○ bin/kibana-plugin install x-pack
    ● Settings renamed
    ○ watcher.enabled ➔ xpack.watcher.enabled
    ○ shield.authc.realms.ldap1.url ➔ xpack.security.authc.realms.ldap1.url
    ● Endpoints deprecated and renamed
    ○ /_shield/user ➔ /_xpack/security
    ● Role format changes
    ● Native is now the default realm
    ● Java keystores no longer required
    Upgrading from 2.x
    X-Pack >> Breaking Changes
    https://www.elastic.co/guide/en/x-pack/current/migrating-to-xpack.html
    

    View Slide

28. • Changes for rpm/deb installation:
    ▪ New repo location: https://artifacts.elastic.co
    ▪ Binaries now at /usr/share/logstash
    • Elasticsearch output plugin
    ▪ workers setting replaced by pipeline.workers
    ▪ Auto-select logstash template for Elasticsearch version
    • Kafka input/output plugins
    ▪ Compatible with Kafka 0.10 (requires Kafka broker 0.10)
    • Ruby filter plugin
    ▪ Java event class requires refactoring
    o tmp = event['field'] -> tmp = event.get('field')
    o event['field'] = tmp -> event.set('field',tmp)
    • All plugins download package removed
    ▪ Use https://www.elastic.co/guide/en/logstash/current/offline-plugins.html
    Upgrading from 2.x
    Logstash >> Breaking Changes
    https://www.elastic.co/guide/en/logstash/current/breaking-changes.html
    

    View Slide

29. • Migrate existing configs (scripts/migrate_beat_config_1_x_to_5_0.py)
    • Topbeat deprecation
    • Replaced by Metricbeat system module
    • Topbeat data is not compatible with 5.0 dashboards
    • rpm/deb
    • New repo location: https://artifacts.elastic.co
    • New binaries at /usr/share/
    • Update beats index template using output.elasticsearch.template.overwrite
    Upgrading from 2.x
    Beats >> Breaking Changes
    https://www.elastic.co/guide/en/beats/libbeat/current/breaking-changes-5.0.html
    

    View Slide

30. Upgrading from 1.x
    Elastic Stack
    30
    

    View Slide

31. • Option: Upgrade to 2.x first
    • Run migration plugin and deprecation logging
    • Reindex to upgrade
    • Option: “Start from scratch”
    • Reindex directly to 5.0 (Reindex API)
    • Lack of migration helpers
    • Requires separate cluster
    • Upgrading from 2.x > Elasticsearch section from above applies here
    Upgrading from 1.x
    Elasticsearch
    https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
    

    View Slide

32. • Kibana 3 not upgradable
    ▪ Recreate all visualizations and dashboards
    • Upgrading from Kibana 4.0 and 4.1 (compatible with ES 1.x) requires reindexing
    of .kibana
    • Upgrading from 2.x > Kibana section from above applies here
    Upgrading from 1.x
    Kibana
    https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
    

    View Slide

33. • Reindex .watches, .triggered_watches and watch history indices
    • Marvel indices not compatible with X-pack Monitoring
    • Upgrading from 2.x > x-pack section from above applies here
    Upgrading from 1.x
    X-Pack
    https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
    

    View Slide

34. • Elasticsearch output plugin (node or transport protocol)
    • Plugin that enables this feature now community-supported
    • Use default (since 2.x) http protocol
    • Upgrading from 2.x > Logstash section from above applies here
    Upgrading from 1.x
    Logstash
    https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
    

    View Slide

35. • Filebeat replaces Logstash forwarder (EOL)
    • Logstash forwarder + lumberjack input -> Filebeat + beats input
    • Reindex existing indices created in ES 1.x
    • Topbeat indices not compatible with 5.0 dashboards
    • Upgrading from 2.x > Beats section from above applies here
    Upgrading from 1.x
    Beats
    https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html
    

    View Slide

36. Upgrading in Elastic Cloud
    36
    

    View Slide

37. 1. Restore latest snapshot of indices to a test cluster running the same version
    Upgrading in Elastic Cloud
    

    View Slide

38. 2. Run the migration plugin and enable deprecation logging in the test cluster with the
    restored snapshot
    3. Review breaking changes documentation
    Upgrading in Elastic Cloud
    

    View Slide

39. 4. Address “red” issues reported by the migration plugin
    5. Upgrade (1 click!) Elasticsearch and Kibana (in that order)
    6. Validate upgraded test cluster before upgrading production cluster.
    Upgrading in Elastic Cloud
    

    View Slide

40. Best Practices & Caveats
    Examples from the real-world
    40
    

    View Slide

41. • Run compatible/supported versions of Elastic products
    • https://www.elastic.co/support/matrix
    • Backup indices before upgrading/reindexing
    • Review breaking changes and implement necessary remediation changes
    • Migration plugin and deprecation logging are only “helper” tools
    • Set up separate 5.0 cluster with parallel indexing
    • Full cluster restart inherently requires downtime/maintenance window
    • Local reindexing requires additional disk space.
    Best Practices
    

    View Slide

42. • All nodes must be upgraded to the same version
    • Upgrade commercial and community plugins
    • Java transport/node clients and NEST not compatible across major versions
    ▪ For Java applications, check out the new Java REST Client
    (https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_features.html)
    • Upgrade to the latest version
    Best Practices
    

    View Slide

43. • TEST! TEST! TEST!
    Best Practices
    

    View Slide

44. Upgrading Resources
    44
    

    View Slide

45. • Cross stack upgrade guide
    • https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html
    • Release highlights (per product)
    • https://www.elastic.co/blog/category/releases
    • Support Matrix ( https://www.elastic.co/support/matrix )
    • Upcoming End of Life (EOL) Dates ( https://www.elastic.co/support/eol )
    • 1.6.x - Dec 9, 2016
    • 1.7.x - Jan 16, 2017
    Resources
    

    View Slide

46. • Training
    • Courses in your area: http://training.elastic.co/classes
    • Consulting
    • For on-site, time-sensitive assistance
    • https://www.elastic.co/services_policy#upgrade-strategy
    • Support
    • https://www.elastic.co/subscriptions
    Resources
    

    View Slide

47. 47
    { }
    “{support engineer} proactively contacted me as our upgrade has been delayed
    twice now. We are finally getting back on track on this end to the reach out was
    very timely and appreciated. {support engineer} provided some great advice in
    terms of technical specifics and a roadmap for information we will have to share
    during the actual implementation to make it a success!”
    

    View Slide

48. 48
    { }
    “Very detailed tailored to our site upgrade instruction by {support engineer}
    worked flawlessly.”
    

    View Slide

49. 49
    { }
    “In the length of time of using Elastic Support, you guys have saved my {fill in the
    blank} 2 major times. The first time was the migration from ES1.x to ES2.x … The
    only reason I was able to complete my maintenance without impact was due to
    the work of {support engineer} and team. You guys rock!!
    

    View Slide

50. • Discuss
    • https://discuss.elastic.co/
    • Github
    • https://github.com/elastic
    • Stackoverflow
    • http://stackoverflow.com/questions/tagged/elasticsearch
    • IRC
    • #elasticsearch, #logstash, #kibana, #beats
    Resources
    

    View Slide

51. Questions?
    51
    Log into IRC to ask questions
    • #elastic-webinar @ Freenode
    • Click ”Join the Chat” link, create an IRC account
    

    View Slide

52. Thank You
    52
    

    View Slide