Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Level Up Your Cluster

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
March 19, 2015
1.1k

Level Up Your Cluster

For some, upgrading can be the bane of their job. Does the latest version fix any bugs that affect my cluster? Will the new minor release break my system?

Often, users decide to upgrade based on a new feature they need, or a bug fix that affects them. But upgrading can have other benefits too, like improved performance and decreased index size. In this talk, we will explore why upgrading can help you, what questions to ask yourself before upgrading, and how to actually perform that pesky upgrade with minimal misery.

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

March 19, 2015
Tweet

Transcript

  1. Level Up Your Cluster Ryan Ernst

  2. { } CC-BY-ND 4.0 Why Upgrade? 2

  3. { } CC-BY-ND 4.0 What is improving in Lucene? 3

    • Bug fixes! • Less heap usage! • Faster!
  4. { } CC-BY-ND 4.0 What is Improving in Elasticsearch? •

    Bug fixes! • Less heap usage! • Faster! 4
  5. { } CC-BY-ND 4.0 Lucene Versioning Overview 5

  6. { } CC-BY-ND 4.0 What’s in a version? • MAJOR.MINOR.BUGFIX

    • Compatibility Questions – Which index formats can it read? – Did the APIs change? – Did my analyzers change? 6
  7. { } CC-BY-ND 4.0 Indexing 7 Buffer Segment 1 Docs

    Buffer Docs Buffer Docs Segment 2 Segment 3
  8. { } CC-BY-ND 4.0 Commits 8 Segment 1 Segment 2

    Segment 3 Commit 1 4.9 4.9 4.9 4.9
  9. { } CC-BY-ND 4.0 Commits can span versions 9 Segment

    1 Segment 2 Segment 3 Commit 1 4.9 4.9 4.9 4.9 Commit 2 4.10 Segment 4 4.10
  10. { } CC-BY-ND 4.0 Legacy segments 10 Segment 1 Segment

    2 Segment 3 4.9 4.9 4.9 Commit 2 4.10 Segment 4 4.10 Segment 0 3.6
  11. { } CC-BY-ND 4.0 11

  12. { } CC-BY-ND 4.0 Open in Lucene 5 12 Segment

    1 Segment 2 Segment 3 4.9 4.9 4.9 Commit 2 4.10 Segment 4 4.10 Segment 0 3.6 IndexFormatTooOldException
  13. { } CC-BY-ND 4.0 Merging 13 Segment 1 Segment 2

    4.9 4.9 Segment 0 3.6 Segment 5 4.10
  14. { } CC-BY-ND 4.0 Merging 14 Commit 3 4.10 Segment

    3 4.9 Segment 4 4.10 Segment 5 4.10 Segments 0-2 Commit 2 4.10
  15. { } CC-BY-ND 4.0 When do we merge? • Background

    – join together smaller segments – remove deleted documents • Explicitly – force merge – upgrade tool 15
  16. { } CC-BY-ND 4.0 Analyzers 16 Elastic{ON} elastic on Indexing

    Query Index Elastic{ON} elastic on Query English Analyzer English Analyzer
  17. { } CC-BY-ND 4.0 Analysis Change 17 Elastic{ON} elasticon Query

    Hypothetical Change English Analyzer ?? elastic on Index
  18. { } CC-BY-ND 4.0 Analysis Compatibility 18 Elastic{ON} elastic on

    Query English Analyzer 4.2 Force Old Behavior elastic on Index
  19. { } CC-BY-ND 4.0 Lucene Version Compatibility 19 Index Format

    API Analyzers Major N - 1 No Guarantees N - 1 Minor Compatible Binary Compatible N - 1 Bugfix Same Binary Compatible N - 1
  20. { } CC-BY-ND 4.0 Elasticsearch Versioning Overview 20

  21. { } CC-BY-ND 4.0 What’s in an Elasticsearch version? •

    Roughly same version format – Betas and RCs are effectively minor versions • Compatibility Questions – Can it read my existing index? – Did the REST API change? – Can it read my mappings? – Can it talk directly to my existing cluster? 21
  22. { } CC-BY-ND 4.0 Index format • Same policy as

    Lucene – Elasticsearch 1.x uses Lucene 4.x – Elasticsearch 2.x uses Lucene 5.x – Pattern should continue in the future 22
  23. { } CC-BY-ND 4.0 Wire format 23 Node 1 Node

    2 1.4 1.4 1.4
  24. { } CC-BY-ND 4.0 Mixed cluster 24 Node 1 Node

    2 Node 3 1.4 1.2 1.4 1.4 1.2 1.2
  25. { } CC-BY-ND 4.0 Analyzers Compatibility 25 { “analysis”: {

    “analyzer”: { “standard”: { “type”: “standard”, “version”: 4.3 } } } }
  26. { } CC-BY-ND 4.0 Elasticsearch Version Compatibility 26 Index Format

    API Mapping Wire Format Major N - 1 No Guarantees N - 1 No Guarantees Minor Compatible Compatible N - 1 Compatible Bugfix Same Compatible N - 1 Same
  27. { } CC-BY-ND 4.0 Upgrading Best Practices 27

  28. { } CC-BY-ND 4.0 Before Upgrading • Backup • Test

    it out 28 PUT /_snapshot/my_repository/backup_before_upgrade Test  Cluster Production  Cluster
  29. { } CC-BY-ND 4.0 29

  30. { } CC-BY-ND 4.0 General upgrade guidelines 30 • Do

    a flush - automatic on shutdown for 1.5+ • Stop updates if you can
  31. { } CC-BY-ND 4.0 Rolling Restart Without this, when nodes

    stop, Elasticsearch will try to shuffle data around to maintain the desired replicas. 31 Step 1: Disable Allocation PUT /_cluster/settings { "transient" : { "cluster.routing.allocation.enable" : "none" } }
  32. { } CC-BY-ND 4.0 Rolling Restart Use the Shutdown API

    instead of killing the process directly. 32 Step 2: Shutdown a Single Node POST /_cluster/nodes/_local/_shutdown
  33. { } CC-BY-ND 4.0 Rolling Restart After upgrading the Elasticsearch

    software, start the node and wait for it to join the cluster. 33 Step 3: Start the upgraded node GET /_cluster/health
  34. { } CC-BY-ND 4.0 Rolling Restart This allows any problems

    with the data on the node that was restarted to be resolved via recovery. 34 Step 4: Enable allocation PUT /_cluster/settings { "transient" : { "cluster.routing.allocation.enable" : "all" } }
  35. { } CC-BY-ND 4.0 Rolling Restart Allow any recoveries to

    complete. 35 Step 5: Wait for Green status GET /_cluster/health GET /_cat/shards GET /_cat/recovery
  36. { } CC-BY-ND 4.0 Rolling Restart 36 Step 6: Rinse

    and Repeat
  37. { } CC-BY-ND 4.0 Problems with Rolling Restart 37 •

    Any indexing eliminates the possibility to downgrade - explicitly indexing docs - recovery from translog • Nodes talk with common format • Past bugs • Getting better
  38. { } CC-BY-ND 4.0 Full Restart 38 • Same as

    a rolling restart but – shutdown all nodes in step 2 – start all nodes in step 3
  39. { } CC-BY-ND 4.0 What about upgrading indexes? • Upgrading

    Elasticsearch is only the software • Still need to upgrade Lucene segments – will happen in background with merges mostly – sometimes need/want to force segments to upgrade 39
  40. { } CC-BY-ND 4.0 Use the Upgrade API 40 GET

    /MYINDEX/_upgrade?human { “MYINDEX" : { "size" : “21gb”, “size_in_bytes” : “21000040500”, “size_to_upgrade” : “10gb”, “size_to_upgrade_in_bytes” : “10000000”, } }
  41. { } CC-BY-ND 4.0 Use the Upgrade API 41 POST

    /MYINDEX/_upgrade 1.4.3+ POST /MYINDEX/_upgrade?wait_for_completion=true 1.4.0-1.4.2 < 1.4.0 ?
  42. { } Questions? Thank you! twitter.com/ryanjernst

  43. { } This work is licensed under the Creative Commons

    Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0