Level Up Your Cluster

For some, upgrading can be the bane of their job. Does the latest version fix any bugs that affect my cluster? Will the new minor release break my system?

Often, users decide to upgrade based on a new feature they need, or a bug fix that affects them. But upgrading can have other benefits too, like improved performance and decreased index size. In this talk, we will explore why upgrading can help you, what questions to ask yourself before upgrading, and how to actually perform that pesky upgrade with minimal misery.

Transcript

1. Level Up Your Cluster Ryan Ernst
2. { } CC-BY-ND 4.0 Why Upgrade? 2
3. { } CC-BY-ND 4.0 What is improving in Lucene? 3 • Bug fixes! • Less heap usage! • Faster!
4. { } CC-BY-ND 4.0 What is Improving in Elasticsearch? • Bug fixes! • Less heap usage! • Faster! 4
5. { } CC-BY-ND 4.0 Lucene Versioning Overview 5
6. { } CC-BY-ND 4.0 What’s in a version? • MAJOR.MINOR.BUGFIX • Compatibility Questions – Which index formats can it read? – Did the APIs change? – Did my analyzers change? 6
7. { } CC-BY-ND 4.0 Indexing 7 Buffer Segment 1 Docs Buffer Docs Buffer Docs Segment 2 Segment 3
8. { } CC-BY-ND 4.0 Commits 8 Segment 1 Segment 2 Segment 3 Commit 1 4.9 4.9 4.9 4.9
9. { } CC-BY-ND 4.0 Commits can span versions 9 Segment 1 Segment 2 Segment 3 Commit 1 4.9 4.9 4.9 4.9 Commit 2 4.10 Segment 4 4.10
10. { } CC-BY-ND 4.0 Legacy segments 10 Segment 1 Segment 2 Segment 3 4.9 4.9 4.9 Commit 2 4.10 Segment 4 4.10 Segment 0 3.6
11. { } CC-BY-ND 4.0 11
12. { } CC-BY-ND 4.0 Open in Lucene 5 12 Segment 1 Segment 2 Segment 3 4.9 4.9 4.9 Commit 2 4.10 Segment 4 4.10 Segment 0 3.6 IndexFormatTooOldException
13. { } CC-BY-ND 4.0 Merging 13 Segment 1 Segment 2 4.9 4.9 Segment 0 3.6 Segment 5 4.10
14. { } CC-BY-ND 4.0 Merging 14 Commit 3 4.10 Segment 3 4.9 Segment 4 4.10 Segment 5 4.10 Segments 0-2 Commit 2 4.10
15. { } CC-BY-ND 4.0 When do we merge? • Background – join together smaller segments – remove deleted documents • Explicitly – force merge – upgrade tool 15
16. { } CC-BY-ND 4.0 Analyzers 16 Elastic{ON} elastic on Indexing Query Index Elastic{ON} elastic on Query English Analyzer English Analyzer
17. { } CC-BY-ND 4.0 Analysis Change 17 Elastic{ON} elasticon Query Hypothetical Change English Analyzer ?? elastic on Index
18. { } CC-BY-ND 4.0 Analysis Compatibility 18 Elastic{ON} elastic on Query English Analyzer 4.2 Force Old Behavior elastic on Index
19. { } CC-BY-ND 4.0 Lucene Version Compatibility 19 Index Format API Analyzers Major N - 1 No Guarantees N - 1 Minor Compatible Binary Compatible N - 1 Bugfix Same Binary Compatible N - 1
20. { } CC-BY-ND 4.0 Elasticsearch Versioning Overview 20
21. { } CC-BY-ND 4.0 What’s in an Elasticsearch version? • Roughly same version format – Betas and RCs are effectively minor versions • Compatibility Questions – Can it read my existing index? – Did the REST API change? – Can it read my mappings? – Can it talk directly to my existing cluster? 21
22. { } CC-BY-ND 4.0 Index format • Same policy as Lucene – Elasticsearch 1.x uses Lucene 4.x – Elasticsearch 2.x uses Lucene 5.x – Pattern should continue in the future 22
23. { } CC-BY-ND 4.0 Wire format 23 Node 1 Node 2 1.4 1.4 1.4
24. { } CC-BY-ND 4.0 Mixed cluster 24 Node 1 Node 2 Node 3 1.4 1.2 1.4 1.4 1.2 1.2
25. { } CC-BY-ND 4.0 Analyzers Compatibility 25 { “analysis”: { “analyzer”: { “standard”: { “type”: “standard”, “version”: 4.3 } } } }
26. { } CC-BY-ND 4.0 Elasticsearch Version Compatibility 26 Index Format API Mapping Wire Format Major N - 1 No Guarantees N - 1 No Guarantees Minor Compatible Compatible N - 1 Compatible Bugfix Same Compatible N - 1 Same
27. { } CC-BY-ND 4.0 Upgrading Best Practices 27
28. { } CC-BY-ND 4.0 Before Upgrading • Backup • Test it out 28 PUT /_snapshot/my_repository/backup_before_upgrade Test  Cluster Production  Cluster
29. { } CC-BY-ND 4.0 29
30. { } CC-BY-ND 4.0 General upgrade guidelines 30 • Do a flush - automatic on shutdown for 1.5+ • Stop updates if you can
31. { } CC-BY-ND 4.0 Rolling Restart Without this, when nodes stop, Elasticsearch will try to shuffle data around to maintain the desired replicas. 31 Step 1: Disable Allocation PUT /_cluster/settings { "transient" : { "cluster.routing.allocation.enable" : "none" } }
32. { } CC-BY-ND 4.0 Rolling Restart Use the Shutdown API instead of killing the process directly. 32 Step 2: Shutdown a Single Node POST /_cluster/nodes/_local/_shutdown
33. { } CC-BY-ND 4.0 Rolling Restart After upgrading the Elasticsearch software, start the node and wait for it to join the cluster. 33 Step 3: Start the upgraded node GET /_cluster/health
34. { } CC-BY-ND 4.0 Rolling Restart This allows any problems with the data on the node that was restarted to be resolved via recovery. 34 Step 4: Enable allocation PUT /_cluster/settings { "transient" : { "cluster.routing.allocation.enable" : "all" } }
35. { } CC-BY-ND 4.0 Rolling Restart Allow any recoveries to complete. 35 Step 5: Wait for Green status GET /_cluster/health GET /_cat/shards GET /_cat/recovery
36. { } CC-BY-ND 4.0 Rolling Restart 36 Step 6: Rinse and Repeat
37. { } CC-BY-ND 4.0 Problems with Rolling Restart 37 • Any indexing eliminates the possibility to downgrade - explicitly indexing docs - recovery from translog • Nodes talk with common format • Past bugs • Getting better
38. { } CC-BY-ND 4.0 Full Restart 38 • Same as a rolling restart but – shutdown all nodes in step 2 – start all nodes in step 3
39. { } CC-BY-ND 4.0 What about upgrading indexes? • Upgrading Elasticsearch is only the software • Still need to upgrade Lucene segments – will happen in background with merges mostly – sometimes need/want to force segments to upgrade 39
40. { } CC-BY-ND 4.0 Use the Upgrade API 40 GET /MYINDEX/_upgrade?human { “MYINDEX" : { "size" : “21gb”, “size_in_bytes” : “21000040500”, “size_to_upgrade” : “10gb”, “size_to_upgrade_in_bytes” : “10000000”, } }
41. { } CC-BY-ND 4.0 Use the Upgrade API 41 POST /MYINDEX/_upgrade 1.4.3+ POST /MYINDEX/_upgrade?wait_for_completion=true 1.4.0-1.4.2 < 1.4.0 ?
42. { } Questions? Thank you! twitter.com/ryanjernst
43. { } This work is licensed under the Creative Commons Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0