Upgrading Log-Analytics Clusters to OpenSearch (Amitai Stern, LogzIO) | RTA Summit 2023

Here at Logz.io, the open-source observability and security company, we run ElasticSearch for over 1300 companies in highly scalable multi-cloud deployments. With the license changes of 2021 we needed to migrate to an open-source platform, and OpenSearch was where we were going to contribute and what we wanted to run in production.

Many equate upgrading to OpenSearch from Elasticsearch in production as changing the tires on a moving bus. Upgrading has many risks, and if the cluster is in continuous production use, ingesting terabytes of data daily, the risks can seem overbearing.

In this talk, we will cover multiple upgrade strategies, including version requirements, and their pros and cons. Additionally, we will cover a different option, which is the way we, at Logz.io, upgraded all our clusters to OpenSearch without significant extra costs while minimizing risk. Not only did we upgrade to OpenSearch, but we also migrated our AWS workloads to Graviton2 instances.

Transcript

1. Upgrading log-analytics clusters to OpenSearch @amitaistern Amitai Stern Software Engineer and Telemetry Storage Team Lead at Logz.io
3. 6.8 7.0.0 7.10 8.7 7.11 1.0.0 2.6
4. January 2021 6.8 7.0.0 7.10 8.7 7.11 1.0.0 2.6
5. July 2021 6.8 7.0.0 7.10 8.7 7.11 1.0.0 2.6 January 2021
6. Apache 2.0-licensed open source Server Side Public License (SSPL) 6.8 7.0.0 7.10 8.7 7.11 1.0.0 2.6
7. Supports both AMD64 and ARM64 architectures Supports AMD64 architecture 6.8 7.0.0 7.10 8.7 7.11 1.0.0 2.6
8. 6.8 7.0.0 7.10 8.7 7.11 1.0.0 2.6
9. Log-Engine Application Kibana Query-Service Other Microservices search ingest Log Analytics Cluster Architecture Amazon S3 (cluster snapshots)
10. Log-Engine Application Kibana Query-Service Other Microservices search ingest Log Analytics Cluster Architecture Amazon S3 (cluster snapshots)
11. Preparing for the upgrade Index versions Deprecated APIs Breaking changes Test env Cluster Settings thresholds
12. Common Upgrading Strategies
13. Common Upgrading Strategies: Blue/Green Cluster Cluster read write
14. Common Upgrading Strategies: Blue/Green Cluster Cluster read write
15. Common Upgrading Strategies: In-Place Data nodes Coordinator nodes Cluster manager nodes Cluster
16. Balancing Risk, Cost, and Speed
17. The Drain Method Data nodes PUT _cluster/settings { "persistent": { "cluster.routing.allocation.exclude._ip": "172.22.4.9" } } 172.22.4.9
18. The Drain Method Data nodes PUT _cluster/settings { "persistent": { "cluster.routing.allocation.exclude._ip": "172.22.4.9", "indices.recovery.max_bytes_per_sec": "150mb" } } 172.22.4.9
19. The Drain Method Data nodes PUT _cluster/settings { "persistent": { "cluster.routing.allocation.include._ip": "172.33.14.1,172.22.4.9" } } 172.22.4.9 172.33.14.1
20. The Drain Method Data nodes PUT _cluster/settings { "persistent": { "cluster.routing.allocation.include._ip": "172.33.14.1,172.22.4.9" } }
21. The Drain Method: Upgrade Process Overview PUT _cluster/settings { "persistent": { "cluster.routing.allocation.include._ip": "<Elasticsearch IPs>" } } Data nodes Coordinator nodes Cluster manager nodes
22. The Drain Method: Upgrade Process Overview PUT _cluster/settings { "persistent": { "cluster.routing.allocation.include._ip": "<Elasticsearch IPs>", "cluster.routing.allocation.exclude._ip": "<OpenSearch IPs>" } } Data nodes Coordinator nodes Cluster manager nodes
23. The Drain Method: Upgrade Process Overview PUT _cluster/settings { "persistent": { "cluster.routing.allocation.include._ip": "<OpenSearch IPs>", "cluster.routing.allocation.exclude._ip": "<Elasticsearch IPs>" } } Data nodes Coordinator nodes Cluster manager nodes
24. The Drain Method: Upgrade Process Overview PUT _cluster/settings { "persistent": { "cluster.routing.allocation.include._ip": "<OpenSearch IPs>", "cluster.routing.allocation.exclude._ip": "<Elasticsearch IPs>" } } Data nodes Coordinator nodes Cluster manager nodes "indices.recovery.max_bytes_per_sec": "300mb"
25. The Drain Method: Upgrade Process Overview PUT _cluster/settings { "persistent": { "cluster.routing.allocation.include._ip": "<OpenSearch IPs>", "cluster.routing.allocation.exclude._ip": "<Elasticsearch IPs>" } } Data nodes Coordinator nodes Cluster manager nodes "indices.recovery.max_bytes_per_sec": "0mb"
26. The Drain Method: Upgrade Process Overview PUT _cluster/settings { "persistent": { "cluster.routing.allocation.include._ip": null, "cluster.routing.allocation.exclude._ip": null } } Data nodes Coordinator nodes Cluster manager nodes
27. The Drain Method: Upgrade Process Overview | Data nodes Coordinator nodes Cluster manager nodes load balancer DNS record
28. The Drain Method: Upgrade Process Overview - New LB - OpenSearch Coordinator Nodes Data nodes load balancer Coordinator nodes Cluster manager nodes load balancer DNS record
29. - New LB - OpenSearch Coordinator Nodes - Override DNS record DNS record The Drain Method: Upgrade Process Overview Data nodes load balancer Coordinator nodes Cluster manager nodes load balancer
30. The Drain Method: Upgrade Process Overview - New LB - OpenSearch Coordinator Nodes - Override DNS record - Remove old Coordinating Nodes Data nodes Coordinator nodes Cluster manager nodes load balancer
31. | The Drain Method: Upgrade Process Overview Data nodes Coordinator nodes Cluster manager nodes
32. The Drain Method: Upgrade Process Overview - Add 3 more Cluster manager Nodes Data nodes Coordinator nodes Cluster manager nodes
33. The Drain Method: Upgrade Process Overview - Add 3 more Cluster manager Nodes - Remove the old ones one at a time (elected one last) Data nodes Coordinator nodes Cluster manager nodes
34. The Drain Method: Upgrade Process Overview - Add 3 more Cluster manager Nodes - Remove the old ones one at a time (elected one last) Data nodes Coordinator nodes Cluster manager nodes
35. The Drain Method: Upgrade Process Overview - Add 3 more Cluster manager Nodes - Remove the old ones one at a time (elected one last) Data nodes Coordinator nodes Cluster manager nodes
36. The Drain Method: Upgrade Process Overview - Add 3 more Cluster manager Nodes - Remove the old ones one at a time (elected one last) Data nodes Coordinator nodes Cluster manager nodes
37. The Drain Method: Upgrade Process Overview - Add 3 more Cluster manager Nodes - Remove the old ones one at a time (elected one last) - Await Cluster Manager Node reelection Data nodes Coordinator nodes ??? Cluster manager nodes
38. DONE :) The Drain Method: Upgrade Process Overview Data nodes Coordinator nodes Cluster manager nodes
39. | The Drain Method: Upgrade Process Overview Data nodes Coordinator nodes Cluster manager nodes
40. The Drain Method: Managing risk Log-Engine Application Kibana Query-Service search ingest Amazon S3 (cluster snapshots)
41. Backup cluster Backup Log-Engine Application Kibana Query-Service search The Drain Method: Managing risk Log-Engine ingest Amazon S3 (cluster snapshots)
42. Backup cluster Application Kibana Query-Service search The Drain Method: Managing risk Log-Engine ingest Amazon S3 (cluster snapshots) Backup Log-Engine
43. The Drain Method: Managing risk Log-Engine Application Kibana Query-Service search ingest Amazon S3 (cluster snapshots)
44. Backup cluster The Drain Method: Managing risk Log-Engine ingest Amazon S3 (cluster snapshots) Backup Log-Engine Application Kibana Query-Service search
45. Backup cluster Application Kibana Query-Service search The Drain Method: Managing risk Log-Engine ingest Amazon S3 (cluster snapshots) Backup Log-Engine
46. Backup cluster Backup Log-Engine Application Kibana Query-Service search The Drain Method: Managing risk ingest Amazon S3 (cluster snapshots)
47. Backup Log-Engine Application Kibana Query-Service search The Drain Method: Managing risk ingest Amazon S3 (cluster snapshots) Restore from Snapshot Backup cluster
48. Backup cluster Backup Log-Engine Application Kibana Query-Service search The Drain Method: Managing risk ingest Amazon S3 (cluster snapshots)
49. Blue/Green In Place Drain Pros Fully revertable (instantly) Can replace hardware as well Fast (within a few hours) Cheap (0 extra nodes) Fully revertable (within hours) Rather fast (many hours) Can replace hardware as well Cheaper than Blue/Green Cons Slow upgrade (days/weeks) Complexity grows over time Double the cluster cost for the duration No rolling back No hardware change Costs more than In Place Complex upgrade process Complex rollback Summary Drain
50. Upgrading log-analytics clusters to OpenSearch Q&A @amitaistern Amitai Stern Software Engineer and Telemetry Storage Team Lead at Logz.io