Scale Your Elasticsearch Cluster

Elasticsearch is highly scalable, but some recent additions make it easier to tie everything together the right way. In this talk we discuss:
* Frozen Indices for much higher storage density per Elasticsearch node and its tradeoffs.
* Index Lifecycle Management or how to automate having a multi-tier architecture and how this actually works in the background.
* Rollups or how to aggregate metrics over time.

The goal of this talk is to make your cluster management less complex and more cost effective.

Transcript

1. SCALE YOUR CLUSTER PHILIPP KRENN̴̴̴̴@XERAA
2. DEVELOPER
3. AGENDA INDEX LIFECYCLE MANAGEMENT FROZEN INDICES ROLLUPS
4. START DEMO SETUP
5. INDEX LIFECYCLE MANAGEMENT
6. ! "
7. FUTURE: SNAPSHOT LIFECYCLE MANAGEMENT FOR NOW ELASTIC CURATOR
8. FEATURES & ORDER https://github.com/elastic/elasticsearch/blob/7.0/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/TimeseriesLifecycleType.java static final List<String> ORDERED_VALID_HOT_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, RolloverAction.NAME ); static final List<String> ORDERED_VALID_WARM_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, ReadOnlyAction.NAME, AllocateAction.NAME, ShrinkAction.NAME, ForceMergeAction.NAME ); static final List<String> ORDERED_VALID_COLD_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, AllocateAction.NAME, FreezeAction.NAME ); static final List<String> ORDERED_VALID_DELETE_ACTIONS = Arrays.asList( DeleteAction.NAME );
9. DEMO
10. PS: 7.0 IMPROVEMENT index.search.idle.after: 30s IFF DEFAULT index.refresh_interval
11. FROZEN INDICES
12. RATIO HEAP : STORAGE INDEX > FROZEN INDEX > CLOSED INDEX
13. FROZEN INDEX READ-ONLY NO MEMORY
14. THROTTLED THREAD POOL 1 PARALLEL SEARCH / NODE 100 IN QUEUE
15. DEMO
16. ROLLUPS
18. DEMO
19. CONCLUSION
20. CODE https://github.com/xeraa/scale-elasticsearch
21. RECAP INDEX LIFECYCLE MANAGEMENT FROZEN INDICES ROLLUPS
22. Questions? PHILIPP KRENN̴̴̴̴@XERAA