Scale Your Elasticsearch Cluster

Scale Your Elasticsearch Cluster

Elasticsearch is highly scalable, but some recent additions make it easier to tie everything together the right way. In this talk we discuss:
* Frozen Indices for much higher storage density per Elasticsearch node and its tradeoffs.
* Index Lifecycle Management or how to automate having a multi-tier architecture and how this actually works in the background.
* Rollups or how to aggregate metrics over time.

The goal of this talk is to make your cluster management less complex and more cost effective.

Ce4685da897c912aa41a815435b40a5a?s=128

Philipp Krenn

April 29, 2019
Tweet

Transcript

  1. SCALE YOUR CLUSTER PHILIPP KRENN̴̴̴̴@XERAA

  2. DEVELOPER

  3. AGENDA INDEX LIFECYCLE MANAGEMENT FROZEN INDICES ROLLUPS

  4. START DEMO SETUP

  5. INDEX LIFECYCLE MANAGEMENT

  6. ! "

  7. FUTURE: SNAPSHOT LIFECYCLE MANAGEMENT FOR NOW ELASTIC CURATOR

  8. FEATURES & ORDER https://github.com/elastic/elasticsearch/blob/7.0/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/TimeseriesLifecycleType.java static final List<String> ORDERED_VALID_HOT_ACTIONS = Arrays.asList(

    SetPriorityAction.NAME, UnfollowAction.NAME, RolloverAction.NAME ); static final List<String> ORDERED_VALID_WARM_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, ReadOnlyAction.NAME, AllocateAction.NAME, ShrinkAction.NAME, ForceMergeAction.NAME ); static final List<String> ORDERED_VALID_COLD_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, AllocateAction.NAME, FreezeAction.NAME ); static final List<String> ORDERED_VALID_DELETE_ACTIONS = Arrays.asList( DeleteAction.NAME );
  9. DEMO

  10. PS: 7.0 IMPROVEMENT index.search.idle.after: 30s IFF DEFAULT index.refresh_interval

  11. FROZEN INDICES

  12. RATIO HEAP : STORAGE INDEX > FROZEN INDEX > CLOSED

    INDEX
  13. FROZEN INDEX READ-ONLY NO MEMORY

  14. THROTTLED THREAD POOL 1 PARALLEL SEARCH / NODE 100 IN

    QUEUE
  15. DEMO

  16. ROLLUPS

  17. None
  18. DEMO

  19. CONCLUSION

  20. CODE https://github.com/xeraa/scale-elasticsearch

  21. RECAP INDEX LIFECYCLE MANAGEMENT FROZEN INDICES ROLLUPS

  22. Questions? PHILIPP KRENN̴̴̴̴@XERAA