Scale Your Elasticsearch Cluster

Elasticsearch is highly scalable, but some recent additions make it easier to tie everything together the right way. In this talk we discuss:
* Frozen Indices for much higher storage density per Elasticsearch node and its tradeoffs.
* Index Lifecycle Management or how to automate having a multi-tier architecture and how this actually works in the background.
* Rollups or how to aggregate metrics over time.

The goal of this talk is to make your cluster management less complex and more cost effective.

Transcript

1. SCALE
   YOUR CLUSTER
   PHILIPP KRENN̴̴̴̴@XERAA
   

   View Slide

2. DEVELOPER
   

   View Slide

3. AGENDA
   INDEX LIFECYCLE MANAGEMENT
   FROZEN INDICES
   ROLLUPS
   

   View Slide

4. START DEMO
   SETUP
   

   View Slide

5. INDEX LIFECYCLE
   MANAGEMENT
   

   View Slide

6. ! "
   

   View Slide

7. FUTURE: SNAPSHOT
   LIFECYCLE MANAGEMENT
   FOR NOW ELASTIC CURATOR
   

   View Slide

8. FEATURES & ORDER
   https://github.com/elastic/elasticsearch/blob/7.0/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/TimeseriesLifecycleType.java
   static final List ORDERED_VALID_HOT_ACTIONS = Arrays.asList(
   SetPriorityAction.NAME, UnfollowAction.NAME, RolloverAction.NAME
   );
   static final List ORDERED_VALID_WARM_ACTIONS = Arrays.asList(
   SetPriorityAction.NAME, UnfollowAction.NAME, ReadOnlyAction.NAME,
   AllocateAction.NAME, ShrinkAction.NAME, ForceMergeAction.NAME
   );
   static final List ORDERED_VALID_COLD_ACTIONS = Arrays.asList(
   SetPriorityAction.NAME, UnfollowAction.NAME, AllocateAction.NAME, FreezeAction.NAME
   );
   static final List ORDERED_VALID_DELETE_ACTIONS = Arrays.asList(
   DeleteAction.NAME
   );
   

   View Slide

9. DEMO
   

   View Slide

10. PS: 7.0 IMPROVEMENT
    index.search.idle.after: 30s
    IFF DEFAULT index.refresh_interval
    

    View Slide

11. FROZEN INDICES
    

    View Slide

12. RATIO HEAP : STORAGE
    INDEX > FROZEN INDEX > CLOSED INDEX
    

    View Slide

13. FROZEN INDEX
    READ-ONLY
    NO MEMORY
    

    View Slide

14. THROTTLED THREAD POOL
    1 PARALLEL SEARCH / NODE
    100 IN QUEUE
    

    View Slide

15. DEMO
    

    View Slide

16. ROLLUPS
    

    View Slide

17. View Slide

18. DEMO
    

    View Slide

19. CONCLUSION
    

    View Slide

20. CODE
    https://github.com/xeraa/scale-elasticsearch
    

    View Slide

21. RECAP
    INDEX LIFECYCLE MANAGEMENT
    FROZEN INDICES
    ROLLUPS
    

    View Slide

22. Questions?
    PHILIPP KRENN̴̴̴̴@XERAA
    

    View Slide