Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scale Your Elasticsearch Cluster

Scale Your Elasticsearch Cluster

Elasticsearch is highly scalable, but some recent additions make it easier to tie everything together the right way. In this talk we discuss:
* Frozen Indices for much higher storage density per Elasticsearch node and its tradeoffs.
* Index Lifecycle Management or how to automate having a multi-tier architecture and how this actually works in the background.
* Rollups or how to aggregate metrics over time.

The goal of this talk is to make your cluster management less complex and more cost effective.

Philipp Krenn

April 29, 2019
Tweet

More Decks by Philipp Krenn

Other Decks in Programming

Transcript

  1. SCALE
    YOUR CLUSTER
    PHILIPP KRENN̴̴̴̴@XERAA

    View full-size slide

  2. AGENDA
    INDEX LIFECYCLE MANAGEMENT
    FROZEN INDICES
    ROLLUPS

    View full-size slide

  3. START DEMO
    SETUP

    View full-size slide

  4. INDEX LIFECYCLE
    MANAGEMENT

    View full-size slide

  5. FUTURE: SNAPSHOT
    LIFECYCLE MANAGEMENT
    FOR NOW ELASTIC CURATOR

    View full-size slide

  6. FEATURES & ORDER
    https://github.com/elastic/elasticsearch/blob/7.0/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/TimeseriesLifecycleType.java
    static final List ORDERED_VALID_HOT_ACTIONS = Arrays.asList(
    SetPriorityAction.NAME, UnfollowAction.NAME, RolloverAction.NAME
    );
    static final List ORDERED_VALID_WARM_ACTIONS = Arrays.asList(
    SetPriorityAction.NAME, UnfollowAction.NAME, ReadOnlyAction.NAME,
    AllocateAction.NAME, ShrinkAction.NAME, ForceMergeAction.NAME
    );
    static final List ORDERED_VALID_COLD_ACTIONS = Arrays.asList(
    SetPriorityAction.NAME, UnfollowAction.NAME, AllocateAction.NAME, FreezeAction.NAME
    );
    static final List ORDERED_VALID_DELETE_ACTIONS = Arrays.asList(
    DeleteAction.NAME
    );

    View full-size slide

  7. PS: 7.0 IMPROVEMENT
    index.search.idle.after: 30s
    IFF DEFAULT index.refresh_interval

    View full-size slide

  8. FROZEN INDICES

    View full-size slide

  9. RATIO HEAP : STORAGE
    INDEX > FROZEN INDEX > CLOSED INDEX

    View full-size slide

  10. FROZEN INDEX
    READ-ONLY
    NO MEMORY

    View full-size slide

  11. THROTTLED THREAD POOL
    1 PARALLEL SEARCH / NODE
    100 IN QUEUE

    View full-size slide

  12. CODE
    https://github.com/xeraa/scale-elasticsearch

    View full-size slide

  13. RECAP
    INDEX LIFECYCLE MANAGEMENT
    FROZEN INDICES
    ROLLUPS

    View full-size slide

  14. Questions?
    PHILIPP KRENN̴̴̴̴@XERAA

    View full-size slide