Snapshot/Restore in Elasticsearch v1.0

Elasticsearch-NL User Group, Amsterdam

Transcript

1. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   Snapshot/Restore
   in Elasticsearch v1.0
   Elasticsearch-NL User Group
   Amsterdam - Oct 10, 2013
   Friday, October 11, 13
   

   View Slide

2. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   About Me
   • Igor Motov
   • Developer at Elasticsearch Inc.
   • Github: imotov
   • Twitter: @imotov
   Friday, October 11, 13
   

   View Slide

3. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   Snapshot/Restore
   Friday, October 11, 13
   

   View Slide

4. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   Snapshot/Restore?
   Backup
   Friday, October 11, 13
   

   View Slide

5. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   Snapshot/Restore?
   Backup
   Rollback to
   previous
   state
   Friday, October 11, 13
   

   View Slide

6. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   Snapshot/Restore?
   Backup
   Rollback to
   previous
   state
   Index copy
   Friday, October 11, 13
   

   View Slide

7. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   Snapshot/Restore?
   Backup
   Cluster to cluster
   copy
   Rollback to
   previous
   state
   Index copy
   Friday, October 11, 13
   

   View Slide

8. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   Snapshot/Restore?
   Backup
   Friday, October 11, 13
   

   View Slide

9. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   Gateways
   Gateways
   S3
   Shared File
   System
   HDFS
   Local
   Friday, October 11, 13
   

   View Slide

10. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Gateways
    Gateways
    S3
    Shared File
    System
    HDFS
    Local
    Deprecated
    Friday, October 11, 13
    

    View Slide

11. Backup v0.90
    1. Disable flush
    2. Find all primary shard location (optional)
    3. Copy files from primary shards (rsync)
    4. Enable flush
    Friday, October 11, 13
    

    View Slide

12. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Backup v1.0
    $ curl -XPUT localhost:9200/_snapshot/my_backup/snapshot_20131010
    repository snapshot
    name
    Friday, October 11, 13
    

    View Slide

13. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Repositories
    • Snapshot Storage
    • Shared File System - v1.0
    • S3 - v1.0
    • HDFS
    • Google Compute Engine
    • Microsoft Azure
    • ...
    Friday, October 11, 13
    

    View Slide

14. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Register Repository
    $ curl -XPUT "localhost:9200/_snapshot/my_backup" -d '{
    "type": "fs",
    "settings": {
    "location":"/mnt/es-test-repo"
    }
    }'
    repository
    name
    type
    location
    Friday, October 11, 13
    

    View Slide

15. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Start Snapshot
    $ curl -XPUT "localhost:9200/_snapshot/my_backup/snapshot_20131010" -d '{
    "indices":"+test_*,-test_4"
    }'
    repository snapshot
    name
    list of indices
    (optional)
    Friday, October 11, 13
    

    View Slide

16. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Incremental Snapshot
    1
    2
    3
    4
    5
    6
    Snapshot
    2013-10-09
    Friday, October 11, 13
    

    View Slide

17. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Incremental Snapshot
    1
    2
    3
    4
    5
    6
    7
    8
    9
    Snapshot
    2013-10-09
    Friday, October 11, 13
    

    View Slide

18. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Incremental Snapshot
    1
    2
    3
    4
    5
    6
    7
    8
    9
    Snapshot
    2013-10-09
    Friday, October 11, 13
    

    View Slide

19. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Incremental Snapshot
    1
    2
    3
    4
    5
    6
    7
    8
    9
    Snapshot
    2013-10-09
    Snapshot
    2013-10-10
    Friday, October 11, 13
    

    View Slide

20. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Restore
    Friday, October 11, 13
    

    View Slide

21. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Restore Index v0.90
    1. Close the index (shutdown the cluster)
    2. Find all existing index shards
    3. Replace all index shards with data from backup
    4. Open the index (start the cluster)
    Friday, October 11, 13
    

    View Slide

22. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Restore Index v1.0
    $ curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_20131010" -d '{
    "indices":"test_*"
    }'
    $ curl -XPOST "localhost:9200/test_*/_close"
    close all indices
    that start with test_
    repository
    name
    snapshot
    name
    list of indices
    to restore
    Friday, October 11, 13
    

    View Slide

23. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Restore Index v1.0 - cont.
    $ curl -XPOST "localhost:9200/_snapshot/my_backup/my_snapshot_20131010" -d '
    {
    "indices":"test_*",
    "rename_pattern": "test_(.)+",
    "rename_replacement": "restored_$1",
    "restore_global_state": true
    }'
    Friday, October 11, 13
    

    View Slide

24. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    https://github.com/elasticsearch/elasticsearch/issues/3826
    Friday, October 11, 13
    

    View Slide

25. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Available in v1.0
    Friday, October 11, 13
    

    View Slide

26. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Questions?
    Friday, October 11, 13
    

    View Slide