Snapshot/Restore in Elasticsearch 1.0

Lightning talk covering the Snapshot/Restore functionality forthcoming in the Elasticsearch 1.0 release.

Transcript

1. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Snapshot/Restore in Elasticsearch v1.0 Elasticsearch-NL User Group Amsterdam - Oct 10, 2013 Friday, October 11, 13
2. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited About Me • Igor Motov • Developer at Elasticsearch Inc. • Github: imotov • Twitter: @imotov Friday, October 11, 13
3. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Snapshot/Restore Friday, October 11, 13
4. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Snapshot/Restore? Backup Friday, October 11, 13
5. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Snapshot/Restore? Backup Rollback to previous state Friday, October 11, 13
6. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Snapshot/Restore? Backup Rollback to previous state Index copy Friday, October 11, 13
7. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Snapshot/Restore? Backup Cluster to cluster copy Rollback to previous state Index copy Friday, October 11, 13
8. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Snapshot/Restore? Backup Friday, October 11, 13
9. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Gateways Gateways S3 Shared File System HDFS Local Friday, October 11, 13
10. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Gateways Gateways S3 Shared File System HDFS Local Deprecated Friday, October 11, 13
11. Backup v0.90 1. Disable flush 2. Find all primary shard location (optional) 3. Copy files from primary shards (rsync) 4. Enable flush Friday, October 11, 13
12. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Backup v1.0 $ curl -XPUT localhost:9200/_snapshot/my_backup/snapshot_20131010 repository snapshot name Friday, October 11, 13
13. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Repositories • Snapshot Storage • Shared File System - v1.0 • S3 - v1.0 • HDFS • Google Compute Engine • Microsoft Azure • ... Friday, October 11, 13
14. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Register Repository $ curl -XPUT "localhost:9200/_snapshot/my_backup" -d '{ "type": "fs", "settings": { "location":"/mnt/es-test-repo" } }' repository name type location Friday, October 11, 13
15. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Start Snapshot $ curl -XPUT "localhost:9200/_snapshot/my_backup/snapshot_20131010" -d '{ "indices":"+test_*,-test_4" }' repository snapshot name list of indices (optional) Friday, October 11, 13
16. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Incremental Snapshot 1 2 3 4 5 6 Snapshot 2013-10-09 Friday, October 11, 13
17. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Incremental Snapshot 1 2 3 4 5 6 7 8 9 Snapshot 2013-10-09 Friday, October 11, 13
18. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Incremental Snapshot 1 2 3 4 5 6 7 8 9 Snapshot 2013-10-09 Friday, October 11, 13
19. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Incremental Snapshot 1 2 3 4 5 6 7 8 9 Snapshot 2013-10-09 Snapshot 2013-10-10 Friday, October 11, 13
20. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Restore Friday, October 11, 13
21. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Restore Index v0.90 1. Close the index (shutdown the cluster) 2. Find all existing index shards 3. Replace all index shards with data from backup 4. Open the index (start the cluster) Friday, October 11, 13
22. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Restore Index v1.0 $ curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_20131010" -d '{ "indices":"test_*" }' $ curl -XPOST "localhost:9200/test_*/_close" close all indices that start with test_ repository name snapshot name list of indices to restore Friday, October 11, 13
23. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Restore Index v1.0 - cont. $ curl -XPOST "localhost:9200/_snapshot/my_backup/my_snapshot_20131010" -d ' { "indices":"test_*", "rename_pattern": "test_(.)+", "rename_replacement": "restored_$1", "restore_global_state": true }' Friday, October 11, 13
24. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited https://github.com/elasticsearch/elasticsearch/issues/3826 Friday, October 11, 13
25. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Available in v1.0 Friday, October 11, 13
26. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Questions? Friday, October 11, 13