Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to reindex 1B documents in 1 hour?

Efa70d43d8e57623d0d5fafaeef1109f?s=47 Qaiser Abbasi
December 13, 2018
Business
0
350

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/

Efa70d43d8e57623d0d5fafaeef1109f?s=128

Qaiser Abbasi

December 13, 2018
Tweet

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Efa70d43d8e57623d0d5fafaeef1109f?s=48 qabbasi
0
31
Efa70d43d8e57623d0d5fafaeef1109f?s=48 qabbasi
0
29

Other Decks in Business

See All in Business
532348a1aba5a909e283624c3d2a9a95?s=48 shift_evolve
0
210
0d35336bfe58d7ed5024591bb5deb68c?s=48 projectofficeguu
0
680
Bfc5999de4cceded50d98ee54781db08?s=48 arm0020
0
10k
2ce86e6d959967388c454e179c7a7091?s=48 leaner_tech
7
35k
D64faebdac14f88d4b6e541a3198b434?s=48 iwacci
2
990
3852b63d665b7e5f78337bfcf7f4e045?s=48 astarnetwork
0
160
Ea7f7af2b8f62b7abf8cf0c9af394cc0?s=48 applibot
0
3k
77ed591988bb10b2ee576ce296b4af7a?s=48 nakatown006
0
340
944a10f2c7ba844b2bfc7b7f624b204c?s=48 lwug
0
140
45c8b5292f6615b315f080b87c7b29b6?s=48 enechain
0
3.9k
93a4fb6dacd2519838dbed9f8cb7bad3?s=48 megumiaow
0
200
A645e7c3a6635ead8fa323c583769591?s=48 hololab
0
1.3k

Featured

See All Featured
5b9fe87ec1faa67a4599782930f45ec9?s=48 sstephenson
149
12k
Dad095ea7038f89f760419ce475d5d14?s=48 michaelherold
227
8.8k
2bb923c57a1fdc3a2eb484bb8d565fd2?s=48 ufuk
60
7.1k
61857dafbd287b3027c4dcea9008ad3c?s=48 carmenhchung
40
1.9k
B3d6434763caa0ef5dc4b792662c49f7?s=48 smashingmag
235
18k
44b54d2ffcb7857004071cc6795dde11?s=48 cassininazir
349
20k
433acaea1012b25d97ae66da9b998dad?s=48 malarkey
121
16k
027d1ebf66cc039a0bd3b55eeadbe75d?s=48 sachag
269
17k
D8fc2580cfaca035f666d9e4ee79a7f7?s=48 mongodb
24
4k
5f50f94346fbcb23706f0707169317a4?s=48 aarron
261
36k
462dad0dd24c568a32dcdb0ae895ae1b?s=48 rasmusluckow
318
19k
20bfe76b3d6105641f879fe45cfc9272?s=48 bkeepers
PRO
55
4.6k

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU rene.treffer@soundcloud.com qaiser.abbasi@soundcloud.com