Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to reindex 1B documents in 1 hour?

Qaiser Abbasi
December 13, 2018
Business
0
490

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/

Qaiser Abbasi

December 13, 2018
Tweet

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
qabbasi
0
35
Testing-Darwinismus
qabbasi
0
37

Other Decks in Business

See All in Business
InfoShell
infoshell_company
0
160
SPOT 虎ノ門 利用ガイド
holder_spot
0
7.2k
株式会社ラクミー採用資料
itokenv
0
130
スクラムPJを通して気づいた当たり前で大切なこと
mihofujita
0
310
Company as a Product
koic
0
180
220609_Flyle運用を組織実装し、ポジティブフィードバックが回るように頑張った話
sugihara_tatsuya
0
280
CrossLog Introduction
tomoshin
1
480
Yondemy-Recruiting Deck
yondemy
0
340
Tangerine採用ピッチ資料/We are hiring
3yasuda
4
2.3k
company-introduction
balmuda
0
150
オシロ株式会社の会社紹介_スライド_ver.6.0.pdf
nally_osiro
1
3.7k
DIGGLE紹介資料
diggle_hr
1
5.1k

Featured

See All Featured
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
182
15k
How STYLIGHT went responsive
nonsquared
85
4k
Docker and Python
trallard
27
1.7k
Fontdeck: Realign not Redesign
paulrobertlloyd
73
4.2k
Writing Fast Ruby
sferik
612
57k
How GitHub (no longer) Works
holman
298
140k
Become a Pro
speakerdeck
PRO
3
2.3k
Build The Right Thing And Hit Your Dates
maggiecrowley
21
1.3k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
21
1.5k
How To Stay Up To Date on Web Technology
chriscoyier
780
250k
Thoughts on Productivity
jonyablonski
45
2.5k
Making Projects Easy
brettharned
100
4.4k

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU rene.treffer@soundcloud.com qaiser.abbasi@soundcloud.com