Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to reindex 1B documents in 1 hour?

Avatar for Qaiser Abbasi Qaiser Abbasi
December 13, 2018
Business
0
770

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/

Avatar for Qaiser Abbasi

Qaiser Abbasi

December 13, 2018
Tweet

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
Avatar for Qaiser Abbasi qabbasi
0
65
Testing-Darwinismus
Avatar for Qaiser Abbasi qabbasi
0
62

Other Decks in Business

See All in Business
【SRE Kaigi 2026】認知負荷を最小化するオブザーバビリティとSLOの導入 ―4名SREが200名のコードエンジニアを支援
Avatar for HIGUCHI.Takashi higuchi_takashi
2
1.3k
株式会社CINC 会社案内/Company introduction
Avatar for M.Mitsuo cinchr
6
74k
malna-recruiting-pitch
Avatar for malna malna
0
14k
LW_brochure_engineer
Avatar for linc-well lincwellhr
0
40k
Eco-Pork Impact Report 2026.02.09 EN
Avatar for Eco-Pork ecopork
0
280
40代データ人材のキャリア戦略
Avatar for Jun Ernesto Okumura pacocat
4
4k
Startup Research : Challenges and solutions for female startup founders in Japan
Avatar for MPower Partners mpower_partners
PRO
0
290
アットウェア 会社説明資料
Avatar for atware atware
0
14k
MEEM_Company_Deck202512.pdf
Avatar for meem info_meem
0
3.9k
【Progmat】Monthly-ST-Market-Report-2026-Jan.
Avatar for progmat progmat
0
340
株式会社Oxxx Culture Deck
Avatar for 株式会社Oxxx oxxxinc
0
690
VISASQ: ABOUT DEV TEAM
Avatar for VISASQ eikohashiba
6
41k

Featured

See All Featured
Primal Persuasion: How to Engage the Brain for Learning That Lasts
Avatar for Mike Taylor tmiket
0
260
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
59
6.3k
How to Ace a Technical Interview
Avatar for Jacob Kaplan-Moss jacobian
281
24k
Winning Ecommerce Organic Search in an AI Era - #searchnstuff2025
Avatar for Aleyda Solis aleyda
1
1.9k
Navigating Team Friction
Avatar for Lara Hogan lara
192
16k
Technical Leadership for Architectural Decision Making
Avatar for Kenny Baas-Schwegler baasie
2
250
We Analyzed 250 Million AI Search Results: Here's What I Found
Avatar for Josh Blyskal joshbly
1
750
Imperfection Machines: The Place of Print at Facebook
Avatar for Scott Boms scottboms
269
14k
Taking LLMs out of the black box: A practical guide to human-in-the-loop distillation
Avatar for Ines Montani inesmontani
PRO
3
2k
SEO Brein meetup: CTRL+C is not how to scale international SEO
Avatar for Linda Hogenes lindahogenes
0
2.4k
DBのスキルで生き残る技術 - AI時代におけるテーブル設計の勘所
Avatar for soudai sone soudai
PRO
62
50k
Fireside Chat
Avatar for paigeccino paigeccino
41
3.8k

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU [email protected] [email protected]