Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to reindex 1B documents in 1 hour?

Avatar for Qaiser Abbasi Qaiser Abbasi
December 13, 2018
Business
0
780

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/

Avatar for Qaiser Abbasi

Qaiser Abbasi

December 13, 2018
Tweet

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
Avatar for Qaiser Abbasi qabbasi
0
66
Testing-Darwinismus
Avatar for Qaiser Abbasi qabbasi
0
63

Other Decks in Business

See All in Business
Mercari-Fact-book_en
Avatar for Mercari, Inc. mercari_inc
2
33k
イークラウド会社紹介
Avatar for イークラウド株式会社 ecrowdinc
0
220
インキュデータ会社紹介資料
Avatar for Recruit@INCUDATA okitsu
3
51k
株式会社SAFELY 会社紹介 / Company
Avatar for 株式会社SAFELY safely_pr
1
5.8k
株式会社IDOM_FACT BOOK 2026
Avatar for 株式会社IDOM idompr
0
240
enechain company deck_english
Avatar for enechain Recruitment Team enechain
PRO
0
290
【スライド150枚】優秀層獲得のための新卒採用マニュアル
Avatar for 袴田 優斗 yuto_hakamada
0
200
「自我を出さなかった」私がアジャイルに出会って─冷笑を捨て、自分の人生を「経験主義」で動かした話
Avatar for Kaedeyamazaki kaedeyamazaki0820
1
410
イオンモール新利府・デジタル証券 ～仙台近郊～徹底解説セミナー
Avatar for 三井物産デジタル・アセットマネジメント株式会社 c0rp_mdm
PRO
0
1.9k
about-oha
Avatar for OHA oha
0
20k
ARグラスが当たり前になったら、デザインってどう変わる？Spectrum Tokyo Festival 2026 #spectrumfest26
Avatar for arisan arisan
0
140
透明性レポート（2025年下半期）
Avatar for Mercari, Inc. mercari_inc
0
590

Featured

See All Featured
Leveraging LLMs for student feedback in introductory data science courses - posit::conf(2025)
Avatar for Mine Cetinkaya-Rundel minecr
1
190
So, you think you're a good person
Avatar for Per Axbom axbom
PRO
2
1.9k
Learning to Love Humans: Emotional Interface Design
Avatar for Aarron Walter aarron
275
41k
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
13
1.1k
How to train your dragon (web standard)
Avatar for Monica Dinculescu notwaldorf
97
6.5k
AI Search:
Where Are We & What Can We Do About It?
Avatar for Aleyda Solis aleyda
0
7.1k
HDC tutorial
Avatar for Michiel Stock michielstock
1
500
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
135
9.8k
Bridging the Design Gap: How Collaborative Modelling removes blockers to flow between stakeholders and teams @FastFlow conf
Avatar for Kenny Baas-Schwegler baasie
0
470
Applied NLP in the Age of Generative AI
Avatar for Ines Montani inesmontani
PRO
4
2.1k
AI Search: Implications for SEO and How to Move Forward - #ShenzhenSEOConference
Avatar for Aleyda Solis aleyda
1
1.1k
It's Worth the Effort
Avatar for 3n 3n
188
29k

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU [email protected] [email protected]