Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to reindex 1B documents in 1 hour?

Avatar for Qaiser Abbasi Qaiser Abbasi
December 13, 2018
Business
780
0

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/

Avatar for Qaiser Abbasi

Qaiser Abbasi

December 13, 2018

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
Avatar for Qaiser Abbasi qabbasi
0
66
Testing-Darwinismus
Avatar for Qaiser Abbasi qabbasi
0
66

Other Decks in Business

See All in Business
LW_brochure_business
Avatar for linc-well lincwellhr
1
80k
白金鉱業meetup発表資料
Avatar for tetsuroito tetsuroito
1
110
営業職／新卒向け会社紹介資料（テックファーム株式会社）
Avatar for Techfirm techfirm
1
1.5k
Company Profile
Avatar for Katsuhiko Eguchi katsuegu23
2
14k
介護休業ガイドブック（スパイダープラス）
Avatar for スパイダープラス株式会社 spiderplus_cb
0
290
採用ピッチデック
Avatar for M&A Cloud macloud
4
87k
Palette Cloud Company Deck
Avatar for PaletteCloud palettecloud
0
12k
インキュデータ会社紹介資料
Avatar for Recruit@INCUDATA okitsu
3
53k
ele&company_companydeck
Avatar for eleand eleand
0
3.5k
enechain company deck_english
Avatar for enechain Recruitment Team enechain
PRO
1
370
20260401_UPDATER_companysummary
Avatar for UPDATER PR updater_pr
0
120k
（48枚）絶対達成新入社員研修の特徴と2つのスタイル（リアル研修とオンライン教材）
Avatar for 横山信弘／絶対達成スライド館 nyattx
PRO
0
190

Featured

See All Featured
Leveraging Curiosity to Care for An Aging Population
Avatar for Cassini Nazir cassininazir
1
220
Paper Plane
Avatar for Katie Cordina Lindsey katiecoart
PRO
1
49k
We Are The Robots
Avatar for Honza Javorek honzajavorek
0
210
Redefining SEO in the New Era of Traffic Generation
Avatar for Szymon Słowik szymonslowik
1
280
世界の人気アプリ100個を分析して見えたペイウォール設計の心得
Avatar for Akihiro Kokubo akihiro_kokubo
PRO
69
38k
SERP Conf. Vienna - Web Accessibility: Optimizing for Inclusivity and SEO
Avatar for Sara Fernández Carmona sarafernandez
2
1.4k
How People are Using Generative and Agentic AI to Supercharge Their Products, Projects, Services and Value Streams Today
Avatar for Helen Beal helenjbeal
1
160
SEO in 2025: How to Prepare for the Future of Search
Avatar for Michael King ipullrank
3
3.4k
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
51
52k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1370
200k
Writing Fast Ruby
Avatar for Erik Berlin sferik
630
63k
Documentation Writing (for coders)
Avatar for Carmen Chung carmenintech
77
5.3k

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU [email protected] [email protected]