Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to reindex 1B documents in 1 hour?

Avatar for Qaiser Abbasi Qaiser Abbasi
December 13, 2018
Business
0
760

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/

Avatar for Qaiser Abbasi

Qaiser Abbasi

December 13, 2018
Tweet

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
Avatar for Qaiser Abbasi qabbasi
0
64
Testing-Darwinismus
Avatar for Qaiser Abbasi qabbasi
0
60

Other Decks in Business

See All in Business
ナウビレッジ株式会社_会社紹介資料_20250821
Avatar for NOWVILLAGE nowvill
0
12k
営業職／新卒向け会社紹介資料（テックファーム株式会社）
Avatar for Techfirm techfirm
1
930
Corporate Story (GA technologies Co., Ltd.)
Avatar for GA technologies gatechnologies
0
170
Fracta Leap 会社紹介資料
Avatar for Fracta Leap 株式会社 fracta_leap
PRO
0
110
CREによる顧客のキャッチアップを加速する仕組み作り / Creating a mechanism to accelerate customer catch-up through CRE
Avatar for Koda Shuhei woody_kawagoe
1
250
定義のない仕事 / Undefined Work
Avatar for nrs nrslib
9
4.1k
物流の専門家がお客様に伴走するサブスク型コンサルティング
Avatar for MCLOGI mclogi
0
380
20250816 「アジャイル」って？～"Do Agile"から"Be Agile"へ～
Avatar for Takumi Abe east_takumi
0
3k
家族アルバム みてね 事業紹介 / Our Business
Avatar for FamilyAlbum familyalbum
6
46k
【テックファームホールディングス】中途採用向け会社説明資料
Avatar for Techfirm techfirm
0
410
ブラインドスクエア＆キーパンチ
Avatar for HEART QUAKE Inc chibanba1982
PRO
0
280
2011年 「ぼっちが懇親会でするべき97のこと」 #97bocchi から積み重ねた【令和7年最新】技術コミュニティ交流戦略2025
Avatar for bash0C7 bash0c7
0
480

Featured

See All Featured
The Pragmatic Product Professional
Avatar for Laura Van Doore lauravandoore
36
6.9k
Designing for Performance
Avatar for Lara Hogan lara
610
69k
Practical Orchestrator
Avatar for Shlomi Noach shlominoach
190
11k
Designing Experiences People Love
Avatar for Jonathan Moore moore
142
24k
StorybookのUI Testing Handbookを読んだ
Avatar for Shingo Yamazaki zakiyama
31
6.1k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1370
200k
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
37
2.8k
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
72
11k
Rails Girls Zürich Keynote
Avatar for Gregor Martynus gr2m
95
14k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
Avatar for Chris Coyier chriscoyier
507
140k
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
135
9.5k
Art, The Web, and Tiny UX
Avatar for Lynn Fisher lynnandtonic
302
21k

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU [email protected] [email protected]