Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to reindex 1B documents in 1 hour?

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Qaiser Abbasi Qaiser Abbasi
December 13, 2018
Business
0
780

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/

Avatar for Qaiser Abbasi

Qaiser Abbasi

December 13, 2018
Tweet

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
Avatar for Qaiser Abbasi qabbasi
0
66
Testing-Darwinismus
Avatar for Qaiser Abbasi qabbasi
0
64

Other Decks in Business

See All in Business
採用ピッチ資料
Avatar for 株式会社メイクスビュー kasamatsu123
0
220
「きっかけ作り」から始めるKiro定着の軌跡
Avatar for itsme iamme
0
200
DATUM STUDIO - 会社紹介資料
Avatar for DATUM STUDIO Co. Ltd. datumstudio
0
1.8k
2025年度ICT職専門研修（海外派遣研修）報告書 No.1
Avatar for 東京都デジタル人材確保・育成 tokyo_metropolitan_gov_digital_hr
0
300
人々にとってかけがえのないプロダクトを作るには ～顧客の日常に紛れる "not not" を見つけろ！～ #pdmyy
Avatar for Takeo Imai bonotake
2
220
（41枚）目標管理の全スキル 目標の立て方・課題の設定の仕方・計画の立て方・仕組みの作り方・進捗管理のやり方等すべてを解説
Avatar for 横山信弘／絶対達成スライド館 nyattx
PRO
2
820
Ambientnavi Company Deck
Avatar for Ambientnavi ambientnavi0329
0
130
動機は不純、だがそれがいい
Avatar for Naoki Iwanaga newrice
0
270
2025年度ICT職専門研修（海外派遣研修）報告書 No.4
Avatar for 東京都デジタル人材確保・育成 tokyo_metropolitan_gov_digital_hr
0
310
透明性レポート（2025年下半期）
Avatar for Mercari, Inc. mercari_inc
0
5.3k
セーフィー株式会社（Safie Inc.） 会社紹介資料
Avatar for Safie safie_recruit
7
420k
特定領域から複数領域へ、そのとき何を求められるのか？縦と横、2つの影響力：統合型を目指す大規模な開発組織での実践
Avatar for keitatomozawa keitatomozawa
3
510

Featured

See All Featured
Reality Check: Gamification 10 Years Later
Avatar for Sebastian Deterding codingconduct
0
2.1k
Getting science done with accelerated Python computing platforms
Avatar for Jacob Tomlinson jacobtomlinson
2
150
Avoiding the “Bad Training, Faster” Trap in the Age of AI
Avatar for Mike Taylor tmiket
0
110
Test your architecture with Archunit
Avatar for Yoan thirion
1
2.2k
Into the Great Unknown - MozCon
Avatar for Noah Learner thekraken
40
2.3k
The B2B funnel & how to create a winning content strategy
Avatar for Katarina Dahlin katarinadahlin
PRO
1
310
Un-Boring Meetings
Avatar for Sebastian Deterding codingconduct
0
240
Why You Should Never Use an ORM
Avatar for John Nunemaker jnunemaker
PRO
61
9.8k
The Art of Delivering Value - GDevCon NA Keynote
Avatar for David Neal reverentgeek
16
1.9k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
Avatar for mongodb mongodb
49
9.9k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
38
2.8k
Sharpening the Axe: The Primacy of Toolmaking
Avatar for Bryan Cantrill bcantrill
46
2.7k

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU [email protected] [email protected]