Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to reindex 1B documents in 1 hour?

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Qaiser Abbasi Qaiser Abbasi
December 13, 2018
Business
0
770

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/

Avatar for Qaiser Abbasi

Qaiser Abbasi

December 13, 2018
Tweet

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
Avatar for Qaiser Abbasi qabbasi
0
65
Testing-Darwinismus
Avatar for Qaiser Abbasi qabbasi
0
62

Other Decks in Business

See All in Business
RDRAモデルからFP・工数・金額につなぐ定量見積り
Avatar for BPStudy bpstudy
1
210
Nulab Fun Deck 〜チームワークが、世界をもっと『おもしろく』する〜
Avatar for 株式会社ヌーラボ nulabinc
PRO
1
2.8k
セーフィー株式会社（Safie Inc.） 会社紹介資料
Avatar for Safie safie_recruit
7
410k
Eco-Pork Impact Report 2026.02.09 EN
Avatar for Eco-Pork ecopork
0
280
ネクストビート 新卒向け会社紹介資料
Avatar for nextbeat nextbeat
1
520
株式会社CINC 会社案内/Company introduction
Avatar for M.Mitsuo cinchr
6
74k
RDRAで価値を可視化する
Avatar for 神崎 kanzaki
2
370
MEEM_Company_Deck202512.pdf
Avatar for meem info_meem
0
3.9k
ノッカリアドベントカレンダー全記録まとめ
Avatar for すずくま szkm555
0
130
-生きる-AI時代におけるライターの生存戦略
Avatar for Mimu mimuhayashi
0
240
Mercari-Fact-book_en
Avatar for Mercari, Inc. mercari_inc
2
32k
NewsPicks Expert説明資料 / NewsPicks Expert Introduction
Avatar for NewsPicks Expert mimir
0
22k

Featured

See All Featured
What does AI have to do with Human Rights?
Avatar for Per Axbom axbom
PRO
0
2k
Skip the Path - Find Your Career Trail
Avatar for Mark Kilby mkilby
0
59
Six Lessons from altMBA
Avatar for Skipper Chong Warson skipperchong
29
4.2k
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
74
11k
How to build a perfect <img>
Avatar for Jono Alderson jonoalderson
1
4.9k
ラッコキーワード サービス紹介資料
Avatar for ラッコ株式会社 rakko
1
2.3M
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
Avatar for Dan Newman danielanewman
231
22k
Fantastic passwords and where to find them - at NoRuKo
Avatar for Phil Nash philnash
52
3.6k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
133
19k
Un-Boring Meetings
Avatar for Sebastian Deterding codingconduct
0
200
16th Malabo Montpellier Forum Presentation
Avatar for AKADEMIYA2063 akademiya2063
PRO
0
52
Stop Working from a Prison Cell
Avatar for Chris Michel hatefulcrawdad
273
21k

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU [email protected] [email protected]