Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to reindex 1B documents in 1 hour?

Avatar for Qaiser Abbasi Qaiser Abbasi
December 13, 2018
Business
0
770

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/

Avatar for Qaiser Abbasi

Qaiser Abbasi

December 13, 2018
Tweet

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
Avatar for Qaiser Abbasi qabbasi
0
65
Testing-Darwinismus
Avatar for Qaiser Abbasi qabbasi
0
62

Other Decks in Business

See All in Business
スタートアップ調査：女性起業家を取り巻く課題と解決策
Avatar for MPower Partners mpower_partners
PRO
0
600
【Progmat】Monthly-ST-Market-Report-2026-Jan.
Avatar for progmat progmat
0
340
LRM株式会社 - ピッチ資料2026
Avatar for myojin@lrm.jp lrm
0
180
re:Invent2025 re:Cap 〜技術的負債解消と AWS Transform Customと わたし〜
Avatar for maijun maijun
0
140
株式会社EventHub 会社紹介資料
Avatar for EventHub eventhub
1
44k
giftee_Company introduction Febrary 2026
Avatar for giftee recruit_giftee
1
610
Akatsuki AI Technologies Company Deck
Avatar for Akatsuki AI Technologies akatsuki_ai_technologies
0
670
株式会社Gizumo_会社紹介資料（2026.1更新）
Avatar for Gizumo gizumo
0
650
enechain company deck
Avatar for enechain Recruitment Team enechain
PRO
10
160k
about-oha
Avatar for OHA oha
0
20k
【新卒向け】株式会社リブに興味のある方へ
Avatar for 株式会社リブ/LiB,Inc. libinc
0
11k
2025 サステナビリティレポート
Avatar for MPower Partners mpower_partners
PRO
1
110

Featured

See All Featured
Neural Spatial Audio Processing for Sound Field Analysis and Control
Avatar for NII S. Koyama's Lab skoyamalab
0
170
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
47
7.9k
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
330
40k
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
Avatar for Aki / @LoveIdahoBurger aki_iinuma
128
55k
My Coaching Mixtape
Avatar for mlcsv mlcsv
0
49
The Pragmatic Product Professional
Avatar for Laura Van Doore lauravandoore
37
7.1k
Avoiding the “Bad Training, Faster” Trap in the Age of AI
Avatar for Mike Taylor tmiket
0
79
YesSQL, Process and Tooling at Scale
Avatar for Rocio Delgado rocio
174
15k
How To Speak Unicorn (iThemes Webinar)
Avatar for Michelle Schulp Hunt marktimemedia
1
380
CSS Pre-Processors: Stylus, Less & Sass
Avatar for Bermon Painter bermonpainter
359
30k
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
141
7.3k
Why Your Marketing Sucks and What You Can Do About It - Sophie Logan
Avatar for Sophie Logan marketingsoph
0
77

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU [email protected] [email protected]