Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

How to reindex 1B documents in 1 hour?

Avatar for Qaiser Abbasi Qaiser Abbasi
December 13, 2018
Business
840
0

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/

Avatar for Qaiser Abbasi

Qaiser Abbasi

December 13, 2018

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
Avatar for Qaiser Abbasi qabbasi
0
72
Testing-Darwinismus
Avatar for Qaiser Abbasi qabbasi
0
81

Other Decks in Business

See All in Business
三井物産グループのデジタル証券～福岡・天神～徹底解説セミナー
Avatar for 三井物産デジタル・アセットマネジメント株式会社 c0rp_mdm
PRO
0
1.3k
増え続ける運用を、 Skill化する"型"でさばく
Avatar for mori morimori3
0
350
セーフィー株式会社（Safie Inc.） 会社紹介資料
Avatar for Safie safie_recruit
8
470k
LW_brochure_business
Avatar for linc-well lincwellhr
1
92k
[2026.09] カルチャーデック｜株式会社サイバーリンクス
Avatar for 株式会社サイバーリンクス cyberlinks
0
350
ダイサンデジタル事業部
Avatar for 北本絵美 daisan123
0
800
社外CTOパートナー(2026.09版) - ORDINARY LLC
Avatar for 西山修 - ORDINARY LLC ordy_nishiyama
0
120
正解が消えた開発現場で、どう後輩を育てればいいのか？~ AI時代における「若手のストレス」と「メンターの限界」に向き合う ~
Avatar for みきてぃ / きたはら (mkitahara) mkitahara01985
1
140
株式会社うるる エンジニア向け採用資料
Avatar for ULURU-HR uluru_hr
3
140k
なんとかする力 〜 Android エンジニアからマネージャー、さらにその先へ？ 〜
Avatar for m.coder m_coder
2
510
Webinar 23.09.2026 NeuroDate III - El impacto de la guerra de Irán en los precios del gas y la electricidad
Avatar for Neuroenergía neuroenergia
PRO
0
110
株式会社アシスト_会社紹介資料
Avatar for ashisuto_career ashisuto_career
3
190k

Featured

See All Featured
Efficient Content Optimization with Google Search Console & Apps Script
Avatar for Katarina Dahlin katarinadahlin
PRO
1
890
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
55
8.3k
CSS Pre-Processors: Stylus, Less & Sass
Avatar for Bermon Painter bermonpainter
360
31k
The untapped power of vector embeddings
Avatar for Frank van Dijk frankvandijk
2
1.9k
Data-driven link building: lessons from a $708K investment (BrightonSEO talk)
Avatar for Szymon Słowik szymonslowik
1
1.3k
How Software Deployment tools have changed in the past 20 years
Avatar for Geshan Manandhar geshan
2
34k
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
13
1.5k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
331
25k
Breaking role norms: Why Content Design is so much more than writing copy - Taylor Woolridge
Avatar for UX Y'all uxyall
1
430
Redefining SEO in the New Era of Traffic Generation
Avatar for Szymon Słowik szymonslowik
1
450
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
39
3.5k
Building an army of robots
Avatar for Kyle Neath kneath
307
47k

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU [email protected] [email protected]