Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

How to reindex 1B documents in 1 hour?

Avatar for Qaiser Abbasi Qaiser Abbasi
December 13, 2018
Business
0
770

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/

Avatar for Qaiser Abbasi

Qaiser Abbasi

December 13, 2018
Tweet

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
Avatar for Qaiser Abbasi qabbasi
0
65
Testing-Darwinismus
Avatar for Qaiser Abbasi qabbasi
0
62

Other Decks in Business

See All in Business
[NGA] カンパニーデック202511Ver.
Avatar for NGA Ltd. ngaltd
PRO
1
700
～モブ、まだいけるよな？～2025年をふりかえってみて_20251126
Avatar for モブエンジニア（Masaki Okuda） masakiokuda
0
150
【DearOne】Dear Newest Member
Avatar for DearOne hrm
2
14k
セーフィー株式会社（Safie Inc.） 会社紹介資料
Avatar for Safie safie_recruit
6
400k
TORICO Ethereum_companydeck_20251217
Avatar for 株式会社TORICO torico
0
650
曖昧なLLMの出力をプロダクト価値へつなげる、要求の具体化と評価
Avatar for Higuchi kokoro zerebom
4
410
インキュデータ会社紹介資料
Avatar for Recruit@INCUDATA okitsu
3
47k
強みのデザイン入門 / Introduction to Strengths Design
Avatar for tbpgr tbpgr
0
400
不感対策ソリューション 詳細資料
Avatar for 日本通信エンジニアリングサービス株式会社 jtes
0
360
pmconf2025_-_現役教師のたこ焼き屋さん___現役PMの駄菓子屋さんが未来に挑む___ユーザーコミュニティ主導のプロダクトマネジメント_.pdf
Avatar for NAGASHIMA Hiroki mindman
0
2.7k
週4社員しながら個人開発にベットする / Betting on Personal Projects While Working a Four-Day Week
Avatar for kohii kohii00
4
2.9k
『業務設計の教科書』の概要
Avatar for Shunsuke Takeuchi shunsuke_takeuchi
PRO
3
7.3k

Featured

See All Featured
Imperfection Machines: The Place of Print at Facebook
Avatar for Scott Boms scottboms
269
13k
From Legacy to Launchpad: Building Startup-Ready Communities
Avatar for Dug Song dugsong
0
110
Writing Fast Ruby
Avatar for Erik Berlin sferik
630
62k
Design in an AI World
Avatar for tapps tapps
0
89
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
37
2.7k
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
Avatar for Aki / @LoveIdahoBurger aki_iinuma
128
54k
Designing for Performance
Avatar for Lara Hogan lara
610
69k
Easily Structure & Communicate Ideas using Wireframe
Avatar for Afnizar Nur Ghifari afnizarnur
194
17k
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
463
34k
Context Engineering - Making Every Token Count
Avatar for Addy Osmani addyosmani
9
540
Scaling GitHub
Avatar for Zach Holman holman
464
140k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
Avatar for Addy Osmani addyosmani
9
1k

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU [email protected] [email protected]