Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

How to reindex 1B documents in 1 hour?

Avatar for Qaiser Abbasi Qaiser Abbasi
December 13, 2018
Business
0
770

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/

Avatar for Qaiser Abbasi

Qaiser Abbasi

December 13, 2018
Tweet

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
Avatar for Qaiser Abbasi qabbasi
0
65
Testing-Darwinismus
Avatar for Qaiser Abbasi qabbasi
0
62

Other Decks in Business

See All in Business
明和不動産会社概要
Avatar for 明和不動産 prkoho
0
3.9k
業務設計のいろは
Avatar for Shunsuke Takeuchi shunsuke_takeuchi
PRO
2
510
プロダクト本部カジュアル面談資料
Avatar for 10xinc 10xinc
0
140
ログラス会社紹介資料 / Loglass Company Deck
Avatar for 株式会社ログラス loglass2019
12
470k
[NGA] カンパニーデック202511Ver.
Avatar for NGA Ltd. ngaltd
PRO
1
450
Company Profile
Avatar for Katsuhiko Eguchi katsuegu23
2
11k
「なんか好き」を設計する 情緒的価値をPMの武器にする3つのポイント
Avatar for inagaki inagakikay
6
6.2k
現場とIT部門の橋渡しをして3000人の開発者を救った話 / Talk. Collaborate. Support. Lessons from Bridging Field and IT
Avatar for NTT docomo Business nttcom
2
1.3k
Mercari-Fact-book_jp
Avatar for Mercari, Inc. mercari_inc
7
180k
malna-recruiting-pitch
Avatar for malna malna
0
12k
フロントエンドにおける「型」の責任分解に対する1つのアプローチ
Avatar for kinocoboy kinocoboy2
4
1.5k
株式会社TVer 会社紹介資料
Avatar for TVer Inc. techtver
PRO
9
96k

Featured

See All Featured
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
Avatar for Dan Newman danielanewman
231
22k
Stop Working from a Prison Cell
Avatar for Chris Michel hatefulcrawdad
273
21k
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
55
9.3k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
37
2.6k
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
348
40k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
Avatar for Irina Nazarova irinanazarova
9
1.1k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
Avatar for Sam Stephenson sstephenson
162
15k
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
234
17k
Save Time (by Creating Custom Rails Generators)
Avatar for Garrett Dimon garrettdimon
PRO
32
1.8k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
Avatar for Michelle Schulp Hunt marktimemedia
31
2.6k
Bootstrapping a Software Product
Avatar for Garrett Dimon garrettdimon
PRO
307
120k
What’s in a name? Adding method to the madness
Avatar for Product Marketing Alliance productmarketing
PRO
24
3.8k

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU [email protected] [email protected]