Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to reindex 1B documents in 1 hour?

Avatar for Qaiser Abbasi Qaiser Abbasi
December 13, 2018
Business
0
760

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/

Avatar for Qaiser Abbasi

Qaiser Abbasi

December 13, 2018
Tweet

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
Avatar for Qaiser Abbasi qabbasi
0
64
Testing-Darwinismus
Avatar for Qaiser Abbasi qabbasi
0
60

Other Decks in Business

See All in Business
KENCOPA Company Dec
Avatar for NK kurodaaaaaa
0
160
「なんとなく使いにくい」を論理的に説明する方法 〜プロダクトエンジニアとしてUXを議論できる第一歩〜
Avatar for みきてぃ / きたはら (mkitahara) mkitahara01985
0
330
数字で見る松岡会計事務所
Avatar for Matsuoka Kaikei wf714201
0
370
Sales Marker Culture Book(English)
Avatar for Sales Marker salesmarker
PRO
2
6k
merpay-Overview
Avatar for Mercari, Inc. mercari_inc
8
180k
仕事の傍ら4歳児のパパでも、アプリをリリースできた理由
Avatar for Ken.T kent_code3
1
220
enechain company deck
Avatar for enechain Recruitment Team enechain
PRO
9
130k
エンジニア職／新卒向け会社紹介資料（テックファーム株式会社）
Avatar for Techfirm techfirm
1
4.6k
アークエル株式会社 会社説明資料
Avatar for アークエル株式会社 aakel
1
6k
キャッチアップ会社紹介
Avatar for catchup catchup
2
57k
Sapeet Recruiting materials
Avatar for Sapeet sapeet
0
5.2k
Agentic AI による新時代の IBP (Intelligent Business Planning)
Avatar for MIKIO KUBO mickey_kubo
1
130

Featured

See All Featured
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
29
1.8k
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
Avatar for Aki / @LoveIdahoBurger aki_iinuma
126
53k
How to train your dragon (web standard)
Avatar for Monica Dinculescu notwaldorf
96
6.2k
The Web Performance Landscape in 2024 [PerfNow 2024]
Avatar for Tammy Everts tammyeverts
9
770
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
Avatar for Eileen M. Uchitelle eileencodes
139
34k
BBQ
Avatar for Matthew Crist matthewcrist
89
9.8k
How To Stay Up To Date on Web Technology
Avatar for Chris Coyier chriscoyier
790
250k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
Avatar for Vitaly Friedman smashingmag
251
21k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
83
9.1k
Git: the NoSQL Database
Avatar for Brandon Keepers bkeepers
PRO
431
65k
How to Ace a Technical Interview
Avatar for Jacob Kaplan-Moss jacobian
279
23k
The Language of Interfaces
Avatar for Des Traynor destraynor
160
25k

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU [email protected] [email protected]