Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to reindex 1B documents in 1 hour?

Avatar for Qaiser Abbasi Qaiser Abbasi
December 13, 2018
Business
0
750

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/
Avatar for Qaiser Abbasi

Qaiser Abbasi

December 13, 2018
Tweet

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
Avatar for Qaiser Abbasi qabbasi
0
62
Testing-Darwinismus
Avatar for Qaiser Abbasi qabbasi
0
58

Other Decks in Business

See All in Business
消防設備について：2720 JAPAN O.K. ロータリーEクラブ ・(有)タナカ消防設備 専務取締役 田中 省吾 会員
Avatar for 2720 Japan O.K. ロータリーEクラブ 2720japanoke
0
720
tokyo_dbt_meetup_#14_意志ある羅針盤たれ＜データサイド＞
Avatar for 山口貴矢 t_yamaguchi
2
340
採用説明資料
Avatar for 株式会社ミツカル recruit_mitsukaru
0
140
アジャイル開発組織における KA法実践の意義
Avatar for Hiroyasu Yoneyama hynym
PRO
0
110
ZEIN株式会社 会社説明資料
Avatar for ZEIN zein
0
590
アシスト 会社紹介資料
Avatar for ashisuto_career ashisuto_career
3
120k
株式会社 Laboro.AI 会社紹介資料
Avatar for Laboro.AI laboroai2016
0
450
フルカイテン株式会社 採用資料
Avatar for FULL KAITEN fullkaiten
0
65k
BALLAS 事業紹介資料
Avatar for 株式会社BALLAS ballas_inc
0
14k
M3 Career Culture Deck（セールス＆コンサルティング職）
Avatar for エムスリーキャリア m3c
1
280k
アディクシィ株式会社 会社資料
Avatar for ADiXi（アディクシィ） adixi
0
400
アッテル会社紹介資料/culture deck
Avatar for Attelu attelu
10
14k

Featured

See All Featured
It's Worth the Effort
Avatar for 3n 3n
184
28k
Dealing with People You Can't Stand - Big Design 2015
Avatar for Cassini Nazir cassininazir
367
26k
A better future with KSS
Avatar for Kyle Neath kneath
239
17k
Making Projects Easy
Avatar for Brett Harned brettharned
116
6.2k
Art, The Web, and Tiny UX
Avatar for Lynn Fisher lynnandtonic
299
21k
Scaling GitHub
Avatar for Zach Holman holman
459
140k
Become a Pro
Avatar for Speaker Deck speakerdeck
PRO
28
5.4k
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
29
2.7k
The World Runs on Bad Software
Avatar for Brandon Keepers bkeepers
PRO
68
11k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
35
2.3k
Why You Should Never Use an ORM
Avatar for John Nunemaker jnunemaker
PRO
56
9.4k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
Avatar for Irina Nazarova irinanazarova
8
780

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU [email protected] [email protected]