Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to reindex 1B documents in 1 hour?

Avatar for Qaiser Abbasi Qaiser Abbasi
December 13, 2018
Business
0
750

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/
Avatar for Qaiser Abbasi

Qaiser Abbasi

December 13, 2018
Tweet

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
Avatar for Qaiser Abbasi qabbasi
0
63
Testing-Darwinismus
Avatar for Qaiser Abbasi qabbasi
0
58

Other Decks in Business

See All in Business
ベタートラップと夏
Avatar for mosa mosa_siru
8
3.4k
Leading Mark新卒採用資料
Avatar for Leading Mark unno
0
2.5k
ユウミ会社説明資料
Avatar for yu-mi yumi2023
0
1.1k
タケウチグループRecruit
Avatar for タケウチグループ takeuchigroup
0
6.9k
LW_brochure_engineer
Avatar for linc-well lincwellhr
0
34k
アッテル会社紹介資料/culture deck
Avatar for Attelu attelu
10
15k
VISASQ: ABOUT US
Avatar for VISASQ eikohashiba
15
500k
Sales Marker Culture book
Avatar for Sales Marker salesmarker
PRO
36
58k
GMOフィナンシャルHD 会社紹介資料
Avatar for GMO Financial Holdings, Inc. gmofh_hr_team
0
50k
採用ピッチ資料｜SBペイメントサービス株式会社
Avatar for SBPS sbps
0
30k
【全ポジション共通】㈱エグゼクション/会社紹介資料
Avatar for EXE HR exe_recruit
1
1.3k
特別講義　理系のための法学入門
Avatar for SEKO_Shuhei seko_shuhei
2
2.4k

Featured

See All Featured
Docker and Python
Avatar for Tania Allard trallard
44
3.5k
Fireside Chat
Avatar for paigeccino paigeccino
37
3.5k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
130
19k
Building Better People: How to give real-time feedback that sticks.
Avatar for Will Jessup wjessup
367
19k
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
45
7.5k
Large-scale JavaScript Application Architecture
Avatar for Addy Osmani addyosmani
512
110k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
Avatar for mongodb mongodb
46
9.6k
Product Roadmaps are Hard
Avatar for C. Todd Lombardo iamctodd
PRO
54
11k
Music & Morning Musume
Avatar for Bryan Veloso bryan
46
6.6k
Code Review Best Practice
Avatar for Trisha Gee trishagee
69
19k
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
10
970
Six Lessons from altMBA
Avatar for Skipper Chong Warson skipperchong
28
3.9k

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU [email protected] [email protected]