Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to reindex 1B documents in 1 hour?

Avatar for Qaiser Abbasi Qaiser Abbasi
December 13, 2018
Business
830
0

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/

Avatar for Qaiser Abbasi

Qaiser Abbasi

December 13, 2018

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
Avatar for Qaiser Abbasi qabbasi
0
72
Testing-Darwinismus
Avatar for Qaiser Abbasi qabbasi
0
80

Other Decks in Business

See All in Business
株式会社WePush 採用候補者様向け会社説明資料
Avatar for 株式会社WePush wepush
1
990
どちらも「ユーザーのため」だった　—価値の対立を仮説検証に変えて #Scrumfest Osaka 2026
Avatar for Aki Tanaka mamedai55
0
470
GOドライブ会社紹介資料
Avatar for go_drive_corp go_drive_corp
2
4.7k
株式会社TableCheck - 会社紹介 Company Profile
Avatar for Acquisition TableCheck tablecheckac
0
9k
セーフィー株式会社（Safie Inc.） 会社紹介資料
Avatar for Safie safie_recruit
7
460k
スマサテ 会社紹介資料
Avatar for Sumasate sumasate
0
2.4k
「受託脳」から「提案脳」へ〜「一緒に悩んで、いいものつくる」を目指して
Avatar for Yoshihito Kuranuki / 倉貫義人 kuranuki
3
1.6k
「エンジニアのための自己管理入門」のインセプションデッキ/Inception Deck of Self-Management beginner's guide book
Avatar for 小田中育生 ikuodanaka
2
550
Akatsuki AI Technologies Company Deck
Avatar for Akatsuki AI Technologies akatsuki_ai_technologies
0
120
Recruiting Deck 2607
Avatar for Strategic Decision sd_official
0
100
RECRUIT DECK 株式会社KOBIRA 会社説明資料
Avatar for 株式会社KOBIRA kobira_official
PRO
0
6.9k
なぜ勉強会だけでは浸透しないのか ～数百人規模の組織でコーディングエージェントを当たり前にした戦略とその結果～
Avatar for Yusuke Sugiura sigma_sig
0
210

Featured

See All Featured
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
330
40k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
234
19k
Embracing the Ebb and Flow
Avatar for Simon Collison colly
88
5.1k
Music & Morning Musume
Avatar for Bryan Veloso bryan
47
7.3k
SEO Brein meetup: CTRL+C is not how to scale international SEO
Avatar for Linda Hogenes lindahogenes
1
2.8k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
Avatar for Tammy Everts tammyeverts
11
990
Dominate Local Search Results - an insider guide to GBP, reviews, and Local SEO
Avatar for Greg Gifford greggifford
PRO
0
300
Leveraging LLMs for student feedback in introductory data science courses - posit::conf(2025)
Avatar for Mine Cetinkaya-Rundel minecr
1
350
DBのスキルで生き残る技術 - AI時代におけるテーブル設計の勘所
Avatar for soudai sone soudai
PRO
68
56k
Building Experiences: Design Systems, User Experience, and Full Site Editing
Avatar for Michelle Schulp Hunt marktimemedia
0
580
Imperfection Machines: The Place of Print at Facebook
Avatar for Scott Boms scottboms
270
14k
Self-Hosted WebAssembly Runtime for Runtime-Neutral Checkpoint/Restore in Edge–Cloud Continuum
Avatar for Yuki Nakata chikuwait chikuwait
0
710

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU [email protected] [email protected]