Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to reindex 1B documents in 1 hour?

Avatar for Qaiser Abbasi Qaiser Abbasi
December 13, 2018
Business
790
0

How to reindex 1B documents in 1 hour?

Talk given at www.meetup.com/Elasticsearch-Berlin/

Avatar for Qaiser Abbasi

Qaiser Abbasi

December 13, 2018

More Decks by Qaiser Abbasi

See All by Qaiser Abbasi
Java User Group Frankfurt – CDI BeanTesting
Avatar for Qaiser Abbasi qabbasi
0
71
Testing-Darwinismus
Avatar for Qaiser Abbasi qabbasi
0
72

Other Decks in Business

See All in Business
エンジニアのためのコミュニケーション術
Avatar for zashii zashii
0
190
DMM.com コーポレートブック
Avatar for DMM.com dmm
2
480k
インキュデータ会社紹介資料
Avatar for Recruit@INCUDATA okitsu
3
55k
株式会社ELYZA（イライザ） 採用情報資料 / RECRUIT PITCH
Avatar for 株式会社ELYZA elyza
2
750k
三井物産グループのデジタル証券～イオン大宮～徹底解説セミナー
Avatar for 三井物産デジタル・アセットマネジメント株式会社 c0rp_mdm
PRO
0
1.1k
introduce_backoffice_coordinate
Avatar for Yuki Yano yuki_yano
PRO
1
540
エージェントスキルによる最適化
Avatar for MIKIO KUBO mickey_kubo
2
120
BASE株式会社 統合報告書2026
Avatar for BASE, Inc. base
PRO
0
1.1k
【キャリア採用】NEC会社紹介資料
Avatar for NEC RECRUITING nec_recruiting
0
140
Codexを安心して業務活用するには？──「権限・接続・実行」の考え方
Avatar for hima2b4 hima2b4
0
220
Smart Share Recruiting Deck
Avatar for smarshare smartshare
0
160
複雑なシステムから大学職員を救う自律型エージェント「だっこくん」
Avatar for Mick Etoh micknerd
0
160

Featured

See All Featured
BBQ
Avatar for Matthew Crist matthewcrist
89
10k
GitHub's CSS Performance
Avatar for Jon Rohan jonrohan
1033
470k
We Analyzed 250 Million AI Search Results: Here's What I Found
Avatar for Josh Blyskal joshbly
1
1.3k
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
59
6.6k
Amusing Abliteration
Avatar for ianozsvald ianozsvald
1
180
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
239
140k
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
141
7.4k
Why Your Marketing Sucks and What You Can Do About It - Sophie Logan
Avatar for Sophie Logan marketingsoph
0
150
Easily Structure & Communicate Ideas using Wireframe
Avatar for Afnizar Nur Ghifari afnizarnur
194
17k
Into the Great Unknown - MozCon
Avatar for Noah Learner thekraken
41
2.5k
Noah Learner - AI + Me: how we built a GSC Bulk Export data pipeline
Avatar for Tech SEO Connect techseoconnect
PRO
0
180
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
Avatar for Dan Newman danielanewman
231
23k

Transcript

  1. Rene Treffer, Qaiser Abbasi How to reindex 1B documents in

    1 hour?

  2. Search @ SoundCloud

  3. Powered by ElasticSearch

  4. Typical search document

  5. Clusters of 30 nodes

  6. Clusters of 30 nodes data size * replication = 120%

    * total memory

  7. Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case

  8. Problems?

  9. Lead time of features and bugfixes Problems?

  10. Indexing

  11. Indexing 1. Extract

  12. Indexing 1. Extract 2. Build ES documents

  13. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES

  14. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates

  15. Indexing 1. Extract 2. Build ES documents 3. Load into

    ES 0. Live updates Kafka Kafka +

  16. Kafka historic current compaction Cluster 1 Cluster 2 shipper 1

    shipper 2 indexer

  17. Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit

  18. ES cluster lifecycle Reindex Live Maintenance

  19. Reindex settings 1. Shards 2. Replication settings 3. Async Translog

    4. Refresh Interval

  20. Finish reindexing 1. Merge into one segment*** 2. Set #

    replicas

  21. Throughput ≈ 600K OP/s ≈ 30 Mins

  22. 4X faster for 95% ≈ 40ms for 50%

  23. 4X Reindexing in 1 Sprint

  24. Summary • Solved initial problem • Enablement in daily life

  25. Future work

  26. Q & A

  27. Sounds interesting? Come and talk to us!

  28. THANK YOU [email protected] [email protected]