How to reindex 1B documents in 1 hour?

Efa70d43d8e57623d0d5fafaeef1109f?s=47 Qaiser Abbasi
December 13, 2018

How to reindex 1B documents in 1 hour?

Efa70d43d8e57623d0d5fafaeef1109f?s=128

Qaiser Abbasi

December 13, 2018
Tweet

Transcript

  1. 7.

    Cluster 2 Cluster 3 Cluster 1 Cluster 2 Cluster 3

    Cluster 1 Multiple clusters per use-case
  2. 10.
  3. 15.
  4. 17.

    Kafka for ES documents 1. Enable compaction 2. Use fast

    compression 3. Use enough partitions 4. Use SSDs + 10GBit
  5. 26.