The Do's and Don’ts of Elasticsearch Scalability and Performance

Slide 1

Slide 1 text

codecentric AG Patrick Peschlow The Do's and Don’ts of Elasticsearch Scalability and Performance

Slide 2

Slide 2 text

codecentric AG Think hard about your mapping

Slide 3

Slide 3 text

codecentric AG Think hard about your mapping −Which fields to analyze? How to analyze them? ! −Need term frequencies, positions, offsets? Field norms? ! −Which fields to not analyze or not index/enable? ! −_all ! −_source vs stored fields

Slide 4

Slide 4 text

codecentric AG Think hard about your mapping −Dynamic mapping/templates −Excessive number of fields? ! −Index-time vs. query-time solutions ! −Multi field, copy to, transform script ! −Relations: parent-child/nested

Slide 5

Slide 5 text

codecentric AG Design for scale

Slide 6

Slide 6 text

codecentric AG Design for scale Shard 1 Shard 2 Shard M ... Search

Slide 7

Slide 7 text

codecentric AG Design for scale 2015-05-30 2015-05-31 2015-06-01 Search for „last 3 days“ 2014-11-25 ...

Slide 8

Slide 8 text

codecentric AG Design for scale routing for user 1 Shard 1 Shard 2 Shard M ... User 2 User 1 User 5 User 3 User 4 User 6 User 7 User 8 Search by user 1

Slide 9

Slide 9 text

codecentric AG Design for scale −Can documents/access be partitioned in a natural way? ! −Need to find documents by ID (update/delete/get)? ! −Know the relevant features −Routing, aliases, multi-index search ! −Indices don’t come for free ! −Measure the impact of distributed search

Slide 10

Slide 10 text

codecentric AG Don’t create more  shards than you need

Slide 11

Slide 11 text

codecentric AG Don’t create more shards than you need −More shards −Enable larger indices −Scale operations on individual documents ! −But shards don’t come for free ! −Measure how many shards you need −When unsure, overallocate a little

Slide 12

Slide 12 text

codecentric AG Don’t treat all nodes as equal

Slide 13

Slide 13 text

codecentric AG Don’t treat all nodes as equal −Cluster nodes −Master nodes, data nodes, client/aggregator nodes ! −Client applications −HTTP? −Transport protocol? −Join the cluster as a client node? −In Java: HTTP client vs TransportClient vs NodeClient

Slide 14

Slide 14 text

codecentric AG Don’t run wasteful queries

Slide 15

Slide 15 text

codecentric AG Don’t run wasteful queries −Only request as many hits as you need ! −Avoid deep pagination ! −Use scan+scroll to iterate without sorting ! −Only query indices/shards that may contain hits

Slide 16

Slide 16 text

codecentric AG Engineer queries

Slide 17

Slide 17 text

codecentric AG Engineer queries −Measure performance −Set up production-like cluster and data ! −Use filters ! −Check and tune filter caching ! −Reduce work for heavyweight filters −Order them, consider accelerators

Slide 18

Slide 18 text

codecentric AG Care about field data

Slide 19

Slide 19 text

codecentric AG Care about field data −Used for sorting, aggregation, parent-child, scripts, … ! −High memory consumption or OutOfMemoryError −Cache limit, circuit breakers avoid the worst ! −Evaluate field data requirements in advance ! −Use „doc values“ to store expensive field data on disk

Slide 20

Slide 20 text

codecentric AG Be prepared for reindexing

Slide 21

Slide 21 text

codecentric AG Be prepared for reindexing −Reasons for reindexing −Mapping changes −Index/shard reaches its capacity −Reduce number of indices/shards

Slide 22

Slide 22 text

codecentric AG Be prepared for reindexing −Reindexing procedure depends on many factors −Data source? −Zero downtime? −Update API usage? −Possible deletes? −Designated component (queue) for indexing?

Slide 23

Slide 23 text

codecentric AG Be prepared for reindexing −Use existing tooling ! −Do it yourself? Use scan+scroll and bulk indexing ! −Follow best practices −Use aliases −Disable refresh −Decrease number of replicas

Slide 24

Slide 24 text

codecentric AG Don’t use the defaults

Slide 25

Slide 25 text

codecentric AG Don’t use the defaults −Cluster settings −cluster name, discovery, minimum_master_nodes −recovery ! −Number of shards and replicas ! −Refresh interval ! −Thread pool and cache configuration

Slide 26

Slide 26 text

codecentric AG Monitor

Slide 27

Slide 27 text

codecentric AG Monitor −Cluster health, split brains ! −Thread pools and caches ! −Garbage collection (the actual JVM output) ! −Slow log ! −Hot threads

Slide 28

Slide 28 text

codecentric AG Follow the production recommendations

Slide 29

Slide 29 text

codecentric AG Follow the production recommendations −A good start would be to read/research them at all ! −Just to mention a few −The more memory, the better −Isolate as much as possible −SSDs and local storage recommended

Slide 30

Slide 30 text

codecentric AG Don’t test in production

Slide 31

Slide 31 text

codecentric AG Don’t test in production −Use a test environment ! −Test the cluster −Single node restarts, rolling upgrades, node loss −Full cluster restarts ! −Test behavior under expected load −Queries −Indexing

Slide 32

Slide 32 text

codecentric AG Read the guide

Slide 33

Slide 33 text

codecentric AG Read the guide −Elasticsearch: The Definitive Guide − https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html ! −Elasticsearch Reference − https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

Slide 34

Slide 34 text

codecentric AG Questions? Dr. rer. nat. Patrick Peschlow  codecentric AG  Merscheider Straße 1  42699 Solingen    tel +49 (0) 212.23 36 28 54  fax +49 (0) 212.23 36 28 79  [email protected]    www.codecentric.de