Elasticsearch & Marvel

Elasticsearch & Marvel

Transcript

1. ELASTICSEARCH & MARVEL EKO KURNIAWAN KHANNEDY
2. ELASTICSEARCH & MARVEL EKO KURNIAWAN KHANNEDY ▸ Principal Software Development Engineer at Blibli. ▸ Part of Research and Development Team at Blibli. ▸ Code Scala & Java, but sometimes code Ruby (In this demo we use Ruby) ▸ https://www.linkedin.com/in/khannedy
3. ELASTICSEARCH & MARVEL AGENDA ▸ What is Elasticsearch? ▸ Cluster ▸ Shard & Replication ▸ Distributed Document Store ▸ Distributed Search ▸ Marvel ▸ DEMO ▸ Setup Elasticsearch ▸ Scale Horizontally ▸ Data In & Data Out ▸ Zero Downtime Migration ▸ Marvel Monitoring Tool
4. WHAT IS ELASTICSEARCH? EKO KURNIAWAN KHANNEDY
5. YOU KNOW, FOR SEARCH
6. CLUSTER ELASTICSEARCH & MARVEL
7. ELASTICSEARCH & MARVEL ELASTICSEARCH ▸ Elasticsearch is build to be always available, and to scale with your needs. Scale can come from buying bigger servers (vertical scale) or from buying more servers (horizontal scale). ▸ Real scalability comes from horizontal scale - the ability to add more nodes to the cluster and to spread load and reliability between them. ▸ Elasticsearch is distributed by nature; it knows how to manage multiple nodes to provide scale and high availability. This also means that your application doesn’t need to care about it.
8. ELASTICSEARCH & MARVEL ELASTICSEARCH CLUSTER ▸ A Node is a running instance of Elasticsearch. ▸ A Cluster consists of one or more nodes with the same cluster.name that working together to share their data and workloads. ▸ As nodes are added to or removed from the cluster, the cluster reorganizes itself to spread the data evenly.
9. ELASTICSEARCH & MARVEL CLUSTER HEALTH ▸ GREEN : All primary and replicas shards are active. ▸ YELLOW : All primary shards are active, but not all replicas shards are active. ▸ RED : Not all primary shards are active.
10. ELASTICSEARCH & MARVEL AN EMPTY CLUSTER
11. ELASTICSEARCH & MARVEL ADD AN INDEX
12. ELASTICSEARCH & MARVEL A TWO-NODE CLUSTER
13. ELASTICSEARCH & MARVEL SCALE HORIZONTALLY
14. ELASTICSEARCH & MARVEL CLUSTER AFTER KILLING ONE NODE
15. SHARD & REPLICA ELASTICSEARCH & MARVEL
16. ELASTICSEARCH & MARVEL ELASTICSEARCH SHARD & REPLICA ▸ By default Elasticsearch will give you 5 shards and 1 Replica per index. ▸ You can change the replica size in runtime without downtime. ▸ But we can not change the shard size. If we want to change the shard size, we need to create new index and migrate old index to new index.
17. ELASTICSEARCH & MARVEL 3 SHARDS WITH NO REPLICA
18. ELASTICSEARCH & MARVEL 3 SHARDS WITH 1 REPLICA
19. ELASTICSEARCH & MARVEL 3 SHARDS WITH 2 REPLICAS
20. DISTRIBUTED DOCUMENT STORE ELASTICSEARCH & MARVEL
21. ELASTICSEARCH & MARVEL ROUTING DOCUMENT TO A SHARD shard = hash(routing) % number_of_primary_shards
22. ELASTICSEARCH & MARVEL CREATING, INDEXING AND DELETING A DOCUMENT
23. ELASTICSEARCH & MARVEL RETRIEVING A DOCUMENT
24. ELASTICSEARCH & MARVEL UPDATING A DOCUMENT
25. ELASTICSEARCH & MARVEL IMMUTABILITY ▸ The inverted index that is written to disk is immutable: it doesn’t change. Ever. This immutability has important benefits. ▸ There is no need for locking. If you never have to update the index, you never have to worry about multiple processes trying to make changes at the same time.
26. ELASTICSEARCH & MARVEL DELETE AND UPDATES ▸ Because document is immutable, so the document cannot be removed, nor can be updated to a newer version of the document. ▸ Every commit point includes a .del file that lists which documents have been deleted. ▸ When a document deleted, it is actually marked as deleted in the .del file. ▸ Document updates work in similar way: when a document is updated, the old version of the document is marked as deleted, and the new version of the document is indexed in a new segment.
27. DISTRIBUTED SEARCH ELASTICSEARCH & MARVEL
28. ELASTICSEARCH & MARVEL DISTRIBUTED SEARCH ▸ Search require a more complicated execution model because we don’t know which documents will match the query: they could be on any shard in the cluster. ▸ Finding all matching documents is only half the story. Result from multiple shards must be combined into single sorted list before return the results. ▸ For this reason, search executed in two-phase process called “query then fetch”
29. ELASTICSEARCH & MARVEL QUERY PHASE
30. ELASTICSEARCH & MARVEL FETCH PHASE
31. ELASTICSEARCH & MARVEL DEEP PAGINATION ▸ Remember that each shard must build a priority queue of length from + size, all of which need to be passed back to the coordinating node. And coordinating node needs to sort through number_or_shards + (from + size) documents in order to find the correct size documents. ▸ With big-enough from values, the sorting process can become very heavy indeed, using vast amount of CPU, memory and bandwidth. ▸ For this reason, we strongly advice against deep paging. ▸ As alternative, we can use Scan & Scroll API for deep pagination.
32. MARVEL ELASTICSEARCH & MARVEL
33. ELASTICSEARCH & MARVEL WHAT IS MARVEL? ▸ Marvel is Elasticsearch Monitoring Tool. ▸ Marvel can monitoring all nodes in Elasticsearch Cluster. ▸ Marvel running on top Kibana.
34. ELASTICSEARCH & MARVEL CLUSTER METRICS
35. ELASTICSEARCH & MARVEL INDEX METRICS
36. ELASTICSEARCH & MARVEL NODE METRICS
37. DEMO
38. THANKS
39. ELASTICSEARCH & MARVEL REFERENCES ▸ https://www.elastic.co/ ▸ https://www.elastic.co/products/elasticsearch ▸ https://www.elastic.co/products/marvel ▸ https://www.elastic.co/learn