Monitoring Elasticsearch for Fun, Profit and Not Getting Woken Up at 3am

Given at the first Elasticsearch Sydney meetup in November 2013.

Transcript

1. Beyond Traffic Lights lessons learned monitoring Elasticsearch for fun, profit and not getting woken up at 03:00
2. n commandments for cluster monitoring • Monitor everything. • Only alert on metrics the customer (dev team/users/et cetera) cares about. • Metrics should be defined for the cluster as a whole, not just individual nodes. • Automate recovery wherever possible.
3. Monitor ALL the things • You never have enough data. • The stats API is your friend. • Store everything.
4. Cluster-wide metrics • Necessary for e.g., splitbrain checks • Not well-suited for host-based monitoring systems
5. Detecting splitbrains nagios check_splitbrain check_topology check_topology NRPE NRPE es1 es2 [ … ]
6. Shockingly, this type of check doesn't work too well without concurrency topologies := make([]string, nNodes) masters := make(map[string]bool) c := make(chan string, nNodes) for _, node := range nodes { go getTopology(node, c) } for i, _ := range nodes { topologies[i] = <-c } masterList := make([]string, 0) for _, topology := range topologies { topologyMaster := getMaster(topology, nodes) if _, ok := masters[topologyMaster]; !ok { masterList = append(masterList, topologyMaster) } masters[topologyMaster] = true }
7. On not getting woken up at 3am • Loss of redundancy is not (always) a failure condition. – So you don't need to alert whenever you lose a node. – Automating alert response can reduce operational workload and human error. Splitbrain alert? - log into every node in the cluster - set cluster.blocks.read_only = true - send out a notification - sysadmin rectifies the partition - minimal data loss
8. Benchmarks and performance tuning • Uncontrolled tests are worse than useless. • So act like data scientists. – actual production queries – one change at a time – every benchmark should be reproducible Sometimes graphs aren't enough.
9. Benchmarks and performance tuning └─> cat *_facet_results_times | numbers 1q: 239.000000 3q: 537.500000 99p: 962.500000 mean: 405.655410 median: 335.000000 std: 208.026379 var: 43274.974297 Sometimes you need the actual numbers.
10. Fin. Queries? Sharif Olorin sio@tesser.org