Elastic @ Target - Scaling Log Aggregation

A presentation given at August 2016 Minneapolis Elasticsearch meetup, about Target's journey in scaling log aggregation using Elasticsearch and Kafka

Transcript

4. kafka app app app app logstash logstash logstash elasticsearch elasticsearch elasticsearch All nodes - data + master eligible kibana
5. kafka app app app app logstash logstash logstash elasticsearch elasticsearch elasticsearch masters kibana m m m ssd ssd ssd ssd ssd ssd
6. kafka app app app app logstash logstash logstash elasticsearch elasticsearch elasticsearch masters kibana m m m ssd ssd ssd hdd hdd hdd curator
7. kafka app app app app logstash logstash logstash elasticsearch elasticsearch elasticsearch kibana m m m ssd ssd ssd hdd hdd hdd external dc target dc openstack physical servers curator
10. # ./kafka-consumer-offset-checker.sh --zookeeper elk-zookeeper.service.consul:2181 --group digital-logging --topic digital_logging_logging Group Topic Pid Offset logSize Lag Owner digital-logging digital_logging_logging 0 129073173 129073469 296 digital-logging_ip-10-48-128-233-1464396793891-9f47996d-0 digital-logging digital_logging_logging 1 119656095 119656095 0 digital-logging_ip-10-48-128-46-1464394771806-6e17033a-0 digital-logging digital_logging_logging 2 116099802 116099802 0 digital-logging_ip-10-48-129-100-1464402868873-fc3ce488-0 digital-logging digital_logging_logging 3 131113050 131124910 11860 digital-logging_ip-10-48-129-229-1464394750510-900ed19c-0 digital-logging digital_logging_logging 4 125142784 125142917 133 digital-logging_ip-10-48-129-230-1464394785403-1e4af488-0 digital-logging digital_logging_logging 5 124061050 124061135 85 digital-logging_ip-10-48-130-225-1468969568171-6d40e0e9-0 digital-logging digital_logging_logging 6 127284883 127284883 0 digital-logging_ip-10-48-130-35-1464394790468-411958eb-0 digital-logging digital_logging_logging 7 122454788 122460697 5909 digital-logging_ip-10-48-131-73-1464394729757-a1102b55-0 digital-logging digital_logging_logging 8 123639897 123640016 119 digital-logging_ip-10-48-131-74-1464394731291-469c29de-0 digital-logging digital_logging_logging 9 124663119 124663199 80 digital-logging_ip-10-48-131-75-1464394735263-7a206b96-0 # ./kafka-consumer-offset-checker.sh --zookeeper elk-zookeeper.service.consul:2181 --group digital-logging --topic digital_logging_search Group Topic Pid Offset logSize Lag Owner digital-logging digital_logging_search 0 800247398 800247486 88 digital-logging_ip-10-48-128-233-1464396793891-9f47996d-0 digital-logging digital_logging_search 1 845027776 845027832 56 digital-logging_ip-10-48-128-46-1464394771806-6e17033a-0 digital-logging digital_logging_search 2 823387607 823387674 67 digital-logging_ip-10-48-129-100-1464402868873-fc3ce488-0 digital-logging digital_logging_search 3 792826837 792826908 71 digital-logging_ip-10-48-129-229-1464394750510-900ed19c-0 digital-logging digital_logging_search 4 833472383 833472468 85 digital-logging_ip-10-48-129-230-1464394785403-1e4af488-0 digital-logging digital_logging_search 5 810161527 810161617 90 digital-logging_ip-10-48-130-225-1468969568171-6d40e0e9-0 digital-logging digital_logging_search 6 795554676 795554888 212 digital-logging_ip-10-48-130-35-1464394790468-411958eb-0 digital-logging digital_logging_search 7 840447062 840447181 119 digital-logging_ip-10-48-131-73-1464394729757-a1102b55-0 digital-logging digital_logging_search 8 825843130 825843308 178 digital-logging_ip-10-48-131-74-1464394731291-469c29de-0 digital-logging digital_logging_search 9 803764087 803764589 502 digital-logging_ip-10-48-131-75-1464394735263-7a206b96-0 input { kafka { zk_connect => ["stable.elk-zookeeper.service.consul:2181"] white_list => ["^digital_logging_[^A-Z].*$"] group_id => ["digital-logging"] } } output { elasticsearch { hosts => ["stable.elk-elasticsearch.service.consul"] index => "logstash-%{[type]}-%{+YYYY.MM.dd}" } }
13. kafka app app app app tribe kibana external dc target dc logstash logstash logstash elastic1 elastic2 elastic3
14. kafka app app app app external dc target dc logstash logstash logstash elastic1 elastic2 elastic3 elastic1a tribe kibana