ELK Stack

Hands-on tour on how to install Elasticsearch 2.0, Logstash 2.0, and Kibana 4.2 on CentOS as well as how to monitor nginx with it.

Transcript

1. Philipp Krenn @xeraa
3. Normally Automate all the things Ansible
4. Vagrant Box $ vagrant init bento/centos-7.1 $ vagrant up --provider virtualbox $ vagrant ssh
5. JDK8 Installation $ cd ~ $ wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u65-b17/jdk-8u65-linux-x64.rpm" $ sudo yum -y localinstall jdk-8u65-linux-x64.rpm $ java -version java version "1.8.0_65" Java(TM) SE Runtime Environment (build 1.8.0_65-b17) Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode) $ sudo tee -a /etc/profile.d/java_home.sh >/dev/null <<'EOF' export JAVA_HOME=/usr/java/default/ EOF $ echo $JAVA_HOME /usr/java/default/
6. Elasticsearch Repository $ sudo rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch $ sudo tee -a /etc/yum.repos.d/elasticsearch.repo >/dev/null <<'EOF' [elasticsearch-2.x] name=Elasticsearch repository for 2.x packages baseurl=http://packages.elastic.co/elasticsearch/2.x/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 EOF
7. Elasticsearch Installation $ sudo yum -y install elasticsearch $ sudo systemctl start elasticsearch $ sudo systemctl enable elasticsearch Only listening on localhost Reconfigure in /etc/elasticsearch/elasticsearch.yml 1.x listened on all interfaces
8. Elasticsearch Test $ curl -X GET http://localhost:9200 { "name" : "Meteor Man", "cluster_name" : "elasticsearch", "version" : { "number" : "2.0.0", "build_hash" : "de54438d6af8f9340d50c5c786151783ce7d6be5", "build_timestamp" : "2015-10-22T08:09:48Z", "build_snapshot" : false, "lucene_version" : "5.2.1" }, "tagline" : "You Know, for Search" }
9. Kibana Repository $ sudo tee -a /etc/yum.repos.d/kibana.repo >/dev/null <<'EOF' [kibana-4.2] name=Kibana repository for 4.2.x packages baseurl=http://packages.elastic.co/kibana/4.2/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 EOF 4.2 not yet in the repository
10. Kibana Installation $ sudo yum -y install kibana $ sudo systemctl start kibana $ sudo chkconfig kibana on Configuration in /opt/kibana/config/kibana.yml systemctl enable kibana not supported
11. Kibana Manual Installaion Elasticsearch 2.0 requires Kibana 4.2 $ wget https://download.elastic.co/kibana/kibana/kibana-4.2.0-linux-x64.tar.gz $ tar xzfv kibana-4.2.0-linux-x64.tar.gz $ kibana-4.2.0-linux-x64/bin/kibana
12. How to reach Kibana on localhost?
13. nginx Installation $ sudo yum -y install epel-release $ sudo yum -y install nginx httpd-tools $ sudo htpasswd -c /etc/nginx/htpasswd.users kibanaadmin $ sudo vi /etc/nginx/nginx.conf
14. server { Delete listen 80 default_server; listen [::]:80 default_server; server_name _; root /usr/share/nginx/html; Keep include /etc/nginx/default.d/*.conf;
15. Kibana Configuration $ sudo tee -a /etc/nginx/conf.d/kibana.conf >/dev/null <<'EOF' server { listen 80; server_name localhost; auth_basic "Restricted Access"; auth_basic_user_file /etc/nginx/htpasswd.users; location / { proxy_pass http://localhost:5601; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } } EOF
16. nginx Installation $ sudo systemctl start nginx $ sudo systemctl enable nginx
17. Logstash Repository $ sudo tee -a /etc/yum.repos.d/logstash.repo >/dev/null <<'EOF' [logstash-2.0] name=Logstash repository for 2.0.x packages baseurl=http://packages.elasticsearch.org/logstash/2.0/centos gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1 EOF
18. Logstash Installation $ sudo yum -y install logstash
19. Certificate Creation $ cd /etc/pki/tls $ sudo openssl req -subj '/CN=localhost/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
20. Logstash Configuration $ sudo tee -a /etc/logstash/conf.d/01-lumberjack-input.conf >/dev/null <<'EOF' input { lumberjack { port => 5043 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" } } EOF
21. https://www.elastic.co/guide/en/ logstash/current/plugins-inputs- lumberjack.html
22. Logstash Configuration $ sudo tee -a /etc/logstash/conf.d/10-syslog.conf >/dev/null <<'EOF' filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } syslog_pri { } date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } } } EOF Without the linebreaks in "message"
23. Grok Regular expressions Pattern: %{PATTERN:IDENTIFIER} "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" Nov 1 01:51:10 localhost nginx: nginx: configuration file /etc/nginx/nginx.conf test is successful Nov 1 01:53:43 localhost yum[2616]: Installed: 1:logstash-2.0.0-1.noarch
24. Logstash Configuration $ sudo tee -a /etc/logstash/conf.d/30-lumberjack-output.conf >/dev/null <<'EOF' output { elasticsearch { hosts => localhost } stdout { codec => rubydebug } } EOF $ sudo systemctl restart logstash hosts was host in Logstash 1.x
25. Logstash Forwarder On each client server $ scp /etc/pki/tls/certs/logstash-forwarder.crt user@client-server:/tmp $ sudo cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/ $ sudo rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch $ sudo tee -a /etc/yum.repos.d/logstash-forwarder.repo >/dev/null <<'EOF' [logstash-forwarder] name=logstash-forwarder repository baseurl=http://packages.elastic.co/logstashforwarder/centos gpgcheck=1 gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch enabled=1 EOF
26. Logstash Forwarder On each client server $ sudo yum -y install logstash-forwarder $ sudo vi /etc/logstash-forwarder.conf { "network": { "servers": [ "localhost:5043" ], "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt", "timeout": 15 }, "files": [ { "paths": [ "/var/log/messages", "/var/log/secure" ], "fields": { "type": "syslog" } } ] }
27. Logstash Forwarder On each client server $ sudo systemctl start logstash-forwarder $ sudo chkconfig logstash-forwarder on systemctl enable logstash-forwarder not supported
28. Vagrant Configuration $ vi Vagrantfile config.vm.network "forwarded_port", guest: 80, host: 8088 $ vagrant reload http://localhost:8088/ kibanaadmin
30. Debug $ tail /var/log/logstash-forwarder/logstash-forwarder.err $ tail /var/log/logstash/logstash.log
32. Elasticsearch Indexes $ curl -X GET localhost:9200/_cat/indices?v health status index pri rep docs.count docs.deleted store.size pri.store.size yellow open .kibana 1 1 2 0 12.4kb 12.4kb yellow open logstash-2015.10.31 5 1 318 0 612.5kb 612.5kb yellow open logstash-2015.10.06 5 1 4606 0 2.5mb 2.5mb yellow open logstash-2015.11.01 5 1 351 0 557.7kb 557.7kb
34. Goal Monitor nginx Access Log
35. Logstash Forwarder $ sudo vi /etc/logstash-forwarder.conf , { "paths": [ "/var/log/nginx/access.log" ], "fields": { "type": "nginx-access" } } $ sudo service logstash-forwarder restart $ tail /var/log/logstash-forwarder/logstash-forwarder.err
36. Logstash Patterns Grok patterns $ sudo mkdir -p /opt/logstash/patterns $ sudo chown logstash:logstash /opt/logstash/patterns $ sudo tee -a /opt/logstash/patterns/nginx >/dev/null <<'EOF' NGUSERNAME [a-zA-Z\.\@\-\+_%]+ NGUSER %{NGUSERNAME} NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident}%{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent} EOF $ sudo chown logstash:logstash /opt/logstash/patterns/nginx Without the linebreaks in NGINXACCESS
37. Logstash Filter How the server parses the relevant log files $ sudo tee -a /etc/logstash/conf.d/11-nginx.conf >/dev/null <<'EOF' filter { if [type] == "nginx-access" { grok { match => { "message" => "%{NGINXACCESS}" } } } } EOF $ sudo service logstash restart
45. Thanks! Questions? @xeraa