Logging Analytics e Machine Learning com Elastic Stack

Transcript

1. 1 Thiago Souza Support Engineer 5 de Agosto, 2017 Logging Analytics e Machine Learning com Elastic Stack
2. 2 Um pouco sobre mim • Desenvolvedor há mais de 10 anos • Trabalho com Elasticsearch desde 2010 (em produção desde 2013) • Support Engineer @ Elastic
3. 2014 Millions of Downloads 40. 100. 2016 2015 2012 2013 Cumulative downloads of the Elastic Stack (Elasticsearch, Kibana, Beats, Logstash) and X-Pack 3 100M+ Downloads 3,000+ Customers 92,000+ Community
4. 4 4 Global Customer Base Tech Finance Telco Consumer
5. 5 Logging Analytics e Machine Learning com Elastic Stack Alguns anos atrás...
6. 6 Logging Analytics e Machine Learning com Elastic Stack Alguns anos atrás...
7. 7 Logging Analytics e Machine Learning com Elastic Stack Alguns anos atrás... AppServer
8. 8 Logging Analytics e Machine Learning com Elastic Stack Alguns anos atrás... AppServer DBServer
9. 9 Logging Analytics e Machine Learning com Elastic Stack Alguns anos atrás... AppServer DBServer FTP app.log (20MB)
10. 10 Logging Analytics e Machine Learning com Elastic Stack Alguns anos atrás... at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:548) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:595) Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket WARNING: Restarting endpoint Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket SEVERE: Endpoint null shutdown due to exception: java.net.BindException: Address already in use: JVM_Bind:8080 java.net.BindException: Address already in use: JVM_Bind:8080 at org.apache.tomcat.util.net.PoolTcpEndpoint.initEndpoint(PoolTcpEndpoint.java:264) at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:441) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:548) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:595) Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run SEVERE: Caught exception (java.lang.ThreadDeath) executing org.apache.tomcat.util.net.TcpWorkerThread@12c5431, terminating thread Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket WARNING: Restarting endpoint Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket SEVERE: Endpoint null shutdown due to exception: java.net.BindException: Address already in use: JVM_Bind:8080 java.net.BindException: Address already in use: JVM_Bind:8080 at org.apache.tomcat.util.net.PoolTcpEndpoint.initEndpoint(PoolTcpEndpoint.java:264) at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:441) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:548) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
11. 11 Logging Analytics e Machine Learning com Elastic Stack Alguns anos atrás... at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:548) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:595) Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket WARNING: Restarting endpoint Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket SEVERE: Endpoint null shutdown due to exception: java.net.BindException: Address already in use: JVM_Bind:8080 java.net.BindException: Address already in use: JVM_Bind:8080 at org.apache.tomcat.util.net.PoolTcpEndpoint.initEndpoint(PoolTcpEndpoint.java:264) at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:441) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:548) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:595) Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run SEVERE: Caught exception (java.lang.ThreadDeath) executing org.apache.tomcat.util.net.TcpWorkerThread@12c5431, terminating thread Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket WARNING: Restarting endpoint Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket SEVERE: Endpoint null shutdown due to exception: java.net.BindException: Address already in use: JVM_Bind:8080 java.net.BindException: Address already in use: JVM_Bind:8080 at org.apache.tomcat.util.net.PoolTcpEndpoint.initEndpoint(PoolTcpEndpoint.java:264) at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:441) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:548) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
12. 12 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer
13. 13 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer
14. 14 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer
15. 15 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer
16. 16 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer
17. 17 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer Microservices! Attaaaaack!!!!
18. 18 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer Microservices! Attaaaaack!!!! app.log (20MB)
19. 19 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer Microservices! Attaaaaack!!!! app.log (20MB) service-1.log (10GB) server-20.log (30GB) microsrv-100.log (31GB) service-202.log (80GB) … potencialmente: +100GB
20. 20 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer Microservices! Attaaaaack!!!! app.log (20MB) service-1.log (10GB) server-20.log (30GB) microsrv-100.log (31GB) service-202.log (80GB) … potencialmente: +100GB
21. 21 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer Microservices! Attaaaaack!!!! app.log (20MB) service-1.log (10GB) server-20.log (30GB) microsrv-100.log (31GB) service-202.log (80GB) … potencialmente: +100GB
22. 22 Logging Analytics e Machine Learning com Elastic Stack Centralized Logging Centralized Logging
23. 23 Logging Analytics e Machine Learning com Elastic Stack Centralized Logging Elastic Stack
24. 24 Logging Analytics e Machine Learning com Elastic Stack Centralized Logging app.log (20MB) service-1.log (10GB) server-20.log (30GB) microsrv-100.log (31GB) service-202.log (80GB) … potencialmente: +100GB
25. 25 Logging Analytics e Machine Learning com Elastic Stack Centralized Logging app.log (20MB) service-1.log (10GB) server-20.log (30GB) microsrv-100.log (31GB) service-202.log (80GB) … potencialmente: +100GB
26. 26 Logging Analytics e Machine Learning com Elastic Stack Centralized Logging app.log (20MB) service-1.log (10GB) server-20.log (30GB) microsrv-100.log (31GB) service-202.log (80GB) … potencialmente: +100GB
27. Elastic Stack 5.x All new versions. All aligned.
28. 28 Logging Analytics e Machine Learning com Elastic Stack Beats => Elasticsearch => Kibana Beats Elasticsearch Kibana • Forma mais simples de começar. • Dados estruturados (ex. métricas) são indexados diretamente
29. 29 Logging Analytics e Machine Learning com Elastic Stack Beats => Elasticsearch => Kibana Beats Elasticsearch Kibana • Para dados não-estruturados (ex. logs) é preciso usar o Ingest Node. • Filebeat modules automatiza tudo! Ingest Node
30. 30 Logging Analytics e Machine Learning com Elastic Stack Beats => Elasticsearch => Kibana Beats Elasticsearch Kibana metricbeat.yml metricbeat.modules: - module: system metricsets: - cpu - filesystem - memory output.elasticsearch: hosts: ["elastic:9200"]
31. 31 Logging Analytics e Machine Learning com Elastic Stack Beats => Elasticsearch => Kibana Beats Elasticsearch Kibana Ingest Node filebeat.yml filebeat.modules: - module: nginx output.elasticsearch: hosts: ["elastic:9200"] Configura o pipeline
32. 32 Logging Analytics e Machine Learning com Elastic Stack Beats => Logstash => Elasticsearch => Kibana Beats Elasticsearch Logstash Kibana • Maior flexibilidade de processamento. • Logstash Persistent Queues (v5.4)
33. 33 Logging Analytics e Machine Learning com Elastic Stack Porém... Beats Logstash Elasticsearch Microservices! Attaaaaack!!!!
34. 34 Logging Analytics e Machine Learning com Elastic Stack Porém...
35. 35 Logging Analytics e Machine Learning com Elastic Stack Porém...
36. 36 Logging Analytics e Machine Learning com Elastic Stack Porém...
37. 37 Logging Analytics e Machine Learning com Elastic Stack Machine Learning
38. 38 Logging Analytics e Machine Learning com Elastic Stack Machine Learning
39. 39 Logging Analytics e Machine Learning com Elastic Stack Machine Learning
40. 40 Muito Obrigado! DÚVIDAS? Thiago Souza thiago@elastic.co Elastic{ON}'17 https://www.elastic.co/elasticon/conf/2017/sf Elastic Community https://discuss.elastic.co Elastic Careers https://elastic.co/careers