Tweet
Tweet

Slide 1

Slide 1 text

1 Thiago Souza Support Engineer 5 de Agosto, 2017 Logging Analytics e Machine Learning com Elastic Stack

Slide 2

Slide 2 text

2 Um pouco sobre mim • Desenvolvedor há mais de 10 anos • Trabalho com Elasticsearch desde 2010 (em produção desde 2013) • Support Engineer @ Elastic

Slide 3

Slide 3 text

2014 Millions of Downloads 40. 100. 2016 2015 2012 2013 Cumulative downloads of the Elastic Stack (Elasticsearch, Kibana, Beats, Logstash) and X-Pack 3 100M+ Downloads 3,000+ Customers 92,000+ Community

Slide 4

Slide 4 text

4 4 Global Customer Base Tech Finance Telco Consumer

Slide 5

Slide 5 text

5 Logging Analytics e Machine Learning com Elastic Stack Alguns anos atrás...

Slide 6

Slide 6 text

6 Logging Analytics e Machine Learning com Elastic Stack Alguns anos atrás...

Slide 7

Slide 7 text

7 Logging Analytics e Machine Learning com Elastic Stack Alguns anos atrás... AppServer

Slide 8

Slide 8 text

8 Logging Analytics e Machine Learning com Elastic Stack Alguns anos atrás... AppServer DBServer

Slide 9

Slide 9 text

9 Logging Analytics e Machine Learning com Elastic Stack Alguns anos atrás... AppServer DBServer FTP app.log (20MB)

Slide 10

Slide 10 text

10 Logging Analytics e Machine Learning com Elastic Stack Alguns anos atrás... at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:548) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:595) Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket WARNING: Restarting endpoint Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket SEVERE: Endpoint null shutdown due to exception: java.net.BindException: Address already in use: JVM_Bind:8080 java.net.BindException: Address already in use: JVM_Bind:8080 at org.apache.tomcat.util.net.PoolTcpEndpoint.initEndpoint(PoolTcpEndpoint.java:264) at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:441) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:548) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:595) Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run SEVERE: Caught exception (java.lang.ThreadDeath) executing org.apache.tomcat.util.net.TcpWorkerThread@12c5431, terminating thread Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket WARNING: Restarting endpoint Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket SEVERE: Endpoint null shutdown due to exception: java.net.BindException: Address already in use: JVM_Bind:8080 java.net.BindException: Address already in use: JVM_Bind:8080 at org.apache.tomcat.util.net.PoolTcpEndpoint.initEndpoint(PoolTcpEndpoint.java:264) at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:441) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:548) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)

Slide 11

Slide 11 text

11 Logging Analytics e Machine Learning com Elastic Stack Alguns anos atrás... at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:548) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:595) Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket WARNING: Restarting endpoint Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket SEVERE: Endpoint null shutdown due to exception: java.net.BindException: Address already in use: JVM_Bind:8080 java.net.BindException: Address already in use: JVM_Bind:8080 at org.apache.tomcat.util.net.PoolTcpEndpoint.initEndpoint(PoolTcpEndpoint.java:264) at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:441) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:548) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:595) Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run SEVERE: Caught exception (java.lang.ThreadDeath) executing org.apache.tomcat.util.net.TcpWorkerThread@12c5431, terminating thread Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket WARNING: Restarting endpoint Dec 7, 2004 7:09:35 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket SEVERE: Endpoint null shutdown due to exception: java.net.BindException: Address already in use: JVM_Bind:8080 java.net.BindException: Address already in use: JVM_Bind:8080 at org.apache.tomcat.util.net.PoolTcpEndpoint.initEndpoint(PoolTcpEndpoint.java:264) at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:441) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:548) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)

Slide 12

Slide 12 text

12 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer

Slide 13

Slide 13 text

13 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer

Slide 14

Slide 14 text

14 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer

Slide 15

Slide 15 text

15 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer

Slide 16

Slide 16 text

16 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer

Slide 17

Slide 17 text

17 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer Microservices! Attaaaaack!!!!

Slide 18

Slide 18 text

18 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer Microservices! Attaaaaack!!!! app.log (20MB)

Slide 19

Slide 19 text

19 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer Microservices! Attaaaaack!!!! app.log (20MB) service-1.log (10GB) server-20.log (30GB) microsrv-100.log (31GB) service-202.log (80GB) … potencialmente: +100GB

Slide 20

Slide 20 text

20 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer Microservices! Attaaaaack!!!! app.log (20MB) service-1.log (10GB) server-20.log (30GB) microsrv-100.log (31GB) service-202.log (80GB) … potencialmente: +100GB

Slide 21

Slide 21 text

21 Logging Analytics e Machine Learning com Elastic Stack Hoje em dia... AppServer DBServer Microservices! Attaaaaack!!!! app.log (20MB) service-1.log (10GB) server-20.log (30GB) microsrv-100.log (31GB) service-202.log (80GB) … potencialmente: +100GB

Slide 22

Slide 22 text

22 Logging Analytics e Machine Learning com Elastic Stack Centralized Logging Centralized Logging

Slide 23

Slide 23 text

23 Logging Analytics e Machine Learning com Elastic Stack Centralized Logging Elastic Stack

Slide 24

Slide 24 text

24 Logging Analytics e Machine Learning com Elastic Stack Centralized Logging app.log (20MB) service-1.log (10GB) server-20.log (30GB) microsrv-100.log (31GB) service-202.log (80GB) … potencialmente: +100GB

Slide 25

Slide 25 text

25 Logging Analytics e Machine Learning com Elastic Stack Centralized Logging app.log (20MB) service-1.log (10GB) server-20.log (30GB) microsrv-100.log (31GB) service-202.log (80GB) … potencialmente: +100GB

Slide 26

Slide 26 text

26 Logging Analytics e Machine Learning com Elastic Stack Centralized Logging app.log (20MB) service-1.log (10GB) server-20.log (30GB) microsrv-100.log (31GB) service-202.log (80GB) … potencialmente: +100GB

Slide 27

Slide 27 text

Elastic Stack 5.x All new versions. All aligned.

Slide 28

Slide 28 text

28 Logging Analytics e Machine Learning com Elastic Stack Beats => Elasticsearch => Kibana Beats Elasticsearch Kibana • Forma mais simples de começar. • Dados estruturados (ex. métricas) são indexados diretamente

Slide 29

Slide 29 text

29 Logging Analytics e Machine Learning com Elastic Stack Beats => Elasticsearch => Kibana Beats Elasticsearch Kibana • Para dados não-estruturados (ex. logs) é preciso usar o Ingest Node. • Filebeat modules automatiza tudo! Ingest Node

Slide 30

Slide 30 text

30 Logging Analytics e Machine Learning com Elastic Stack Beats => Elasticsearch => Kibana Beats Elasticsearch Kibana metricbeat.yml metricbeat.modules: - module: system metricsets: - cpu - filesystem - memory output.elasticsearch: hosts: ["elastic:9200"]

Slide 31

Slide 31 text

31 Logging Analytics e Machine Learning com Elastic Stack Beats => Elasticsearch => Kibana Beats Elasticsearch Kibana Ingest Node filebeat.yml filebeat.modules: - module: nginx output.elasticsearch: hosts: ["elastic:9200"] Configura o pipeline

Slide 32

Slide 32 text

32 Logging Analytics e Machine Learning com Elastic Stack Beats => Logstash => Elasticsearch => Kibana Beats Elasticsearch Logstash Kibana • Maior flexibilidade de processamento. • Logstash Persistent Queues (v5.4)

Slide 33

Slide 33 text

33 Logging Analytics e Machine Learning com Elastic Stack Porém... Beats Logstash Elasticsearch Microservices! Attaaaaack!!!!

Slide 34

Slide 34 text

34 Logging Analytics e Machine Learning com Elastic Stack Porém...

Slide 35

Slide 35 text

35 Logging Analytics e Machine Learning com Elastic Stack Porém...

Slide 36

Slide 36 text

36 Logging Analytics e Machine Learning com Elastic Stack Porém...

Slide 37

Slide 37 text

37 Logging Analytics e Machine Learning com Elastic Stack Machine Learning

Slide 38

Slide 38 text

38 Logging Analytics e Machine Learning com Elastic Stack Machine Learning

Slide 39

Slide 39 text

39 Logging Analytics e Machine Learning com Elastic Stack Machine Learning

Slide 40

Slide 40 text

40 Muito Obrigado! DÚVIDAS? Thiago Souza thiago@elastic.co Elastic{ON}'17 https://www.elastic.co/elasticon/conf/2017/sf Elastic Community https://discuss.elastic.co Elastic Careers https://elastic.co/careers