Tweet
Tweet

Slide 1

Slide 1 text

Awesome Logging Infrastructure Using The Stack Philipp Krenn@xeraa 1

Slide 2

Slide 2 text

Infrastructure | Developer Advocate 2

Slide 3

Slide 3 text

Disclaimer This is not a training https://www.elastic.co/training 3

Slide 4

Slide 4 text

Who is using Elasticsearch Logstash and Kibana Beats X-Pack 4

Slide 5

Slide 5 text

Starting point https://github.com/xeraa/vagrant-elastic-stack 5

Slide 6

Slide 6 text

USB Sticks 6

Slide 7

Slide 7 text

Box Vagrant Ansible Provisioner 7

Slide 8

Slide 8 text

Credentials vagrant & vagrant 8

Slide 9

Slide 9 text

SSH $ ssh [email protected] -p 2222 -o PreferredAuthentications=password Windows: http://www.putty.org 9

Slide 10

Slide 10 text

Ansible $ cd /elastic-stack/ $ ls 10

Slide 11

Slide 11 text

11

Slide 12

Slide 12 text

12

Slide 13

Slide 13 text

REST $ curl -XGET -u "elastic:changeme" http://localhost:9200/ 13

Slide 14

Slide 14 text

14

Slide 15

Slide 15 text

Login http://localhost:5601 Login: elastic & changeme 15

Slide 16

Slide 16 text

Overview GET / GET _cat GET _cat/indices?v 16

Slide 17

Slide 17 text

Insert data PUT /movies/movie/1 { "title": "The Godfather", "director": "Francis Ford Coppola", "year": 1972 } GET /movies/movie/1 GET /movies/_mapping 17

Slide 18

Slide 18 text

Replace data PUT /movies/movie/1 { "title": "The Godfather", "director": "Francis Ford Coppola", "year": 1972, "genres": ["Crime", "Drama"] } GET /movies/movie/1 18

Slide 19

Slide 19 text

More data PUT /movies/movie/2 { "title": "Lawrence of Arabia", "director": "David Lean", "year": 1962, "genres": ["Adventure", "Biography", "Drama"] } PUT /movies/movie/3 { "title": "Apocalypse Now", "director": "Francis Ford Coppola", "year": 1979, "genres": ["Drama", "War"] } 19

Slide 20

Slide 20 text

Query endpoints /_search /movies/_search /movies/movie/_search 20

Slide 21

Slide 21 text

Queries POST /movies/_search { "query": { "query_string": { "query": "ford" } } } 21

Slide 22

Slide 22 text

Filter POST /movies/_search { "query": { "bool": { "filter": { "term": { "year": 1972 } } } } } 22

Slide 23

Slide 23 text

Cleanup DELETE /movies GET /movies/movie/1 23

Slide 24

Slide 24 text

Insert test data $ java -jar /opt/injector.jar 100000 1000 24

Slide 25

Slide 25 text

Overview GET _cat/indices?v 25

Slide 26

Slide 26 text

Search GET /person/person/_search { "query": { "match": { "address.country": "Germany" } } } 26

Slide 27

Slide 27 text

More complex search GET /person/person/_search { "query": { "bool": { "must": [ { "match": { "address.country": "Germany" } }, { "range": { "dateOfBirth": { "from": "1970", "to": "1971" } } } ] } } } 27

Slide 28

Slide 28 text

Aggregation GET /person/person/_search { "size": 0, "aggs": { "by_country": { "terms": { "field": "address.country" } } } } 28

Slide 29

Slide 29 text

Index pattern in Kibana Index name person Time-field name dateOfBirth 29

Slide 30

Slide 30 text

Kibana Discover 30

Slide 31

Slide 31 text

Kibana Visualize Vertical bar chart with a date histogram Save 31

Slide 32

Slide 32 text

Kibana Visualize Pie chart split on the gender Save 32

Slide 33

Slide 33 text

Kibana Visualize Pie chart split on the country and then city Save 33

Slide 34

Slide 34 text

Kibana Visualize Tile map Save 34

Slide 35

Slide 35 text

Kibana Dashboard Combine all the saved visualizations 35

Slide 36

Slide 36 text

36

Slide 37

Slide 37 text

Filebeat 37

Slide 38

Slide 38 text

Filebeat Modules 38

Slide 39

Slide 39 text

/var/log/kibana/kibana.log JSON messages Limit Kibana view to its type 39

Slide 40

Slide 40 text

Kibana Discover Limit Kibana view to the nginx-access type 40

Slide 41

Slide 41 text

41

Slide 42

Slide 42 text

/var/log/syslog Logstash filter 42

Slide 43

Slide 43 text

Collect nginx in /etc/filebeat/filebeat.yml - input_type: log paths: - /var/log/nginx/access.log document_type: nginx-access 43

Slide 44

Slide 44 text

Disable Filebeat modules #filebeat.modules: #- module: system #- module: nginx 44

Slide 45

Slide 45 text

Enable the Logstash output output.logstash: hosts: ["localhost:5044"] username: "elastic" password: "changeme" 45

Slide 46

Slide 46 text

Disable the Elasticsearch output #output.elasticsearch: # hosts: ["localhost:9200"] # username: "elastic" # password: "changeme" 46

Slide 47

Slide 47 text

Filebeat Restart $ sudo service filebeat restart 47

Slide 48

Slide 48 text

Logstash pattern /opt/logstash/patterns/nginx 48

Slide 49

Slide 49 text

Logstash Filter $ sudo tee -a /etc/logstash/conf.d/11-nginx-filter.conf >/dev/null <<'EOF' filter { if [type] == "nginx-access" { grok { patterns_dir => ["/opt/logstash/patterns"] match => { "message" => "%{NGINXACCESS}" } } } } EOF $ sudo service logstash restart 49

Slide 50

Slide 50 text

Debug Logstash $ less /var/log/logstash/logstash-plain.log 50

Slide 51

Slide 51 text

Metricbeat 51

Slide 52

Slide 52 text

Metricbeat System 52

Slide 53

Slide 53 text

Metricbeat Service 53

Slide 54

Slide 54 text

Visual Builder Docker network traffic 54

Slide 55

Slide 55 text

Packetbeat 55

Slide 56

Slide 56 text

Protocols 56

Slide 57

Slide 57 text

Flows Application layer: Unsupported / encrypted (TLS) protocols IP / TCP / UDP Number of packets & bytes Retransmissions Temporal flow 57

Slide 58

Slide 58 text

Heartbeat 58

Slide 59

Slide 59 text

Heartbeat Open Heartbeat dashboard and set to auto-refresh $ sudo service nginx stop 59

Slide 60

Slide 60 text

Winlogbeat 60

Slide 61

Slide 61 text

libbeat https://github.com/elastic/beats/tree/master/generate/beat 61

Slide 62

Slide 62 text

62

Slide 63

Slide 63 text

X-Pack Monitoring Graph Reporting Alerting Machine Learning 63

Slide 64

Slide 64 text

X-Pack Basic 64

Slide 65

Slide 65 text

Monitoring GET _nodes/stats { "_nodes" : { "total" : 1, "successful" : 1, "failed" : 0 }, "cluster_name" : "elasticsearch", "nodes" : { "Koy5OmQ5RoiuFlL_TQ2Ngg" : { "timestamp" : 1492549940504, "name" : "Koy5OmQ", ... 65

Slide 66

Slide 66 text

66

Slide 67

Slide 67 text

Search Profiler GET /person "profile": true, "size": 0, "query": { ... } ... "searches" : [ { "query" : [ { "type" : "BooleanQuery", "description" : "...", "time" : "2.366942000ms", "time_in_nanos" : 2366942, "breakdown" : { "score" : 0, "build_scorer_count" : 5, ... 67

Slide 68

Slide 68 text

68

Slide 69

Slide 69 text

69

Slide 70

Slide 70 text

Install $ bin/elasticsearch-plugin install x-pack $ bin/kibana-plugin install x-pack $ bin/logstash-plugin install x-pack 70

Slide 71

Slide 71 text

GET _xpack/license { "license" : { "status" : "active", "uid" : "...", "type" : "trial", "issue_date" : "2017-04-18T20:41:50.233Z", "issue_date_in_millis" : 1492548110233, "expiry_date" : "2017-05-18T20:41:50.233Z", "expiry_date_in_millis" : 1495140110233, "max_nodes" : 1000, "issued_to" : "elasticsearch", "issuer" : "elasticsearch", "start_date_in_millis" : -1 } } 71

Slide 72

Slide 72 text

72

Slide 73

Slide 73 text

73

Slide 74

Slide 74 text

Install $ curl -XPUT -u elastic:changeme 'http://localhost:9200/_xpack/license?acknowledge=true' -H "Content-Type: application/json" -d @philipp-krenn-...-v5.json 74

Slide 75

Slide 75 text

GET _xpack/license { "license" : { "status" : "active", "uid" : "...", "type" : "basic", "issue_date" : "2017-04-18T00:00:00.000Z", "issue_date_in_millis" : 1492473600000, "expiry_date" : "2018-04-18T23:59:59.999Z", "expiry_date_in_millis" : 1524095999999, "max_nodes" : 100, "issued_to" : "Philipp Krenn (Elastic)", "issuer" : "Web Form", "start_date_in_millis" : 1492473600000 } } 75

Slide 76

Slide 76 text

Conclusion 76

Slide 77

Slide 77 text

77

Slide 78

Slide 78 text

78

Slide 79

Slide 79 text

79

Slide 80

Slide 80 text

80

Slide 81

Slide 81 text

Opbeat 81

Slide 82

Slide 82 text

PS: More Open Source https://www.elastic.co/training Development Support Consulting Production Support 82

Slide 83

Slide 83 text

Thanks! Questions? Philipp Krenn@xeraa 83

Slide 84

Slide 84 text

Container ship: https://flic.kr/p/hjxW62 https://flic.kr/p/2AzAVJ Wooden logs: https://flic.kr/p/9vvbKE Files: https://flic.kr/p/2EFcQ Metric: https://flic.kr/p/9g5h3f Packages: https://flic.kr/p/cJFDLN Windows: https://flic.kr/p/94Z6y Library: https://flic.kr/p/fiXcBj 84