Monitor MongoDB with the Elastic Stack

How to use the Elastic Stack (previously called ELK Stack) to monitor logs is widely known. But it can also give you a complete picture of your MongoDB installation:
* System metrics: Keep track of network traffic and system load.
* Logs: Collect and parse MongoDB logs.
* MongoDB metrics: Gather the most relevant attributes with the dedicated Metricbeat module.
* Queries: Monitor your queries on the wire with Packetbeat.

And we will do all of that together since it is so easy and quick to set up.

Transcript

1. Monitor with the Stack Philipp Krenn
   
   
   
   
   @xeraa 1
2. Infrastructure | Developer Advocate 2
3. Disclaimer This is not a training https://www.elastic.co/training 3
4. Who Is Using Elasticsearch Logstash and Kibana Beats 4
8. Starting Point https://github.com/xeraa/mongodb-monitoring 8
9. USB Sticks 9
10. Box Vagrant Ansible Provisioner 10
11. Credentials vagrant & vagrant 11
12. SSH $ ssh vagrant@127.0.0.1 -p 2222 -o PreferredAuthentications=password Windows: http://www.putty.org 12
13. Ansible $ cd /elastic-stack/ $ ls 13
15. REST $ curl -XGET -u "elastic:changeme" http://localhost:9200/ 15
17. Login http://localhost:5601 elastic & changeme 17
19. Filebeat 19
20. Filebeat Modules 20
21. System Dashboards 21
23. MongoDB Logs /var/log/mongodb/mongod.log 23
24. /etc/filebeat/filebeat.yml filebeat.prospectors: - input_type: log paths: - /var/log/mongodb/mongod.log document_type: mongodb 24
25. PS: Multiline Logs 25
26. /etc/filebeat/filebeat.yml filebeat.prospectors: - input_type: log paths: - /var/log/java-app/*.log document_type: java multiline.pattern: '^[[:space:]]' multiline.negate: false multiline.match: after 26
27. Test $ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20 27
28. Kibana Discover Limit Kibana view to the mongodb type 28
30. Grok Patterns https://github.com/logstash-plugins/logstash-patterns- core/blob/master/patterns/grok-patterns 30
31. MongoDB Patterns https://github.com/logstash-plugins/logstash-patterns- core/blob/master/patterns/mongodb 31
32. Building Patterns Grokdebug in Kibana 5.5+ (X-Pack Basic) https://grokdebug.herokuapp.com 32
34. /etc/filebeat/filebeat.yml output.elasticsearch: hosts: ["localhost:9200"] username: "{{ elastic_user }}" password: "{{ elastic_password }}" pipeline: "mongodb_log" 34
35. Console Pattern in a single line PUT _ingest/pipeline/mongodb_log { "description": "Ingest pipeline for MongoDB logs", "processors": [ { "grok": { "field": "message", "patterns": [ "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{MONGO3_SEVERITY:mongodb.severity} %{SPACE}%{MONGO3_COMPONENT:mongodb.component}%{SPACE} (?:[%{DATA:mongodb.context}])?%{SPACE}%{GREEDYDATA:message}" ] } } ] } 35
36. Filebeat Restart $ sudo service filebeat restart 36
37. Test $ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20 37
38. Refresh Field List Management → Index Patterns → filebeat-* → Refresh field list 38
39. Visualize mongodb.component of log events 39
42. Fetch the Slow Log 42
43. /etc/logstash/conf.d/00-mongodb- input.conf input { mongodb { uri => 'mongodb://127.0.0.1:27017/POCDB' placeholder_db_dir => '/var/local/logstash-mongodb/' collection => 'system.profile' batch_size => 500 generateId => true } } 43
44. /etc/logstash/conf.d/20-elasticsearch- output.conf output { elasticsearch { hosts => ["localhost:9200"] manage_template => false index => "mongodb-%{+YYYY.MM.dd}" document_type => "slowlog" user => "{{ elastic_user }}" password => "{{ elastic_password }}" } } 44
45. Logstash Restart $ sudo service logstash restart 45
46. Debug Logstash $ less /var/log/logstash/logstash-plain.log 46
47. Doesn't Work Getting data from system collections https://github.com/phutchins/logstash-input-mongodb/issues/8 47
48. Extension Idea Build an exec filter to run slow queries against MongoDB's .explain() 48
49. Metricbeat 49
50. Metricbeat System 50
51. Metricbeat Service 51
52. /etc/metricbeat/metricbeat.yml - module: mongodb metricsets: ["dbstats", "status"] hosts: ["localhost:27017"] 52
53. Metricbeat Restart $ sudo service metricbeat restart 53
54. Test $ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20 54
56. Visual Builder mongodb.status.network.in.bytes vs mongodb.status.network.out.bytes 56
58. Packetbeat 58
59. Protocols 59
60. Flows Application layer: Unsupported / encrypted (TLS) protocols IP / TCP / UDP Number of packets & bytes Retransmissions Temporal flow 60
61. /etc/packetbeat/packetbeat.yml packetbeat.protocols.mongodb: ports: [27017] 61
62. Packetbeat Restart $ sudo service packetbeat restart 62
63. Test $ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20 63
65. Heartbeat 65
66. Heartbeat ICMP, TCP, HTTP, HTTPS 66
67. /etc/heartbeat/heartbeat.yml heartbeat.monitors: - type: tcp hosts: ["127.0.0.1:27017"] schedule: '@every 10s' 67
68. Heartbeat Restart $ sudo service heartbeat restart 68
69. Test $ sudo service mongod stop $ sudo service mongod start 69
70. Visualize Up or down and TCP response times 70
73. Dashboard Combining visualizations 73
75. Winlogbeat 75
76. libbeat https://github.com/elastic/beats/tree/master/generate/beat 76
78. X-Pack Security Monitoring Graph Reporting Alerting Machine Learning 78
79. X-Pack Basic 79
80. Conclusion 80
84. Thanks! Questions? Philipp Krenn
    
    
    
    
    @xeraa 84