Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Monitor MongoDB with the Elastic Stack

Philipp Krenn
September 25, 2017

Monitor MongoDB with the Elastic Stack

How to use the Elastic Stack (previously called ELK Stack) to monitor logs is widely known. But it can also give you a complete picture of your MongoDB installation:
* System metrics: Keep track of network traffic and system load.
* Logs: Collect and parse MongoDB logs.
* MongoDB metrics: Gather the most relevant attributes with the dedicated Metricbeat module.
* Queries: Monitor your queries on the wire with Packetbeat.

And we will do all of that together since it is so easy and quick to set up.

Philipp Krenn

September 25, 2017
Tweet

More Decks by Philipp Krenn

Other Decks in Programming

Transcript

  1. 5

  2. 6

  3. 7

  4. 14

  5. 16

  6. 18

  7. 22

  8. /etc/filebeat/filebeat.yml filebeat.prospectors: - input_type: log paths: - /var/log/java-app/*.log document_type: java

    multiline.pattern: '^[[:space:]]' multiline.negate: false multiline.match: after 26
  9. 29

  10. 33

  11. Console Pattern in a single line PUT _ingest/pipeline/mongodb_log { "description":

    "Ingest pipeline for MongoDB logs", "processors": [ { "grok": { "field": "message", "patterns": [ "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{MONGO3_SEVERITY:mongodb.severity} %{SPACE}%{MONGO3_COMPONENT:mongodb.component}%{SPACE} (?:[%{DATA:mongodb.context}])?%{SPACE}%{GREEDYDATA:message}" ] } } ] } 35
  12. 40

  13. 41

  14. /etc/logstash/conf.d/00-mongodb- input.conf input { mongodb { uri => 'mongodb://127.0.0.1:27017/POCDB' placeholder_db_dir

    => '/var/local/logstash-mongodb/' collection => 'system.profile' batch_size => 500 generateId => true } } 43
  15. /etc/logstash/conf.d/20-elasticsearch- output.conf output { elasticsearch { hosts => ["localhost:9200"] manage_template

    => false index => "mongodb-%{+YYYY.MM.dd}" document_type => "slowlog" user => "{{ elastic_user }}" password => "{{ elastic_password }}" } } 44
  16. 55

  17. 57

  18. Flows Application layer: Unsupported / encrypted (TLS) protocols IP /

    TCP / UDP Number of packets & bytes Retransmissions Temporal flow 60
  19. 64

  20. 71

  21. 72

  22. 74

  23. 77

  24. 81

  25. 82

  26. 83