Introduction to Docker Monitoring

Datadog's Docker Monitoring Tutorial from DockerCon 2016. Presented by Matt Williams

Transcript

1. Introduction to Docker Monitoring Matt Williams - @technovangelist - mattw@datadoghq.com Ilan Rabinovitch - @irabinovitch - ilan@datadoghq.com
2. Intro / Agenda • Datadog • Monitoring 101 • Docker Overview • Build the App Start by cloning the demo repository on github: git clone https://github.com/DataDog/dockercon- 2016-tutorial
3. Get started with the app 1. Open readme.md at the root of the repo 2. Install the Docker Toolbox 3. Create a docker-machine 4. Run the docker pull commands at the bottom of the file
4. •SaaS based infrastructure and app monitoring •Open Source Agent •Time series data (metrics and events) •Processing nearly a trillion data points per day •Intelligent Alerting •We’re hiring! (www.datadoghq.com/careers/) Datadog Overview
7. Using and Sharing the same metrics and measurements across teams is key to avoiding misunderstandings.
8. Culture Automation Metrics Sharing Damon Edwards & John Willis DevOps Day LA
9. Collecting data is cheap; not having it when you need it can be expensive
10. Instrument all the things!
14. Photo : Bundesarchiv, B 145 Bild-F038788-0020 / Schaack, Lothar / CC-BY-SA 3.0
17. Examples: NGINX - Metrics Work Metrics: • Requests Per Second Dropped Connections • Request Time • Error Rates (4xx or 5xx) • Success (2xx) Resource Metrics: • Disk I/O • Memory • CPU • Queue Length
18. Examples: NGINX - Events • Configuration Change • Code Deployment • Service Started / Stopped • etc
21. Why Use Docker 1. Rapid application deployment 2. Portability across machines 3. Version control and component reuse 4. Sharing 5. Lightweight footprint and minimal overhead 6. Simplified maintenance Source: Redhat Documentation
22. Docker Architecture https://docs.docker.com/v1.9/engine/introduction/understanding-docker/
23. Docker Hub
24. Docker Hub
25. Docker Hub
26. Application Architecture Twitter API Custom App RabbitMQ Logstash Elastic Search Kibana Datadog Agent
27. Session Process 1. I will describe high level concepts, talk about commands and tools you will use 2. When we get to the slide: Step xx Hands On, it’s your turn a. Go to the folder for the step in the repo you cloned b. Open the readme file, this will be useful if you want to copy/paste commands c. I will be doing the steps on the big screen i. You can either just watch or follow along
28. Using the Docker commands docker run ... docker ps docker logs ... docker stop ... docker rm ...
29. Overview of Rabbit Message Broker software/ Message Oriented Middleware Implements Advanced Message Queuing Protocol (AMQP) https://www.cloudamqp.com/blog/2015-05-18-part1-rabbitmq-for- beginners-what-is-rabbitmq.html
30. Step 01 Hands On 1. docker run -d --hostname rabbit --name rabbit rabbitmq:3 2. docker ps 3. docker logs <container-id> 4. docker stop <container-id> 5. docker run --hostname rabbit --name rabbit rabbitmq:3 6. docker rm <container-id> 7. docker run --hostname rabbit --name rabbit rabbitmq:3-mng
31. Using docker-compose commands docker-compose files are easier to share and easier to link containers docker run is easier to work with single containers docker-compose up docker-compose stop docker-compose rm
32. Our first docker-compose.yaml
33. Step 02 Hands On 1. docker-compose up 2. docker-compose stop [<service>] 3. docker-compose rm
34. Step 03 Code Review - main.go
35. Step 03 Code Review - main.go
36. Step 03 Code Review - main.go
37. Step 03 Code Review - main.go
38. Step 03 - Dockerfile
39. Step 03 - docker-compose.yaml
40. Step 03 - docker-compose.yaml cont’d
41. Dockerize
42. Step 03 - Dockerfile2
43. Step 03 Hands On 1. Create a Twitter app, set the environment vars in your shell for the 4 keys 2. docker-compose up 3. docker-compose -f docker-compose2.yaml up 4. docker-compose -f docker-compose2.yaml build dc2016golangapp
44. Step 04 - Get Started with Datadog 1) If you do not have an account, sign up for free at http://datadog.com 2) Login and find your API key a) Select Integrations->APIs
45. RabbitMQ Metrics Work Metrics: Resource Metrics:
46. RabbitMQ Metrics Work Metrics: • total messages in the queue • Message Rate (published vs delivered) • Active_consumers • Redeliver Rate • messages_unacknowledged Resource Metrics: • Used file descriptors • Memory used by queue • Disk I/O • Network I/O
47. Step 04 - docker-compose.yaml
48. Step 04 - docker-compose.yaml
49. Datadog Integrations
50. Integrations Find the tile you want to set up Follow the instructions
51. Step 04 Hands On 1. Create a Datadog trial account, set the API Key as an environment variable 2. docker-compose up 3. docker ps -f “name=_datadog”; docker exec -it <id> bash 4. service datadog-agent info 5. docker exec -it `docker ps -f “name=_datadog” -q` service datadog-agent info 6. docker-compose exec datadog service datadog-agent info 7. docker-compose rm -f; docker-compose up
52. Logstash overview Collects, processes, and forwards events and log messages Part of the ELK (ElasticSearch, Logstash, Kibana) Stack “Provides the ability to filter, massage, and shape the data so that it’s easier to work with” (logz.io tutorial on logstash)
53. Logstash configuration input { rabbitmq { host => "rabbit" queue => "TweetQ" tags => ["test"]}} filter { grok { match => ["message", "^%{TIMESTAMP...REEDYDATA:tweet}"] } }
54. Logstash Dockerfile
55. docker-compose
56. Step 05 Hands On 1. docker-compose up 2. docker-compose build logstash 3. docker images 4. docker-compose -f docker-compose2.yaml up
57. Elasticsearch overview Elasticsearch is an open-source, broadly-distributable, readily-scalable, enterprise-grade search engine. Accessible through an extensive and elaborate API, Elasticsearch can power extremely fast searches that support your data discovery applications. https://qbox.io/blog/what-is-elasticsearch
58. Kibana overview We are using Kibana as a front end for Elasticsearch
59. Logstash Configuration
60. docker-compose
61. elastic.yaml
62. Step 06 Hands On 1. docker-compose up
63. Instrumenting the app You can instrument the key metrics and events from your app in Datadog Go to http://docs.datadoghq.com/libraries/ to see which libraries are available We will use https://github.com/DataDog/datadog-go in this session
64. Dogstatsd vs API Two choices for sending metrics to Datadog. RESTful API or Dogstatsd API sends each metric directly to Datadog Dogstatsd relays it via the Datadog agent
65. Available Libraries
66. Using the golang lib Create the client - statsd.New(host:8125) Specify Namespace and Tags Add a Count - client.Count(name, value, tags, rate) Add a Gauge - client.Gauge(name, value, tags, rate) Add an Event - client.Event(statsd.Event{title, text, time, host, aggkey, priority, source, alert type, tags)
67. main.go
68. main.go
69. main.go
70. Tags All the Way Down
75. Demo - Creating Dashboards
76. Demo - Creating Monitors
77. Introduction to Docker Monitoring Matt Williams - @technovangelist - mattw@datadoghq.com Ilan Rabinovitch - @irabinovitch - ilan@datadoghq.com