Introduction to Docker Monitoring

Slide 1

Slide 1 text

Introduction to Docker Monitoring Matt Williams - @technovangelist - [email protected] Ilan Rabinovitch - @irabinovitch - [email protected]

Slide 2

Slide 2 text

Intro / Agenda • Datadog • Monitoring 101 • Docker Overview • Build the App Start by cloning the demo repository on github: git clone https://github.com/DataDog/dockercon- 2016-tutorial

Slide 3

Slide 3 text

Get started with the app 1. Open readme.md at the root of the repo 2. Install the Docker Toolbox 3. Create a docker-machine 4. Run the docker pull commands at the bottom of the file

Slide 4

Slide 4 text

•SaaS based infrastructure and app monitoring •Open Source Agent •Time series data (metrics and events) •Processing nearly a trillion data points per day •Intelligent Alerting •We’re hiring! (www.datadoghq.com/careers/) Datadog Overview

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Using and Sharing the same metrics and measurements across teams is key to avoiding misunderstandings.

Slide 8

Slide 8 text

Culture Automation Metrics Sharing Damon Edwards & John Willis DevOps Day LA

Slide 9

Slide 9 text

Collecting data is cheap; not having it when you need it can be expensive

Slide 10

Slide 10 text

Instrument all the things!

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

Photo : Bundesarchiv, B 145 Bild-F038788-0020 / Schaack, Lothar / CC-BY-SA 3.0

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

Examples: NGINX - Metrics Work Metrics: • Requests Per Second Dropped Connections • Request Time • Error Rates (4xx or 5xx) • Success (2xx) Resource Metrics: • Disk I/O • Memory • CPU • Queue Length

Slide 18

Slide 18 text

Examples: NGINX - Events • Configuration Change • Code Deployment • Service Started / Stopped • etc

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

Why Use Docker 1. Rapid application deployment 2. Portability across machines 3. Version control and component reuse 4. Sharing 5. Lightweight footprint and minimal overhead 6. Simplified maintenance Source: Redhat Documentation

Slide 22

Slide 22 text

Docker Architecture https://docs.docker.com/v1.9/engine/introduction/understanding-docker/

Slide 23

Slide 23 text

Docker Hub

Slide 24

Slide 24 text

Docker Hub

Slide 25

Slide 25 text

Docker Hub

Slide 26

Slide 26 text

Application Architecture Twitter API Custom App RabbitMQ Logstash Elastic Search Kibana Datadog Agent

Slide 27

Slide 27 text

Session Process 1. I will describe high level concepts, talk about commands and tools you will use 2. When we get to the slide: Step xx Hands On, it’s your turn a. Go to the folder for the step in the repo you cloned b. Open the readme file, this will be useful if you want to copy/paste commands c. I will be doing the steps on the big screen i. You can either just watch or follow along

Slide 28

Slide 28 text

Using the Docker commands docker run ... docker ps docker logs ... docker stop ... docker rm ...

Slide 29

Slide 29 text

Overview of Rabbit Message Broker software/ Message Oriented Middleware Implements Advanced Message Queuing Protocol (AMQP) https://www.cloudamqp.com/blog/2015-05-18-part1-rabbitmq-for- beginners-what-is-rabbitmq.html

Slide 30

Slide 30 text

Step 01 Hands On 1. docker run -d --hostname rabbit --name rabbit rabbitmq:3 2. docker ps 3. docker logs 4. docker stop 5. docker run --hostname rabbit --name rabbit rabbitmq:3 6. docker rm 7. docker run --hostname rabbit --name rabbit rabbitmq:3-mng

Slide 31

Slide 31 text

Using docker-compose commands docker-compose files are easier to share and easier to link containers docker run is easier to work with single containers docker-compose up docker-compose stop docker-compose rm

Slide 32

Slide 32 text

Our first docker-compose.yaml

Slide 33

Slide 33 text

Step 02 Hands On 1. docker-compose up 2. docker-compose stop [] 3. docker-compose rm

Slide 34

Slide 34 text

Step 03 Code Review - main.go

Slide 35

Slide 35 text

Step 03 Code Review - main.go

Slide 36

Slide 36 text

Step 03 Code Review - main.go

Slide 37

Slide 37 text

Step 03 Code Review - main.go

Slide 38

Slide 38 text

Step 03 - Dockerfile

Slide 39

Slide 39 text

Step 03 - docker-compose.yaml

Slide 40

Slide 40 text

Step 03 - docker-compose.yaml cont’d

Slide 41

Slide 41 text

Dockerize

Slide 42

Slide 42 text

Step 03 - Dockerfile2

Slide 43

Slide 43 text

Step 03 Hands On 1. Create a Twitter app, set the environment vars in your shell for the 4 keys 2. docker-compose up 3. docker-compose -f docker-compose2.yaml up 4. docker-compose -f docker-compose2.yaml build dc2016golangapp

Slide 44

Slide 44 text

Step 04 - Get Started with Datadog 1) If you do not have an account, sign up for free at http://datadog.com 2) Login and find your API key a) Select Integrations->APIs

Slide 45

Slide 45 text

RabbitMQ Metrics Work Metrics: Resource Metrics:

Slide 46

Slide 46 text

RabbitMQ Metrics Work Metrics: ● total messages in the queue ● Message Rate (published vs delivered) ● Active_consumers ● Redeliver Rate ● messages_unacknowledged Resource Metrics: ● Used file descriptors ● Memory used by queue ● Disk I/O ● Network I/O

Slide 47

Slide 47 text

Step 04 - docker-compose.yaml

Slide 48

Slide 48 text

Step 04 - docker-compose.yaml

Slide 49

Slide 49 text

Datadog Integrations

Slide 50

Slide 50 text

Integrations Find the tile you want to set up Follow the instructions

Slide 51

Slide 51 text

Step 04 Hands On 1. Create a Datadog trial account, set the API Key as an environment variable 2. docker-compose up 3. docker ps -f “name=_datadog”; docker exec -it bash 4. service datadog-agent info 5. docker exec -it `docker ps -f “name=_datadog” -q` service datadog-agent info 6. docker-compose exec datadog service datadog-agent info 7. docker-compose rm -f; docker-compose up

Slide 52

Slide 52 text

Logstash overview Collects, processes, and forwards events and log messages Part of the ELK (ElasticSearch, Logstash, Kibana) Stack “Provides the ability to filter, massage, and shape the data so that it’s easier to work with” (logz.io tutorial on logstash)

Slide 53

Slide 53 text

Logstash configuration input { rabbitmq { host => "rabbit" queue => "TweetQ" tags => ["test"]}} filter { grok { match => ["message", "^%{TIMESTAMP...REEDYDATA:tweet}"] } }

Slide 54

Slide 54 text

Logstash Dockerfile

Slide 55

Slide 55 text

docker-compose

Slide 56

Slide 56 text

Step 05 Hands On 1. docker-compose up 2. docker-compose build logstash 3. docker images 4. docker-compose -f docker-compose2.yaml up

Slide 57

Slide 57 text

Elasticsearch overview Elasticsearch is an open-source, broadly-distributable, readily-scalable, enterprise-grade search engine. Accessible through an extensive and elaborate API, Elasticsearch can power extremely fast searches that support your data discovery applications. https://qbox.io/blog/what-is-elasticsearch

Slide 58

Slide 58 text

Kibana overview We are using Kibana as a front end for Elasticsearch

Slide 59

Slide 59 text

Logstash Configuration

Slide 60

Slide 60 text

docker-compose

Slide 61

Slide 61 text

elastic.yaml

Slide 62

Slide 62 text

Step 06 Hands On 1. docker-compose up

Slide 63

Slide 63 text

Instrumenting the app You can instrument the key metrics and events from your app in Datadog Go to http://docs.datadoghq.com/libraries/ to see which libraries are available We will use https://github.com/DataDog/datadog-go in this session

Slide 64

Slide 64 text

Dogstatsd vs API Two choices for sending metrics to Datadog. RESTful API or Dogstatsd API sends each metric directly to Datadog Dogstatsd relays it via the Datadog agent

Slide 65

Slide 65 text

Available Libraries

Slide 66

Slide 66 text

Using the golang lib Create the client - statsd.New(host:8125) Specify Namespace and Tags Add a Count - client.Count(name, value, tags, rate) Add a Gauge - client.Gauge(name, value, tags, rate) Add an Event - client.Event(statsd.Event{title, text, time, host, aggkey, priority, source, alert type, tags)

Slide 67

Slide 67 text

main.go

Slide 68

Slide 68 text

main.go

Slide 69

Slide 69 text

main.go

Slide 70

Slide 70 text

Tags All the Way Down

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

No content

Slide 73

Slide 73 text

No content

Slide 74

Slide 74 text

No content

Slide 75

Slide 75 text

Demo - Creating Dashboards

Slide 76

Slide 76 text

Demo - Creating Monitors

Slide 77

Slide 77 text

Introduction to Docker Monitoring Matt Williams - @technovangelist - [email protected] Ilan Rabinovitch - @irabinovitch - [email protected]