VISUALIZATION OF DOCKER
METRICS
2015/12/09 by janus_wel
Slide 2
Slide 2 text
WHO?
高木健介 janus_wel
archtect/programmer for full-stack
Slide 3
Slide 3 text
PURPOSE OF
VISUALIZATION ?
Slide 4
Slide 4 text
TO CHECK LIFE OR DEATH ?
should be covered by orchestration
ex) consul
Slide 5
Slide 5 text
TO USE RESOURCES EFFICIENTLY !!
figure out bottlenecks
decision a number of containers on a machine
Slide 6
Slide 6 text
TO MAKE DOCUMENTS
for customers
for executives
Slide 7
Slide 7 text
TO FOLLOW THE PDCA CYCLE
figure out influences of changes
Slide 8
Slide 8 text
COLLECTING METRICS USUALLY
is important to trace back
Slide 9
Slide 9 text
TOOLS
Docker remote API
InfluxDB
Slide 10
Slide 10 text
DOCKER REMOTE API
https://docs.docker.com/engine/reference/api/docker_remote_
Slide 11
Slide 11 text
REST LIKE API
The API tends to be REST, but for some
complex commands, like attach or pull, the
HTTP connection is hijacked to transport
STDOUT, STDIN, and STDERR.
Slide 12
Slide 12 text
AVAILABILITY
maybe, all operations in commands are available.
https://docs.docker.com/engine/reference/api/docker_remote_
Slide 13
Slide 13 text
FOR EXAMPLE
List containers
Create a container
Start a container
Stop a container
Remove a container
List Images
Build image from a Dockerfile
Create an image
Display system-wide information
Show the docker version information
...
Slide 14
Slide 14 text
REQUIREMENTS
needs some settings for access control
$DOCKER_OPTS in /etc/init/docker.conf on Ubuntu
$OPTIONS in /etc/sysconfig/docker on other Linux
if 192.0.2.1 is a client,
D
O
C
K
E
R
_
O
P
T
S
=
"
-
H 1
9
2
.
0
.
2
.
1
:
2
3
7
6 -
H u
n
i
x
:
/
/
/
v
a
r
/
r
u
n
/
d
o
c
k
e
r
.
s
o
c
k
"
O
P
T
I
O
N
S
=
"
-
H 1
9
2
.
0
.
2
.
1
:
2
3
7
6 -
H u
n
i
x
:
/
/
/
v
a
r
/
r
u
n
/
d
o
c
k
e
r
.
s
o
c
k
"
https://docs.docker.com/engine/userguide/basics/#bind-
docker-to-another-host-port-or-a-unix-socket
Slide 15
Slide 15 text
NOTE
make sure to hear from unix domain socket.
O
P
T
I
O
N
S
=
"
-
H 1
9
2
.
0
.
2
.
1
:
2
3
7
6 -
H u
n
i
x
:
/
/
/
v
a
r
/
r
u
n
/
d
o
c
k
e
r
.
s
o
c
k
"
# ^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
or, docker commands are not available.
Slide 16
Slide 16 text
FOR DOCKER MACHINE
machines created by docker-machine needs https
by default, in your ~/.docker/machine/machines/(machine
name)/
cert.pem
key.pem
this path is set by "$(docker-machine env (machine name))"
as $DOCKER_CERT_PATH
Slide 17
Slide 17 text
INFO
w
g
e
t h
t
t
p
:
/
/
(
m
a
c
h
i
n
e i
p
)
:
2
3
7
6
/
i
n
f
o -
O - -
q | j
q .
for docker-machine
w
g
e
t \
-
-
n
o
-
c
h
e
c
k
-
c
e
r
t
i
f
i
c
a
t
e \
-
-
c
e
r
t
i
f
i
c
a
t
e
=
$
{
D
O
C
K
E
R
_
C
E
R
T
_
P
A
T
H
}
/
c
e
r
t
.
p
e
m \
-
-
p
r
i
v
a
t
e
-
k
e
y
=
$
{
D
O
C
K
E
R
_
C
E
R
T
_
P
A
T
H
}
/
k
e
y
.
p
e
m \
h
t
t
p
s
:
/
/
$
(
d
o
c
k
e
r
-
m
a
c
h
i
n
e i
p (
m
a
c
h
i
n
e n
a
m
e
)
)
:
2
3
7
6
/
i
n
f
o \
-
O - -
q | j
q .
"curl" can't handle non-checking local cert, so use "wget"
Slide 18
Slide 18 text
No content
Slide 19
Slide 19 text
STATS API
to get container's metrics, use stats API
GET /containers/(id)/stats
https://docs.docker.com/engine/reference/api/docker_remote_
container-stats-based-on-resource-usage
Slide 20
Slide 20 text
No content
Slide 21
Slide 21 text
INFLUXDB
https://influxdb.com/
Slide 22
Slide 22 text
No content
Slide 23
Slide 23 text
POINTS
schemaless
SQL like query
REST like interface
Slide 24
Slide 24 text
SCHEMALESS
there is no need to define somethings
when writing data, InfluxDB needs only
TABLE NAME and VALUE
these are optional
timestamp
tags
https://influxdb.com/docs/v0.9/concepts/schema_and_data_la
Slide 25
Slide 25 text
SQL LIKE QUERY
to read
S
E
L
E
C
T * F
R
O
M c
p
u
_
l
o
a
d
_
s
h
o
r
t W
H
E
R
E h
o
s
t
=
s
e
r
v
e
r
0
1 A
N
D r
e
g
i
o
n
=
u
s
-
w
e
s
t
-
- ^
^
^
^
^
^
^
^
^
^
^
^
^
^ ^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
^
-
- t
a
b
l
e n
a
m
e t
a
g
s
https://influxdb.com/docs/v0.9/query_language/data_explor
Slide 26
Slide 26 text
REST LIKE INTERFACE
# C
R
E
A
T
E D
A
T
A
B
A
S
E
c
u
r
l -
G h
t
t
p
:
/
/
l
o
c
a
l
h
o
s
t
:
8
0
8
6
/
q
u
e
r
y -
-
d
a
t
a
-
u
r
l
e
n
c
o
d
e "
q
=
C
R
E
A
T
E D
A
T
A
B
A
S
E m
y
d
b
"
# I
N
S
E
R
T
c
u
r
l -
i -
X
P
O
S
T '
h
t
t
p
:
/
/
l
o
c
a
l
h
o
s
t
:
8
0
8
6
/
w
r
i
t
e
?
d
b
=
m
y
d
b
' -
-
d
a
t
a
-
b
i
n
a
r
y '
c
p
u
_
l
o
a
d
_
# S
E
L
E
C
T
c
u
r
l -
G '
h
t
t
p
:
/
/
l
o
c
a
l
h
o
s
t
:
8
0
8
6
/
q
u
e
r
y
?
d
b
=
m
y
d
b
' -
-
d
a
t
a
-
u
r
l
e
n
c
o
d
e "
q
=
S
E
L
E
C
T v
a
l
u
https://influxdb.com/docs/v0.9/guides/writing_data.html
https://influxdb.com/docs/v0.9/guides/querying_data.html
Slide 27
Slide 27 text
SYSTEM
pull archtecture
collect metrics minutely
run with various OSs
Linux
Windows
others can run Docker
for now, test only in Linux...
Slide 28
Slide 28 text
No content
Slide 29
Slide 29 text
METRICS COLLECTOR
get metrics from Docker remote API and insert them
InfluxDB
in Ruby
requires gem "clockwork"
runs a collection minutely
Slide 30
Slide 30 text
VISUALIZE
in HTML and JavaScript
use chart.js
use some EcmaScript 2015 features
"let" statement
Promise
Object.assign
http://www.chartjs.org/
now available with only Google Chrome
Slide 31
Slide 31 text
DEMO
http://52.193.71.92/
Slide 32
Slide 32 text
METRICS
legend description type
rx_bytes received bytes counter
tx_bytes transferred bytes counter
total_cpu_usage cpu usage time % counter
PROBLEMS
load
stats API is heavy
pull architecture
calling API is proportional to number of containers
handling mass containers
id is not human-friendly
Slide 35
Slide 35 text
PUSH ARCHITECTURE AND AGENT
to collect with low load, we need inspect /sys
c
a
t /
s
y
s
/
f
s
/
c
g
r
o
u
p
/
m
e
m
o
r
y
/
d
o
c
k
e
r
/
<
c
o
n
t
a
i
n
e
r
-
i
d
>
/
m
e
m
o
r
y
.
s
t
a
t
c
a
t /
s
y
s
/
f
s
/
c
g
r
o
u
p
/
c
p
u
a
c
c
t
/
d
o
c
k
e
r
/
<
c
o
n
t
a
i
n
e
r
-
i
d
>
/
c
p
u
a
c
c
t
.
s
t
a
t
pull architecture is unfit to cat local files
agent is needed
https://docs.docker.com/engine/articles/runmetrics/
Slide 36
Slide 36 text
AFTER
increase collectable metrices
I don't know Docker metrics means what is...
agent-ize collector