Microservices
with Clojure
Microservices Meetup Berlin
Michael Vitz & Joy Clark
Slide 2
Slide 2 text
Who are we?
Michael Vitz
michael.vitz@innoq.com
@michaelvitz
Joy Clark
joy.clark@innoq.com
@iamjoyclark
Slide 3
Slide 3 text
~80 people in Germany (Monheim, Berlin,
Offenbach, Munich)
~20 people in Switzerland (Zürich, Cham)
Slide 4
Slide 4 text
Who are you?
Slide 5
Slide 5 text
Microservices?
Various aspects!
http://martinfowler.com/articles/microservices.html
Slide 6
Slide 6 text
“Do one thing and do it well!”
Slide 7
Slide 7 text
Easier to understand
$ m
a
n u
n
i
q
.
.
.
D
E
S
C
R
I
P
T
I
O
N
T
h
e u
n
i
q u
t
i
l
i
t
y r
e
a
d
s t
h
e s
p
e
c
i
f
i
e
d i
n
p
u
t
_
f
i
l
e c
o
m
p
a
r
i
n
g a
d
j
a
c
e
n
t l
i
n
e
s
, a
n
d
w
r
i
t
e
s a c
o
p
y o
f e
a
c
h u
n
i
q
u
e i
n
p
u
t l
i
n
e t
o t
h
e o
u
t
p
u
t
_
f
i
l
e
.
.
.
.
Slide 8
Slide 8 text
Understanding a monolith...
$ m
a
n m
y
-
h
u
g
e
-
m
o
n
o
l
i
t
h
<
e
n
d
l
e
s
s t
e
x
t
>
Slide 9
Slide 9 text
Composability
p
s -
e
f |
g
r
e
p p
r
o
g
r
a
m
-
i
-
d
o
n
t
-
l
i
k
e |
g
r
e
p -
v g
r
e
p |
a
w
k '
{
p
r
i
n
t $
2
}
' |
k
i
l
l
Slide 10
Slide 10 text
“Do one thing and do it well!”?
Yes!
Slide 11
Slide 11 text
“One thing” - which?
Slide 12
Slide 12 text
“Organize around
business capabilities!”
Slide 13
Slide 13 text
Don’t build thinks like “frontend”,
“api” and “database access”
–
instead, build “fulfillment”
and “payment”.
Slide 14
Slide 14 text
“Organize around
business capabilities!”?
Yes!
Slide 15
Slide 15 text
All of this could be done with
Libraries.
Would that be Microservices?
Slide 16
Slide 16 text
Code Abstractions
Function
(Class)
Library
Component
Service
Slide 17
Slide 17 text
Service
Communicates via network
Independently deployable
Technologically independent
Clear module boundaries
Slide 18
Slide 18 text
Microservices promise
very loose coupling
but
you have to handle
the complexity of
distributed systems!
Slide 19
Slide 19 text
This requires rules and
conventions across systems.
Slide 20
Slide 20 text
Good starting point:
http://12factor.net/
Slide 21
Slide 21 text
Rules & conventions
Project setup is done more than once
has to be easy
Deployment is done all the time
must be automated
Many apps are configured and write logs
should be simple for every app, common format,
Splunk or ELK
Slide 22
Slide 22 text
Rules & conventions
Many services are talking to each other
standardize on protocols and data formats
The network is not reliable
asynchronous programming and stability
patterns for decoupling and resilience
It’s easy to lose track of what’s actually happening
gather metrics for monitoring
Slide 23
Slide 23 text
Given such rules,
how to play nice using Clojure?
Slide 24
Slide 24 text
Clojure Crash Course
(
p
r
i
n
t
l
n "
H
e
l
l
o B
e
r
l
i
n
!
"
)
Lisp + JVM
Functional programming language
Simple programming model
Slide 25
Slide 25 text
Clojure Crash Course
(
p
r
i
n
t
l
n "
H
e
l
l
o B
e
r
l
i
n
!
"
)
Lisp + JVM
Functional programming language
Simple programming model
Slide 26
Slide 26 text
Clojure Crash Course
{
:
n
a
m
e "
C
l
o
j
u
r
e
"
:
f
e
a
t
u
r
e
s [
:
f
u
n
c
t
i
o
n
a
l :
j
v
m :
p
a
r
e
n
s
]
:
c
r
e
a
t
o
r "
R
i
c
h H
i
c
k
e
y
"
:
s
t
a
b
l
e
-
v
e
r
s
i
o
n {
:
n
u
m
b
e
r "
1
.
8
.
0
"
:
r
e
l
e
a
s
e "
2
0
1
6
/
0
1
/
1
9
"
}
}
Slide 27
Slide 27 text
Clojure Crash Course
(
d
e
f
n m
a
i
n [
a
r
g
s
] (
p
r
i
n
t
l
n "
H
e
l
l
o W
o
r
l
d
!
"
)
)
vs.
p
u
b
l
i
c s
t
a
t
i
c v
o
i
d m
a
i
n
(
S
t
r
i
n
g
[
] a
r
g
s
) {
S
y
s
t
e
m
.
o
u
t
.
p
r
i
n
t
l
n
(
"
H
e
l
l
o W
o
r
l
d
!
"
)
;
}
Slide 28
Slide 28 text
Clojure Crash Course
(
+ 1 2 3
)
> 6
(
:
c
i
t
y {
:
n
a
m
e "
i
n
n
o
Q
"
:
c
i
t
y "
B
e
r
l
i
n
"
}
)
> "
B
e
r
l
i
n
"
(
m
a
p i
n
c [
1 2 3
]
)
> (
2 3 4
)
Slide 29
Slide 29 text
Example
contacts
simple-calendar
noti cations
Slide 30
Slide 30 text
Project setup
derivative of by
“Garuda Takeoff” Simon_sees (CC BY 2.0)
Slide 31
Slide 31 text
Leiningen
Alternative to Maven
Describes Clojure project with generic data
structures
Maven repository compatibility
Offers plugin system
Slide 32
Slide 32 text
Leiningen
(
d
e
f
p
r
o
j
e
c
t s
i
m
p
l
e
-
c
a
l
e
n
d
a
r "
0
.
1
.
0
-
S
N
A
P
S
H
O
T
"
:
d
e
s
c
r
i
p
t
i
o
n "
s
i
m
p
l
e c
a
l
e
n
d
a
r a
p
p
"
:
u
r
l "
h
t
t
p
s
:
/
/
g
i
t
h
u
b
.
c
o
m
/
i
n
n
o
q
/
s
i
m
p
l
e
-
c
a
l
e
n
d
a
r
"
:
d
e
p
e
n
d
e
n
c
i
e
s [
[
o
r
g
.
c
l
o
j
u
r
e
/
c
l
o
j
u
r
e "
1
.
8
.
0
"
]
[
c
o
m
p
o
j
u
r
e "
1
.
4
.
0
"
]
[
r
i
n
g "
1
.
4
.
0
"
]
[
r
i
n
g
/
r
i
n
g
-
j
s
o
n "
0
.
4
.
0
"
]
.
.
.
[
e
n
v
i
r
o
n "
1
.
0
.
2
"
]
]
:
p
l
u
g
i
n
s [
[
l
e
i
n
-
r
i
n
g "
0
.
9
.
7
"
]
[
l
e
i
n
-
e
n
v
i
r
o
n "
1
.
0
.
2
"
]
]
:
r
i
n
g {
:
h
a
n
d
l
e
r s
i
m
p
l
e
-
c
a
l
e
n
d
a
r
.
c
o
r
e
/
w
e
b
a
p
p
:
i
n
i
t s
i
m
p
l
e
-
c
a
l
e
n
d
a
r
.
c
o
r
e
/
i
n
i
t
}
:
p
r
o
f
i
l
e
s {
:
u
b
e
r
j
a
r {
:
a
o
t :
a
l
l
}
}
)
Slide 33
Slide 33 text
Automated deployment
derivative of by
“Launch Button -- SMASH Rocket Club 5-9-09 4” Steven Depolo (CC BY 2.0)
Slide 34
Slide 34 text
Automated deployment
Not really a language thing ...
... but good tooling can help a lot
Leiningen makes building fat JARs easy
WAR files can be generated as well
Slide 35
Slide 35 text
Logging & configuration
by
“27 Jan 2007 (Flickr)” wonderferret (CC BY 2.0)
Slide 36
Slide 36 text
Logging
Don’t let every app handle log files
Just write everything to stdout
Let some external tool handle storage
Standardize on log format
Slide 37
Slide 37 text
org.clojure/tools.logging
Macros delegating to different implementations
Supports slf4j, Apache commons-logging, log4j
and java.util.logging
Logging levels: :
t
r
a
c
e :
d
e
b
u
g :
i
n
f
o :
w
a
r
n
:
e
r
r
o
r :
f
a
t
a
l
Slide 38
Slide 38 text
org.clojure/tools.logging
(
n
s s
i
m
p
l
e
-
c
a
l
e
n
d
a
r
.
c
o
r
e
(
:
r
e
q
u
i
r
e [
c
l
o
j
u
r
e
.
t
o
o
l
s
.
l
o
g
g
i
n
g :
a
s l
o
g
]
)
(
d
e
f
n u
p
d
a
t
e
-
c
o
n
t
a
c
t
! [
u
r
l
]
.
.
.
(
l
o
g
/
i
n
f
o "
U
p
d
a
t
e
d u
s
e
r
" i
d n
e
w
-
e
m
a
i
l
)
)
p
r
o
j
e
c
t
.
c
l
j
:
:
d
e
p
e
n
d
e
n
c
i
e
s [
[
o
r
g
.
c
l
o
j
u
r
e
/
t
o
o
l
s
.
l
o
g
g
i
n
g "
0
.
3
.
1
"
]
[
l
o
g
4
j "
1
.
2
.
1
7
" :
e
x
c
l
u
s
i
o
n
s [
j
a
v
a
x
.
m
a
i
l
/
m
a
i
l
j
a
v
a
x
.
j
m
s
/
j
m
s
c
o
m
.
s
u
n
.
j
d
m
k
/
j
m
x
t
o
o
l
s
c
o
m
.
s
u
n
.
j
m
x
/
j
m
x
r
i
]
]
[
o
r
g
.
s
l
f
4
j
/
s
l
f
4
j
-
l
o
g
4
j
1
2 "
1
.
7
.
1
8
"
]
.
.
.
]
Slide 39
Slide 39 text
org.clojure/tools.logging
l
o
g
4
j
.
p
r
o
p
e
r
t
i
e
s
:
l
o
g
4
j
.
r
o
o
t
L
o
g
g
e
r
=
I
N
F
O
, s
t
a
n
d
a
r
d
l
o
g
4
j
.
a
p
p
e
n
d
e
r
.
s
t
a
n
d
a
r
d
=
o
r
g
.
a
p
a
c
h
e
.
l
o
g
4
j
.
C
o
n
s
o
l
e
A
p
p
e
n
d
e
r
l
o
g
4
j
.
a
p
p
e
n
d
e
r
.
s
t
a
n
d
a
r
d
.
T
a
r
g
e
t
=
S
y
s
t
e
m
.
o
u
t
l
o
g
4
j
.
a
p
p
e
n
d
e
r
.
s
t
a
n
d
a
r
d
.
l
a
y
o
u
t
=
o
r
g
.
a
p
a
c
h
e
.
l
o
g
4
j
.
P
a
t
t
e
r
n
L
a
y
o
u
t
l
o
g
4
j
.
a
p
p
e
n
d
e
r
.
s
t
a
n
d
a
r
d
.
l
a
y
o
u
t
.
C
o
n
v
e
r
s
i
o
n
P
a
t
t
e
r
n
=
%
d
{
y
y
y
y
-
m
m
-
d
d H
H
:
m
m
:
s
s
,
S
S
S
} [
%
p
] %
c - %
m
%
n
s
t
d
o
u
t
:
2
0
1
5
-
1
1
-
1
8 1
3
:
1
1
:
5
4
,
4
6
8 [
I
N
F
O
] s
i
m
p
l
e
-
c
a
l
e
n
d
a
r
.
c
o
r
e - U
p
d
a
t
e
d u
s
e
r 5
f
5
6
5
0
4
0 e
v
e
@
e
x
a
m
p
l
e
.
o
r
g
2
0
1
5
-
1
1
-
1
8 1
3
:
1
1
:
5
4
,
4
7
6 [
I
N
F
O
] s
i
m
p
l
e
-
c
a
l
e
n
d
a
r
.
c
o
r
e - U
p
d
a
t
e
d u
s
e
r 7
8
6
4
9
4
e
f b
o
b
@
e
x
a
m
p
l
e
.
o
r
g
Slide 40
Slide 40 text
Configuration
Store configuration in the environment, not in the
codebase
Use mechanism that works for every application,
e.g., environment variables
Slide 41
Slide 41 text
environ
Manages environment settings following 12factor
Reads values from
Java system properties
Environment variables
.
l
e
i
n
-
e
n
v (created via Leiningen plugin from
p
r
o
f
i
l
e
s
.
c
l
j )
Slide 42
Slide 42 text
environ
(
d
e
f c
o
n
t
a
c
t
s
-
f
e
e
d (
e
n
v :
c
o
n
t
a
c
t
s
-
f
e
e
d
)
)
> j
a
v
a -
D
c
o
n
t
a
c
t
s
.
f
e
e
d
=
h
t
t
p
:
/
/
c
o
n
t
a
c
t
s
.
e
x
a
m
p
l
e
.
o
r
g
/
f
e
e
d
-
j
a
r s
t
a
n
d
a
l
o
n
e
.
j
a
r
> C
O
N
T
A
C
T
S
_
F
E
E
D
=
h
t
t
p
:
/
/
c
o
n
t
a
c
t
s
.
e
x
a
m
p
l
e
.
o
r
g
/
f
e
e
d
l
e
i
n r
i
n
g s
e
r
v
e
r
-
h
e
a
d
l
e
s
s
> l
e
i
n r
i
n
g s
e
r
v
e
r
-
h
e
a
d
l
e
s
s
p
r
o
f
i
l
e
s
.
c
l
j
:
{
:
d
e
v {
:
e
n
v {
:
c
o
n
t
a
c
t
s
-
f
e
e
d "
h
t
t
p
:
/
/
c
o
n
t
a
c
t
s
.
e
x
a
m
p
l
e
.
o
r
g
/
f
e
e
d
"
}
}
}
Slide 43
Slide 43 text
Protocols & data formats
derivative of by
“Free ports” Markus Reinhardt (CC BY 2.0)
Slide 44
Slide 44 text
Protocols & data formats
Avoid using different protocols everywhere
Standardize on the outside:
HTTP
JSON
XML
Atom
Slide 45
Slide 45 text
HTTP server
Ring for HTTP basics
Compojure for routing
Request & response are data
A web app is a function which takes a request and
returns a response
https://github.com/ring-clojure/ring/blob/master/SPEC
Slide 46
Slide 46 text
Ring
(
d
e
f e
x
a
m
p
l
e
-
r
e
q
u
e
s
t {
:
u
r
i "
/
c
o
n
t
a
c
t
s
"
:
r
e
q
u
e
s
t
-
m
e
t
h
o
d :
g
e
t
:
h
e
a
d
e
r
s {
"
A
c
c
e
p
t
" "
t
e
x
t
/
p
l
a
i
n
"
}
}
)
(
d
e
f
n e
x
a
m
p
l
e
-
a
p
p [
r
e
q
]
{
:
s
t
a
t
u
s 2
0
0
:
b
o
d
y (
s
t
r "
H
e
l
l
o a
t " (
:
u
r
i r
e
q
)
)
}
)
(
e
x
a
m
p
l
e
-
a
p
p e
x
a
m
p
l
e
-
r
e
q
u
e
s
t
)
> {
:
s
t
a
t
u
s 2
0
0
:
b
o
d
y "
H
e
l
l
o a
t /
c
o
n
t
a
c
t
s
"
}
Slide 47
Slide 47 text
Compojure
(
d
e
f
r
o
u
t
e
s c
o
n
t
a
c
t
s
(
G
E
T "
/
c
o
n
t
a
c
t
s
/
:
i
d
" [
i
d
]
(
g
e
t
-
c
o
n
t
a
c
t i
d
)
)
(
P
U
T "
/
c
o
n
t
a
c
t
s
/
:
i
d
" [
i
d :
a
s r
e
q
u
e
s
t
]
(
u
p
d
a
t
e
-
c
o
n
t
a
c
t
! i
d (
:
b
o
d
y r
e
q
u
e
s
t
)
)
)
(
P
O
S
T "
/
c
o
n
t
a
c
t
s
" r
e
q
u
e
s
t
(
a
d
d
-
c
o
n
t
a
c
t
! (
:
b
o
d
y r
e
q
u
e
s
t
)
)
)
(
G
E
T "
/
f
e
e
d
" [
]
{
:
s
t
a
t
u
s 2
0
0
:
h
e
a
d
e
r
s {
"
C
o
n
t
e
n
t
-
T
y
p
e
" "
a
p
p
l
i
c
a
t
i
o
n
/
a
t
o
m
+
x
m
l
"
}
:
b
o
d
y (
c
o
n
t
a
c
t
s
-
f
e
e
d
)
}
)
)
)
Slide 48
Slide 48 text
Frameworks
Quickly get a web app up and running.
Duct
Luminus
Pedestal
Modularity
tesla-microservices
Slide 49
Slide 49 text
org.clojure/data.json
Clojure data structures are JSON superset
(
j
s
o
n
/
w
r
i
t
e
-
s
t
r {
:
n
a
m
e "
A
l
i
c
e M
i
l
l
e
r
"
:
t
e
a
m
s [
"
T
e
a
m A
"
, "
T
e
a
m B
"
]
}
)
> "
{
\
"
n
a
m
e
\
"
:
\
"
A
l
i
c
e M
i
l
l
e
r
\
"
,
\
"
t
e
a
m
s
\
"
:
[
\
"
T
e
a
m A
\
"
,
\
"
T
e
a
m B
\
"
]
}
"
(
j
s
o
n
/
r
e
a
d
-
s
t
r "
{
\
"
i
d
\
"
: 1
2
3
,
\
"
n
a
m
e
\
"
: \
"
A
l
i
c
e M
i
l
l
e
r
\
"
}
"
:
k
e
y
-
f
n k
e
y
w
o
r
d
)
> {
:
i
d 1
2
3
, :
n
a
m
e "
A
l
i
c
e M
i
l
l
e
r
"
}
Slide 50
Slide 50 text
HTML
HTML is represented with generic data structures
This allows processing HTML with standard Clojure
functions
There are different formats, a popular one is called
“hiccup”
Slide 51
Slide 51 text
HTML
<
h
t
m
l
>
<
b
o
d
y
>
<
h
1
>
h
e
l
l
o
!
<
/
h
1
>
<
i
m
g s
r
c
=
"
s
o
m
e
.
j
p
g
" /
>
<
i
m
g s
r
c
=
"
a
n
o
t
h
e
r
.
j
p
g
" /
>
<
/
b
o
d
y
>
<
/
h
t
m
l
>
[
:
h
t
m
l {
}
[
:
b
o
d
y {
}
[
:
h
1 {
} "
h
e
l
l
o
!
"
]
[
:
i
m
g {
:
s
r
c "
s
o
m
e
.
j
p
g
"
}
]
[
:
i
m
g {
:
s
r
c "
a
n
o
t
h
e
r
.
j
p
g
"
}
]
]
]
Slide 52
Slide 52 text
Atom
XML format for representing feeds of data
Useful for doing pub/sub without middleware
Slide 53
Slide 53 text
Creating Atom Feeds
(
d
e
f
n e
n
t
r
y [
e
v
e
n
t
]
[
:
e
n
t
r
y
[
:
t
i
t
l
e (
-
> e
v
e
n
t :
t
y
p
e n
a
m
e
)
]
[
:
u
p
d
a
t
e
d (
:
t
i
m
e
s
t
a
m
p e
v
e
n
t
)
]
[
:
a
u
t
h
o
r [
:
n
a
m
e "
c
o
n
t
a
c
t
s s
e
r
v
i
c
e
"
]
]
[
:
i
d (
s
t
r "
u
r
n
:
c
o
n
t
a
c
t
s
:
f
e
e
d
:
e
v
e
n
t
:
" (
:
i
d e
v
e
n
t
)
)
]
[
:
c
o
n
t
e
n
t {
:
t
y
p
e "
j
s
o
n
"
} (
j
s
o
n
/
g
e
n
e
r
a
t
e
-
s
t
r
i
n
g e
v
e
n
t
)
]
]
)
(
d
e
f
n a
t
o
m
-
f
e
e
d [
e
v
e
n
t
s u
r
l
]
(
c
l
o
j
u
r
e
.
d
a
t
a
.
x
m
l
/
e
m
i
t
-
s
t
r
(
c
l
o
j
u
r
e
.
d
a
t
a
.
x
m
l
/
s
e
x
p
-
a
s
-
e
l
e
m
e
n
t
[
:
f
e
e
d {
:
x
m
l
n
s "
h
t
t
p
:
/
/
w
w
w
.
w
3
.
o
r
g
/
2
0
0
5
/
A
t
o
m
"
}
[
:
i
d "
u
r
n
:
c
o
n
t
a
c
t
s
:
f
e
e
d
"
]
[
:
u
p
d
a
t
e
d (
-
> e
v
e
n
t
s l
a
s
t :
t
i
m
e
s
t
a
m
p
)
]
[
:
t
i
t
l
e {
:
t
y
p
e "
t
e
x
t
"
} "
c
o
n
t
a
c
t
s e
v
e
n
t
s
"
]
[
:
l
i
n
k {
:
r
e
l "
s
e
l
f
" :
h
r
e
f u
r
l
}
]
(
m
a
p e
n
t
r
y e
v
e
n
t
s
)
]
)
)
)
Slide 54
Slide 54 text
adamwynne/feedparser-clj
Retrieves and Parses RSS/Atom feeds
(
d
e
f f
(
f
e
e
d
p
a
r
s
e
r
/
p
a
r
s
e
-
f
e
e
d "
h
t
t
p
s
:
/
/
w
w
w
.
i
n
n
o
q
.
c
o
m
/
d
e
/
p
o
d
c
a
s
t
.
r
s
s
"
)
)
(
:
t
i
t
l
e f
)
> "
i
n
n
o
Q P
o
d
c
a
s
t
"
(
c
o
u
n
t (
:
e
n
t
r
i
e
s f
)
)
> 1
8
Library for consuming feeds based on feedparser:
Feedworker
Slide 55
Slide 55 text
Lightweight messaging
Just a side note:
Kafka (clj-kafka)
RabbitMQ (Langohr)
...
Slide 56
Slide 56 text
Problems of distributed systems:
Stability
derivative of by
“the Jenga” Ed Garcia (CC BY 2.0)
Slide 57
Slide 57 text
Cascading failures
dash-
board
app
web
service
data-
base
Slide 58
Slide 58 text
Cascading failures
dash-
board
app
web
service
data-
base
Slide 59
Slide 59 text
Cascading failures
dash-
board
app
web
service
data-
base
Slide 60
Slide 60 text
Cascading failures
dash-
board
app
web
service
data-
base
Slide 61
Slide 61 text
Cascading failures
dash-
board
app
web
service
data-
base
Slide 62
Slide 62 text
How to achieve stability?
Slide 63
Slide 63 text
Bulkheads
derivative of by (Public Domain)
“Titanic sinking” Willy Stöwer
Slide 64
Slide 64 text
Asynchronous communication
Helps to decouple and separate different
application parts
E-mail notifications do not need to be sent
synchronously
Slide 65
Slide 65 text
org.clojure/core.async
Supports asynchronous programming and
communications
Messages can be sent to and read from channels
Channels can be buffered or unbuffered
Blocking and non-blocking operations possible
Slide 66
Slide 66 text
org.clojure/core.async
(
d
e
f n
o
t
i
f
i
c
a
t
i
o
n
s (
c
h
a
n 1
0
0
0
)
)
Slide 67
Slide 67 text
org.clojure/core.async
(
d
e
f n
o
t
i
f
i
c
a
t
i
o
n
s (
c
h
a
n 1
0
0
0
)
)
Slide 68
Slide 68 text
org.clojure/core.async
(
d
e
f n
o
t
i
f
i
c
a
t
i
o
n
s (
c
h
a
n 1
0
0
0
)
)
(
d
e
f
n s
e
n
d
-
n
o
t
i
f
i
c
a
t
i
o
n [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
(
g
o (
>
! n
o
t
i
f
i
c
a
t
i
o
n
s {
:
e
m
a
i
l e
m
a
i
l
:
e
v
e
n
t
-
l
i
n
k e
v
e
n
t
-
l
i
n
k
}
)
)
)
Slide 69
Slide 69 text
org.clojure/core.async
(
d
e
f n
o
t
i
f
i
c
a
t
i
o
n
s (
c
h
a
n 1
0
0
0
)
)
(
d
e
f
n s
e
n
d
-
n
o
t
i
f
i
c
a
t
i
o
n [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
(
g
o (
>
! n
o
t
i
f
i
c
a
t
i
o
n
s {
:
e
m
a
i
l e
m
a
i
l
:
e
v
e
n
t
-
l
i
n
k e
v
e
n
t
-
l
i
n
k
}
)
)
)
Slide 70
Slide 70 text
org.clojure/core.async
(
d
e
f n
o
t
i
f
i
c
a
t
i
o
n
s (
c
h
a
n 1
0
0
0
)
)
(
d
e
f
n s
e
n
d
-
n
o
t
i
f
i
c
a
t
i
o
n [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
(
g
o (
>
! n
o
t
i
f
i
c
a
t
i
o
n
s {
:
e
m
a
i
l e
m
a
i
l
:
e
v
e
n
t
-
l
i
n
k e
v
e
n
t
-
l
i
n
k
}
)
)
)
Slide 71
Slide 71 text
org.clojure/core.async
(
d
e
f n
o
t
i
f
i
c
a
t
i
o
n
s (
c
h
a
n 1
0
0
0
)
)
(
d
e
f
n s
e
n
d
-
n
o
t
i
f
i
c
a
t
i
o
n [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
(
g
o (
>
! n
o
t
i
f
i
c
a
t
i
o
n
s {
:
e
m
a
i
l e
m
a
i
l
:
e
v
e
n
t
-
l
i
n
k e
v
e
n
t
-
l
i
n
k
}
)
)
)
(
d
e
f
n n
o
t
i
f
y
-
u
s
e
r [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
.
.
.
)
(
d
e
f
n s
t
a
r
t
-
n
o
t
i
f
i
e
r [
]
(
g
o
-
l
o
o
p [
m
e
s
s
a
g
e (
<
! n
o
t
i
f
i
c
a
t
i
o
n
s
)
]
(
n
o
t
i
f
y
-
u
s
e
r (
:
e
m
a
i
l m
e
s
s
a
g
e
) (
:
e
v
e
n
t
-
l
i
n
k m
e
s
s
a
g
e
)
)
(
r
e
c
u
r (
<
! n
o
t
i
f
i
c
a
t
i
o
n
s
)
)
)
)
Slide 72
Slide 72 text
org.clojure/core.async
(
d
e
f n
o
t
i
f
i
c
a
t
i
o
n
s (
c
h
a
n 1
0
0
0
)
)
(
d
e
f
n s
e
n
d
-
n
o
t
i
f
i
c
a
t
i
o
n [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
(
g
o (
>
! n
o
t
i
f
i
c
a
t
i
o
n
s {
:
e
m
a
i
l e
m
a
i
l
:
e
v
e
n
t
-
l
i
n
k e
v
e
n
t
-
l
i
n
k
}
)
)
)
(
d
e
f
n n
o
t
i
f
y
-
u
s
e
r [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
.
.
.
)
(
d
e
f
n s
t
a
r
t
-
n
o
t
i
f
i
e
r [
]
(
g
o
-
l
o
o
p [
m
e
s
s
a
g
e (
<
! n
o
t
i
f
i
c
a
t
i
o
n
s
)
]
(
n
o
t
i
f
y
-
u
s
e
r (
:
e
m
a
i
l m
e
s
s
a
g
e
) (
:
e
v
e
n
t
-
l
i
n
k m
e
s
s
a
g
e
)
)
(
r
e
c
u
r (
<
! n
o
t
i
f
i
c
a
t
i
o
n
s
)
)
)
)
Slide 73
Slide 73 text
org.clojure/core.async
(
d
e
f n
o
t
i
f
i
c
a
t
i
o
n
s (
c
h
a
n 1
0
0
0
)
)
(
d
e
f
n s
e
n
d
-
n
o
t
i
f
i
c
a
t
i
o
n [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
(
g
o (
>
! n
o
t
i
f
i
c
a
t
i
o
n
s {
:
e
m
a
i
l e
m
a
i
l
:
e
v
e
n
t
-
l
i
n
k e
v
e
n
t
-
l
i
n
k
}
)
)
)
(
d
e
f
n n
o
t
i
f
y
-
u
s
e
r [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
.
.
.
)
(
d
e
f
n s
t
a
r
t
-
n
o
t
i
f
i
e
r [
]
(
g
o
-
l
o
o
p [
m
e
s
s
a
g
e (
<
! n
o
t
i
f
i
c
a
t
i
o
n
s
)
]
(
n
o
t
i
f
y
-
u
s
e
r (
:
e
m
a
i
l m
e
s
s
a
g
e
) (
:
e
v
e
n
t
-
l
i
n
k m
e
s
s
a
g
e
)
)
(
r
e
c
u
r (
<
! n
o
t
i
f
i
c
a
t
i
o
n
s
)
)
)
)
Slide 74
Slide 74 text
Circuit Breaker
derivative of by
“switch-fuse” Mark_K_ (CC BY 2.0)
Slide 75
Slide 75 text
com.netflix.hystrix/hystrix-clj
Idiomatic Clojure wrapper for Hystrix
(
h
y
s
t
r
i
x
/
d
e
f
c
o
m
m
a
n
d n
o
t
i
f
y
-
u
s
e
r [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
(
c
l
i
e
n
t
/
p
o
s
t n
o
t
i
f
i
c
a
t
i
o
n
-
s
e
r
v
i
c
e
{
:
c
o
n
t
e
n
t
-
t
y
p
e :
j
s
o
n
:
b
o
d
y .
.
. }
)
)
Will throw exception in case of timeout or other
failure
Slide 76
Slide 76 text
com.netflix.hystrix/hystrix-clj
; r
e
t
u
r
n
s t
r
u
e i
f s
u
c
e
s
s
f
u
l
, f
a
l
s
e i
f c
i
r
c
u
i
t
-
b
r
e
a
k
e
r o
p
e
n
(
h
y
s
t
r
i
x
/
d
e
f
c
o
m
m
a
n
d n
o
t
i
f
y
-
u
s
e
r
{
:
h
y
s
t
r
i
x
/
f
a
l
l
b
a
c
k
-
f
n n
o
t
i
f
y
-
f
a
l
l
b
a
c
k
}
[
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
(
c
l
i
e
n
t
/
p
o
s
t n
o
t
i
f
i
c
a
t
i
o
n
-
s
e
r
v
i
c
e
{
:
c
o
n
t
e
n
t
-
t
y
p
e :
j
s
o
n
:
b
o
d
y .
.
. }
)
t
r
u
e
)
; r
e
t
u
r
n
s f
a
l
s
e i
f c
i
r
c
u
i
t
-
b
r
e
a
k
e
r o
p
e
n
(
d
e
f
n n
o
t
i
f
y
-
f
a
l
l
b
a
c
k [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
(
l
e
t [
i
s
O
p
e
n (
.
i
s
C
i
r
c
u
i
t
B
r
e
a
k
e
r
O
p
e
n h
y
s
t
r
i
x
/
*
c
o
m
m
a
n
d
*
)
]
; a
d
d m
e
s
s
a
g
e t
o q
u
e
u
e a
g
a
i
n
(
s
e
n
d
-
n
o
t
i
f
i
c
a
t
i
o
n e
m
a
i
l e
v
e
n
t
-
l
i
n
k
)
(
n
o
t i
s
O
p
e
n
)
)
)
Slide 77
Slide 77 text
com.netflix.hystrix/hystrix-clj
(
d
e
f
n s
t
a
r
t
-
n
o
t
i
f
i
e
r [
]
(
g
o
-
l
o
o
p [
m
e
s
s
a
g
e (
<
! n
o
t
i
f
i
c
a
t
i
o
n
s
)
]
(
i
f
-
n
o
t (
n
o
t
i
f
y
-
u
s
e
r (
:
e
m
a
i
l m
e
s
s
a
g
e
)
(
:
e
v
e
n
t
-
l
i
n
k m
e
s
s
a
g
e
)
)
(
d
o
(
l
o
g
/
e
r
r
o
r "
C
a
n
n
o
t r
e
a
c
h n
o
t
i
f
i
c
a
t
i
o
n s
e
r
v
i
c
e
"
"
- w
i
l
l w
a
i
t u
n
t
i
l n
e
x
t t
r
y
"
)
(
<
! (
t
i
m
e
o
u
t 5
0
0
0
)
)
)
)
(
r
e
c
u
r (
<
! n
o
t
i
f
i
c
a
t
i
o
n
s
)
)
)
)
hystrix-event-stream-clj available as well
Slide 78
Slide 78 text
Monitoring
derivative of by
“Space Shuttle Endeavour's Control Panels” Steve Jurvetson (CC BY 2.0)
Slide 79
Slide 79 text
Questions
How many HTTP errors are occurring?
Are database queries failing?
Is that backend service slow again?
How many jobs are in the queue?
Slide 80
Slide 80 text
Metrics
Logging provides a stream of events
Metrics provide aggregated state
Popular library: Dropwizard Metrics
Two steps: collect metrics, then publish them
Slide 81
Slide 81 text
Gauges
by
“Old gauge, old style” speredenn (CC BY 2.0)
Slide 82
Slide 82 text
Gauges
Current state of one single value
Number of jobs in the queue
Some configured value
Ratio of cache hits to misses
Slide 83
Slide 83 text
Gauges
(
d
e
f m
e
t
r
i
c
s
-
r
e
g
i
s
t
r
y (
n
e
w
-
r
e
g
i
s
t
r
y
)
)
(
g
a
u
g
e
-
f
n m
e
t
r
i
c
s
-
r
e
g
i
s
t
r
y "
j
o
b
s
-
r
e
a
d
y
"
#
(
q
u
e
r
y d
a
t
a
b
a
s
e
"
s
e
l
e
c
t c
o
u
n
t
(
*
)
f
r
o
m j
o
b
s
w
h
e
r
e s
t
a
t
u
s = '
r
e
a
d
y
'
"
)
)
Slide 84
Slide 84 text
Counters
derivative of by
“The US National Debt clock / counter, New York” Ben Sutherland (CC BY 2.0)
Slide 85
Slide 85 text
Counters
Single numeric value
For example, number of logged in users
Slide 86
Slide 86 text
Counters
(
d
e
f l
o
g
g
e
d
-
i
n
-
u
s
e
r
s (
c
o
u
n
t
e
r m
e
t
r
i
c
s
-
r
e
g
i
s
t
r
y "
l
o
g
g
e
d
-
i
n
-
u
s
e
r
s
"
)
)
(
P
O
S
T "
/
l
o
g
i
n
" [
n
a
m
e p
w
d
]
.
.
.
(
i
n
c
! l
o
g
g
e
d
-
i
n
-
u
s
e
r
s
)
.
.
.
)
(
P
O
S
T "
/
l
o
g
o
u
t
" [
n
a
m
e
]
.
.
.
(
d
e
c
! l
o
g
g
e
d
-
i
n
-
u
s
e
r
s
)
.
.
.
)
Slide 87
Slide 87 text
Histograms
derivative of by
“Number of cat posts on Metafilter per month, as a percentage of the total number of posts” Steven Taschuk (CC BY 2.0)
Slide 88
Slide 88 text
Histograms
Distribution of numerical data (min, max, mean,
standard deviation, quantiles)
For example, number of search results
Different value “reservoirs”, e.g.,
Entire application lifetime
Last N searches
Last N minutes
Slide 89
Slide 89 text
Histograms
(
d
e
f n
u
m
b
e
r
-
o
f
-
r
e
s
u
l
t
s
(
h
i
s
t
o
g
r
a
m m
e
t
r
i
c
s
-
r
e
g
i
s
t
r
y "
n
u
m
b
e
r
-
o
f
-
s
e
a
r
c
h
-
r
e
s
u
l
t
s
"
)
)
(
d
e
f
n s
e
a
r
c
h [
q
u
e
r
y
]
(
l
e
t [
r
e
s
u
l
t
s (
e
x
e
c
u
t
e q
u
e
r
y
)
]
(
u
p
d
a
t
e
! n
u
m
b
e
r
-
o
f
-
r
e
s
u
l
t
s (
c
o
u
n
t r
e
s
u
l
t
s
)
)
r
e
s
u
l
t
s
)
)
Slide 90
Slide 90 text
Meters
derivative of by
“Hosp_1011” Justin Taylor (CC BY 2.0)
Slide 91
Slide 91 text
Meters
Rate of an event (per second)
Total count of events
Average rate over application lifetime
Rate in the last 1, 5 and 15 minutes
For example, incoming requests
Slide 92
Slide 92 text
Meters
(
d
e
f i
n
c
-
r
e
q
u
e
s
t
s
-
m
e
t
e
r
(
m
e
t
e
r
s
/
m
e
t
e
r m
e
t
r
i
c
s
-
r
e
g
i
s
t
r
y "
i
n
c
o
m
i
n
g
-
r
e
q
u
e
s
t
s
-
m
e
t
e
r
"
)
)
(
d
e
f
n r
e
q
u
e
s
t
s
-
m
e
t
e
r [
h
a
n
d
l
e
r
]
(
f
n [
r
e
q
]
(
m
e
t
e
r
s
/
m
a
r
k
! i
n
c
-
r
e
q
u
e
s
t
s
-
m
e
t
e
r
)
(
h
a
n
d
l
e
r r
e
q
)
)
)
Slide 93
Slide 93 text
Timers
by
“Stopwatch” William Warby (CC BY 2.0)
Slide 94
Slide 94 text
Timers
Histogram & meter
Histogram of duration of an activity, meter of
occurrence of the activity
For example, specific type of database query
Slide 95
Slide 95 text
Timers
(
d
e
f i
n
c
-
r
e
q
u
e
s
t
s
-
t
i
m
e
r
(
t
i
m
e
r
s
/
t
i
m
e
r
! m
e
t
r
i
c
s
-
r
e
g
i
s
t
r
y "
i
n
c
o
m
i
n
g
-
r
e
q
u
e
s
t
s
-
t
i
m
e
r
"
)
)
(
d
e
f
n r
e
q
u
e
s
t
s
-
t
i
m
e
r [
h
a
n
d
l
e
r
]
(
f
n [
r
e
q
]
(
t
i
m
e
r
s
/
t
i
m
e
! i
n
c
-
r
e
q
u
e
s
t
s
-
t
i
m
e
r
(
h
a
n
d
l
e
r r
e
q
)
)
)
)
Slide 96
Slide 96 text
Reporting Metrics
Slide 97
Slide 97 text
Reporting
Metrics stores current state in its registry
Should be reported to external tool
Visualization
Historical data
Reporters for console, JMX, Ganglia, Graphite, CSV
etc. included
Slide 98
Slide 98 text
Conclusion
Organize around business capabilities
Rules & conventions needed
Standardize interfaces, logging and configuration
Different solutions to improve stability
Monitor your systems
Good support in Clojure
Slide 99
Slide 99 text
Example Microservices
Simple Calendar:
Simple Contacts:
https://github.com/innoq/simple-
calendar
https://github.com/innoq/simple-
contacts