Microservices_with_Clojure.pdf

Microservices
with Clojure
Microservices Meetup Berlin
Michael Vitz & Joy Clark

View Slide

Who are we?
Michael Vitz
[email protected]
@michaelvitz
Joy Clark
[email protected]
@iamjoyclark

View Slide

~80 people in Germany (Monheim, Berlin,
Oﬀenbach, Munich)
~20 people in Switzerland (Zürich, Cham)

View Slide

Who are you?

View Slide

Microservices?
Various aspects!
http://martinfowler.com/articles/microservices.html

View Slide

“Do one thing and do it well!”

View Slide

Easier to understand
$ m
a
n u
n
i
q
.
.
.
D
E
S
C
R
I
P
T
I
O
N
T
h
e u
n
i
q u
t
i
l
i
t
y r
e
a
d
s t
h
e s
p
e
c
i
f
i
e
d i
n
p
u
t
_
f
i
l
e c
o
m
p
a
r
i
n
g a
d
j
a
c
e
n
t l
i
n
e
s
, a
n
d
w
r
i
t
e
s a c
o
p
y o
f e
a
c
h u
n
i
q
u
e i
n
p
u
t l
i
n
e t
o t
h
e o
u
t
p
u
t
_
f
i
l
e
.
.
.
.

View Slide

Understanding a monolith...
$ m
a
n m
y
-
h
u
g
e
-
m
o
n
o
l
i
t
h
<
e
n
d
l
e
s
s t
e
x
t
>

View Slide

Composability
p
s -
e
f |
g
r
e
p p
r
o
g
r
a
m
-
i
-
d
o
n
t
-
l
i
k
e |
g
r
e
p -
v g
r
e
p |
a
w
k '
{
p
r
i
n
t $
2
}
' |
k
i
l
l

View Slide

“Do one thing and do it well!”?
Yes!

View Slide

“One thing” - which?

View Slide

“Organize around
business capabilities!”

View Slide

Don’t build thinks like “frontend”,
“api” and “database access”
–
instead, build “fulﬁllment”
and “payment”.

View Slide

“Organize around
business capabilities!”?
Yes!

View Slide

All of this could be done with
Libraries.
Would that be Microservices?

View Slide

Code Abstractions
Function
(Class)
Library
Component
Service

View Slide

Service
Communicates via network
Independently deployable
Technologically independent
Clear module boundaries

View Slide

Microservices promise
very loose coupling
but
you have to handle
the complexity of
distributed systems!

View Slide

This requires rules and
conventions across systems.

View Slide

Good starting point:
http://12factor.net/

View Slide

Rules & conventions
Project setup is done more than once
 has to be easy
Deployment is done all the time
 must be automated
Many apps are conﬁgured and write logs
 should be simple for every app, common format,
Splunk or ELK

View Slide

Rules & conventions
Many services are talking to each other
 standardize on protocols and data formats
The network is not reliable
 asynchronous programming and stability
patterns for decoupling and resilience
It’s easy to lose track of what’s actually happening
 gather metrics for monitoring

View Slide

Given such rules,
how to play nice using Clojure?

View Slide

Clojure Crash Course
(
p
r
i
n
t
l
n "
H
e
l
l
o B
e
r
l
i
n
!
"
)
Lisp + JVM
Functional programming language
Simple programming model

View Slide

{
:
n
a
m
e "
C
l
o
j
u
r
e
"
:
f
e
a
t
u
r
e
s [
:
f
u
n
c
t
i
o
n
a
l :
j
v
m :
p
a
r
e
n
s
]
:
c
r
e
a
t
o
r "
R
i
c
h H
i
c
k
e
y
"
:
s
t
a
b
l
e
-
v
e
r
s
i
o
n {
:
n
u
m
b
e
r "
1
.
8
.
0
"
:
r
e
l
e
a
s
e "
2
0
1
6
/
0
1
/
1
9
"
}
}

View Slide

(
d
e
f
n m
a
i
n [
a
r
g
s
] (
p
r
i
n
t
l
n "
H
e
l
l
o W
o
r
l
d
!
"
)
)
vs.
p
u
b
l
i
c s
t
a
t
i
c v
o
i
d m
a
i
n
(
S
t
r
i
n
g
[
] a
r
g
s
) {
S
y
s
t
e
m
.
o
u
t
.
p
r
i
n
t
l
n
(
"
H
e
l
l
o W
o
r
l
d
!
"
)
;
}

View Slide

(
+ 1 2 3
)
> 6
(
:
c
i
t
y {
:
n
a
m
e "
i
n
n
o
Q
"
:
c
i
t
y "
B
e
r
l
i
n
"
}
)
> "
B
e
r
l
i
n
"
(
m
a
p i
n
c [
1 2 3
]
)
> (
2 3 4
)

View Slide

Example
contacts
simple-calendar
noti cations

View Slide

Project setup
derivative of by
“Garuda Takeoﬀ” Simon_sees (CC BY 2.0)

View Slide

Leiningen
Alternative to Maven
Describes Clojure project with generic data
structures
Maven repository compatibility
Oﬀers plugin system

View Slide

Leiningen
(
d
e
f
p
r
o
j
e
c
t s
i
m
p
l
e
-
c
a
l
e
n
d
a
r "
0
.
1
.
0
-
S
N
A
P
S
H
O
T
"
:
d
e
s
c
r
i
p
t
i
o
n "
s
i
m
p
l
e c
a
l
e
n
d
a
r a
p
p
"
:
u
r
l "
h
t
t
p
s
:
/
/
g
i
t
h
u
b
.
c
o
m
/
i
n
n
o
q
/
s
i
m
p
l
e
-
c
a
l
e
n
d
a
r
"
:
d
e
p
e
n
d
e
n
c
i
e
s [
[
o
r
g
.
c
l
o
j
u
r
e
/
c
l
o
j
u
r
e "
1
.
8
.
0
"
]
[
c
o
m
p
o
j
u
r
e "
1
.
4
.
0
"
]
[
r
i
n
g "
1
.
4
.
0
"
]
[
r
i
n
g
/
r
i
n
g
-
j
s
o
n "
0
.
4
.
0
"
]
.
.
.
[
e
n
v
i
r
o
n "
1
.
0
.
2
"
]
]
:
p
l
u
g
i
n
s [
[
l
e
i
n
-
r
i
n
g "
0
.
9
.
7
"
]
[
l
e
i
n
-
e
n
v
i
r
o
n "
1
.
0
.
2
"
]
]
:
r
i
n
g {
:
h
a
n
d
l
e
r s
i
m
p
l
e
-
c
a
l
e
n
d
a
r
.
c
o
r
e
/
w
e
b
a
p
p
:
i
n
i
t s
i
m
p
l
e
-
c
a
l
e
n
d
a
r
.
c
o
r
e
/
i
n
i
t
}
:
p
r
o
f
i
l
e
s {
:
u
b
e
r
j
a
r {
:
a
o
t :
a
l
l
}
}
)

View Slide

Automated deployment
derivative of by
“Launch Button -- SMASH Rocket Club 5-9-09 4” Steven Depolo (CC BY 2.0)

View Slide

Automated deployment
Not really a language thing ...
... but good tooling can help a lot
Leiningen makes building fat JARs easy
WAR ﬁles can be generated as well

View Slide

Logging & conﬁguration
by
“27 Jan 2007 (Flickr)” wonderferret (CC BY 2.0)

View Slide

Logging
Don’t let every app handle log ﬁles
Just write everything to stdout
Let some external tool handle storage
Standardize on log format

View Slide

org.clojure/tools.logging
Macros delegating to diﬀerent implementations
Supports slf4j, Apache commons-logging, log4j
and java.util.logging
Logging levels: :
t
r
a
c
e :
d
e
b
u
g :
i
n
f
o :
w
a
r
n
:
e
r
r
o
r :
f
a
t
a
l

View Slide

(
n
s s
i
m
p
l
e
-
c
a
l
e
n
d
a
r
.
c
o
r
e
(
:
r
e
q
u
i
r
e [
c
l
o
j
u
r
e
.
t
o
o
l
s
.
l
o
g
g
i
n
g :
a
s l
o
g
]
)
(
d
e
f
n u
p
d
a
t
e
-
c
o
n
t
a
c
t
! [
u
r
l
]
.
.
.
(
l
o
g
/
i
n
f
o "
U
p
d
a
t
e
d u
s
e
r
" i
d n
e
w
-
e
m
a
i
l
)
)
p
r
o
j
e
c
t
.
c
l
j
:
:
d
e
p
e
n
d
e
n
c
i
e
s [
[
o
r
g
.
c
l
o
j
u
r
e
/
t
o
o
l
s
.
l
o
g
g
i
n
g "
0
.
3
.
1
"
]
[
l
o
g
4
j "
1
.
2
.
1
7
" :
e
x
c
l
u
s
i
o
n
s [
j
a
v
a
x
.
m
a
i
l
/
m
a
i
l
j
a
v
a
x
.
j
m
s
/
j
m
s
c
o
m
.
s
u
n
.
j
d
m
k
/
j
m
x
t
o
o
l
s
c
o
m
.
s
u
n
.
j
m
x
/
j
m
x
r
i
]
]
[
o
r
g
.
s
l
f
4
j
/
s
l
f
4
j
-
l
o
g
4
j
1
2 "
1
.
7
.
1
8
"
]
.
.
.
]

View Slide

l
o
g
4
j
.
p
r
o
p
e
r
t
i
e
s
:
l
o
g
4
j
.
r
o
o
t
L
o
g
g
e
r
=
I
N
F
O
, s
t
a
n
d
a
r
d
l
o
g
4
j
.
a
p
p
e
n
d
e
r
.
s
t
a
n
d
a
r
d
=
o
r
g
.
a
p
a
c
h
e
.
l
o
g
4
j
.
C
o
n
s
o
l
e
A
p
p
e
n
d
e
r
l
o
g
4
j
.
a
p
p
e
n
d
e
r
.
s
t
a
n
d
a
r
d
.
T
a
r
g
e
t
=
S
y
s
t
e
m
.
o
u
t
l
o
g
4
j
.
a
p
p
e
n
d
e
r
.
s
t
a
n
d
a
r
d
.
l
a
y
o
u
t
=
o
r
g
.
a
p
a
c
h
e
.
l
o
g
4
j
.
P
a
t
t
e
r
n
L
a
y
o
u
t
l
o
g
4
j
.
a
p
p
e
n
d
e
r
.
s
t
a
n
d
a
r
d
.
l
a
y
o
u
t
.
C
o
n
v
e
r
s
i
o
n
P
a
t
t
e
r
n
=
%
d
{
y
y
y
y
-
m
m
-
d
d H
H
:
m
m
:
s
s
,
S
S
S
} [
%
p
] %
c - %
m
%
n
s
t
d
o
u
t
:
2
0
1
5
-
1
1
-
1
8 1
3
:
1
1
:
5
4
,
4
6
8 [
I
N
F
O
] s
i
m
p
l
e
-
c
a
l
e
n
d
a
r
.
c
o
r
e - U
p
d
a
t
e
d u
s
e
r 5
f
5
6
5
0
4
0 e
v
e
@
e
x
a
m
p
l
e
.
o
r
g
2
0
1
5
-
1
1
-
1
8 1
3
:
1
1
:
5
4
,
4
7
6 [
I
N
F
O
] s
i
m
p
l
e
-
c
a
l
e
n
d
a
r
.
c
o
r
e - U
p
d
a
t
e
d u
s
e
r 7
8
6
4
9
4
e
f b
o
b
@
e
x
a
m
p
l
e
.
o
r
g

View Slide

Conﬁguration
Store conﬁguration in the environment, not in the
codebase
Use mechanism that works for every application,
e.g., environment variables

View Slide

environ
Manages environment settings following 12factor
Reads values from
Java system properties
Environment variables
.
l
e
i
n
-
e
n
v (created via Leiningen plugin from
p
r
o
f
i
l
e
s
.
c
l
j )

View Slide

environ
(
d
e
f c
o
n
t
a
c
t
s
-
f
e
e
d (
e
n
v :
c
o
n
t
a
c
t
s
-
f
e
e
d
)
)
> j
a
v
a -
D
c
o
n
t
a
c
t
s
.
f
e
e
d
=
h
t
t
p
:
/
/
c
o
n
t
a
c
t
s
.
e
x
a
m
p
l
e
.
o
r
g
/
f
e
e
d
-
j
a
r s
t
a
n
d
a
l
o
n
e
.
j
a
r
> C
O
N
T
A
C
T
S
_
F
E
E
D
=
h
t
t
p
:
/
/
c
o
n
t
a
c
t
s
.
e
x
a
m
p
l
e
.
o
r
g
/
f
e
e
d
l
e
i
n r
i
n
g s
e
r
v
e
r
-
h
e
a
d
l
e
s
s
> l
e
i
n r
i
n
g s
e
r
v
e
r
-
h
e
a
d
l
e
s
s
p
r
o
f
i
l
e
s
.
c
l
j
:
{
:
d
e
v {
:
e
n
v {
:
c
o
n
t
a
c
t
s
-
f
e
e
d "
h
t
t
p
:
/
/
c
o
n
t
a
c
t
s
.
e
x
a
m
p
l
e
.
o
r
g
/
f
e
e
d
"
}
}
}

View Slide

Protocols & data formats
derivative of by
“Free ports” Markus Reinhardt (CC BY 2.0)

View Slide

Protocols & data formats
Avoid using diﬀerent protocols everywhere
Standardize on the outside:
HTTP
JSON
XML
Atom

View Slide

HTTP server
Ring for HTTP basics
Compojure for routing
Request & response are data
A web app is a function which takes a request and
returns a response
https://github.com/ring-clojure/ring/blob/master/SPEC

View Slide

Ring
(
d
e
f e
x
a
m
p
l
e
-
r
e
q
u
e
s
t {
:
u
r
i "
/
c
o
n
t
a
c
t
s
"
:
r
e
q
u
e
s
t
-
m
e
t
h
o
d :
g
e
t
:
h
e
a
d
e
r
s {
"
A
c
c
e
p
t
" "
t
e
x
t
/
p
l
a
i
n
"
}
}
)
(
d
e
f
n e
x
a
m
p
l
e
-
a
p
p [
r
e
q
]
{
:
s
t
a
t
u
s 2
0
0
:
b
o
d
y (
s
t
r "
H
e
l
l
o a
t " (
:
u
r
i r
e
q
)
)
}
)
(
e
x
a
m
p
l
e
-
a
p
p e
x
a
m
p
l
e
-
r
e
q
u
e
s
t
)
> {
:
s
t
a
t
u
s 2
0
0
:
b
o
d
y "
H
e
l
l
o a
t /
c
o
n
t
a
c
t
s
"
}

View Slide

Compojure
(
d
e
f
r
o
u
t
e
s c
o
n
t
a
c
t
s
(
G
E
T "
/
c
o
n
t
a
c
t
s
/
:
i
d
" [
i
d
]
(
g
e
t
-
c
o
n
t
a
c
t i
d
)
)
(
P
U
T "
/
c
o
n
t
a
c
t
s
/
:
i
d
" [
i
d :
a
s r
e
q
u
e
s
t
]
(
u
p
d
a
t
e
-
c
o
n
t
a
c
t
! i
d (
:
b
o
d
y r
e
q
u
e
s
t
)
)
)
(
P
O
S
T "
/
c
o
n
t
a
c
t
s
" r
e
q
u
e
s
t
(
a
d
d
-
c
o
n
t
a
c
t
! (
:
b
o
d
y r
e
q
u
e
s
t
)
)
)
(
G
E
T "
/
f
e
e
d
" [
]
{
:
s
t
a
t
u
s 2
0
0
:
h
e
a
d
e
r
s {
"
C
o
n
t
e
n
t
-
T
y
p
e
" "
a
p
p
l
i
c
a
t
i
o
n
/
a
t
o
m
+
x
m
l
"
}
:
b
o
d
y (
c
o
n
t
a
c
t
s
-
f
e
e
d
)
}
)
)
)

View Slide

Frameworks
Quickly get a web app up and running.
Duct
Luminus
Pedestal
Modularity
tesla-microservices

View Slide

org.clojure/data.json
Clojure data structures are JSON superset
(
j
s
o
n
/
w
r
i
t
e
-
s
t
r {
:
n
a
m
e "
A
l
i
c
e M
i
l
l
e
r
"
:
t
e
a
m
s [
"
T
e
a
m A
"
, "
T
e
a
m B
"
]
}
)
> "
{
\
"
n
a
m
e
\
"
:
\
"
A
l
i
c
e M
i
l
l
e
r
\
"
,
\
"
t
e
a
m
s
\
"
:
[
\
"
T
e
a
m A
\
"
,
\
"
T
e
a
m B
\
"
]
}
"
(
j
s
o
n
/
r
e
a
d
-
s
t
r "
{
\
"
i
d
\
"
: 1
2
3
,
\
"
n
a
m
e
\
"
: \
"
A
l
i
c
e M
i
l
l
e
r
\
"
}
"
:
k
e
y
-
f
n k
e
y
w
o
r
d
)
> {
:
i
d 1
2
3
, :
n
a
m
e "
A
l
i
c
e M
i
l
l
e
r
"
}

View Slide

HTML
HTML is represented with generic data structures
This allows processing HTML with standard Clojure
functions
There are diﬀerent formats, a popular one is called
“hiccup”

View Slide

HTML
<
h
t
m
l
>
<
b
o
d
y
>
<
h
1
>
h
e
l
l
o
!
<
/
h
1
>
<
i
m
g s
r
c
=
"
s
o
m
e
.
j
p
g
" /
>
<
i
m
g s
r
c
=
"
a
n
o
t
h
e
r
.
j
p
g
" /
>
<
/
b
o
d
y
>
<
/
h
t
m
l
>
[
:
h
t
m
l {
}
[
:
b
o
d
y {
}
[
:
h
1 {
} "
h
e
l
l
o
!
"
]
[
:
i
m
g {
:
s
r
c "
s
o
m
e
.
j
p
g
"
}
]
[
:
i
m
g {
:
s
r
c "
a
n
o
t
h
e
r
.
j
p
g
"
}
]
]
]

View Slide

Atom
XML format for representing feeds of data
Useful for doing pub/sub without middleware

View Slide

Creating Atom Feeds
(
d
e
f
n e
n
t
r
y [
e
v
e
n
t
]
[
:
e
n
t
r
y
[
:
t
i
t
l
e (
-
> e
v
e
n
t :
t
y
p
e n
a
m
e
)
]
[
:
u
p
d
a
t
e
d (
:
t
i
m
e
s
t
a
m
p e
v
e
n
t
)
]
[
:
a
u
t
h
o
r [
:
n
a
m
e "
c
o
n
t
a
c
t
s s
e
r
v
i
c
e
"
]
]
[
:
i
d (
s
t
r "
u
r
n
:
c
o
n
t
a
c
t
s
:
f
e
e
d
:
e
v
e
n
t
:
" (
:
i
d e
v
e
n
t
)
)
]
[
:
c
o
n
t
e
n
t {
:
t
y
p
e "
j
s
o
n
"
} (
j
s
o
n
/
g
e
n
e
r
a
t
e
-
s
t
r
i
n
g e
v
e
n
t
)
]
]
)
(
d
e
f
n a
t
o
m
-
f
e
e
d [
e
v
e
n
t
s u
r
l
]
(
c
l
o
j
u
r
e
.
d
a
t
a
.
x
m
l
/
e
m
i
t
-
s
t
r
(
c
l
o
j
u
r
e
.
d
a
t
a
.
x
m
l
/
s
e
x
p
-
a
s
-
e
l
e
m
e
n
t
[
:
f
e
e
d {
:
x
m
l
n
s "
h
t
t
p
:
/
/
w
w
w
.
w
3
.
o
r
g
/
2
0
0
5
/
A
t
o
m
"
}
[
:
i
d "
u
r
n
:
c
o
n
t
a
c
t
s
:
f
e
e
d
"
]
[
:
u
p
d
a
t
e
d (
-
> e
v
e
n
t
s l
a
s
t :
t
i
m
e
s
t
a
m
p
)
]
[
:
t
i
t
l
e {
:
t
y
p
e "
t
e
x
t
"
} "
c
o
n
t
a
c
t
s e
v
e
n
t
s
"
]
[
:
l
i
n
k {
:
r
e
l "
s
e
l
f
" :
h
r
e
f u
r
l
}
]
(
m
a
p e
n
t
r
y e
v
e
n
t
s
)
]
)
)
)

View Slide

adamwynne/feedparser-clj
Retrieves and Parses RSS/Atom feeds
(
d
e
f f
(
f
e
e
d
p
a
r
s
e
r
/
p
a
r
s
e
-
f
e
e
d "
h
t
t
p
s
:
/
/
w
w
w
.
i
n
n
o
q
.
c
o
m
/
d
e
/
p
o
d
c
a
s
t
.
r
s
s
"
)
)
(
:
t
i
t
l
e f
)
> "
i
n
n
o
Q P
o
d
c
a
s
t
"
(
c
o
u
n
t (
:
e
n
t
r
i
e
s f
)
)
> 1
8
Library for consuming feeds based on feedparser:
Feedworker

View Slide

Lightweight messaging
Just a side note:
Kafka (clj-kafka)
RabbitMQ (Langohr)
...

View Slide

Problems of distributed systems:
Stability
derivative of by
“the Jenga” Ed Garcia (CC BY 2.0)

View Slide

Cascading failures
dash-
board
app
web
service
data-
base

View Slide

How to achieve stability?

View Slide

Bulkheads
derivative of by (Public Domain)
“Titanic sinking” Willy Stöwer

View Slide

Asynchronous communication
Helps to decouple and separate diﬀerent
application parts
E-mail notiﬁcations do not need to be sent
synchronously

View Slide

org.clojure/core.async
Supports asynchronous programming and
communications
Messages can be sent to and read from channels
Channels can be buﬀered or unbuﬀered
Blocking and non-blocking operations possible

View Slide

(
d
e
f n
o
t
i
f
i
c
a
t
i
o
n
s (
c
h
a
n 1
0
0
0
)
)

View Slide

(
d
e
f n
o
t
i
f
i
c
a
t
i
o
n
s (
c
h
a
n 1
0
0
0
)
)
(
d
e
f
n s
e
n
d
-
n
o
t
i
f
i
c
a
t
i
o
n [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
(
g
o (
>
! n
o
t
i
f
i
c
a
t
i
o
n
s {
:
e
m
a
i
l e
m
a
i
l
:
e
v
e
n
t
-
l
i
n
k e
v
e
n
t
-
l
i
n
k
}
)
)
)

View Slide

(
d
e
f n
o
t
i
f
i
c
a
t
i
o
n
s (
c
h
a
n 1
0
0
0
)
)
(
d
e
f
n s
e
n
d
-
n
o
t
i
f
i
c
a
t
i
o
n [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
(
g
o (
>
! n
o
t
i
f
i
c
a
t
i
o
n
s {
:
e
m
a
i
l e
m
a
i
l
:
e
v
e
n
t
-
l
i
n
k e
v
e
n
t
-
l
i
n
k
}
)
)
)
(
d
e
f
n n
o
t
i
f
y
-
u
s
e
r [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
.
.
.
)
(
d
e
f
n s
t
a
r
t
-
n
o
t
i
f
i
e
r [
]
(
g
o
-
l
o
o
p [
m
e
s
s
a
g
e (
<
! n
o
t
i
f
i
c
a
t
i
o
n
s
)
]
(
n
o
t
i
f
y
-
u
s
e
r (
:
e
m
a
i
l m
e
s
s
a
g
e
) (
:
e
v
e
n
t
-
l
i
n
k m
e
s
s
a
g
e
)
)
(
r
e
c
u
r (
<
! n
o
t
i
f
i
c
a
t
i
o
n
s
)
)
)
)

View Slide

Circuit Breaker
derivative of by
“switch-fuse” Mark_K_ (CC BY 2.0)

View Slide

com.netﬂix.hystrix/hystrix-clj
Idiomatic Clojure wrapper for Hystrix
(
h
y
s
t
r
i
x
/
d
e
f
c
o
m
m
a
n
d n
o
t
i
f
y
-
u
s
e
r [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
(
c
l
i
e
n
t
/
p
o
s
t n
o
t
i
f
i
c
a
t
i
o
n
-
s
e
r
v
i
c
e
{
:
c
o
n
t
e
n
t
-
t
y
p
e :
j
s
o
n
:
b
o
d
y .
.
. }
)
)
Will throw exception in case of timeout or other
failure

View Slide

; r
e
t
u
r
n
s t
r
u
e i
f s
u
c
e
s
s
f
u
l
, f
a
l
s
e i
f c
i
r
c
u
i
t
-
b
r
e
a
k
e
r o
p
e
n
(
h
y
s
t
r
i
x
/
d
e
f
c
o
m
m
a
n
d n
o
t
i
f
y
-
u
s
e
r
{
:
h
y
s
t
r
i
x
/
f
a
l
l
b
a
c
k
-
f
n n
o
t
i
f
y
-
f
a
l
l
b
a
c
k
}
[
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
(
c
l
i
e
n
t
/
p
o
s
t n
o
t
i
f
i
c
a
t
i
o
n
-
s
e
r
v
i
c
e
{
:
c
o
n
t
e
n
t
-
t
y
p
e :
j
s
o
n
:
b
o
d
y .
.
. }
)
t
r
u
e
)
; r
e
t
u
r
n
s f
a
l
s
e i
f c
i
r
c
u
i
t
-
b
r
e
a
k
e
r o
p
e
n
(
d
e
f
n n
o
t
i
f
y
-
f
a
l
l
b
a
c
k [
e
m
a
i
l e
v
e
n
t
-
l
i
n
k
]
(
l
e
t [
i
s
O
p
e
n (
.
i
s
C
i
r
c
u
i
t
B
r
e
a
k
e
r
O
p
e
n h
y
s
t
r
i
x
/
*
c
o
m
m
a
n
d
*
)
]
; a
d
d m
e
s
s
a
g
e t
o q
u
e
u
e a
g
a
i
n
(
s
e
n
d
-
n
o
t
i
f
i
c
a
t
i
o
n e
m
a
i
l e
v
e
n
t
-
l
i
n
k
)
(
n
o
t i
s
O
p
e
n
)
)
)

View Slide

(
d
e
f
n s
t
a
r
t
-
n
o
t
i
f
i
e
r [
]
(
g
o
-
l
o
o
p [
m
e
s
s
a
g
e (
<
! n
o
t
i
f
i
c
a
t
i
o
n
s
)
]
(
i
f
-
n
o
t (
n
o
t
i
f
y
-
u
s
e
r (
:
e
m
a
i
l m
e
s
s
a
g
e
)
(
:
e
v
e
n
t
-
l
i
n
k m
e
s
s
a
g
e
)
)
(
d
o
(
l
o
g
/
e
r
r
o
r "
C
a
n
n
o
t r
e
a
c
h n
o
t
i
f
i
c
a
t
i
o
n s
e
r
v
i
c
e
"
"
- w
i
l
l w
a
i
t u
n
t
i
l n
e
x
t t
r
y
"
)
(
<
! (
t
i
m
e
o
u
t 5
0
0
0
)
)
)
)
(
r
e
c
u
r (
<
! n
o
t
i
f
i
c
a
t
i
o
n
s
)
)
)
)
hystrix-event-stream-clj available as well

View Slide

Monitoring
derivative of by
“Space Shuttle Endeavour's Control Panels” Steve Jurvetson (CC BY 2.0)

View Slide

Questions
How many HTTP errors are occurring?
Are database queries failing?
Is that backend service slow again?
How many jobs are in the queue?

View Slide

Metrics
Logging provides a stream of events
Metrics provide aggregated state
Popular library: Dropwizard Metrics
Two steps: collect metrics, then publish them

View Slide

Gauges
by
“Old gauge, old style” speredenn (CC BY 2.0)

View Slide

Gauges
Current state of one single value
Number of jobs in the queue
Some conﬁgured value
Ratio of cache hits to misses

View Slide

Gauges
(
d
e
f m
e
t
r
i
c
s
-
r
e
g
i
s
t
r
y (
n
e
w
-
r
e
g
i
s
t
r
y
)
)
(
g
a
u
g
e
-
f
n m
e
t
r
i
c
s
-
r
e
g
i
s
t
r
y "
j
o
b
s
-
r
e
a
d
y
"
#
(
q
u
e
r
y d
a
t
a
b
a
s
e
"
s
e
l
e
c
t c
o
u
n
t
(
*
)
f
r
o
m j
o
b
s
w
h
e
r
e s
t
a
t
u
s = '
r
e
a
d
y
'
"
)
)

View Slide

Counters
derivative of by
“The US National Debt clock / counter, New York” Ben Sutherland (CC BY 2.0)

View Slide

Counters
Single numeric value
For example, number of logged in users

View Slide

Counters
(
d
e
f l
o
g
g
e
d
-
i
n
-
u
s
e
r
s (
c
o
u
n
t
e
r m
e
t
r
i
c
s
-
r
e
g
i
s
t
r
y "
l
o
g
g
e
d
-
i
n
-
u
s
e
r
s
"
)
)
(
P
O
S
T "
/
l
o
g
i
n
" [
n
a
m
e p
w
d
]
.
.
.
(
i
n
c
! l
o
g
g
e
d
-
i
n
-
u
s
e
r
s
)
.
.
.
)
(
P
O
S
T "
/
l
o
g
o
u
t
" [
n
a
m
e
]
.
.
.
(
d
e
c
! l
o
g
g
e
d
-
i
n
-
u
s
e
r
s
)
.
.
.
)

View Slide

Histograms
derivative of by
“Number of cat posts on Metaﬁlter per month, as a percentage of the total number of posts” Steven Taschuk (CC BY 2.0)

View Slide

Histograms
Distribution of numerical data (min, max, mean,
standard deviation, quantiles)
For example, number of search results
Diﬀerent value “reservoirs”, e.g.,
Entire application lifetime
Last N searches
Last N minutes

View Slide

Histograms
(
d
e
f n
u
m
b
e
r
-
o
f
-
r
e
s
u
l
t
s
(
h
i
s
t
o
g
r
a
m m
e
t
r
i
c
s
-
r
e
g
i
s
t
r
y "
n
u
m
b
e
r
-
o
f
-
s
e
a
r
c
h
-
r
e
s
u
l
t
s
"
)
)
(
d
e
f
n s
e
a
r
c
h [
q
u
e
r
y
]
(
l
e
t [
r
e
s
u
l
t
s (
e
x
e
c
u
t
e q
u
e
r
y
)
]
(
u
p
d
a
t
e
! n
u
m
b
e
r
-
o
f
-
r
e
s
u
l
t
s (
c
o
u
n
t r
e
s
u
l
t
s
)
)
r
e
s
u
l
t
s
)
)

View Slide

Meters
derivative of by
“Hosp_1011” Justin Taylor (CC BY 2.0)

View Slide

Meters
Rate of an event (per second)
Total count of events
Average rate over application lifetime
Rate in the last 1, 5 and 15 minutes
For example, incoming requests

View Slide

Meters
(
d
e
f i
n
c
-
r
e
q
u
e
s
t
s
-
m
e
t
e
r
(
m
e
t
e
r
s
/
m
e
t
e
r m
e
t
r
i
c
s
-
r
e
g
i
s
t
r
y "
i
n
c
o
m
i
n
g
-
r
e
q
u
e
s
t
s
-
m
e
t
e
r
"
)
)
(
d
e
f
n r
e
q
u
e
s
t
s
-
m
e
t
e
r [
h
a
n
d
l
e
r
]
(
f
n [
r
e
q
]
(
m
e
t
e
r
s
/
m
a
r
k
! i
n
c
-
r
e
q
u
e
s
t
s
-
m
e
t
e
r
)
(
h
a
n
d
l
e
r r
e
q
)
)
)

View Slide

Timers
by
“Stopwatch” William Warby (CC BY 2.0)

View Slide

Timers
Histogram & meter
Histogram of duration of an activity, meter of
occurrence of the activity
For example, speciﬁc type of database query

View Slide

Timers
(
d
e
f i
n
c
-
r
e
q
u
e
s
t
s
-
t
i
m
e
r
(
t
i
m
e
r
s
/
t
i
m
e
r
! m
e
t
r
i
c
s
-
r
e
g
i
s
t
r
y "
i
n
c
o
m
i
n
g
-
r
e
q
u
e
s
t
s
-
t
i
m
e
r
"
)
)
(
d
e
f
n r
e
q
u
e
s
t
s
-
t
i
m
e
r [
h
a
n
d
l
e
r
]
(
f
n [
r
e
q
]
(
t
i
m
e
r
s
/
t
i
m
e
! i
n
c
-
r
e
q
u
e
s
t
s
-
t
i
m
e
r
(
h
a
n
d
l
e
r r
e
q
)
)
)
)

View Slide

Reporting Metrics

View Slide

Reporting
Metrics stores current state in its registry
Should be reported to external tool
Visualization
Historical data
Reporters for console, JMX, Ganglia, Graphite, CSV
etc. included

View Slide

Conclusion
Organize around business capabilities
Rules & conventions needed
Standardize interfaces, logging and conﬁguration
Diﬀerent solutions to improve stability
Monitor your systems
Good support in Clojure

View Slide

Example Microservices
Simple Calendar:
Simple Contacts:
https://github.com/innoq/simple-
calendar
https://github.com/innoq/simple-
contacts

View Slide

Thank you! Questions?
https://www.innoq.com/de/talks/2016/04/microservice-meetup-clojure/

View Slide

Microservices_with_Clojure.pdf

Microservices_with_Clojure.pdf

Joy Heron

More Decks by Joy Heron

Other Decks in Technology

Featured

Transcript