Berlin 2013 - Session - Alex Petrov

Monitorin’ it
Friday, September 20, 13

View Slide


View Slide

@ifesdjeen
tweet right along the talk if you have a question

View Slide

This talk is rather
philosophical

View Slide

Hobbyist monitoring geek
(in free time, of course)

View Slide

Several rewrites from
scratch

View Slide

Tons of experiences

View Slide

Many extracted
components
already available as open-source solutions

View Slide

Needs your opinion

View Slide

Monitoring is not about
software

View Slide

It’s all about
insight

View Slide

Current state of art:
So easy to get started
Just as easy to hit the limit
Just a tiny bit too hard to change tooling
And even harder to extend existing one

View Slide

General path:
Try out some existing tool
Find ways to implement more complex scenarios
Eventually give in and use what’s already there
Keep ranting and saying `#monitoringsucks`

View Slide

But wait, I thought there
was...

View Slide

#monitoringlove
#monitoringlove

View Slide

Put on your
monitoring gloves
monitoring hats
monitoring socks (you got ‘em)
We’re going out for a journey

View Slide

When I hear Monitoring, I’m like

View Slide

In my world everyone's a pony
and they all eat rainbows and poop butterflies

View Slide

Let’s try to redefine it all but first let’s
simplify it

View Slide

Ad-hoc vs Post-hoc

View Slide


View Slide

I once wanted to
understand what’s going on on my website

View Slide

...and then I was like
monitoring.increment "page_load_#{response.status}_count"
monitoring.timing "page_load_#{response.status}_load_time", time
monitoring.increment "page_load_#{request.user_agent}_count"
monitoring.gauge "page_load_#{response.status}_time_gauge", time

View Slide

And deployed it to
a hundred something servers (rofl, right?)

View Slide

And then I wanted
to add more

View Slide

And had to deploy it to
a hundred something servers (lmao, right?)

View Slide

And then I wanted to...
well, you get the idea

View Slide

Anything that
a server
can do
server
should do

View Slide

What if you want a suit...

View Slide

Let’s turn it other way
around

View Slide

Package everything related to a
single event
into
single payload
and figure out which metrics you need on the server side

View Slide

What it gives you
granularity
simple, stupid client
no need to think in advance*
rethink your metrics any time you want
add more rollups and aggregates as you need them
(of course, you’d still have to think, you can just take it much more easy) *

View Slide

What would monitoring
system look like?

View Slide


View Slide

reporter
Processing
unit
Persistent Store
State Machine Alerts
Reduce Engine
(ad-hoc queries and analytics)
History Graphs
Table Data
Real-time graphs
Real-time table data
Latest Events
(capped in-memory store)
reporter
reporter
reporter
reporter
UDP,
TCP,
AMQP
Console
(nrepl, custom consumers,
real-time analysis)
Collector Collector

View Slide


View Slide

Processing
unit

View Slide

Simple Scenario
Even / Odd
splitter
Even numbers Odd numbers

View Slide

add counts for both
Even / Odd
splitter
Multicast Multicast
Even numbers
count
Even numbers
buffer
Odd numbers
count
Even numbers
buffer

View Slide

calculate sum for each 10
Even / Odd
splitter
Multicast Multicast
Even numbers
count
Even numbers
buffer
Odd numbers
count
Even numbers
buffer
Summarizer
Sum buffer
Summarizer
Sum buffer

View Slide

Even / Odd
splitter
Multicast Multicast
Even numbers
count
Even numbers
buffer
Odd numbers
count
Even numbers
buffer
Summarizer
Sum buffer

View Slide

Even / Odd
splitter
Multicast Multicast
Even numbers
count
Even numbers
buffer
Summarizer
Sum buffer

View Slide

Event-based
Create simple, independent parts (aggregate, filter, multicast, transfomer etc)
Define dependencies between them (routing)
Parts are completely decoupled
Every part can have it’s own state
Routing is dynamic, and can be changed in runtime
Graphs are expressive, easy to understand

View Slide

Even more applicable to monitoring

View Slide

Windowed operations

View Slide

Why matters?
Raw numbers are too much
Window is an easy way to accumulate and summarize
Flexible and very comosable

View Slide

Sliding window
t0 t1 t2 (emit)
+---+ +---+---+ +---+---+---+
| 1 | | 1 | 2 | | 1 | 2 | 3 | <6>
+---+ +---+---+ +---+---+---+
t4 (emit)
-...+---+---+---+ -...+---+---+---+
: 1 : 2 | 3 | 4 | <9> : 2 : 3 | 4 | 5 | <12>
-...+---+---+---+ -...+---+---+---+

View Slide

Sliding window
Accumulates items in buffer
When full:
- emits all contents
- drops an oldest value

View Slide

Tumbling window
t0 t1 t2 (emit)
+---+ +---+---+ +---+---+---+ -...-...-...-
| 1 | | 1 | 2 | | 1 | 2 | 3 | <6> : 1 : 2 : 3 :
+---+ +---+---+ +---+---+---+ -...-...-...-
t3 t4 t5 (emit)
+---+ +---+---+ +---+---+---+ -...-...-...-
| 4 | | 4 | 5 | | 4 | 5 | 6 | <15> : 4 : 5 : 6 :
+---+ +---+---+ +---+---+---+ -...-...-...-

View Slide

Tumbling window
Accumulates items in buffer
When full:
- emits all contents
- drops all values

View Slide

Clock
Control wether window should or should not yet (emit)
Clocks in windows are arbitrary
Can be
- monotonic
- wall clock
- arbitrary (business clock)

View Slide

Summing up

View Slide

Idea is simple:
• Everything that’s coming in is an event
• Events are split by triplet (application/env/event type)
• Every event can have multiple metrics
• Metric has a key and value
• And filter
• And several rollups (tumbling window)
• Rollup has an aggregate function triggered on overflow
• And sliding window with last N values
• And visualization (area, line, barchart) attached

View Slide

Incoming payload
{:application "my-app"
:environment "production"
:type “page_load”
:execution_time 542
:user_agent “Mozilla...”
:host "web001"
:status 200}

View Slide

:execution_time 542
:host "web001"
:status 200}
Identification (splitter)

View Slide

:execution_time 542
:host "web001"
:status 200}
key ->
value ->
aggregate: median

View Slide

:execution_time 542
:host "web001"
:status 200}
key ->
value ->
aggregate: max

View Slide

:execution_time 542
:host "web001"
:status 200}
key ->
aggregate: count

View Slide

:type “exception”
:stacktrace “NullPointer.”
:host "web001"}
key ->
aggregate: count

View Slide

Idea is simple:
• Everything that’s coming in is an event
• Events are split by triplet (application/env/event type)
• Every event can have multiple metrics
• Metric has a key and value
• And filter
• And several rollups (tumbling window)
• Rollup has an aggregate function triggered on overflow
• And sliding window with last N values
• And visualization (area, line, barchart) attached

View Slide

Please pardon my lisp

View Slide

(event :my_web_app :production :page_load
(metric :page-load
(group #(get-in % [:additional_info :status]))
(value [:additional_info :execution_time])
(rollup 1 :second
(aggregate :max #(apply max %)
(store-last 60)
(visualize :line :y "Max response time"))
(aggregate :mean stats/mean
(store-last 60)
(visualize :line :y "Mean response time"))
(aggregate :count #(apply min %)
(store-last 60)
(visualize :line :y "Min response time")))))

View Slide

:execution_time 542
:host "web001"
:status 200}

View Slide

(metric :page-load
(group #(get-in % [:status]))
(value [:execution_time])
(rollup 1 :second
(store-last 60)
(store-last 60)
(store-last 60)

View Slide

:execution_time 542
:host "web001"
:status 200}

View Slide

(metric :page-load
(group #(get-in % [:additional_info :status]))
(value [:additional_info :execution_time])
(rollup 1 :second
(store-last 60)
(store-last 60)
(store-last 60)

View Slide

Wait, I want to see
correlations!

View Slide


View Slide

Boxplots?

View Slide

You got em’

View Slide

Linear regression?

View Slide

Sure, sir!

View Slide


View Slide

Built on open source
EEP for event emitter & windows https://github.com/clojurewerkz/eep
Meltdown for anonymous topologies https://github.com/clojurewerkz/eep
Eventoverse-graphs for graphs https://github.com/ifesdjeen/eventoverse-graphs
Clj-push for websockets https://github.com/ifesdjeen/clj-pushr
Cascalog for map/reduce https://github.com/nathanmarz/cascalog

View Slide

Available soon under
@clojurewerkz

View Slide


View Slide

@ifesdjeen

View Slide

Berlin 2013 - Session - Alex Petrov

Berlin 2013 - Session - Alex Petrov

Monitorama

More Decks by Monitorama

Featured

Transcript