ElixirConf 2016 - Measuring your elixir App

Slide 1

Slide 1 text

Measuring your Elixir application Renan Ranelli

Slide 2

Slide 2 text

Senior software engineer @ Milhouse (@renanranelli)

Slide 3

Slide 3 text

Milhouse (@renanranelli)

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Why should I care?

Slide 7

Slide 7 text

Why should I care? We write code.

Slide 8

Slide 8 text

Why should I care? We write code. We do it because it generate business value.

Slide 9

Slide 9 text

Why should I care? We write code. We do it because it generate business value. But this business value is generated only when our code *runs*. *NOT* when it is written

Slide 10

Slide 10 text

Why should I care? In order to make good decisions about something, we must understand it.

Slide 11

Slide 11 text

Why should I care? In order to make good decisions about something, we must understand it. (And we want to make good decisions because we like getting paid)

Slide 12

Slide 12 text

But... there is a problem

Slide 13

Slide 13 text

Why should I care? Its hard to reason about code

Slide 14

Slide 14 text

Which one is faster?

Slide 15

Slide 15 text

We don’t know

Slide 16

Slide 16 text

Which is faster?

Slide 17

Slide 17 text

The mental model we have of our system is *always* flawed.

Slide 18

Slide 18 text

But wait! There's still hope.

Slide 19

Slide 19 text

When we measure stuff, the confusion fades:

Slide 20

Slide 20 text

When we measure stuff, the confusion fades:

Slide 21

Slide 21 text

This is *SCIENCE* !

Slide 22

Slide 22 text

This is *SCIENCE* !

Slide 23

Slide 23 text

Types of metrics & how we look at them

Slide 24

Slide 24 text

The metrics we’re interested are **time series**:

Slide 25

Slide 25 text

The metrics we’re interested are **time series**:

Slide 26

Slide 26 text

Metrics – average value

Slide 27

Slide 27 text

Metrics – average value

Slide 28

Slide 28 text

Metrics – average value ← Time

Slide 29

Slide 29 text

Metrics – average value First thing you should ask: Which time period does this average refer to ??? Averages completely destroy the **variability** and **recency**.

Slide 30

Slide 30 text

Metrics – average value A common thing to do is add a “decay” component to the average calculation:

Slide 31

Slide 31 text

Metrics – average value A common thing to do is add a “decay” component to the average calculation:

Slide 32

Slide 32 text

Metrics – distribution

Slide 33

Slide 33 text

Metrics – distribution

Slide 34

Slide 34 text

Histograms Histograms are like averages on steroids. As such, you **have** to pay attention to The aggregation period it refers to

Slide 35

Slide 35 text

Metrics – resolution

Slide 36

Slide 36 text

Metrics – resolution

Slide 37

Slide 37 text

How do I get started in the real world?

Slide 38

Slide 38 text

You have to solve these parts: ❑ Collect ❑ Store ❑ Query, Analyze & Transform ❑ Visualize ❑ … ❑ MAKE DECISIONS (!)

Slide 39

Slide 39 text

Our metrics collection architecture Log parsing Exometer & Elixometer Influxdb plugin “report api” Query & Visualize Transform & Analyze STORE Collect Store View

Slide 40

Slide 40 text

Our (boring) choices ❑ Collect ●Exometer & Elixometer are very easy to set up ●We have used collectd extensively in the past ❑ Store (InfluxDB) ●**Very** easy to set up & configure. And free. ●Built-in retention policy configuration ●Awesome query language & Continuous queries ●Plugs seamlessly with grafana

Slide 41

Slide 41 text

Our (boring) choices ❑ View (Grafana) ●We have used it in the past. ●It is awesome and beautiful (you will see!)

Slide 42

Slide 42 text

Our (boring) choices ❑ You might ask: ❑ Why you’re not using solution like Splunk, DataDog, etc?

Slide 43

Slide 43 text

Our (boring) choices ❑ You might ask: Why you’re not using solution like Splunk, DataDog, etc? ❑ Our answer: ❑ We don't like our data outside our walls. Also, NDAs and legal obligations

Slide 44

Slide 44 text

Exometer & Elixometer ❑ Exometer Exometer Reporter Metrics buffer (pre aggregated ) High frequency Metric reports Store & update Aggregated view Collect & convert Low frequency Metrics formated for other backends

Slide 45

Slide 45 text

Exometer & Elixometer ❑ This design is almost omnipresent in metrics world. ❑ Other tools like statsd, telegraf, etc might fit the role of exometer in your case. Use the one you’re most comfortable with. ❑ It is **very** important to be able to control the frequency which your storage receives requests. You don’t want to overload it.

Slide 46

Slide 46 text

DEMONSTRATION ● Measuring all requests to a phoenix app Step by step instructions here: http://milhouseonsoftware.com/2016/05/08/me asuring-your-elixir-application/

Slide 47

Slide 47 text

Other things you should measure

Slide 48

Slide 48 text

Other things you should measure ❑ The Erlang VM is **awesome** (and that’s why we’re here ;P) ❑ There are lots of resources out there on how to measure, monitor and operate it. ❑ But sadly I don’t have enough time to go in depth about it ~ Use vm_stats to gather virtual machine metrics! https://github.com/ferd/vmstats

Slide 49

Slide 49 text

Other things you should measure ❑ The Erlang VM is **awesome** (and that’s why we’re here ;P) ❑ ❑ There are lots of resources out there on how to measure, monitor and operate it. ❑ ❑ But sadly I don’t have enough time to go in depth about the things the instrumentation the VM exposes ~

Slide 50

Slide 50 text

Other things you should measure ❑ The Erlang VM is **awesome** (and that’s why we’re here ;P) ❑ ❑ There are lots of resources out there on how to measure, monitor and operate it.

Slide 51

Slide 51 text

Closing remarks

Slide 52

Slide 52 text

Closing remarks ❑ Find “seams” in your code where you can “plug” measurements. ❑ Try not to spread metrics reporting “everywhere”. It’s best to separat “high level” measurements from “low level” ones. ❑ This is just the beginning. After you have this infrastructure in place, you have to start doing analysis, correlation and other math stuffs. That’s where the real *gold* is.

Slide 53

Slide 53 text

Closing remarks ❑ Bend your metrics to generate alerts. This is **really** awesome. ❑ Another (similar) metrics collection architecture is described here: http://tech.footballaddicts.com/blog/gathering-metrics-in-elixir- applications ❑ Zed Shaw has an awesome post about statistics: ❑https://zedshaw.com/archive/programmers-need-to-learn-statistics- or-i-will-kill-them-all/