The OPS side of DEV

Hello.
My name is Mårten Gustafson

View Slide

...I used to work here...

View Slide

...now I work here...

View Slide

...doing mostly this...

View Slide

...but spend a fair share of my time looking at metrics like this...

View Slide

AUTOMATE
ALL THE TINGS!
...being rabid about this, which makes me a fan of...

View Slide

...DevOps...and it’s general concepts...but...

View Slide

the OPS side of DEV
...I’m talking about this

View Slide

OpsDev
...I think: we (as developers) need to think about this!

View Slide

develop for operations
* I think we need to get better at this

View Slide

develop for production
* I think we need to focus on this, which is why...

View Slide

...I’ll start off with one of the most boring things to most developers...

View Slide

logging

View Slide

* Huge ﬁles
* Messy log format
* Hard to ﬁlter
* Hard to correlate
* Might as well...

View Slide

/bin/my-awesome-service 2&>1 > /dev/null

View Slide

surprisingly hard

View Slide

or...
we’re surprisingly bad

View Slide

so, logging

View Slide

1. do it

View Slide

framework
pick a framework that’s:
* makes sense
* is de-facto standard?
* is ﬂexible
* is lightweight
* is easy to use

View Slide

consistent
try to log in a consistent manner
* think of log messages in terms of operation
* what’s an error (should somebody be woken up?)
* what’s a trace (who’s the audience?)
* etc

View Slide

2. rotation & retention

View Slide

resources are finite
* rotate your log files
* put an upper bound on the size of one log file

View Slide

deﬁne window of interest
* for the local disc: toss anythings that’s older than X, or:
* compress (and then toss when they’re even older)
* will you ever look in compressed log ﬁles?

View Slide

3. formatting

View Slide

0 [main] INFO Main - foo
0 [main] WARN Main - bar
0 [doer] ERROR Worker - gah
java.lang.NullPointerException: gah
! at Doer.worker(Doer.java:13)

View Slide


View Slide

easy on the eyes
* aligned (easier on the eyes)

View Slide

easy on the tools
* tail & grep friendly

View Slide

INFO [2012-02-25 20:24:03] foo.Main - foo
WARN [2012-02-25 20:24:03] foo.Main - bar
ERROR [2012-02-25 20:24:03] foo.Worker - gah
! java.lang.NullPointerException: gah

View Slide

INFO [2012-02-25 20:24:03] foo.Main - foo
WARN [2012-02-25 20:24:03] foo.Main - bar
INFO [2012-02-25 20:24:03] foo.Main - foo
WARN [2012-02-25 20:24:03] foo.Main - bar
INFO [2012-02-25 20:24:03] foo.Main - foo
WARN [2012-02-25 20:24:03] foo.Main - bar
INFO [2012-02-25 20:24:03] foo.Main - foo
WARN [2012-02-25 20:24:03] foo.Main - bar

View Slide

https://github.com/codahale/logula
* have a look at this

View Slide

3. destinations

View Slide

* when this is your reality, you don’t really only want log ﬁles on local machine disk

View Slide

to name a few

View Slide

SMTP ﬁle
syslog
SQL
AMQP
IRC XMPP
to name a few

View Slide

critical
=
“real time”

View Slide

View Slide

audit
=
remote + restricted access

View Slide

mix and match

View Slide

always fallback on local ﬁle
(fallacies of distributed computing)
* Not SAN, NFS, NAS, etc

View Slide

(beware of sensitive data)
* security sensitive: keys, passwords, etc
* integrity sensitive: whatever you’re users might provide that’s not for everyone’s eyes

View Slide

4. separation

View Slide

we usually log most things

View Slide

we usually don’t separate

View Slide

UTILIZE
ALL THE CONTEXTS!

View Slide

multiple logs & context logs

View Slide

* Look at the SiftingAppender in logback-classic for an example

View Slide

traditional log

View Slide

traditional log
userid

View Slide

traditional log
userid
session id

View Slide

5. configuration

View Slide

sane defaults
* location
* rotation

View Slide

per environment
* have a conﬁguration that automatically adapts to the environment
* log everything to stdout in local development
* log everything to ﬁle in test
* log X to Y and Z to FOO in prod

View Slide

re:conﬁgurable
* don’t require a deploy to change a log level
* provide an API
* use JMX
* so that you can tweak logging (enable tracing) right in production when you need to

View Slide

View Slide

metrics

View Slide

let your code speak

View Slide

INSTRUMENT
ALL THE CODE!

View Slide

meters

View Slide

counters
meters
timers
gauges
histograms

View Slide

EXPOSE
ALL THE VALUES!

View Slide

...or whatever makes sense for you
...and yes, it’s Comic Sans for BAYEUX
...and yes, it’s Helvetica for JMX

View Slide

JMX JSON
XML
HTTP
XMPP
AMQP
THRIFT
BAYEUX
RMI
CSV
...or whatever makes sense for you
...and yes, it’s Comic Sans for BAYEUX
...and yes, it’s Helvetica for JMX

View Slide

* put all your values into your tools and services
* BUT DON’T FORGET THE AD-HOC, LOCAL, USAGE (ie JMX)

View Slide

View Slide

self checks!

View Slide

@Override
protected Result check() throws Exception {
if (database.ping()) {
return Result.healthy();
}
return Result.unhealthy("Can't ping database");
}
* databases
* other services
* other dependencies
* make them explicitly invokable

View Slide

trend on them

View Slide

alert on them

View Slide

...render them as markers/buttons/light bulbs/whatever

View Slide

make instrumentation a habit
* just do it

View Slide

ﬁnd optimal usage later
* you’ll never use it if it ain’t there

View Slide

View Slide

http://metrics.codahale.com/

View Slide

View Slide

packaging
* adaptive to different environments

View Slide

one package
* one package, regardless of environment

View Slide

bundle dependencies

View Slide

über jar
(maven shade plugin)
* for example

View Slide

adaptive conﬁguration
* adaptive to different environments

View Slide

isolated
* mocks dependencies with (static) dummy answers

View Slide

org.mockito
mockito-core
test

* maven example

View Slide

org.mockito
mockito-core
compile

* maven example

View Slide

127.0.0.1
* expect everything to be available on 127.0.0.1

View Slide

test / qa / staging / prod
* the usual suspects

View Slide

either detect your environment
* ie, bundle conﬁgurations for all environments

View Slide

or load externalized configuration
* DNS
* ZooKeeper
* CouchDB
* Doozer
* External property/YML/JSON/whatever files
** in one sane specified location (preferably the working directory)

View Slide

strive for zero-touch conﬁguration
* packages should JUST WORK

View Slide

View Slide

the operational aspect needs to be an
integral part of:

View Slide

integral part of: development

View Slide

integral part of: design

View Slide

integral part of: architecture

View Slide

integral part of: reasoning

View Slide

And therefore, a quick comment on...

View Slide

*aaS
....this...

View Slide

...[whatever] as a service
* what ever as a service

View Slide

OUTSOURCE
ALL THE TINGS!

View Slide

to name some
* logging
* alerting

View Slide

Tools! Nice!

View Slide

(new shiny object syndrome)

View Slide

View Slide

aka

View Slide

fallacies of distributed computing

View Slide

View Slide

1. The network is reliable
2. Latency is zero
3. Bandwidth is inﬁnite
4. The network is secure
5. Topology doesn't change
6. There is one administrator
7. Transport cost is zero
8. The network is homogeneous (- James Gosling)
Fallacies of distributed computing
- Peter Deutsch

View Slide

so when using this, we need to seriously consider...

View Slide

1. The network is reliable
2. Latency is zero
3. Bandwidth is inﬁnite
4. The network is secure
5. Topology doesn't change
6. There is one administrator
7. Transport cost is zero
8. The network is homogeneous (- James Gosling)
Fallacies of distributed computing
- Peter Deutsch
...this
* reliability (overall, geo location, connectivity)
* security (communication, retention)
* cost (of using, of not being available)

View Slide

by all means...

View Slide

...use services

View Slide

but not only

View Slide

don’t bet
your operation
on their availability

View Slide

View Slide

responsibility

View Slide

logging

View Slide

YOU

View Slide

metrics

View Slide

YOU

View Slide

packaging

View Slide

YOU

View Slide

conﬁguration

View Slide

YOU

View Slide

sane defaults

View Slide

YOU

View Slide

NOT operations

View Slide

NOT your hosting provider

View Slide

NOT your boss

View Slide

NOT service provider

View Slide

NOT your colleague

View Slide

YOU

View Slide

develop accordingly

View Slide

develop for operations

View Slide

love your operations
@martengustafson
http://marten.gustafson.pp.se/
[email protected]

View Slide

The OPS side of DEV

The OPS side of DEV

Mårten Gustafson

More Decks by Mårten Gustafson

Other Decks in Programming

Featured

Transcript