Tweet
Tweet

Slide 1

Slide 1 text

Sometimes Tools Matter John E. Vincent DevOpsDays Goteborg 2011

Slide 2

Slide 2 text

We all know about DevOps

Slide 3

Slide 3 text

We all know about DevOps I R DEV! I R OPS!

Slide 4

Slide 4 text

So what's the big deal?

Slide 5

Slide 5 text

“With XXXXX you are be able to do easily common task on your local or remote machine. The aim of XXXXX is to become the unique and universal tool language that permit you to make your own brew, apt-get or yum package, same syntax for all your machine.”

Slide 6

Slide 6 text

“With XXXXX you are be able to do easily common task on your local or remote machine. The aim of XXXXX is to become the unique and universal tool language that permit you to make your own brew, apt-get or yum package, same syntax for all your machine.” Not Puppet Not Chef Not CFengine Not Capistrano Not Fabric Not DeployML

Slide 7

Slide 7 text

YANFT (yet another f*ing tool)

Slide 8

Slide 8 text

We're doing something wrong. Something is missing.

Slide 9

Slide 9 text

I'll be over here with Capistrano, kthx You just need to write a Chef recipe and ….

Slide 10

Slide 10 text

You can't solve cultural issues with tools

Slide 11

Slide 11 text

You can't solve cultural issues with tools or can you?

Slide 12

Slide 12 text

Some Issues ● Repository mismatch ● Different languages ● Volatile configuration ● Visibility ● Sensitive information ● Testability ● Packaging

Slide 13

Slide 13 text

Caveats and Comments ● No single tool is going to solve all your problems – sorry. ● You may have already heard of some of these tools. ● You will likely have to “mold” some of these tools to fit your environment ● The tool itself is not the point, it is the end result ● I don't have all the answers...

Slide 14

Slide 14 text

Impedance mismatches

Slide 15

Slide 15 text

Sample Shop ● Operations – Puppet, EC2 ● Development – Maven, Spring ● War files ● Spring properties files ● Metric assloads of XML ● Database changes

Slide 16

Slide 16 text

Issues ● Operations repository isn't application repository ● Developers need to test locally during development ● Properties files are artifacts ● Some settings are environment specific ● Some settings are “sensitive” ● How do new settings get in CM? ● Where do database changes fit? ● What about the rest of the business?

Slide 17

Slide 17 text

Know your Enemy

Slide 18

Slide 18 text

Know your Enemy This sandwich kicked my ass.

Slide 19

Slide 19 text

Know your Enemy This sandwich kicked my ass because I didn't know my enemy

Slide 20

Slide 20 text

The competition isn't between “devops” tools. It's between devops and “shell scripts covered in meat sauce” (with apologies to @littleidea)

Slide 21

Slide 21 text

“shell scripts covered in meat sauce” == inertia

Slide 22

Slide 22 text

Configuration

Slide 23

Slide 23 text

Configuration Competition ● XML files (unless you run bcfg2) ● Key/Value property files (.ini style files) ● YAML ● JSON ● Hard coded “stuff” ● My-effing-SQL, Mongo-effing-DB All of these things are “simpler” to understand.

Slide 24

Slide 24 text

Configuration Champions ● ZooKeeper, Nesoi, Noah ● Moves volatile configuration outside of application ● Can be populated by both operations and development* ● Service discovery ● No immediate need to learn a new DSL/Language ● Can still control access* ● Don't underestimate the power of environment variables ● You still need “proper” configuration management

Slide 25

Slide 25 text

Packaging

Slide 26

Slide 26 text

Packaging Protagonists ● War files ● Lein, Maven/Ivy, Rubygems, Agner, CPAN, Pypi ● RVM ● Homebrew Not always simpler to understand but built into the community.

Slide 27

Slide 27 text

Packaging Princes ● FPM, brew2deb, fpm-cookery ● Makes building OS packages “painless” ● OS packages have value (rollback, versioning, validation) ● Need a single artifact that can be deployed in all environments

Slide 28

Slide 28 text

Development Environments

Slide 29

Slide 29 text

Development Defeat ● Windows/OSX for development, Linux/Solaris for production ● QA different than Production ● Different versions of critical libraries, jars, gems whatever ● Exploded war files vs. Packaged war files “Works on my machine”

Slide 30

Slide 30 text

Development Dreams ● Vagrant, Veewee, Whirr ● Provides “production” in a box ● Developers should be “deploying” locally ● Operations needs to make modules, manifests, cookbooks (whatever) flexible This is one of the hardest to accomplish

Slide 31

Slide 31 text

Visibility

Slide 32

Slide 32 text

Visibility Villains ● uptime, *top, iostat, vmstat, sar* ● ssh ● tail ● Host Obsession Disorder Remember who the competition is...

Slide 33

Slide 33 text

Visibility Victors - part 1 ● Statsd, Graphite, Graylog2, Logstash ● ruby-metrics, codahale metrics for JVM ● “If it moves, graph it” ● Disk is “cheap” ● Dashboards are the bomb. Become one with information radiators. ● Logstash can pipe the shit out of EVERYTHING ● You don't know what you don't need until you know what you have ● Overcompensate initially (but don't over do it!)

Slide 34

Slide 34 text

Visibility Victors - part 2 ● Jenkins, Rundeck, MCollective ● Hosts don't matter, only services ● Sometimes you still need to get on a specific host....to reprovision it. ● Wrap the access for auditing and accountability ● DevOps is not about giving root access to developers ● The “myth” of security.

Slide 35

Slide 35 text

The “other” stuff

Slide 36

Slide 36 text

Shitty Stuff ● Manual database migrations ● Rollbacks ● The myth of the sensitive

Slide 37

Slide 37 text

Smart Stuff ● Flyway, Liquibase, Rails migrations ● Consider NoSQL/Schema-less stores ● Roll forward. Never roll back. ● You can never truly roll back anyway ● “Sensitive” data is a myth (for some values of sensitive)

Slide 38

Slide 38 text

There is, however, ONE tool that can solve almost every technological and cultural problem

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

Just kidding! (sort of)

Slide 41

Slide 41 text

This is the point where you realize I've duped you

Slide 42

Slide 42 text

You are a tool

Slide 43

Slide 43 text

You are THE tool

Slide 44

Slide 44 text

A tale of two tools

Slide 45

Slide 45 text

This is Kelsey

Slide 46

Slide 46 text

Kelsey's Problem ● Operations used Puppet with Cobbler as ENC ● Application configuration was in ERB templates managed by Puppet ● Cultural issues ruled out development actually managing the templates ● System of Record was Cobbler

Slide 47

Slide 47 text

Kelsey's Solution ● Programming motherf**er ● Learned some Java (in a week) ● Wrote some classes, built some jars ● Delivered an XML-RPC library that development could use to query Cobbler for information

Slide 48

Slide 48 text

This is Jordan

Slide 49

Slide 49 text

Jordan's Problem ● Everyone has their own effing packaging format ● Distributions suck ● Debs suck to build ● RPMS suck moderately less to build

Slide 50

Slide 50 text

Jordan's Solution ● Hate Driven Development ● Wrote FPM (eFfing package manager) ● Alien done right ● Converts debs, rpms, pypi, gems, compiled directories to native distro package format

Slide 51

Slide 51 text

Side note: If you ever say “I wonder if someone has done X” stop at the “I” and go to github.com/jordansissel

Slide 52

Slide 52 text

Real Talk ● Step outside your comfort zone ● Be a technologist not a specialist ● SysAdmins: Learn to write some code. Seriously. ● Developers: Learn the OS. Seriously. ● Everyone: Stop blaming other people ● Software can help ease communication but only you can prevent forest fires.

Slide 53

Slide 53 text

Go from this....

Slide 54

Slide 54 text

and this

Slide 55

Slide 55 text

to this

Slide 56

Slide 56 text

Thank You!

Slide 57

Slide 57 text

Keep in touch! ● @lusis - twitter ● lusis – github ● John Vincent – linkedin ● blog.lusis.org ● john@lusis.org