Beyond the Node - Speaker Deck

Slide 1

Slide 1 text

Beyond the Node Advanced 'Arkestration' with Noah PuppetConf 2011

Slide 2

Slide 2 text

.plan  What's the problem?  Orchestration  Overview of Noah  Use Cases  Q&A

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

A Brief History of Time (with apologies to Hawking)

Slide 5

Slide 5 text

Configuration Management ● Manual

Slide 6

Slide 6 text

Configuration Management

Slide 7

Slide 7 text

Configuration Management

Slide 8

Slide 8 text

Configuration Management ● Manual ● Scripted

Slide 9

Slide 9 text

Configuration Management

Slide 10

Slide 10 text

Configuration Management ● Manual ● Scripted ● Intelligent

Slide 11

Slide 11 text

CFEngine

Slide 12

Slide 12 text

Chef

Slide 13

Slide 13 text

Puppet

Slide 14

Slide 14 text

Configuration Management ● Manual ● Scripted ● Intelligent

Slide 15

Slide 15 text

So we've solved this, right?

Slide 16

Slide 16 text

“I think..and I don't know if anyone would agree, that configuration management is a solved problem at this point, right?”

Slide 17

Slide 17 text

“I think..and I don't know if anyone would agree, that configuration management is a solved problem at this point, right?” WTF? WTF?

Slide 18

Slide 18 text

"The point I want to make..Configuration management is not a solved problem...and it's dangerous to make the mistake to think that the way we do things now is the best way to do them..." - Andrew Clay Shafer

Slide 19

Slide 19 text

“what I was attempting to say ... is that the current crop of configuration management tools have reached a usable point where they do enough (for now). What we’re seeing as questions now are 'How do I think beyond the single node where this tool is running?'”

Slide 20

Slide 20 text

“what I was attempting to say (epic fail, I might add) is that the current crop of configuration management tools have reached a usable point where they do enough (for now). What we’re seeing as questions now are 'How do I think beyond the single node where this tool is running?'”

Slide 21

Slide 21 text

Orchestration and Noah Noah attempts to do two things:  Bridge an impedance mismatch between operational, developmental and application configuration  Provide mechanisms for coordination between applications, nodes, services, configuration management and other infrastructure aspects

Slide 22

Slide 22 text

Example Use Case Noah Nagios Puppet (app) Load Balancer 1) Capacity Reached. Tell Noah 2) Noah tells provisioning system 3) Capacity allocated. Tell Noah 4) Noah triggers Puppet on LB 5) Noah triggers Puppet on Nagios

Slide 23

Slide 23 text

“Inspired” by

Slide 24

Slide 24 text

"ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications."

Slide 25

Slide 25 text

Features  Hierarchical name space of data registers similar to a Unix filesystem (znodes)  Leader election, locking, configuration, sequences  Distributed and highly available  Watches Downsides  Persistent connections required  Official bindings are C and Java  Somewhat "complex" to setup and run

Slide 26

Slide 26 text

Noah and ZooKeeper Noah is essentially a re-imagining of ZK, taking the things I liked:  Znode concept  Watches and making it work over HTTP using JSON

Slide 27

Slide 27 text

Noah Basics

Slide 28

Slide 28 text

The Primitives  Hosts Has status (up, down, pending) Can have Services  Services Has status (up, down, pending)  Must belong to a Host   Applications Can have Configurations  Configurations Can belong to many Applications

Slide 29

Slide 29 text

The Not So Primitives  Ephemerals Identified by a “path”  Tags Can be applied to any Primitive or Ephemeral  Links Can be applied to any Primitive, Ephemeral or Tag  Watches Can be applied at any hierarchy or specific object

Slide 30

Slide 30 text

URL Patterns ● Hosts ● /hosts/app01/services/httpd ● Services ● /services/httpd/app02 ● Applications ● /applications/my_app/configurations/log_level ● Configurations ● /configurations/log_level

Slide 31

Slide 31 text

GETting Data

Slide 32

Slide 32 text

{ "foo": { "created_at": "2011-04-20 04:50:41 UTC", "updated_at": "2011-04-20 04:50:41 UTC", "tags": [ ], "id": "4af53f8d-36d2-2158-c4a1-514ff3578048", "links": [ ], "services": { }, "status": "up" }, "bar": { } }

Slide 33

Slide 33 text

GETting Data ● Hosts

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

GETting Data ● Hosts ● Services

Slide 37

Slide 37 text

{ "foo_svc": { "foo": { "created_at": "2011-04-20 05:09:44 UTC", "updated_at": "2011-04-20 05:09:44 UTC", "tags": [ ], "id": "53504526-a5d7-a604-5fe8-1f8fd2e1aef0", "links": [ ], "status": "down" } }, "both": { } } Service name

Slide 38

Slide 38 text

Slide 39

Slide 39 text

GETting Data ● Hosts ● Services ● Applications

Slide 40

Slide 40 text

{ "myrailsapp1": { "created_at": "2011-04-21 06:57:42 UTC", "updated_at": "2011-04-21 06:57:42 UTC", "tags": [ "sample_data", "production" ], "id": "8774fead-e63a-8e5c-0453-1b5330bbd04c", "links": [ "/my_sample_organization" ], "configurations": { "database.yml": { "format": "yaml", "body": "development:\n database: development_database\n adapter: mysql\n username: dev_user\n password: dev_password\n" } } }, "myrestapp1": { } } App Name

Slide 41

Slide 41 text

Slide 42

Slide 42 text

GETting Data ● Hosts ● Services ● Applications ● Configurations *

Slide 43

Slide 43 text

{ "barconf1": { "created_at": "2011-04-27 01:20:37 UTC", "format": "string", "body": "barbody1", "updated_at": "2011-04-27 01:20:37 UTC", "tags": [ ], "id": "aa183dce-30ec-2d7e-c6f9-f3385161a7e6", "links": [ ] }, "barconf2": { } }

Slide 44

Slide 44 text

GETting Data ● Hosts ● Services ● Applications ● Configurations * ● Ephemerals *

Slide 45

Slide 45 text

GETting Data ● Hosts ● Services ● Applications ● Configurations * ● Ephemerals * /ephemerals/my/config/setting returns the data stored there.

Slide 46

Slide 46 text

PUTting Data  /hosts/host_name  /services/service_name/host_name  /applications/application_name  /configurations/configuration_name  /ephemerals/arbitrary/path/to/whatever

Slide 47

Slide 47 text

PUTting Data  /hosts/host_name  /services/service_name/host_name  /applications/application_name  /configurations/configuration_name  /ephemerals/arbitrary/path/to/whatever

Slide 48

Slide 48 text

PUTting Data  Host {“status”:”up”}  Service {“status”:”up”,”host_status”:”up”}  Application {“name”:”app_name”}  Configuration {“format”:”format_type”,”body”:”data”}  Ephemeral raw data

Slide 49

Slide 49 text

DELETEing Data  /hosts/host_name  /services/service_name/host_name  /applications/application_name  /configurations/configuration_name  /ephemerals/arbitrary/path/to/whatever

Slide 50

Slide 50 text

Tagging  Append “tag” to any PUT url with PUT method e.g. '/applications/some_application/tag'  {“tags”:”sometag”} or {“tags”:[“tag1”,”tag2”]}  Call DELETE instead of PUT to remove tags  GET '/tags' for all tags or '/tag/tagname' for specific tag

Slide 51

Slide 51 text

Linking  Append “link” to any PUT url with PUT method e.g. '/applications/some_application/link'  {“link_name”:”my_organization”}  Call DELETE instead of PUT to remove links *  GET '/link_name' for all linked objects *  Designed to allow custom namespaces

Slide 52

Slide 52 text

Watches

Slide 53

Slide 53 text

What are Watches?  A bit different than ZK watches  Async  Persistent *  Pluggable (URI pattern based)  Hierarchical (superset and subset)  Append '/watch' to almost any url path  {“endpoint”:”http://some_host:port/ping_me”}  '/watch' root endpoint as well { “endpoint”:”http://some_host/ping”, ”pattern”:”//noah/applications/app_name” }

Slide 54

Slide 54 text

Create a Watch $ curl -XPUT -d '{“pattern”:”//noah/applications”,"endpoint":" http://host/ping"}' http://noahserver/watches

Slide 55

Slide 55 text

Perform some operation $ curl -XPUT -d '{"name":"fooapp2"}' http://noahserver/applications/fooapp2

Slide 56

Slide 56 text

Message sent to Endpoint { "id":"be9f6a45-0d49-3f00-59a8-c6ad61da24e1", "tags":[], "links":[], "name":"fooapp2", "created_at":"2011-09-15 03:07:39 UTC", "updated_at":"2011-09-15 03:07:39 UTC", "configurations":{}, "action":"create", "pubcategory":"//noah/applications/fooapp2" }

Slide 57

Slide 57 text

Perform another operation $ curl -XPUT -d '{"tags":"footag2"}' http://noahserver/applications/fooapp2/tag

Slide 58

Slide 58 text

Message sent to Endpoint { "id":"be9f6a45-0d49-3f00-59a8-c6ad61da24e1", "tags":["footag"], "links":[], "name":"fooapp2", "created_at":"2011-09-15 03:07:39 UTC", "updated_at":"2011-09-15 03:15:03 UTC", "configurations":{}, "action":"update", "pubcategory":"//noah/applications/fooapp2" }

Slide 59

Slide 59 text

Watches are “plugin” based You can write your own plugins with Ruby.  URI Pattern *  amqp://user:pass@host:port/exchange  rundeck://token@host:port/job_id  puppet://hostname/  statsd://host:port/metric.path.name.action  Currently based on Event Machine  Future based on Actors and worker pools

Slide 60

Slide 60 text

Watch Notes  Creator of the watch is unrelated to receiver  Don't have to use the “official” watch daemon  You can watch the watches (PubSub)

Slide 61

Slide 61 text

Basic Puppet Integration  http://github.com/jamtur01/puppet-noah  ENC, Functions and Facts  Hosts → Nodes  Host's Services → Classes  Configurations → Facts (string types only)

Slide 62

Slide 62 text

So where does it fit? Noah Puppet Apps Scripts Monitoring Deploy CI

Slide 63

Slide 63 text

Noah + Puppet + Applications Noah Puppet Applications

Slide 64

Slide 64 text

Common P+A Use Case  Puppet knows Nodes and Classes  App has configuration file  Configuration file controlled by Puppet  Puppet client run required to update file

Slide 65

Slide 65 text

Common P+A Issues  Puppet repo is not app repo  Devs must learn ERB  Devs might even have to learn Puppet  Introducing new settings and communication  Daemon-mode is exception rather than rule

Slide 66

Slide 66 text

N+P+A Version  Noah knows Nodes and Classes  Move volatile settings into Noah  Configuration file controlled by Puppet  App gets volatile settings from Noah *  Future Puppet runs update Noah  Noah tells App *  Truth vs. Reality  My DB master is foo  My DB master right NOW is bar

Slide 67

Slide 67 text

Volatile Settings?  Not all configuration is created equal  “Elasticity” - Pool of worker node EC2 ips  Ad-hoc configuration – Logging levels  Environment aware – Nodes of class “foo”

Slide 68

Slide 68 text

N+P+A Benefits  Devs don't HAVE to learn ERB  Devs don't HAVE to learn Puppet  Ops and Dev friendly bridge point  No unintended changes released

Slide 69

Slide 69 text

Integration with CI Noah Puppet CI Deploy

Slide 70

Slide 70 text

Integration with CI ● Jenkins asks Noah for nodes of class “foo” ● Jenkins triggers Puppet (or deploy) on appropriate nodes ● Jenkins is always “current”

Slide 71

Slide 71 text

Integration with Monitoring Noah Puppet Monitoring App

Slide 72

Slide 72 text

Integration with Monitoring ● Puppet manages Nagios configs ● Nagios updates Noah status (event handler) ● Noah optionally fires watches based on status change ● Endpoint is application. App “reconfigures”

Slide 73

Slide 73 text

Integration with Scripts Noah Scripts (node 1) Scripts (node 2) Scripts (node 3)

Slide 74

Slide 74 text

Integration with Scripts ● Provide “locking” and “coordination” of cron jobs across hosts ● Stop dealing with lock files on host ● Use ephemerals and curl

Slide 75

Slide 75 text

Gotchas ● Noah itself is NOT distributed * ● Data redundancy based on Redis ● No ACLs * ● Async

Slide 76

Slide 76 text

Other options ● ZooKeeper ● Doozer ● NoSQL stores (Redis + Webdis, Riak) * ● Nesoi * ● Write your own!

Slide 77

Slide 77 text

Questions?

Slide 78

Slide 78 text

Noah  Github: http://github.com/lusis/Noah  IRC: #noah  Blog(s): http://blog.lusis.org/blog/categories/noah http://lusislog.blogspot.com/search/label/noah

Slide 79

Slide 79 text

John E. Vincent  Twitter: @lusis  Github: http://github.com/lusis  IRC: lusis  LinkedIn: http://linkedin.com/in/lusis  Blog(s): http://blog.lusis.org http://lusislog.blogspot.com

Slide 80

Slide 80 text

Thank You!