Cross node orchestration with Chef and Noah

Slide 1

Slide 1 text

Cross-node Orchestration for Chef with Noah

Slide 2

Slide 2 text

Cross-node Orchestration for Chef with Noah (sort of)

Slide 3

Slide 3 text

Let's step back a little

Slide 4

Slide 4 text

We've got this part down { "name": "es_dispatcher_lb", "description": "", "json_class": "Chef::Role", "default_attributes": { }, "override_attributes": { "postfix": { "myhostname": "dispatcher-lb.enstratus.com" } }, "chef_type": "role", "run_list": [ "role[es_loadbalancer]", "recipe[enstratus::dispatcher_lb]", "role[monitored_host]" ], "env_run_lists": { } } {

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Congrats! You've got a load balancer! (now what?)

Slide 7

Slide 7 text

Remember this line? "recipe[enstratus::dispatcher_lb]"

Slide 8

Slide 8 text

The deep Chefster backends = search(:node, "role:#{node[:enstratus] [:dispatcher_lb][:dispatcher_server_role]} AND chef_environment:#{node.chef_environment}") + <% @backends.each do |be| -%> server <%= be.name %> <%= be.ipaddress %>:2023 check inter 5000 <% end -%> = "recipe[enstratus::dispatcher_lb]"

Slide 9

Slide 9 text

Slide 10

Slide 10 text

More appropriately = haproxy backend backend backend backend

Slide 11

Slide 11 text

Congrats! You've got a distributed system!

Slide 12

Slide 12 text

What happens when I... ● Add a new backend? ● Remove a dead backend?

Slide 13

Slide 13 text

You want to... ● Have confguration changes on one system affect change on another system ● Gate confguration changes on one system based on state of another system ● Establish relationships between systems

Slide 14

Slide 14 text

But which changes?

Slide 15

Slide 15 text

But... ● Should adding a new backend node cause a chef- client run on the haproxy nodes? ● If the chef-client run on the haproxy node changes a different setting, what node(s) needs to know that? ● What if THOSE nodes change something? NOW who do we tell?

Slide 16

Slide 16 text

Congrats! You've got a graph!

Slide 17

Slide 17 text

It's a graph....

Slide 18

Slide 18 text

It's a graph....

Slide 19

Slide 19 text

It's a graph....

Slide 20

Slide 20 text

It's a graph!

Slide 21

Slide 21 text

Be careful what you wish for ● Circular dependencies ● Transitive dependencies ● Version conflicts ● Computational cost

Slide 22

Slide 22 text

So about Noah

Slide 23

Slide 23 text

What is it? ● Sinatra + Redis app ● HTTP + JSON ● RESTish ● Watches (callbacks) ● Inspired by, but no replacement for, Apache Zookeeper

Slide 24

Slide 24 text

Data goes in... curl -X PUT -d '{"name":"fooapp"}' \ http://noahserverhost:5678/applications/fooapp {"result":"success", "id":"0d52d942-4580-f30e-45154edda2a2fc3d", "action":"create", "name":"fooapp" }

Slide 25

Slide 25 text

Data comes out... curl -X GET \ http://noahserverhost:5678/applications/fooapp { "id":"0d52d942-4580-f30e-4515-4edda2a2fc3d", "tags":[], "links":[], "name":"fooapp", "created_at":"Thu May 10 02:22:53 UTC 2012", "updated_at":"Thu May 10 02:22:53 UTC 2012", "configurations":{} }

Slide 26

Slide 26 text

Primitives/Data Types ● Host ● Service ● Application ● Configuration These are “get you started” primitives

Slide 27

Slide 27 text

Ephemerals ● Arbitrary paths in the URL scheme namespaced under /ephemerals ● Key is path name /ephemerals/foo/bar/chefconf/ ● Value is...whatever Doesn't have to be JSON. Could be a blob. ● No relation in between keys /foo/bar is unrelated to /foo

Slide 28

Slide 28 text

Watches ● “Pluggable” callbacks ● Think triggers ● Register against any path in the system ● Changes at that path (or below it) execute the registered callback ● Sends JSON dump of the state of change

Slide 29

Slide 29 text

Watch Example (part 1) ● Add some data curl -XPUT -d '1' http://localhost:5678/ephemerals/chef/notify/localhost ● Get a response {"action":"create", "result":"success", "id":"7e743edd-a40f-5062-a5a0-70a8b4df7721", "path":"/chef/notify/localhost", "data":"1", "created_at":"Wed May 09 03:56:25 UTC 2012", "updated_at":"Wed May 09 03:56:25 UTC 2012"}

Slide 30

Slide 30 text

Watch Example (part 2) ● Register a watch curl -XPUT -d '{"pattern":"//noah/ephemerals/chef/notify/localhost", "endpoint":"http://localhost:8080"}' http://localhost:5678/watches/ ● Get a response {"action":"create","result":"success","id":"51828298-ad79-26c3-459b- fd757b85db75","pattern":"//noah/ephemerals/chef/notify/localhost","name" :"Ly9ub2FoL2VwaGVtZXJhbHMvY2hlZi9ub3RpZnkvbG9jYWxob3N0fGh0dHA6Ly9sb2NhbG hvc3Q6ODA4MQ==","endpoint":"http://localhost:8080,"created_at":"Thu May 10 03:02:25 UTC 2012","updated_at":"Thu May 10 03:02:25 UTC 2012"}

Slide 31

Slide 31 text

What just happened? ● We told Noah that when a change happens to /ephemerals/chef/notify/localhost ● send an HTTP POST to http://localhost:8080/

Slide 32

Slide 32 text

Now what? ● Change that data curl -XPUT -d '2' http://localhost:5678/ephemerals/chef/notify/localhost Get a response {"action":"update", "result":"success", "id":"7e743edd-a40f-5062-a5a0-70a8b4df7721", "path":"/chef/notify/localhost", "data":"2", "created_at":"Wed May 09 03:56:25 UTC 2012", "updated_at":"Wed May 09 04:00:00 UTC 2012"}

Slide 33

Slide 33 text

But something else happened this time... I, [2012-05-09T00:01:00.395491 #36180] INFO -- Noah::Agent: Found new watches D, [2012-05-09T00:01:00.395611 #36180] DEBUG -- Noah::Agent: Current watch count: 0 D, [2012-05-09T00:01:00.396681 #36180] DEBUG -- Noah::Agent: New watch count: 1 I, [2012-05-09T00:01:00.396850 #36180] INFO -- Noah::Agents::HttpAgent: Noah::Agents::HttpAgent worker initiated I, [2012-05-09T23:10:36.438180 #38052] INFO -- Noah::Agents::HttpAgent: Message posted to http://localhost:8080 successfully

Slide 34

Slide 34 text

And what did localhost:8080 see? { "id"=>"7e743edd-a40f-5062-a5a0-70a8b4df7721", "tags"=>[], "links"=>[], "path"=>"/ephemerals//chef/notify/localhost", "data"=>"1", "created_at"=>"Wed May 09 03:56:25 UTC 2012", "updated_at"=>"Thu May 10 03:10:36 UTC 2012", "action"=>"update", "pubcategory"=>"//noah/ephemerals/chef/notify/localhost" }

Slide 35

Slide 35 text

What can we do with that? ● Parse the data to effect some change? ● Use it as a signaling mechanism? ● Alert on it? ● More importantly, how can we use it with Chef?

Slide 36

Slide 36 text

Noah cookbook ● http://github.com/lusis-cookbooks/noah ● Installs Noah + Redis ● Adds a noah_get method ● Automatically registers node in Noah (noah::register recipe) ● Also some new resources....

Slide 37

Slide 37 text

Noah LWRP ● Register any primitive ● noah_application ● noah_service ● noah_configuration ● noah_application ● Dump some data in an ephemeral node ● noah_ephemeral ● Block the chef run until some data shows up in Noah ● noah_block

Slide 38

Slide 38 text

Common Scenario haproxy django1 django2 mysql

Slide 39

Slide 39 text

Current Solution ● knife bootstrap -r “role[mysql]” ● Wait ● knife bootstrap -r “role[django1] ● Wait ● knife bootstrap -r “role[django2]” ● Wait ● knife bootstrap -r “role[haproxy]”

Slide 40

Slide 40 text

noah_block haproxy django1 django2 mysql

Slide 41

Slide 41 text

noah_block haproxy django1 django2 mysql

Slide 42

Slide 42 text

noah_block haproxy django1 django2 mysql ● Needs a DB ● Loads DB Needs to wait for initial DB load Needs backends Needs a DB

Slide 43

Slide 43 text

Django1/2 recipe noah_block "wait_for_db" do path "http://noah:5678/ephemerals/#{db_master_role}" timeout 600 data "ready" retry_interval 5 on_failure :retry end

Slide 44

Slide 44 text

Django2 recipe noah_block "wait_for_migration" do path "http://noah:5678/ephemerals/migration-status" timeout 600 data "done" retry_interval 5 on_failure :retry end

Slide 45

Slide 45 text

haproxy recipe noah_block "wait_for_backends" do path "http://noah:5678/ephemerals/backends" timeout 600 data "available" retry_interval 5 on_failure :retry end

Slide 46

Slide 46 text

Triggering Chef Runs ● Use noah_ephemeral at end of the dependency run_list (on the backend node) ● Leverage string interpolation for path name ● Data is unimportant, just the signal

Slide 47

Slide 47 text

Small Problem... There's no “pokeable” Chef client

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

Small Solution ● Small web app ● listens for HTTP POST requests ● runs chef-client

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

Almost.....

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

Demo time

Slide 55

Slide 55 text

Well that was fun ● Obviously needs to be done “right” ● Add security ● Write your own! ● Watches are pluggable. ● AMQP ● ZeroMQ ● HTTPS + Token auth?