Save 37% off PRO during our Black Friday Sale! »

Cross node orchestration with Chef and Noah

6385d06011a9633cd45cab0662ae9eb8?s=47 John Vincent
May 16, 2012
1.6k

Cross node orchestration with Chef and Noah

long-form of my talk at #chefconf

6385d06011a9633cd45cab0662ae9eb8?s=128

John Vincent

May 16, 2012
Tweet

Transcript

  1. Cross-node Orchestration for Chef with Noah

  2. Cross-node Orchestration for Chef with Noah (sort of)

  3. Let's step back a little

  4. We've got this part down { "name": "es_dispatcher_lb", "description": "",

    "json_class": "Chef::Role", "default_attributes": { }, "override_attributes": { "postfix": { "myhostname": "dispatcher-lb.enstratus.com" } }, "chef_type": "role", "run_list": [ "role[es_loadbalancer]", "recipe[enstratus::dispatcher_lb]", "role[monitored_host]" ], "env_run_lists": { } } {
  5. We've got this part down { "name": "es_dispatcher_lb", "description": "",

    "json_class": "Chef::Role", "default_attributes": { }, "override_attributes": { "postfix": { "myhostname": "dispatcher-lb.enstratus.com" } }, "chef_type": "role", "run_list": [ "role[es_loadbalancer]", "recipe[enstratus::dispatcher_lb]", "role[monitored_host]" ], "env_run_lists": { } } Do this Then this Then this
  6. Congrats! You've got a load balancer! (now what?)

  7. Remember this line? "recipe[enstratus::dispatcher_lb]"

  8. The deep Chefster backends = search(:node, "role:#{node[:enstratus] [:dispatcher_lb][:dispatcher_server_role]} AND chef_environment:#{node.chef_environment}")

    + <% @backends.each do |be| -%> server <%= be.name %> <%= be.ipaddress %>:2023 check inter 5000 <% end -%> = "recipe[enstratus::dispatcher_lb]"
  9. The deep Chefster backends = search(:node, "role:#{node[:enstratus] [:dispatcher_lb][:dispatcher_server_role]} AND chef_environment:#{node.chef_environment}")

    + <% @backends.each do |be| -%> server <%= be.name %> <%= be.ipaddress %>:2023 check inter 5000 <% end -%> = "recipe[enstratus::dispatcher_lb]"
  10. More appropriately = haproxy backend backend backend backend

  11. Congrats! You've got a distributed system!

  12. What happens when I... • Add a new backend? •

    Remove a dead backend?
  13. You want to... • Have confguration changes on one system

    affect change on another system • Gate confguration changes on one system based on state of another system • Establish relationships between systems
  14. But which changes?

  15. But... • Should adding a new backend node cause a

    chef- client run on the haproxy nodes? • If the chef-client run on the haproxy node changes a different setting, what node(s) needs to know that? • What if THOSE nodes change something? NOW who do we tell?
  16. Congrats! You've got a graph!

  17. It's a graph....

  18. It's a graph....

  19. It's a graph....

  20. It's a graph!

  21. Be careful what you wish for • Circular dependencies •

    Transitive dependencies • Version conflicts • Computational cost
  22. So about Noah

  23. What is it? • Sinatra + Redis app • HTTP

    + JSON • RESTish • Watches (callbacks) • Inspired by, but no replacement for, Apache Zookeeper
  24. Data goes in... curl -X PUT -d '{"name":"fooapp"}' \ http://noahserverhost:5678/applications/fooapp

    {"result":"success", "id":"0d52d942-4580-f30e-45154edda2a2fc3d", "action":"create", "name":"fooapp" }
  25. Data comes out... curl -X GET \ http://noahserverhost:5678/applications/fooapp { "id":"0d52d942-4580-f30e-4515-4edda2a2fc3d",

    "tags":[], "links":[], "name":"fooapp", "created_at":"Thu May 10 02:22:53 UTC 2012", "updated_at":"Thu May 10 02:22:53 UTC 2012", "configurations":{} }
  26. Primitives/Data Types • Host • Service • Application • Configuration

    These are “get you started” primitives
  27. Ephemerals • Arbitrary paths in the URL scheme namespaced under

    /ephemerals • Key is path name /ephemerals/foo/bar/chefconf/ • Value is...whatever Doesn't have to be JSON. Could be a blob. • No relation in between keys /foo/bar is unrelated to /foo
  28. Watches • “Pluggable” callbacks • Think triggers • Register against

    any path in the system • Changes at that path (or below it) execute the registered callback • Sends JSON dump of the state of change
  29. Watch Example (part 1) • Add some data curl -XPUT

    -d '1' http://localhost:5678/ephemerals/chef/notify/localhost • Get a response {"action":"create", "result":"success", "id":"7e743edd-a40f-5062-a5a0-70a8b4df7721", "path":"/chef/notify/localhost", "data":"1", "created_at":"Wed May 09 03:56:25 UTC 2012", "updated_at":"Wed May 09 03:56:25 UTC 2012"}
  30. Watch Example (part 2) • Register a watch curl -XPUT

    -d '{"pattern":"//noah/ephemerals/chef/notify/localhost", "endpoint":"http://localhost:8080"}' http://localhost:5678/watches/ • Get a response {"action":"create","result":"success","id":"51828298-ad79-26c3-459b- fd757b85db75","pattern":"//noah/ephemerals/chef/notify/localhost","name" :"Ly9ub2FoL2VwaGVtZXJhbHMvY2hlZi9ub3RpZnkvbG9jYWxob3N0fGh0dHA6Ly9sb2NhbG hvc3Q6ODA4MQ==","endpoint":"http://localhost:8080,"created_at":"Thu May 10 03:02:25 UTC 2012","updated_at":"Thu May 10 03:02:25 UTC 2012"}
  31. What just happened? • We told Noah that when a

    change happens to /ephemerals/chef/notify/localhost • send an HTTP POST to http://localhost:8080/
  32. Now what? • Change that data curl -XPUT -d '2'

    http://localhost:5678/ephemerals/chef/notify/localhost Get a response {"action":"update", "result":"success", "id":"7e743edd-a40f-5062-a5a0-70a8b4df7721", "path":"/chef/notify/localhost", "data":"2", "created_at":"Wed May 09 03:56:25 UTC 2012", "updated_at":"Wed May 09 04:00:00 UTC 2012"}
  33. But something else happened this time... I, [2012-05-09T00:01:00.395491 #36180] INFO

    -- Noah::Agent: Found new watches D, [2012-05-09T00:01:00.395611 #36180] DEBUG -- Noah::Agent: Current watch count: 0 D, [2012-05-09T00:01:00.396681 #36180] DEBUG -- Noah::Agent: New watch count: 1 I, [2012-05-09T00:01:00.396850 #36180] INFO -- Noah::Agents::HttpAgent: Noah::Agents::HttpAgent worker initiated I, [2012-05-09T23:10:36.438180 #38052] INFO -- Noah::Agents::HttpAgent: Message posted to http://localhost:8080 successfully
  34. And what did localhost:8080 see? { "id"=>"7e743edd-a40f-5062-a5a0-70a8b4df7721", "tags"=>[], "links"=>[], "path"=>"/ephemerals//chef/notify/localhost",

    "data"=>"1", "created_at"=>"Wed May 09 03:56:25 UTC 2012", "updated_at"=>"Thu May 10 03:10:36 UTC 2012", "action"=>"update", "pubcategory"=>"//noah/ephemerals/chef/notify/localhost" }
  35. What can we do with that? • Parse the data

    to effect some change? • Use it as a signaling mechanism? • Alert on it? • More importantly, how can we use it with Chef?
  36. Noah cookbook • http://github.com/lusis-cookbooks/noah • Installs Noah + Redis •

    Adds a noah_get method • Automatically registers node in Noah (noah::register recipe) • Also some new resources....
  37. Noah LWRP • Register any primitive • noah_application • noah_service

    • noah_configuration • noah_application • Dump some data in an ephemeral node • noah_ephemeral • Block the chef run until some data shows up in Noah • noah_block
  38. Common Scenario haproxy django1 django2 mysql

  39. Current Solution • knife bootstrap -r “role[mysql]” • Wait •

    knife bootstrap -r “role[django1] • Wait • knife bootstrap -r “role[django2]” • Wait • knife bootstrap -r “role[haproxy]”
  40. noah_block haproxy django1 django2 mysql

  41. noah_block haproxy django1 django2 mysql

  42. noah_block haproxy django1 django2 mysql • Needs a DB •

    Loads DB Needs to wait for initial DB load Needs backends Needs a DB
  43. Django1/2 recipe noah_block "wait_for_db" do path "http://noah:5678/ephemerals/#{db_master_role}" timeout 600 data

    "ready" retry_interval 5 on_failure :retry end
  44. Django2 recipe noah_block "wait_for_migration" do path "http://noah:5678/ephemerals/migration-status" timeout 600 data

    "done" retry_interval 5 on_failure :retry end
  45. haproxy recipe noah_block "wait_for_backends" do path "http://noah:5678/ephemerals/backends" timeout 600 data

    "available" retry_interval 5 on_failure :retry end
  46. Triggering Chef Runs • Use noah_ephemeral at end of the

    dependency run_list (on the backend node) • Leverage string interpolation for path name • Data is unimportant, just the signal
  47. Small Problem... There's no “pokeable” Chef client

  48. None
  49. Small Solution • Small web app • listens for HTTP

    POST requests • runs chef-client
  50. None
  51. None
  52. Almost.....

  53. None
  54. Demo time

  55. Well that was fun • Obviously needs to be done

    “right” • Add security • Write your own! • Watches are pluggable. • AMQP • ZeroMQ • HTTPS + Token auth?
  56. Wrap up • Questions? • Comments? • Rude remarks?

  57. Thanks! github.com/lusis twitter.com/lusis blog.lusis.org github.com/lusis/noah