$30 off During Our Annual Pro Sale. View Details »

Cross node orchestration with Chef and Noah

John Vincent
May 16, 2012
1.8k

Cross node orchestration with Chef and Noah

long-form of my talk at #chefconf

John Vincent

May 16, 2012
Tweet

Transcript

  1. Cross-node Orchestration for
    Chef with Noah

    View Slide

  2. Cross-node Orchestration for
    Chef with Noah
    (sort of)

    View Slide

  3. Let's step back a little

    View Slide

  4. We've got this part down
    {
    "name": "es_dispatcher_lb",
    "description": "",
    "json_class": "Chef::Role",
    "default_attributes": {
    },
    "override_attributes": {
    "postfix": {
    "myhostname": "dispatcher-lb.enstratus.com"
    }
    },
    "chef_type": "role",
    "run_list": [
    "role[es_loadbalancer]",
    "recipe[enstratus::dispatcher_lb]",
    "role[monitored_host]"
    ],
    "env_run_lists": {
    }
    } {

    View Slide

  5. We've got this part down
    {
    "name": "es_dispatcher_lb",
    "description": "",
    "json_class": "Chef::Role",
    "default_attributes": {
    },
    "override_attributes": {
    "postfix": {
    "myhostname": "dispatcher-lb.enstratus.com"
    }
    },
    "chef_type": "role",
    "run_list": [
    "role[es_loadbalancer]",
    "recipe[enstratus::dispatcher_lb]",
    "role[monitored_host]"
    ],
    "env_run_lists": {
    }
    }
    Do this
    Then this
    Then this

    View Slide

  6. Congrats! You've got a load
    balancer!
    (now what?)

    View Slide

  7. Remember this line?
    "recipe[enstratus::dispatcher_lb]"

    View Slide

  8. The deep Chefster
    backends = search(:node, "role:#{node[:enstratus]
    [:dispatcher_lb][:dispatcher_server_role]} AND
    chef_environment:#{node.chef_environment}")
    +
    <% @backends.each do |be| -%>
    server <%= be.name %> <%= be.ipaddress %>:2023
    check inter 5000
    <% end -%>
    =
    "recipe[enstratus::dispatcher_lb]"

    View Slide

  9. The deep Chefster
    backends = search(:node, "role:#{node[:enstratus]
    [:dispatcher_lb][:dispatcher_server_role]} AND
    chef_environment:#{node.chef_environment}")
    +
    <% @backends.each do |be| -%>
    server <%= be.name %> <%= be.ipaddress %>:2023
    check inter 5000
    <% end -%>
    =
    "recipe[enstratus::dispatcher_lb]"

    View Slide

  10. More appropriately
    =
    haproxy
    backend
    backend backend
    backend

    View Slide

  11. Congrats! You've got a
    distributed system!

    View Slide

  12. What happens when I...

    Add a new backend?

    Remove a dead backend?

    View Slide

  13. You want to...

    Have confguration changes on one system affect
    change on another system

    Gate confguration changes on one system based
    on state of another system

    Establish relationships between systems

    View Slide

  14. But which changes?

    View Slide

  15. But...

    Should adding a new backend node cause a chef-
    client run on the haproxy nodes?

    If the chef-client run on the haproxy node
    changes a different setting, what node(s) needs to
    know that?

    What if THOSE nodes change something? NOW
    who do we tell?

    View Slide

  16. Congrats! You've got a graph!

    View Slide

  17. It's a graph....

    View Slide

  18. It's a graph....

    View Slide

  19. It's a graph....

    View Slide

  20. It's a graph!

    View Slide

  21. Be careful what you wish for

    Circular dependencies

    Transitive dependencies

    Version conflicts

    Computational cost

    View Slide

  22. So about Noah

    View Slide

  23. What is it?

    Sinatra + Redis app

    HTTP + JSON

    RESTish

    Watches (callbacks)

    Inspired by, but no replacement for, Apache
    Zookeeper

    View Slide

  24. Data goes in...
    curl -X PUT -d '{"name":"fooapp"}' \
    http://noahserverhost:5678/applications/fooapp
    {"result":"success",
    "id":"0d52d942-4580-f30e-45154edda2a2fc3d",
    "action":"create",
    "name":"fooapp"
    }

    View Slide

  25. Data comes out...
    curl -X GET \
    http://noahserverhost:5678/applications/fooapp
    {
    "id":"0d52d942-4580-f30e-4515-4edda2a2fc3d",
    "tags":[],
    "links":[],
    "name":"fooapp",
    "created_at":"Thu May 10 02:22:53 UTC 2012",
    "updated_at":"Thu May 10 02:22:53 UTC 2012",
    "configurations":{}
    }

    View Slide

  26. Primitives/Data Types

    Host

    Service

    Application

    Configuration
    These are “get you started” primitives

    View Slide

  27. Ephemerals

    Arbitrary paths in the URL scheme namespaced under
    /ephemerals

    Key is path name
    /ephemerals/foo/bar/chefconf/

    Value is...whatever
    Doesn't have to be JSON. Could be a blob.

    No relation in between keys
    /foo/bar is unrelated to /foo

    View Slide

  28. Watches

    “Pluggable” callbacks

    Think triggers

    Register against any path in the system

    Changes at that path (or below it) execute the
    registered callback

    Sends JSON dump of the state of change

    View Slide

  29. Watch Example
    (part 1)

    Add some data
    curl -XPUT -d '1'
    http://localhost:5678/ephemerals/chef/notify/localhost

    Get a response
    {"action":"create",
    "result":"success",
    "id":"7e743edd-a40f-5062-a5a0-70a8b4df7721",
    "path":"/chef/notify/localhost",
    "data":"1",
    "created_at":"Wed May 09 03:56:25 UTC 2012",
    "updated_at":"Wed May 09 03:56:25 UTC 2012"}

    View Slide

  30. Watch Example
    (part 2)

    Register a watch
    curl -XPUT -d
    '{"pattern":"//noah/ephemerals/chef/notify/localhost",
    "endpoint":"http://localhost:8080"}'
    http://localhost:5678/watches/

    Get a response
    {"action":"create","result":"success","id":"51828298-ad79-26c3-459b-
    fd757b85db75","pattern":"//noah/ephemerals/chef/notify/localhost","name"
    :"Ly9ub2FoL2VwaGVtZXJhbHMvY2hlZi9ub3RpZnkvbG9jYWxob3N0fGh0dHA6Ly9sb2NhbG
    hvc3Q6ODA4MQ==","endpoint":"http://localhost:8080,"created_at":"Thu May
    10 03:02:25 UTC 2012","updated_at":"Thu May 10 03:02:25 UTC 2012"}

    View Slide

  31. What just happened?

    We told Noah that when a change happens to
    /ephemerals/chef/notify/localhost

    send an HTTP POST to
    http://localhost:8080/

    View Slide

  32. Now what?

    Change that data
    curl -XPUT -d '2'
    http://localhost:5678/ephemerals/chef/notify/localhost
    Get a response
    {"action":"update",
    "result":"success",
    "id":"7e743edd-a40f-5062-a5a0-70a8b4df7721",
    "path":"/chef/notify/localhost",
    "data":"2",
    "created_at":"Wed May 09 03:56:25 UTC 2012",
    "updated_at":"Wed May 09 04:00:00 UTC 2012"}

    View Slide

  33. But something else happened
    this time...
    I, [2012-05-09T00:01:00.395491 #36180] INFO --
    Noah::Agent: Found new watches
    D, [2012-05-09T00:01:00.395611 #36180] DEBUG --
    Noah::Agent: Current watch count: 0
    D, [2012-05-09T00:01:00.396681 #36180] DEBUG --
    Noah::Agent: New watch count: 1
    I, [2012-05-09T00:01:00.396850 #36180] INFO --
    Noah::Agents::HttpAgent: Noah::Agents::HttpAgent
    worker initiated
    I, [2012-05-09T23:10:36.438180 #38052] INFO --
    Noah::Agents::HttpAgent: Message posted to
    http://localhost:8080 successfully

    View Slide

  34. And what did localhost:8080
    see?
    {
    "id"=>"7e743edd-a40f-5062-a5a0-70a8b4df7721",
    "tags"=>[],
    "links"=>[],
    "path"=>"/ephemerals//chef/notify/localhost",
    "data"=>"1",
    "created_at"=>"Wed May 09 03:56:25 UTC 2012",
    "updated_at"=>"Thu May 10 03:10:36 UTC 2012",
    "action"=>"update",
    "pubcategory"=>"//noah/ephemerals/chef/notify/localhost"
    }

    View Slide

  35. What can we do with that?

    Parse the data to effect some change?

    Use it as a signaling mechanism?

    Alert on it?

    More importantly, how can we use it with Chef?

    View Slide

  36. Noah cookbook

    http://github.com/lusis-cookbooks/noah

    Installs Noah + Redis

    Adds a noah_get method

    Automatically registers node in Noah
    (noah::register recipe)

    Also some new resources....

    View Slide

  37. Noah LWRP

    Register any primitive

    noah_application

    noah_service

    noah_configuration

    noah_application

    Dump some data in an ephemeral node

    noah_ephemeral

    Block the chef run until some data shows up in Noah

    noah_block

    View Slide

  38. Common Scenario
    haproxy
    django1 django2
    mysql

    View Slide

  39. Current Solution

    knife bootstrap -r “role[mysql]”

    Wait

    knife bootstrap -r “role[django1]

    Wait

    knife bootstrap -r “role[django2]”

    Wait

    knife bootstrap -r “role[haproxy]”

    View Slide

  40. noah_block
    haproxy
    django1 django2
    mysql

    View Slide

  41. noah_block
    haproxy
    django1 django2
    mysql

    View Slide

  42. noah_block
    haproxy
    django1 django2
    mysql

    Needs a DB

    Loads DB
    Needs to wait for initial DB load
    Needs backends
    Needs a DB

    View Slide

  43. Django1/2 recipe
    noah_block "wait_for_db" do
    path "http://noah:5678/ephemerals/#{db_master_role}"
    timeout 600
    data "ready"
    retry_interval 5
    on_failure :retry
    end

    View Slide

  44. Django2 recipe
    noah_block "wait_for_migration" do
    path "http://noah:5678/ephemerals/migration-status"
    timeout 600
    data "done"
    retry_interval 5
    on_failure :retry
    end

    View Slide

  45. haproxy recipe
    noah_block "wait_for_backends" do
    path "http://noah:5678/ephemerals/backends"
    timeout 600
    data "available"
    retry_interval 5
    on_failure :retry
    end

    View Slide

  46. Triggering Chef Runs

    Use noah_ephemeral at end of the dependency
    run_list (on the backend node)

    Leverage string interpolation for path name

    Data is unimportant, just the signal

    View Slide

  47. Small Problem...
    There's no “pokeable” Chef
    client

    View Slide

  48. View Slide

  49. Small Solution

    Small web app

    listens for HTTP POST requests

    runs chef-client

    View Slide

  50. View Slide

  51. View Slide

  52. Almost.....

    View Slide

  53. View Slide

  54. Demo time

    View Slide

  55. Well that was fun

    Obviously needs to be done “right”

    Add security

    Write your own!

    Watches are pluggable.

    AMQP

    ZeroMQ

    HTTPS + Token auth?

    View Slide

  56. Wrap up

    Questions?

    Comments?

    Rude remarks?

    View Slide

  57. Thanks!
    github.com/lusis
    twitter.com/lusis
    blog.lusis.org
    github.com/lusis/noah

    View Slide