Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Beyond the Node

Beyond the Node

Presentation from PuppetConf 2011
Complimentary video is available here:
http://www.youtube.com/watch?v=uRgf4pzzwsw

John Vincent

October 17, 2011
Tweet

More Decks by John Vincent

Other Decks in Technology

Transcript

  1. Beyond the Node Advanced 'Arkestration' with Noah PuppetConf 2011

  2. .plan  What's the problem?  Orchestration  Overview of

    Noah  Use Cases  Q&A
  3. None
  4. A Brief History of Time (with apologies to Hawking)

  5. Configuration Management • Manual

  6. Configuration Management

  7. Configuration Management

  8. Configuration Management • Manual • Scripted

  9. Configuration Management

  10. Configuration Management • Manual • Scripted • Intelligent

  11. CFEngine

  12. Chef

  13. Puppet

  14. Configuration Management • Manual • Scripted • Intelligent

  15. So we've solved this, right?

  16. “I think..and I don't know if anyone would agree, that

    configuration management is a solved problem at this point, right?”
  17. “I think..and I don't know if anyone would agree, that

    configuration management is a solved problem at this point, right?” WTF? WTF?
  18. "The point I want to make..Configuration management is not a

    solved problem...and it's dangerous to make the mistake to think that the way we do things now is the best way to do them..." - Andrew Clay Shafer
  19. “what I was attempting to say ... is that the

    current crop of configuration management tools have reached a usable point where they do enough (for now). What we’re seeing as questions now are 'How do I think beyond the single node where this tool is running?'”
  20. “what I was attempting to say (epic fail, I might

    add) is that the current crop of configuration management tools have reached a usable point where they do enough (for now). What we’re seeing as questions now are 'How do I think beyond the single node where this tool is running?'”
  21. Orchestration and Noah Noah attempts to do two things: 

    Bridge an impedance mismatch between operational, developmental and application configuration  Provide mechanisms for coordination between applications, nodes, services, configuration management and other infrastructure aspects
  22. Example Use Case Noah Nagios Puppet (app) Load Balancer 1)

    Capacity Reached. Tell Noah 2) Noah tells provisioning system 3) Capacity allocated. Tell Noah 4) Noah triggers Puppet on LB 5) Noah triggers Puppet on Nagios
  23. “Inspired” by

  24. "ZooKeeper is a centralized service for maintaining configuration information, naming,

    providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications."
  25. Features  Hierarchical name space of data registers similar to

    a Unix filesystem (znodes)  Leader election, locking, configuration, sequences  Distributed and highly available  Watches Downsides  Persistent connections required  Official bindings are C and Java  Somewhat "complex" to setup and run
  26. Noah and ZooKeeper Noah is essentially a re-imagining of ZK,

    taking the things I liked:  Znode concept  Watches and making it work over HTTP using JSON
  27. Noah Basics

  28. The Primitives  Hosts Has status (up, down, pending) Can

    have Services  Services Has status (up, down, pending)  Must belong to a Host   Applications Can have Configurations  Configurations Can belong to many Applications
  29. The Not So Primitives  Ephemerals Identified by a “path”

     Tags Can be applied to any Primitive or Ephemeral  Links Can be applied to any Primitive, Ephemeral or Tag  Watches Can be applied at any hierarchy or specific object
  30. URL Patterns • Hosts • /hosts/app01/services/httpd • Services • /services/httpd/app02

    • Applications • /applications/my_app/configurations/log_level • Configurations • /configurations/log_level
  31. GETting Data

  32. { "foo": { "created_at": "2011-04-20 04:50:41 UTC", "updated_at": "2011-04-20 04:50:41

    UTC", "tags": [ ], "id": "4af53f8d-36d2-2158-c4a1-514ff3578048", "links": [ ], "services": { }, "status": "up" }, "bar": { } }
  33. GETting Data • Hosts

  34. { "foo": { "created_at": "2011-04-20 04:50:41 UTC", "updated_at": "2011-04-20 04:50:41

    UTC", "tags": [ ], "id": "4af53f8d-36d2-2158-c4a1-514ff3578048", "links": [ ], "services": { “mysql”:”pending”, “httpd”:”up” }, "status": "up" }, "bar": { } } Host Name
  35. { "foo": { "created_at": "2011-04-20 04:50:41 UTC", "updated_at": "2011-04-20 04:50:41

    UTC", "tags": [ ], "id": "4af53f8d-36d2-2158-c4a1-514ff3578048", "links": [ ], "services": { “mysql”:”pending”, “httpd”:”up” }, "status": "up" }, "bar": { } } Services
  36. GETting Data • Hosts • Services

  37. { "foo_svc": { "foo": { "created_at": "2011-04-20 05:09:44 UTC", "updated_at":

    "2011-04-20 05:09:44 UTC", "tags": [ ], "id": "53504526-a5d7-a604-5fe8-1f8fd2e1aef0", "links": [ ], "status": "down" } }, "both": { } } Service name
  38. { "foo_svc": { "foo": { "created_at": "2011-04-20 05:09:44 UTC", "updated_at":

    "2011-04-20 05:09:44 UTC", "tags": [ ], "id": "53504526-a5d7-a604-5fe8-1f8fd2e1aef0", "links": [ ], "status": "down" } }, "both": { } } Host Name
  39. GETting Data • Hosts • Services • Applications

  40. { "myrailsapp1": { "created_at": "2011-04-21 06:57:42 UTC", "updated_at": "2011-04-21 06:57:42

    UTC", "tags": [ "sample_data", "production" ], "id": "8774fead-e63a-8e5c-0453-1b5330bbd04c", "links": [ "/my_sample_organization" ], "configurations": { "database.yml": { "format": "yaml", "body": "development:\n database: development_database\n adapter: mysql\n username: dev_user\n password: dev_password\n" } } }, "myrestapp1": { } } App Name
  41. { "myrailsapp1": { "created_at": "2011-04-21 06:57:42 UTC", "updated_at": "2011-04-21 06:57:42

    UTC", "tags": [ "sample_data", "production" ], "id": "8774fead-e63a-8e5c-0453-1b5330bbd04c", "links": [ "/my_sample_organization" ], "configurations": { "database.yml": { "format": "yaml", "body": "development:\n database: development_database\n adapter: mysql\n username: dev_user\n password: dev_password\n" } } }, "myrestapp1": { } } Configs
  42. GETting Data • Hosts • Services • Applications • Configurations

    *
  43. { "barconf1": { "created_at": "2011-04-27 01:20:37 UTC", "format": "string", "body":

    "barbody1", "updated_at": "2011-04-27 01:20:37 UTC", "tags": [ ], "id": "aa183dce-30ec-2d7e-c6f9-f3385161a7e6", "links": [ ] }, "barconf2": { } }
  44. GETting Data • Hosts • Services • Applications • Configurations

    * • Ephemerals *
  45. GETting Data • Hosts • Services • Applications • Configurations

    * • Ephemerals * /ephemerals/my/config/setting returns the data stored there.
  46. PUTting Data  /hosts/host_name  /services/service_name/host_name  /applications/application_name  /configurations/configuration_name

     /ephemerals/arbitrary/path/to/whatever
  47. PUTting Data  /hosts/host_name  /services/service_name/host_name  /applications/application_name  /configurations/configuration_name

     /ephemerals/arbitrary/path/to/whatever
  48. PUTting Data  Host {“status”:”up”}  Service {“status”:”up”,”host_status”:”up”}  Application

    {“name”:”app_name”}  Configuration {“format”:”format_type”,”body”:”data”}  Ephemeral raw data
  49. DELETEing Data  /hosts/host_name  /services/service_name/host_name  /applications/application_name  /configurations/configuration_name

     /ephemerals/arbitrary/path/to/whatever
  50. Tagging  Append “tag” to any PUT url with PUT

    method e.g. '/applications/some_application/tag'  {“tags”:”sometag”} or {“tags”:[“tag1”,”tag2”]}  Call DELETE instead of PUT to remove tags  GET '/tags' for all tags or '/tag/tagname' for specific tag
  51. Linking  Append “link” to any PUT url with PUT

    method e.g. '/applications/some_application/link'  {“link_name”:”my_organization”}  Call DELETE instead of PUT to remove links *  GET '/link_name' for all linked objects *  Designed to allow custom namespaces
  52. Watches

  53. What are Watches?  A bit different than ZK watches

     Async  Persistent *  Pluggable (URI pattern based)  Hierarchical (superset and subset)  Append '/watch' to almost any url path  {“endpoint”:”http://some_host:port/ping_me”}  '/watch' root endpoint as well { “endpoint”:”http://some_host/ping”, ”pattern”:”//noah/applications/app_name” }
  54. Create a Watch $ curl -XPUT -d '{“pattern”:”//noah/applications”,"endpoint":" http://host/ping"}' http://noahserver/watches

  55. Perform some operation $ curl -XPUT -d '{"name":"fooapp2"}' http://noahserver/applications/fooapp2

  56. Message sent to Endpoint { "id":"be9f6a45-0d49-3f00-59a8-c6ad61da24e1", "tags":[], "links":[], "name":"fooapp2", "created_at":"2011-09-15

    03:07:39 UTC", "updated_at":"2011-09-15 03:07:39 UTC", "configurations":{}, "action":"create", "pubcategory":"//noah/applications/fooapp2" }
  57. Perform another operation $ curl -XPUT -d '{"tags":"footag2"}' http://noahserver/applications/fooapp2/tag

  58. Message sent to Endpoint { "id":"be9f6a45-0d49-3f00-59a8-c6ad61da24e1", "tags":["footag"], "links":[], "name":"fooapp2", "created_at":"2011-09-15

    03:07:39 UTC", "updated_at":"2011-09-15 03:15:03 UTC", "configurations":{}, "action":"update", "pubcategory":"//noah/applications/fooapp2" }
  59. Watches are “plugin” based You can write your own plugins

    with Ruby.  URI Pattern *  amqp://user:[email protected]:port/exchange  rundeck://[email protected]:port/job_id  puppet://hostname/  statsd://host:port/metric.path.name.action  Currently based on Event Machine  Future based on Actors and worker pools
  60. Watch Notes  Creator of the watch is unrelated to

    receiver  Don't have to use the “official” watch daemon  You can watch the watches (PubSub)
  61. Basic Puppet Integration  http://github.com/jamtur01/puppet-noah  ENC, Functions and Facts

     Hosts → Nodes  Host's Services → Classes  Configurations → Facts (string types only)
  62. So where does it fit? Noah Puppet Apps Scripts Monitoring

    Deploy CI
  63. Noah + Puppet + Applications Noah Puppet Applications

  64. Common P+A Use Case  Puppet knows Nodes and Classes

     App has configuration file  Configuration file controlled by Puppet  Puppet client run required to update file
  65. Common P+A Issues  Puppet repo is not app repo

     Devs must learn ERB  Devs might even have to learn Puppet  Introducing new settings and communication  Daemon-mode is exception rather than rule
  66. N+P+A Version  Noah knows Nodes and Classes  Move

    volatile settings into Noah  Configuration file controlled by Puppet  App gets volatile settings from Noah *  Future Puppet runs update Noah  Noah tells App *  Truth vs. Reality  My DB master is foo  My DB master right NOW is bar
  67. Volatile Settings?  Not all configuration is created equal 

    “Elasticity” - Pool of worker node EC2 ips  Ad-hoc configuration – Logging levels  Environment aware – Nodes of class “foo”
  68. N+P+A Benefits  Devs don't HAVE to learn ERB 

    Devs don't HAVE to learn Puppet  Ops and Dev friendly bridge point  No unintended changes released
  69. Integration with CI Noah Puppet CI Deploy

  70. Integration with CI • Jenkins asks Noah for nodes of

    class “foo” • Jenkins triggers Puppet (or deploy) on appropriate nodes • Jenkins is always “current”
  71. Integration with Monitoring Noah Puppet Monitoring App

  72. Integration with Monitoring • Puppet manages Nagios configs • Nagios

    updates Noah status (event handler) • Noah optionally fires watches based on status change • Endpoint is application. App “reconfigures”
  73. Integration with Scripts Noah Scripts (node 1) Scripts (node 2)

    Scripts (node 3)
  74. Integration with Scripts • Provide “locking” and “coordination” of cron

    jobs across hosts • Stop dealing with lock files on host • Use ephemerals and curl
  75. Gotchas • Noah itself is NOT distributed * • Data

    redundancy based on Redis • No ACLs * • Async
  76. Other options • ZooKeeper • Doozer • NoSQL stores (Redis

    + Webdis, Riak) * • Nesoi * • Write your own!
  77. Questions?

  78. Noah  Github: http://github.com/lusis/Noah  IRC: #noah  Blog(s): http://blog.lusis.org/blog/categories/noah

    http://lusislog.blogspot.com/search/label/noah
  79. John E. Vincent  Twitter: @lusis  Github: http://github.com/lusis 

    IRC: lusis  LinkedIn: http://linkedin.com/in/lusis  Blog(s): http://blog.lusis.org http://lusislog.blogspot.com
  80. Thank You!