Beyond the Node

Presentation from PuppetConf 2011
Complimentary video is available here:
http://www.youtube.com/watch?v=uRgf4pzzwsw

1. Beyond the Node Advanced 'Arkestration' with Noah PuppetConf 2011
2. .plan  What's the problem?  Orchestration  Overview of Noah  Use Cases  Q&A
4. A Brief History of Time (with apologies to Hawking)
5. Configuration Management • Manual
6. Configuration Management
7. Configuration Management
8. Configuration Management • Manual • Scripted
9. Configuration Management
10. Configuration Management • Manual • Scripted • Intelligent
11. CFEngine
12. Chef
13. Puppet
14. Configuration Management • Manual • Scripted • Intelligent
15. So we've solved this, right?
16. “I think..and I don't know if anyone would agree, that configuration management is a solved problem at this point, right?”
17. “I think..and I don't know if anyone would agree, that configuration management is a solved problem at this point, right?” WTF? WTF?
18. "The point I want to make..Configuration management is not a solved problem...and it's dangerous to make the mistake to think that the way we do things now is the best way to do them..." - Andrew Clay Shafer
19. “what I was attempting to say ... is that the current crop of configuration management tools have reached a usable point where they do enough (for now). What we’re seeing as questions now are 'How do I think beyond the single node where this tool is running?'”
20. “what I was attempting to say (epic fail, I might add) is that the current crop of configuration management tools have reached a usable point where they do enough (for now). What we’re seeing as questions now are 'How do I think beyond the single node where this tool is running?'”
21. Orchestration and Noah Noah attempts to do two things:  Bridge an impedance mismatch between operational, developmental and application configuration  Provide mechanisms for coordination between applications, nodes, services, configuration management and other infrastructure aspects
22. Example Use Case Noah Nagios Puppet (app) Load Balancer 1) Capacity Reached. Tell Noah 2) Noah tells provisioning system 3) Capacity allocated. Tell Noah 4) Noah triggers Puppet on LB 5) Noah triggers Puppet on Nagios
23. “Inspired” by
24. "ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications."
25. Features  Hierarchical name space of data registers similar to a Unix filesystem (znodes)  Leader election, locking, configuration, sequences  Distributed and highly available  Watches Downsides  Persistent connections required  Official bindings are C and Java  Somewhat "complex" to setup and run
26. Noah and ZooKeeper Noah is essentially a re-imagining of ZK, taking the things I liked:  Znode concept  Watches and making it work over HTTP using JSON
27. Noah Basics
28. The Primitives  Hosts Has status (up, down, pending) Can have Services  Services Has status (up, down, pending)  Must belong to a Host   Applications Can have Configurations  Configurations Can belong to many Applications
29. The Not So Primitives  Ephemerals Identified by a “path”  Tags Can be applied to any Primitive or Ephemeral  Links Can be applied to any Primitive, Ephemeral or Tag  Watches Can be applied at any hierarchy or specific object
30. URL Patterns • Hosts • /hosts/app01/services/httpd • Services • /services/httpd/app02 • Applications • /applications/my_app/configurations/log_level • Configurations • /configurations/log_level
31. GETting Data
32. { "foo": { "created_at": "2011-04-20 04:50:41 UTC", "updated_at": "2011-04-20 04:50:41 UTC", "tags": [ ], "id": "4af53f8d-36d2-2158-c4a1-514ff3578048", "links": [ ], "services": { }, "status": "up" }, "bar": { } }
33. GETting Data • Hosts
34. { "foo": { "created_at": "2011-04-20 04:50:41 UTC", "updated_at": "2011-04-20 04:50:41 UTC", "tags": [ ], "id": "4af53f8d-36d2-2158-c4a1-514ff3578048", "links": [ ], "services": { “mysql”:”pending”, “httpd”:”up” }, "status": "up" }, "bar": { } } Host Name
35. { "foo": { "created_at": "2011-04-20 04:50:41 UTC", "updated_at": "2011-04-20 04:50:41 UTC", "tags": [ ], "id": "4af53f8d-36d2-2158-c4a1-514ff3578048", "links": [ ], "services": { “mysql”:”pending”, “httpd”:”up” }, "status": "up" }, "bar": { } } Services
36. GETting Data • Hosts • Services
37. { "foo_svc": { "foo": { "created_at": "2011-04-20 05:09:44 UTC", "updated_at": "2011-04-20 05:09:44 UTC", "tags": [ ], "id": "53504526-a5d7-a604-5fe8-1f8fd2e1aef0", "links": [ ], "status": "down" } }, "both": { } } Service name
38. { "foo_svc": { "foo": { "created_at": "2011-04-20 05:09:44 UTC", "updated_at": "2011-04-20 05:09:44 UTC", "tags": [ ], "id": "53504526-a5d7-a604-5fe8-1f8fd2e1aef0", "links": [ ], "status": "down" } }, "both": { } } Host Name
39. GETting Data • Hosts • Services • Applications
40. { "myrailsapp1": { "created_at": "2011-04-21 06:57:42 UTC", "updated_at": "2011-04-21 06:57:42 UTC", "tags": [ "sample_data", "production" ], "id": "8774fead-e63a-8e5c-0453-1b5330bbd04c", "links": [ "/my_sample_organization" ], "configurations": { "database.yml": { "format": "yaml", "body": "development:\n database: development_database\n adapter: mysql\n username: dev_user\n password: dev_password\n" } } }, "myrestapp1": { } } App Name
41. { "myrailsapp1": { "created_at": "2011-04-21 06:57:42 UTC", "updated_at": "2011-04-21 06:57:42 UTC", "tags": [ "sample_data", "production" ], "id": "8774fead-e63a-8e5c-0453-1b5330bbd04c", "links": [ "/my_sample_organization" ], "configurations": { "database.yml": { "format": "yaml", "body": "development:\n database: development_database\n adapter: mysql\n username: dev_user\n password: dev_password\n" } } }, "myrestapp1": { } } Configs
42. GETting Data • Hosts • Services • Applications • Configurations *
43. { "barconf1": { "created_at": "2011-04-27 01:20:37 UTC", "format": "string", "body": "barbody1", "updated_at": "2011-04-27 01:20:37 UTC", "tags": [ ], "id": "aa183dce-30ec-2d7e-c6f9-f3385161a7e6", "links": [ ] }, "barconf2": { } }
44. GETting Data • Hosts • Services • Applications • Configurations * • Ephemerals *
45. GETting Data • Hosts • Services • Applications • Configurations * • Ephemerals * /ephemerals/my/config/setting returns the data stored there.
46. PUTting Data  /hosts/host_name  /services/service_name/host_name  /applications/application_name  /configurations/configuration_name  /ephemerals/arbitrary/path/to/whatever
47. PUTting Data  /hosts/host_name  /services/service_name/host_name  /applications/application_name  /configurations/configuration_name  /ephemerals/arbitrary/path/to/whatever
48. PUTting Data  Host {“status”:”up”}  Service {“status”:”up”,”host_status”:”up”}  Application {“name”:”app_name”}  Configuration {“format”:”format_type”,”body”:”data”}  Ephemeral raw data
49. DELETEing Data  /hosts/host_name  /services/service_name/host_name  /applications/application_name  /configurations/configuration_name  /ephemerals/arbitrary/path/to/whatever
50. Tagging  Append “tag” to any PUT url with PUT method e.g. '/applications/some_application/tag'  {“tags”:”sometag”} or {“tags”:[“tag1”,”tag2”]}  Call DELETE instead of PUT to remove tags  GET '/tags' for all tags or '/tag/tagname' for specific tag
51. Linking  Append “link” to any PUT url with PUT method e.g. '/applications/some_application/link'  {“link_name”:”my_organization”}  Call DELETE instead of PUT to remove links *  GET '/link_name' for all linked objects *  Designed to allow custom namespaces
52. Watches
53. What are Watches?  A bit different than ZK watches  Async  Persistent *  Pluggable (URI pattern based)  Hierarchical (superset and subset)  Append '/watch' to almost any url path  {“endpoint”:”http://some_host:port/ping_me”}  '/watch' root endpoint as well { “endpoint”:”http://some_host/ping”, ”pattern”:”//noah/applications/app_name” }
54. Create a Watch $ curl -XPUT -d '{“pattern”:”//noah/applications”,"endpoint":" http://host/ping"}' http://noahserver/watches
55. Perform some operation $ curl -XPUT -d '{"name":"fooapp2"}' http://noahserver/applications/fooapp2
56. Message sent to Endpoint { "id":"be9f6a45-0d49-3f00-59a8-c6ad61da24e1", "tags":[], "links":[], "name":"fooapp2", "created_at":"2011-09-15 03:07:39 UTC", "updated_at":"2011-09-15 03:07:39 UTC", "configurations":{}, "action":"create", "pubcategory":"//noah/applications/fooapp2" }
57. Perform another operation $ curl -XPUT -d '{"tags":"footag2"}' http://noahserver/applications/fooapp2/tag
58. Message sent to Endpoint { "id":"be9f6a45-0d49-3f00-59a8-c6ad61da24e1", "tags":["footag"], "links":[], "name":"fooapp2", "created_at":"2011-09-15 03:07:39 UTC", "updated_at":"2011-09-15 03:15:03 UTC", "configurations":{}, "action":"update", "pubcategory":"//noah/applications/fooapp2" }
59. Watches are “plugin” based You can write your own plugins with Ruby.  URI Pattern *  amqp://user:pass@host:port/exchange  rundeck://token@host:port/job_id  puppet://hostname/  statsd://host:port/metric.path.name.action  Currently based on Event Machine  Future based on Actors and worker pools
60. Watch Notes  Creator of the watch is unrelated to receiver  Don't have to use the “official” watch daemon  You can watch the watches (PubSub)
61. Basic Puppet Integration  http://github.com/jamtur01/puppet-noah  ENC, Functions and Facts  Hosts → Nodes  Host's Services → Classes  Configurations → Facts (string types only)
62. So where does it fit? Noah Puppet Apps Scripts Monitoring Deploy CI
63. Noah + Puppet + Applications Noah Puppet Applications
64. Common P+A Use Case  Puppet knows Nodes and Classes  App has configuration file  Configuration file controlled by Puppet  Puppet client run required to update file
65. Common P+A Issues  Puppet repo is not app repo  Devs must learn ERB  Devs might even have to learn Puppet  Introducing new settings and communication  Daemon-mode is exception rather than rule
66. N+P+A Version  Noah knows Nodes and Classes  Move volatile settings into Noah  Configuration file controlled by Puppet  App gets volatile settings from Noah *  Future Puppet runs update Noah  Noah tells App *  Truth vs. Reality  My DB master is foo  My DB master right NOW is bar
67. Volatile Settings?  Not all configuration is created equal  “Elasticity” - Pool of worker node EC2 ips  Ad-hoc configuration – Logging levels  Environment aware – Nodes of class “foo”
68. N+P+A Benefits  Devs don't HAVE to learn ERB  Devs don't HAVE to learn Puppet  Ops and Dev friendly bridge point  No unintended changes released
69. Integration with CI Noah Puppet CI Deploy
70. Integration with CI • Jenkins asks Noah for nodes of class “foo” • Jenkins triggers Puppet (or deploy) on appropriate nodes • Jenkins is always “current”
71. Integration with Monitoring Noah Puppet Monitoring App
72. Integration with Monitoring • Puppet manages Nagios configs • Nagios updates Noah status (event handler) • Noah optionally fires watches based on status change • Endpoint is application. App “reconfigures”
73. Integration with Scripts Noah Scripts (node 1) Scripts (node 2) Scripts (node 3)
74. Integration with Scripts • Provide “locking” and “coordination” of cron jobs across hosts • Stop dealing with lock files on host • Use ephemerals and curl
75. Gotchas • Noah itself is NOT distributed * • Data redundancy based on Redis • No ACLs * • Async
76. Other options • ZooKeeper • Doozer • NoSQL stores (Redis + Webdis, Riak) * • Nesoi * • Write your own!
77. Questions?
78. Noah  Github: http://github.com/lusis/Noah  IRC: #noah  Blog(s): http://blog.lusis.org/blog/categories/noah http://lusislog.blogspot.com/search/label/noah
79. John E. Vincent  Twitter: @lusis  Github: http://github.com/lusis  IRC: lusis  LinkedIn: http://linkedin.com/in/lusis  Blog(s): http://blog.lusis.org http://lusislog.blogspot.com
80. Thank You!