Chef on Python and MongoDB

DevOps is the new rage among system administrators, applying agile software development techniques to infrastructure configuration management. In the center of the DevOps movement is the open-source Chef tool, implemented in Ruby atop CouchDB. Unsatisfied with the performance of the open-source and/or hosted Chef server and needing better integration with our Python web application, we set out to build a new implementation in Python atop MongoDB. This talk will give you an overview of Chef, reasons for doing a new implementation, and lots of code examples of how we made it all work together to get a chef server that screams.

Transcript

1. Rick  Copeland  @rick446   Arborian  Consulting,  LLC  
2.   Infrastructure  as  Code     Resources  &  Providers     Cookbooks,  Recipes,  Clients,  and  Nodes  
3. Chef  Server     (API)   RabbitMQ   RabbitMQ   RabbitMQ   CouchDB   Solr   Solr  Indexer   chef-­‐client   knife   Chef  Server   (Web  UI)   HTTP  REST  API  
4. Build,  register,  and  authenticate  the  node   Synchronize  cookbooks   Build  resource  collection  (run  the  recipes  in  order)   Configure  node  (“converge”)   Run  notification  handlers  
5. Chef  Server     (API)   RabbitMQ   RabbitMQ   RabbitMQ   CouchDB   Solr   Solr  Indexer   chef-­‐client   knife   Chef  Server   (Web  UI)   HTTP  REST  API   Ruby   Ruby   Ruby  
6.   Chef  assumes  a  bootstrapped  node  exists     Chef  doesn’t  keep  release  notes     Code  and  infrastructure  are  versioned   differently     Solution:  Web  app  to  manage  deployments  &   generate  release  notes  
7. MongoDB   Solr   Solr  Indexer   chef-­‐client   knife   HTTP  REST  API   MonQ   Chef  Server     (API  +  web)   Ruby   Python  
8.   Reduce  #  of  processes  &  technologies     Don’t  know  Ruby  well     Keep  private  keys  out  of  the  system     Integrate  with  existing  authentication     Performance  
9. /clients   /data   /roles   /nodes   /cookbooks   /environments   /sandboxes   /search  
10.   Mostly  JSON    almost  BSON     References  to  Ruby  files  stored  separately   {      "name":  "allura-­‐0.0.1”,…      "json_class":  "Chef::CookbookVersion",      "attributes":  [          {              "name":  "default.rb",              "url":  "https://s3.amazonaws.com/opscode-­‐platform-­‐production-­‐data/…                                  "checksum":  "749a3a328d6c47a32d134b336183981f",              "path":  "attributes/default.rb",              "specificity":  "default”…   Ruby  files  stored  on  S3  
11. role = collection( 'chef.role', doc_session, Field('_id', S.ObjectId()), Field('account_id', S.ObjectId(if_missing=None)), Field('name', str), Field('description', str), Field('default_attributes', str), Field('override_attributes', str), Field('run_list', [ str ] ), Field('env_run_lists', { str: [ str ]}), Index('account_id', 'name', unique=True)) MongoDB  Validator   Shorthand  with   Python  Types   Embedded   Documents   Index  Definitions  
12. class Role(object): def url(self): return request.relative_url( config.chef_api_root + '/roles/' + self.name) def __json__(self): return dict( chef_type='role', json_class='Chef::Role',! … default_attributes=loads(self.default_attributes),! …) def update(self, d): self.name = d['name'] … self.default_attributes = dumps(d['default_attributes']) self.override_attributes = dumps(d['override_attributes'])! … Models  know   where  they  live   Models  can  be   turned  into  dict  (to   be  JSONified)   Models  can  be   updated  from  dict  
13. class RoleSchema(JSONModelSchema):! model_class=CM.role! chef_type='role’! json_class='Chef::Role’! exclude_fields=['_id', 'account_id']
14.   /foo/bar/baz    root.foo.bar.baz     Lots  of  decorators     Validation  (params/body    **kwargs)     Authorization     Rendering     HTTP  method  lookup     GET  /foo/bar    root.foo.bar._get()  
15. class RolesController(RESTController): @expose(template_engine='json') def _get(self):… @expose(template_engine='json', acl=[CACE.admin(True), ACE.any(False)], schema=cv.RoleSchema) def _post(self, **kwargs):… def __getattr__(self, name): return RoleController(name) Returns  JSON   Only  admins  can   access   Convert  and   Validate  POST   Continue  dotted   lookup  
16. class RoleController(RESTController): @expose(template_engine='json') def _get(self): … @expose(template_engine='json', acl=[CACE.admin(True), ACE.any(False)], schema=cv.RoleSchema) def _put(self, name, **kwargs): … @expose(template_engine='json', acl=[CACE.admin(True), ACE.any(False)]) def _delete(self): … PUT  looks  just  like  a  POST  
17. class RoleController(RESTController):! def __init__(self, name): self._role = CM.Role.query.get( account_id=c.account._id, name=name) if self._role is None: raise AttributeError, name! @expose(template_engine='json') def _get(self): return self._role! AttributeError       HTTP  404  Not  Found   Auto-­‐JSONify  
18. class RoleController(RESTController):! … @expose(template_engine='json', acl=[CACE.admin(True), ACE.any(False)], schema=cv.RoleSchema) def _put(self, name, **kwargs): assert name == self._role.name self._role.update(kwargs) return self._role! @expose(template_engine='json', acl=[CACE.admin(True), ACE.any(False)]) def _delete(self): self._role.delete() return self._role Update  model   from  kwargs  
19.   Don’t  trust  the  docs     Don’t  trust  the  docs   ▪  Don’t  trust  the  docs     Use  fat  models     Framework  support  for  REST  &  JSON     You’re  gonna  have  to  learn  some  Ruby  anyway     JSON  !=  BSON  
20.   Better  test  coverage     Search  support  (SOLR  /  ElasticSearch)     More  testing  with  real-­‐world  deployments     Finalize  integration  with  deployment   manager  
21. Rick  Copeland  @rick446   Arborian  Consulting,  LLC  
22.   http://openmymind.net/2011/10/28/ CouchDB-­‐And-­‐MongoDB-­‐Performance/     MongoDB  is  14x  faster     http://www.snailinaturtleneck.com/blog/ 2009/06/29/couchdb-­‐vs-­‐mongodb-­‐ benchmark/