Ansible quartermaster

This presentation gives an overview of Care.com's infrastructure, Ansible dynamic inventory in general, and the system we developed to standardize our AWS, Rackspace cloud and dedicated, and local dedicated inventories.

Transcript

1. ANSIBLE QUARTERMASTER combining and consolidating dynamic and static inventories github
2. WHO AM I? director of production operations at care.com mcoddington@care.com or coddington@gmail.com twitter @mpc512
3. WHAT THIS TALK IS ABOUT why use dynamic inventories? what comes "out of the box" with ansible? why write your own?
4. CARE.COM INFRASTRUCTURES Rackspace IAD2 (original prod infrastructure) Rackspace ORD1 (soon to be production - hybrid cloud) 2 AWS-based services International sites on AWS (newer than IAD, older than those 2 services) Corporate datacenter at Markley
5. NOT QUITE THIS
6. MORE LIKE THIS
7. WHY? reasons originally, small IT team with no prod ops never the right time for a big change always the right time for a small changes/experimentation but now we have some problems
8. There are only two hard things in Computer Science: cache invalidation and naming things -- Phil Karlton http://martinfowler.com/bliki/TwoHardThings.html
9. PROBLEMS want to share roles, but ec2.py and rax.py have inconsistent naming even different naming in the static infrastructures try not to let limitations of one environment impact others
11. DYNAMIC INVENTORY "OUT OF THE BOX" Ansible Docs - dynamic inventory Ansible Docs - developing dynamic inventory scripts if the inventory file ("hosts") is executable, ansible assumes it is dynamic executable should return json formatted as ansible expects for multiple inventories, just pass -i <dir>
12. --list should return groups, group vars % ./hosts ‐‐list { "databases" : { "hosts" : [ "host1.example.com", "host2.example.com" ], "vars" : { "a" : true } }, "webservers" : [ "host2.example.com", "host3.example.com" ], "5points" : [ "host7.example.com" ] }
13. --host <host> should return host vars % ./hosts ‐‐host moocow.example.com { "favcolor" : "red", "monitoring" : "pack.example.com" }
14. _meta dict item contains hostvars, for efficiency % ./hosts ‐‐list { # results of inventory script as above go here # ... "_meta" : { "hostvars" : { "moocow.example.com" : { "favcolor" : "red", "monitoring": "pack.example.com" } # ... } } }
17. YAML CONFIG FILE SECTIONS hosts (static hosts) groups (groups and group vars) includes (dynamic inventories)
18. HOSTS SECTION hosts: 528713‐orlprdback1: stack: prod systype: back shortname: orlprdback1 common_crypto_gpg_keys: ‐ keyfile: root@cz‐reporting01.carezen.local.pub keydir: gpg‐public‐keys owner: root group: root gpgdir: /root/.gnupg 591583‐orlprdcb1: stack: prod systype: couchbase shortname: orlprdcb1 extra_groups: ['couchbase‐live', 'couchbase‐clustera']
19. GROUPS SECTION groups: api: extra_groups: ['forkprocess'] user_roles: ['users_common'] tomcat_catalina_log4j_log_file: catalina‐log4j.out back: extra_groups: ['dedicated', 'forkprocess'] user_roles: ['users_common'] couchbase: extra_groups: ['dedicated'] user_roles: ['users_common','duosecurity']
20. INCLUDES SECTION includes: raxpy: /ansible/shared/bin/rax.py
21. SO WHAT DOES THIS BUY US? once all the data is structured, we can standardize things we can create groups based on var values (like group_by) stuff like the 'extra_groups' thing described previously we can create new groups based on existing groups and vars (metagroups)
22. RAX_METADATA SPECIAL STUFF all metadata become vars, but any metadata (from rax.py) starting with czgroup_ because a group a la group_by if var == "rax_metadata": for raxvar in hostdict[hostname][var].keys(): if raxvar.startswith("czgroup_"): group = hostdict[hostname][var][raxvar] groupdict.setdefault(group,{}) groupdict[group].setdefault('hosts',set()) groupdict[group]['hosts'].add(hostname) hostdict[hostname][raxvar.replace("czgroup_","",1)] = group
23. EC2_TAG SPECIAL STUFF all tags become vars, but any key (from ec2.py) starting with ec2_tag_czgroup_ becomes a group a la group_by if var.startswith("ec2_tag_czgroup_"): group = hostdict[hostname][var] groupdict.setdefault(group,{}) groupdict[group].setdefault('hosts',set()) groupdict[group]['hosts'].add(hostname) hostdict[hostname][var.replace("ec2_tag_czgroup_","",1)] = group elif var.startswith("ec2_tag_"): thisvar = var.replace("ec2_tag_","",1) # AWS requires capitalized 'Name' to show up in the console so we # use that as a tag, but other places we use this var it is called # name (lowercase) so we make that conversion here. if thisvar == 'Name': thisvar = 'name' hostdict[hostname][thisvar] = hostdict[hostname][var]
24. METAGROUPS create groups based on multiple variable values or on variables and existing group membership metagroups: # Create a $stack‐$systype group for all stacks and systypes, and also # one for $stack‐forkprocess. ‐ [ {variable: stack}, {variable: systype} ] ‐ [ {variable: stack}, {group: forkprocess} ]
25. HOW IS IT WORKING? by march we will have 80% of our environments on this new system knowing where we want to embed the complexity of vars and groups helps move more quickly
26. QUESTIONS/COMMENTS?