Abusing Databags for Fun and Profit

Databags are a handy dandy way of storing state. This session looks at ways of generating, manipulating and using databags to do update and synchronisation of configuration and requirements across multiple nodes in an environment. This talk covers building tools to interact with databags, creating databags within recipes, and cover real world use cases.

Transcript

1. Abusing DataBags for Fun and Profit Friday, 25 May 12
2. @thommay Friday, 25 May 12
3. Friday, 25 May 12 Incubator for scientific startups and developers of tools for scientists
4. Before Friday, 25 May 12 Way back in 2009...
5. We could pull from external data stores... Friday, 25 May 12
6. require 'net/ldap' l = Net::LDAP.new(host: "x.y") l.bind l.search(filter: ...) do |u| homedir = u[:homedirectory].first directory homedir do owner u[:uidNumber].first.to_i group gid mode "0750" recursive true end end Friday, 25 May 12
7. Or we could use attributes Friday, 25 May 12
8. default[:app][:db1][:host] = "" default[:app][:db1][:user] = "" default[:app][:db1][:pass] = "" default[:app][:db1][:db] = "db1" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" attributes/default.rb Friday, 25 May 12
9. default[:app][:db1][:host] = "" default[:app][:db1][:user] = "" default[:app][:db1][:pass] = "" default[:app][:db1][:db] = "db1" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:user] = "appro" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" attributes/default.rb Friday, 25 May 12
10. default[:app][:db1][:host] = "" default[:app][:db1][:user] = "" default[:app][:db1][:pass] = "" default[:app][:db1][:db] = "db1" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:user] = "appro" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" attributes/default.rb production role Friday, 25 May 12
11. default[:app][:db1][:host] = "" default[:app][:db1][:user] = "" default[:app][:db1][:pass] = "" default[:app][:db1][:db] = "db1" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:user] = "appro" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:host] = "x.x.x" default[:app][:db1][:pass] = "2342" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" attributes/default.rb production role Friday, 25 May 12
12. default[:app][:db1][:host] = "" default[:app][:db1][:user] = "" default[:app][:db1][:pass] = "" default[:app][:db1][:db] = "db1" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:user] = "appro" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:host] = "x.x.x" default[:app][:db1][:pass] = "2342" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" attributes/default.rb production role cluster role Friday, 25 May 12
13. default[:app][:db1][:host] = "" default[:app][:db1][:user] = "" default[:app][:db1][:pass] = "" default[:app][:db1][:db] = "db1" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:user] = "appro" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:host] = "x.x.x" default[:app][:db1][:pass] = "2342" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" attributes/default.rb production role cluster role Friday, 25 May 12
14. Enter Data Bags Friday, 25 May 12 First release in chef 0.8. really low key announcement: “allow you to store arbitrary data within the Chef Server, and have it fully indexed for Search”
15. “Data bags provide an arbitrary store of globally available JSON data.” Friday, 25 May 12
16. “Data bags provide an arbitrary store of globally available JSON data.” Friday, 25 May 12
17. “Data bags provide an arbitrary store of globally available JSON data.” Friday, 25 May 12
18. Bags Friday, 25 May 12
19. list of data bags Friday, 25 May 12
20. Of Data Friday, 25 May 12
21. { "level": 6, "id": "some-test-node", "volumes": { "sdg": "/dev/xvdg", "sdh": "/dev/xvdh", "sdi": "/dev/xvdi", "sdj": "/dev/xvdj", "sdk": "/dev/xvdk", "sdl": "/dev/xvdl", "sdm": "/dev/xvdm", "sdn": "/dev/xvdn", }, "mount": "/srv/", "volsize": 250 } Friday, 25 May 12
22. Driving your cookbooks Friday, 25 May 12
23. Load Friday, 25 May 12
24. Load A single item Friday, 25 May 12
25. Load A single item But only if it exists! Friday, 25 May 12
26. ebs = data_bag_item("ebs", node.hostname) xvds = [] ebs["volumes"].each_pair do |vol, real| xvds << real aws_ebs_volume "srv_volume_#{vol}" do aws_access_key node.aws.access_key aws_secret_access_key node.aws.secret_key size ebs["volsize"] device "/dev/#{vol}" action [:create, :attach] end end Friday, 25 May 12
27. ebs = data_bag_item("ebs", node.hostname) xvds = [] ebs["volumes"].each_pair do |vol, real| xvds << real aws_ebs_volume "srv_volume_#{vol}" do aws_access_key node.aws.access_key aws_secret_access_key node.aws.secret_key size ebs["volsize"] device "/dev/#{vol}" action [:create, :attach] end end Friday, 25 May 12
28. Search Friday, 25 May 12
29. Search Data Bags are indexed Friday, 25 May 12
30. Search Data Bags are indexed Search on any field Friday, 25 May 12
31. db = search(:databases, "id:#{dbname}").first if db template "#{base_dir}/shared/config/database.yml" do cookbook "baton" owner site variables({name: db["id"], pass: db[chef_environment]["pass"]}) end end 16:21 ~ knife search databases “id:test” 1 items found chef_type: data_bag_item data_bag: databases id: test creator: Thom May development: pass: foobar Friday, 25 May 12
32. Iterate Friday, 25 May 12
33. data_bag(:ftpusers).each do |v| user = data_bag_item(:ftpusers, v) user user["id"] do password user["password"] shell "/bin/sh" group "metrics" home "/home/#{user["id"]}" supports :manage_home => true action [:create,:manage,:modify] end end Friday, 25 May 12
34. Danger! Friday, 25 May 12
35. Security risks Friday, 25 May 12 Altering data bags from the node when using the Open Source chef-server requires giving the node's API client admin privileges. In most cases, this is not advisable
36. Security risks Race-y Friday, 25 May 12 There’s no guarantee in the API that you’re the only thing writing.
37. Locking Friday, 25 May 12 I’ve not tried this, although CB mentioned it this morning.
38. lock "deploy-foo" #do stuff here under a lock unlock "deploy-foo" Friday, 25 May 12
39. def lock(name) begin item = data_bag_item("locks", name) raise "Lock found" if item rescue Net::HTTPServerException => e raise e unless e.response_code == "404" end item = Chef::DataBagItem.new item.data_bag "locks" item.raw_data = { "id" => name, "holder" => node.fqdn } item.save end def unlock(name) begin item = data_bag_item("locks", name) rescue Net::HTTPServerException => e raise e unless e.response_code == "404" end item.destroy end Friday, 25 May 12
40. Sequencing Friday, 25 May 12 kind of an extension of locking
41. Bring up your database Friday, 25 May 12
42. Bring up your database before your app server Friday, 25 May 12
43. Synchronisation Friday, 25 May 12
44. Shared State Friday, 25 May 12
45. def generate_id Chef::Log.info "Percona server-id: #{node.percona.server_id}" begin item = data_bag_item( "percona", "server-id" ) id = item['last-used'] + 1 rescue Net::HTTPServerException => e id = 1 raise e unless e.response.code == "404" end item = Chef::DataBagItem.new item.data_bag "percona" item.raw_data = { "id" => "server-id", "last-used" => id } item.save node.set.percona.server_id = id node.save end generate_id if node.percona.server_id == 0 Friday, 25 May 12
46. Orchestration Friday, 25 May 12
47. Positives Friday, 25 May 12
48. Positives Lightweight Friday, 25 May 12
49. Positives Uses existing tools Friday, 25 May 12
50. Positives Asynchronous Friday, 25 May 12
51. Negatives Friday, 25 May 12
52. Negatives Asynchronous Friday, 25 May 12
53. Negatives Hard to reason about Friday, 25 May 12
54. Caching Friday, 25 May 12 less interesting with partial search but still handy.
55. Optimise Friday, 25 May 12
56. Spice up your life Friday, 25 May 12
57. Crazy easy Chef API library Friday, 25 May 12
58. uri = URI.parse(conf["chef_url"]) Spice.setup do |s| s.host = uri.host s.port = uri.port.to_s s.url_path = uri.path s.scheme = uri.scheme s.client_name = conf["client_name"] s.key_file = conf["key_file"] end Spice.connect! Spice::DataBag.show(:name => conf["db_name"]).keys.each do |k| item = Spice::DataBag.show_item(:name => conf["db_name"], :id => k) end Friday, 25 May 12
59. github.com/danryan/spice Friday, 25 May 12
60. NoSQL? Friday, 25 May 12
61. bag bucket Friday, 25 May 12
62. item key Friday, 25 May 12
63. 11:13 ~ % knife data bag show databases app1 app2 auth refinery surechem_curi website Friday, 25 May 12
64. Friday, 25 May 12
65. Don’t expect to run Facebook from databags Friday, 25 May 12
66. Or any other site with more than 1 req/s Friday, 25 May 12
67. But awesome for internal apps Friday, 25 May 12
68. Thanks! Friday, 25 May 12