Abusing Databags for Fun and Profit

Transcript

1. Abusing DataBags for Fun and Profit Friday, 25 May 12

2. @thommay Friday, 25 May 12

3. Friday, 25 May 12 Incubator for scientific startups and developers

   of tools for scientists

4. Before Friday, 25 May 12 Way back in 2009...

5. We could pull from external data stores... Friday, 25 May

   12

6. require 'net/ldap' l = Net::LDAP.new(host: "x.y") l.bind l.search(filter: ...) do

   |u| homedir = u[:homedirectory].first directory homedir do owner u[:uidNumber].first.to_i group gid mode "0750" recursive true end end Friday, 25 May 12

7. Or we could use attributes Friday, 25 May 12

8. default[:app][:db1][:host] = "" default[:app][:db1][:user] = "" default[:app][:db1][:pass] = "" default[:app][:db1][:db]

   = "db1" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" attributes/default.rb Friday, 25 May 12

9. default[:app][:db1][:host] = "" default[:app][:db1][:user] = "" default[:app][:db1][:pass] = "" default[:app][:db1][:db]

   = "db1" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:user] = "appro" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" attributes/default.rb Friday, 25 May 12

10. default[:app][:db1][:host] = "" default[:app][:db1][:user] = "" default[:app][:db1][:pass] = "" default[:app][:db1][:db]

    = "db1" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:user] = "appro" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" attributes/default.rb production role Friday, 25 May 12

11. default[:app][:db1][:host] = "" default[:app][:db1][:user] = "" default[:app][:db1][:pass] = "" default[:app][:db1][:db]

    = "db1" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:user] = "appro" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:host] = "x.x.x" default[:app][:db1][:pass] = "2342" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" attributes/default.rb production role Friday, 25 May 12

12. default[:app][:db1][:host] = "" default[:app][:db1][:user] = "" default[:app][:db1][:pass] = "" default[:app][:db1][:db]

    = "db1" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:user] = "appro" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:host] = "x.x.x" default[:app][:db1][:pass] = "2342" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" attributes/default.rb production role cluster role Friday, 25 May 12

13. default[:app][:db1][:host] = "" default[:app][:db1][:user] = "" default[:app][:db1][:pass] = "" default[:app][:db1][:db]

    = "db1" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:user] = "appro" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:host] = "x.x.x" default[:app][:db1][:pass] = "2342" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" attributes/default.rb production role cluster role Friday, 25 May 12

14. Enter Data Bags Friday, 25 May 12 First release in

    chef 0.8. really low key announcement: “allow you to store arbitrary data within the Chef Server, and have it fully indexed for Search”

15. “Data bags provide an arbitrary store of globally available JSON

    data.” Friday, 25 May 12

16. “Data bags provide an arbitrary store of globally available JSON

    data.” Friday, 25 May 12

17. “Data bags provide an arbitrary store of globally available JSON

    data.” Friday, 25 May 12

18. Bags Friday, 25 May 12

19. list of data bags Friday, 25 May 12

20. Of Data Friday, 25 May 12

21. { "level": 6, "id": "some-test-node", "volumes": { "sdg": "/dev/xvdg", "sdh":

    "/dev/xvdh", "sdi": "/dev/xvdi", "sdj": "/dev/xvdj", "sdk": "/dev/xvdk", "sdl": "/dev/xvdl", "sdm": "/dev/xvdm", "sdn": "/dev/xvdn", }, "mount": "/srv/", "volsize": 250 } Friday, 25 May 12

22. Driving your cookbooks Friday, 25 May 12

23. Load Friday, 25 May 12

24. Load A single item Friday, 25 May 12

25. Load A single item But only if it exists! Friday,

    25 May 12

26. ebs = data_bag_item("ebs", node.hostname) xvds = [] ebs["volumes"].each_pair do |vol,

    real| xvds << real aws_ebs_volume "srv_volume_#{vol}" do aws_access_key node.aws.access_key aws_secret_access_key node.aws.secret_key size ebs["volsize"] device "/dev/#{vol}" action [:create, :attach] end end Friday, 25 May 12

27. ebs = data_bag_item("ebs", node.hostname) xvds = [] ebs["volumes"].each_pair do |vol,

    real| xvds << real aws_ebs_volume "srv_volume_#{vol}" do aws_access_key node.aws.access_key aws_secret_access_key node.aws.secret_key size ebs["volsize"] device "/dev/#{vol}" action [:create, :attach] end end Friday, 25 May 12

28. Search Friday, 25 May 12

29. Search Data Bags are indexed Friday, 25 May 12

30. Search Data Bags are indexed Search on any field Friday,

    25 May 12

31. db = search(:databases, "id:#{dbname}").first if db template "#{base_dir}/shared/config/database.yml" do cookbook

    "baton" owner site variables({name: db["id"], pass: db[chef_environment]["pass"]}) end end 16:21 ~ knife search databases “id:test” 1 items found chef_type: data_bag_item data_bag: databases id: test creator: Thom May development: pass: foobar Friday, 25 May 12

32. Iterate Friday, 25 May 12

33. data_bag(:ftpusers).each do |v| user = data_bag_item(:ftpusers, v) user user["id"] do

    password user["password"] shell "/bin/sh" group "metrics" home "/home/#{user["id"]}" supports :manage_home => true action [:create,:manage,:modify] end end Friday, 25 May 12

34. Danger! Friday, 25 May 12

35. Security risks Friday, 25 May 12 Altering data bags from

    the node when using the Open Source chef-server requires giving the node's API client admin privileges. In most cases, this is not advisable

36. Security risks Race-y Friday, 25 May 12 There’s no guarantee

    in the API that you’re the only thing writing.

37. Locking Friday, 25 May 12 I’ve not tried this, although

    CB mentioned it this morning.

38. lock "deploy-foo" #do stuff here under a lock unlock "deploy-foo"

    Friday, 25 May 12

39. def lock(name) begin item = data_bag_item("locks", name) raise "Lock found"

    if item rescue Net::HTTPServerException => e raise e unless e.response_code == "404" end item = Chef::DataBagItem.new item.data_bag "locks" item.raw_data = { "id" => name, "holder" => node.fqdn } item.save end def unlock(name) begin item = data_bag_item("locks", name) rescue Net::HTTPServerException => e raise e unless e.response_code == "404" end item.destroy end Friday, 25 May 12

40. Sequencing Friday, 25 May 12 kind of an extension of

    locking

41. Bring up your database Friday, 25 May 12

42. Bring up your database before your app server Friday, 25

    May 12

43. Synchronisation Friday, 25 May 12

44. Shared State Friday, 25 May 12

45. def generate_id Chef::Log.info "Percona server-id: #{node.percona.server_id}" begin item = data_bag_item(

    "percona", "server-id" ) id = item['last-used'] + 1 rescue Net::HTTPServerException => e id = 1 raise e unless e.response.code == "404" end item = Chef::DataBagItem.new item.data_bag "percona" item.raw_data = { "id" => "server-id", "last-used" => id } item.save node.set.percona.server_id = id node.save end generate_id if node.percona.server_id == 0 Friday, 25 May 12

46. Orchestration Friday, 25 May 12

47. Positives Friday, 25 May 12

48. Positives Lightweight Friday, 25 May 12

49. Positives Uses existing tools Friday, 25 May 12

50. Positives Asynchronous Friday, 25 May 12

51. Negatives Friday, 25 May 12

52. Negatives Asynchronous Friday, 25 May 12

53. Negatives Hard to reason about Friday, 25 May 12

54. Caching Friday, 25 May 12 less interesting with partial search

    but still handy.

55. Optimise Friday, 25 May 12

56. Spice up your life Friday, 25 May 12

57. Crazy easy Chef API library Friday, 25 May 12

58. uri = URI.parse(conf["chef_url"]) Spice.setup do |s| s.host = uri.host s.port

    = uri.port.to_s s.url_path = uri.path s.scheme = uri.scheme s.client_name = conf["client_name"] s.key_file = conf["key_file"] end Spice.connect! Spice::DataBag.show(:name => conf["db_name"]).keys.each do |k| item = Spice::DataBag.show_item(:name => conf["db_name"], :id => k) end Friday, 25 May 12

59. github.com/danryan/spice Friday, 25 May 12

60. NoSQL? Friday, 25 May 12

61. bag bucket Friday, 25 May 12

62. item key Friday, 25 May 12

63. 11:13 ~ % knife data bag show databases app1 app2

    auth refinery surechem_curi website Friday, 25 May 12

64. Friday, 25 May 12

65. Don’t expect to run Facebook from databags Friday, 25 May

    12

66. Or any other site with more than 1 req/s Friday,

    25 May 12

67. But awesome for internal apps Friday, 25 May 12

68. Thanks! Friday, 25 May 12