Abusing Databags for Fun and Profit

Slide 1

Slide 1 text

Abusing DataBags for Fun and Proﬁt Friday, 25 May 12

Slide 2

Slide 2 text

@thommay Friday, 25 May 12

Slide 3

Slide 3 text

Friday, 25 May 12 Incubator for scientiﬁc startups and developers of tools for scientists

Slide 4

Slide 4 text

Before Friday, 25 May 12 Way back in 2009...

Slide 5

Slide 5 text

We could pull from external data stores... Friday, 25 May 12

Slide 6

Slide 6 text

require 'net/ldap' l = Net::LDAP.new(host: "x.y") l.bind l.search(filter: ...) do |u| homedir = u[:homedirectory].first directory homedir do owner u[:uidNumber].first.to_i group gid mode "0750" recursive true end end Friday, 25 May 12

Slide 7

Slide 7 text

Or we could use attributes Friday, 25 May 12

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

default[:app][:db1][:host] = "" default[:app][:db1][:user] = "" default[:app][:db1][:pass] = "" default[:app][:db1][:db] = "db1" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:user] = "appro" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" default[:app][:db1][:host] = "x.x.x" default[:app][:db1][:pass] = "2342" default[:app][:db2][:host] = "x.x.x" default[:app][:db2][:user] = "db2" default[:app][:db2][:pass] = "2342" default[:app][:db2][:db] = "db2" set[:app][:version] = "1.2.3" attributes/default.rb production role Friday, 25 May 12

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Enter Data Bags Friday, 25 May 12 First release in chef 0.8. really low key announcement: “allow you to store arbitrary data within the Chef Server, and have it fully indexed for Search”

Slide 15

Slide 15 text

“Data bags provide an arbitrary store of globally available JSON data.” Friday, 25 May 12

Slide 16

Slide 16 text

“Data bags provide an arbitrary store of globally available JSON data.” Friday, 25 May 12

Slide 17

Slide 17 text

“Data bags provide an arbitrary store of globally available JSON data.” Friday, 25 May 12

Slide 18

Slide 18 text

Bags Friday, 25 May 12

Slide 19

Slide 19 text

list of data bags Friday, 25 May 12

Slide 20

Slide 20 text

Of Data Friday, 25 May 12

Slide 21

Slide 21 text

{ "level": 6, "id": "some-test-node", "volumes": { "sdg": "/dev/xvdg", "sdh": "/dev/xvdh", "sdi": "/dev/xvdi", "sdj": "/dev/xvdj", "sdk": "/dev/xvdk", "sdl": "/dev/xvdl", "sdm": "/dev/xvdm", "sdn": "/dev/xvdn", }, "mount": "/srv/", "volsize": 250 } Friday, 25 May 12

Slide 22

Slide 22 text

Driving your cookbooks Friday, 25 May 12

Slide 23

Slide 23 text

Load Friday, 25 May 12

Slide 24

Slide 24 text

Load A single item Friday, 25 May 12

Slide 25

Slide 25 text

Load A single item But only if it exists! Friday, 25 May 12

Slide 26

Slide 26 text

ebs = data_bag_item("ebs", node.hostname) xvds = [] ebs["volumes"].each_pair do |vol, real| xvds << real aws_ebs_volume "srv_volume_#{vol}" do aws_access_key node.aws.access_key aws_secret_access_key node.aws.secret_key size ebs["volsize"] device "/dev/#{vol}" action [:create, :attach] end end Friday, 25 May 12

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Search Friday, 25 May 12

Slide 29

Slide 29 text

Search Data Bags are indexed Friday, 25 May 12

Slide 30

Slide 30 text

Search Data Bags are indexed Search on any ﬁeld Friday, 25 May 12

Slide 31

Slide 31 text

db = search(:databases, "id:#{dbname}").first if db template "#{base_dir}/shared/config/database.yml" do cookbook "baton" owner site variables({name: db["id"], pass: db[chef_environment]["pass"]}) end end 16:21 ~ knife search databases “id:test” 1 items found chef_type: data_bag_item data_bag: databases id: test creator: Thom May development: pass: foobar Friday, 25 May 12

Slide 32

Slide 32 text

Iterate Friday, 25 May 12

Slide 33

Slide 33 text

data_bag(:ftpusers).each do |v| user = data_bag_item(:ftpusers, v) user user["id"] do password user["password"] shell "/bin/sh" group "metrics" home "/home/#{user["id"]}" supports :manage_home => true action [:create,:manage,:modify] end end Friday, 25 May 12

Slide 34

Slide 34 text

Danger! Friday, 25 May 12

Slide 35

Slide 35 text

Security risks Friday, 25 May 12 Altering data bags from the node when using the Open Source chef-server requires giving the node's API client admin privileges. In most cases, this is not advisable

Slide 36

Slide 36 text

Security risks Race-y Friday, 25 May 12 There’s no guarantee in the API that you’re the only thing writing.

Slide 37

Slide 37 text

Locking Friday, 25 May 12 I’ve not tried this, although CB mentioned it this morning.

Slide 38

Slide 38 text

lock "deploy-foo" #do stuff here under a lock unlock "deploy-foo" Friday, 25 May 12

Slide 39

Slide 39 text

def lock(name) begin item = data_bag_item("locks", name) raise "Lock found" if item rescue Net::HTTPServerException => e raise e unless e.response_code == "404" end item = Chef::DataBagItem.new item.data_bag "locks" item.raw_data = { "id" => name, "holder" => node.fqdn } item.save end def unlock(name) begin item = data_bag_item("locks", name) rescue Net::HTTPServerException => e raise e unless e.response_code == "404" end item.destroy end Friday, 25 May 12

Slide 40

Slide 40 text

Sequencing Friday, 25 May 12 kind of an extension of locking

Slide 41

Slide 41 text

Bring up your database Friday, 25 May 12

Slide 42

Slide 42 text

Bring up your database before your app server Friday, 25 May 12

Slide 43

Slide 43 text

Synchronisation Friday, 25 May 12

Slide 44

Slide 44 text

Shared State Friday, 25 May 12

Slide 45

Slide 45 text

def generate_id Chef::Log.info "Percona server-id: #{node.percona.server_id}" begin item = data_bag_item( "percona", "server-id" ) id = item['last-used'] + 1 rescue Net::HTTPServerException => e id = 1 raise e unless e.response.code == "404" end item = Chef::DataBagItem.new item.data_bag "percona" item.raw_data = { "id" => "server-id", "last-used" => id } item.save node.set.percona.server_id = id node.save end generate_id if node.percona.server_id == 0 Friday, 25 May 12

Slide 46

Slide 46 text

Orchestration Friday, 25 May 12

Slide 47

Slide 47 text

Positives Friday, 25 May 12

Slide 48

Slide 48 text

Positives Lightweight Friday, 25 May 12

Slide 49

Slide 49 text

Positives Uses existing tools Friday, 25 May 12

Slide 50

Slide 50 text

Positives Asynchronous Friday, 25 May 12

Slide 51

Slide 51 text

Negatives Friday, 25 May 12

Slide 52

Slide 52 text

Negatives Asynchronous Friday, 25 May 12

Slide 53

Slide 53 text

Negatives Hard to reason about Friday, 25 May 12

Slide 54

Slide 54 text

Caching Friday, 25 May 12 less interesting with partial search but still handy.

Slide 55

Slide 55 text

Optimise Friday, 25 May 12

Slide 56

Slide 56 text

Spice up your life Friday, 25 May 12

Slide 57

Slide 57 text

Crazy easy Chef API library Friday, 25 May 12

Slide 58

Slide 58 text

uri = URI.parse(conf["chef_url"]) Spice.setup do |s| s.host = uri.host s.port = uri.port.to_s s.url_path = uri.path s.scheme = uri.scheme s.client_name = conf["client_name"] s.key_file = conf["key_file"] end Spice.connect! Spice::DataBag.show(:name => conf["db_name"]).keys.each do |k| item = Spice::DataBag.show_item(:name => conf["db_name"], :id => k) end Friday, 25 May 12

Slide 59

Slide 59 text

github.com/danryan/spice Friday, 25 May 12

Slide 60

Slide 60 text

NoSQL? Friday, 25 May 12

Slide 61

Slide 61 text

bag bucket Friday, 25 May 12

Slide 62

Slide 62 text

item key Friday, 25 May 12

Slide 63

Slide 63 text

11:13 ~ % knife data bag show databases app1 app2 auth refinery surechem_curi website Friday, 25 May 12