Saltstack, Fabric and the Deploy problem

Introduction to the Salt provisioning system and the usafe of Fabric as a bridge between old bash script and the new way of provisioning servers.

Transcript

1. Saltstack Provisioning Systems And Scalable Deploy Strategies Sys App Architect 1
2. Provisioning Old Sys Admin Style Manual conf by hand QA test it and (sometimes) breaks it Writing all the doc to create the env again Forgot something? Fix it QA breaks it again Forgot something? Fix it QA breaks it again … 2 Why you need it
3. Provisisioning It saves time and money Reproducible environment Auto documented Easy scalable Easy to create Dev env like a Prod env Sharing a git repo is sharing the whole env One man can manage 100 servers (1/1200 at Google) 3 Why you need it
4. Provisioning Rules Use a strong naming convention (type-zone-provider-#-prod) Divide your servers by type Find common rules to all server Keep everything as generic as possible Prepare for multi zone and multi provider Small states 4 What you need
5. Saltastack Saltstack history The author: Thomas S. Hatch What needed to be fix: the slowness The solution: ØMQ ! Fast, modular, async Dependencies need to be specified explicitly 5 The revolution
6. Saltstack Design Execution Modules: States Grains Renderers (yaml, mako, jinja) Returners (redis, cassandra, etcd) Runners (custom scripts executed on master) ! Push/Pull, choose what you like 6 Components
7. Saltstack State file {% set home = pillar.get('home', '/home/ubuntu') %}! {% set pythondir = pillar.get('pythondir', 'easyvote') %}! nginx-pkg:! pkg.installed:! - name: nginx! ! start-nginx-server:! service.running:! - name: nginx! - enable: True! - reload: True! - require:! - pkg: nginx-pkg! - watch:! - file: python-nginx-conf! ! python-nginx-conf:! file.managed:! - name: /etc/nginx/sites-available/easyvote-python.conf! - source: salt://nginx/conf/easyvote-python.conf! - template: jinja! - defaults:! pythondir: {{ home }}/{{ pythondir }}! - require: ! - pkg: nginx-pkg! 7 State
8. Saltstack Grains of Salt Info from the minion host: easyvotegandi01! id: easyvote-01-staging! ip_interfaces: {'lo': ['127.0.0.1'], 'eth0': ['46.226.105.41']}! ipv4:! 127.0.0.1! 46.226.105.41! ipv6:! 2001:4b98:dc0:43:216:3eff:fe34:72fe! ::1! fe80::216:3eff:fe34:72fe! kernel: Linux! kernelrelease: 3.2.53-xenU-8869-x86_64! localhost: easyvotegandi01! lsb_distrib_codename: precise! lsb_distrib_description: Ubuntu 12.04.3 LTS! lsb_distrib_id: Ubuntu! lsb_distrib_release: 12.04! 8 Grains
9. Saltstack Pillar, push variables to the minion Pillars are dictionaries (json, yaml, etc.) #!py! ! def run():! c9 = {}! config = subprocess.check_output("/usr/local/bin/node -e ‘console.log(! JSON.stringify(require(\"./deploy\")({revision: \"abcdefg\"}), null, 2))'", ! shell=True, cwd="%s/cloud9infra-config" % dirname)! c9['c9'] = json.loads(config)! return c9! ! ! ! mysql:! ----------! id:! 1! log-basename:! master! user:! ubuntu 9 Pillar
10. Saltstack File templating Jinja + YAML, Mako + YAML, Wempy + YAML, Jinja + json, Mako + json, Wempy + json. ! daemonize yes! pidfile {{ directory }}/redis-infra.pid! port {{ redis.port }}! bind {{ bind }}! timeout {{ redis.timeout }}! loglevel notice! logfile {{ logdir }}/redis-infra.log! databases {{ redis.databases }}! save 900 100! save 300 10000! save 60 100000! stop-writes-on-bgsave-error yes! rdbcompression yes! rdbchecksum yes! dbfilename {{ redis.dbfilename }}! dir {{ directory }}! 10 Renderers
11. Saltastack Save your data output salt '*' test.ping --return mongo_return,redis_return,cassandra_return Write your own import redis! import json! ! def returner(ret):! '''! Return information to a redis server! '''! serv = redis.Redis(! host='redis-serv.example.com',! port=6379,! db='0')! serv.sadd("%(id)s:jobs" % ret, ret['jid'])! serv.set("%(jid)s:%(id)s" % ret, json.dumps(ret['return']))! serv.sadd('jobs', ret['jid'])! serv.sadd(ret['jid'], ret['id'])! 11 Returners
12. Saltstack Targeting minion servers salt 'web[1-5]' test.ping salt ‘*-prod' test.ping base: 'web1-(prod|devel)': - match: pcre - nginx ! nodegroups: group1: 'L@foo.domain.com,bar.domain.com,baz.domain.com or bl*.domain.com' group2: 'G@os:Debian and foo.domain.com’ ! salt -N group1 test.ping ! G is globbing, L list of minions, S subnet, etc. 12 Targeting
13. Saltstack Runners and cli You can define your own commands or run common ones ! salt -L foo.bar.baz,quo.qux cmd.run ‘pgrep foo’ salt -G 'os:Ubuntu' test.ping salt '*' cmd.run 'echo "Hello: $FIRST_NAME"' env='{FIRST_NAME: "Joe"}' 13 Runners
14. Saltstack Highstates and keys Accept a key salt-key -L salt-key -a ! Run all the states salt ‘*’ state.highstate 14 Keys and auto provision
15. Fabric Fabric Set of python libraries to run command remotely ! Parallel execution Remote terminal output Interactive remote shell Plain python 15 From bash to fabric
16. Fabric @task! def install_salt(master='salt.c9.io', name=None, set_hostname=True):! '''! Usage install_salt:master='master.salt.hostname',name='minion-name',set_hostname='yes'! '''! distro = run('lsb_release -s -c')! if distro == 'wheezy':! run('echo deb http://debian.saltstack.com/debian wheezy-saltstack main\! | sudo tee /etc/apt/sources.list.d/saltstack.list')! run('wget -q -O- “http://debian.saltstack.com/debian-salt-team-joehealy.gpg.key”\! | sudo apt-key add -')! ! run('sudo apt-get update')! run('sudo apt-get -y install salt-minion')! ! files.uncomment('/etc/salt/minion', 'master:.*', use_sudo=True)! files.sed('/etc/salt/minion', 'master:.*', 'master: %s' % master, use_sudo=True)! if name:! files.uncomment('/etc/salt/minion', 'id:.*', use_sudo=True)! files.sed('/etc/salt/minion', 'id:.*', 'id: %s' % name, use_sudo=True)! ! if set_hostname:! sudo('hostname %s' % name)! sudo('echo %s > /etc/hostname' % name)! ! run('sudo service salt-minion restart')! 16 fabric for salt
17. Fabric Fabric CLI ! $ fab -l ! Install and configure saltstack minion in a new fresh server! fab -H user@your.server.name install_salt! ! Available commands:! ! install_salt Usage install_salt:master=‘master.salt.hostname',name='...! ! ! Live Demo! 17 Option
18. Fabric Bash + Fabric + Salt = Power KISS: Use bash for script Use fabric for complex script with external software interaction, and for deploy Use Salt to provision ! Play with them, some example: Jenkins uses Fabric to deploy (legacy code) Jenkins uses Salt to deploy (new code) to staging 18 Bash + Fabric
19. Continous delivery The github workflow Develop locally (Vagrant) Solve the issue or implement a feature in a new branch Make a pull request Social review (we are humans, right?) Test it Merge if passes all the steps Deploy to staging automatically 19 Workflow
20. Continous delivery Con. delivery != Con. Deployment Use continuous deployment for staging env Use manual deploy for production env Monitor it Revert if everything is on fire Correlate deploy with metrics Use datadog, now, really! Demo! 20 Manual deploy?
21. Sys App Architect @sysapp @mastrolinux http://sysapparchitect.com we are scaling too 21