Saltstack Provisioning Systems And Scalable Deploy Strategies Sys App Architect 1 

Provisioning Old Sys Admin Style Manual conf by hand QA test it and (sometimes) breaks it Writing all the doc to create the env again Forgot something? Fix it QA breaks it again Forgot something? Fix it QA breaks it again … 2 Why you need it 

Provisisioning It saves time and money Reproducible environment Auto documented Easy scalable Easy to create Dev env like a Prod env Sharing a git repo is sharing the whole env One man can manage 100 servers (1/1200 at Google) 3 Why you need it 

Provisioning Rules Use a strong naming convention (type-zone-provider-#-prod) Divide your servers by type Find common rules to all server Keep everything as generic as possible Prepare for multi zone and multi provider Small states 4 What you need 

Saltastack Saltstack history The author: Thomas S. Hatch What needed to be fix: the slowness The solution: ØMQ ! Fast, modular, async Dependencies need to be specified explicitly 5 The revolution 

Saltstack Design Execution Modules: States Grains Renderers (yaml, mako, jinja) Returners (redis, cassandra, etcd) Runners (custom scripts executed on master) ! Push/Pull, choose what you like 6 Components 

Saltstack State file {% set home = pillar.get('home', '/home/ubuntu') %}! {% set pythondir = pillar.get('pythondir', 'easyvote') %}! nginx-pkg:! pkg.installed:! - name: nginx! ! start-nginx-server:! service.running:! - name: nginx! - enable: True! - reload: True! - require:! - pkg: nginx-pkg! - watch:! - file: python-nginx-conf! ! python-nginx-conf:! file.managed:! - name: /etc/nginx/sites-available/easyvote-python.conf! - source: salt://nginx/conf/easyvote-python.conf! - template: jinja! - defaults:! pythondir: {{ home }}/{{ pythondir }}! - require: ! - pkg: nginx-pkg! 7 State 

Saltstack Grains of Salt Info from the minion host: easyvotegandi01! id: easyvote-01-staging! ip_interfaces: {'lo': ['127.0.0.1'], 'eth0': ['46.226.105.41']}! ipv4:! 127.0.0.1! 46.226.105.41! ipv6:! 2001:4b98:dc0:43:216:3eff:fe34:72fe! ::1! fe80::216:3eff:fe34:72fe! kernel: Linux! kernelrelease: 3.2.53-xenU-8869-x86_64! localhost: easyvotegandi01! lsb_distrib_codename: precise! lsb_distrib_description: Ubuntu 12.04.3 LTS! lsb_distrib_id: Ubuntu! lsb_distrib_release: 12.04! 8 Grains 

Saltstack Pillar, push variables to the minion Pillars are dictionaries (json, yaml, etc.) #!py! ! def run():! c9 = {}! config = subprocess.check_output("/usr/local/bin/node -e ‘console.log(! JSON.stringify(require(\"./deploy\")({revision: \"abcdefg\"}), null, 2))'", ! shell=True, cwd="%s/cloud9infra-config" % dirname)! c9['c9'] = json.loads(config)! return c9! ! ! ! mysql:! ----------! id:! 1! log-basename:! master! user:! ubuntu 9 Pillar 

Saltstack File templating Jinja + YAML, Mako + YAML, Wempy + YAML, Jinja + json, Mako + json, Wempy + json. ! daemonize yes! pidfile {{ directory }}/redis-infra.pid! port {{ redis.port }}! bind {{ bind }}! timeout {{ redis.timeout }}! loglevel notice! logfile {{ logdir }}/redis-infra.log! databases {{ redis.databases }}! save 900 100! save 300 10000! save 60 100000! stop-writes-on-bgsave-error yes! rdbcompression yes! rdbchecksum yes! dbfilename {{ redis.dbfilename }}! dir {{ directory }}! 10 Renderers 

Saltastack Save your data output salt '*' test.ping --return mongo_return,redis_return,cassandra_return Write your own import redis! import json! ! def returner(ret):! '''! Return information to a redis server! '''! serv = redis.Redis(! host='redis-serv.example.com',! port=6379,! db='0')! serv.sadd("%(id)s:jobs" % ret, ret['jid'])! serv.set("%(jid)s:%(id)s" % ret, json.dumps(ret['return']))! serv.sadd('jobs', ret['jid'])! serv.sadd(ret['jid'], ret['id'])! 11 Returners 

Saltstack Targeting minion servers salt 'web[1-5]' test.ping salt ‘*-prod' test.ping base: 'web1-(prod|devel)': - match: pcre - nginx ! nodegroups: group1: '[email protected],bar.domain.com,baz.domain.com or bl*.domain.com' group2: 'G@os:Debian and foo.domain.com’ ! salt -N group1 test.ping ! G is globbing, L list of minions, S subnet, etc. 12 Targeting 

Saltstack Runners and cli You can define your own commands or run common ones ! salt -L foo.bar.baz,quo.qux cmd.run ‘pgrep foo’ salt -G 'os:Ubuntu' test.ping salt '*' cmd.run 'echo "Hello: $FIRST_NAME"' env='{FIRST_NAME: "Joe"}' 13 Runners 

Saltstack Highstates and keys Accept a key salt-key -L salt-key -a ! Run all the states salt ‘*’ state.highstate 14 Keys and auto provision 

Fabric Fabric Set of python libraries to run command remotely ! Parallel execution Remote terminal output Interactive remote shell Plain python 15 From bash to fabric 

Fabric @task! def install_salt(master='salt.c9.io', name=None, set_hostname=True):! '''! Usage install_salt:master='master.salt.hostname',name='minion-name',set_hostname='yes'! '''! distro = run('lsb_release -s -c')! if distro == 'wheezy':! run('echo deb http://debian.saltstack.com/debian wheezy-saltstack main\! | sudo tee /etc/apt/sources.list.d/saltstack.list')! run('wget -q -O- “http://debian.saltstack.com/debian-salt-team-joehealy.gpg.key”\! | sudo apt-key add -')! ! run('sudo apt-get update')! run('sudo apt-get -y install salt-minion')! ! files.uncomment('/etc/salt/minion', 'master:.*', use_sudo=True)! files.sed('/etc/salt/minion', 'master:.*', 'master: %s' % master, use_sudo=True)! if name:! files.uncomment('/etc/salt/minion', 'id:.*', use_sudo=True)! files.sed('/etc/salt/minion', 'id:.*', 'id: %s' % name, use_sudo=True)! ! if set_hostname:! sudo('hostname %s' % name)! sudo('echo %s > /etc/hostname' % name)! ! run('sudo service salt-minion restart')! 16 fabric for salt 

Fabric Fabric CLI ! $ fab -l ! Install and configure saltstack minion in a new fresh server! fab -H [email protected] install_salt! ! Available commands:! ! install_salt Usage install_salt:master=‘master.salt.hostname',name='...! ! ! Live Demo! 17 Option 

Fabric Bash + Fabric + Salt = Power KISS: Use bash for script Use fabric for complex script with external software interaction, and for deploy Use Salt to provision ! Play with them, some example: Jenkins uses Fabric to deploy (legacy code) Jenkins uses Salt to deploy (new code) to staging 18 Bash + Fabric 

Continous delivery The github workflow Develop locally (Vagrant) Solve the issue or implement a feature in a new branch Make a pull request Social review (we are humans, right?) Test it Merge if passes all the steps Deploy to staging automatically 19 Workflow 

Continous delivery Con. delivery != Con. Deployment Use continuous deployment for staging env Use manual deploy for production env Monitor it Revert if everything is on fire Correlate deploy with metrics Use datadog, now, really! Demo! 20 Manual deploy? 

Sys App Architect @sysapp @mastrolinux http://sysapparchitect.com we are scaling too 21