Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Saltstack, Fabric and the Deploy problem

Saltstack, Fabric and the Deploy problem

Introduction to the Salt provisioning system and the usafe of Fabric as a bridge between old bash script and the new way of provisioning servers.


Luca Cipriani

April 03, 2014


  1. Saltstack Provisioning Systems And Scalable Deploy Strategies Sys App Architect

  2. Provisioning Old Sys Admin Style Manual conf by hand QA

    test it and (sometimes) breaks it Writing all the doc to create the env again Forgot something? Fix it QA breaks it again Forgot something? Fix it QA breaks it again … 2 Why you need it
  3. Provisisioning It saves time and money Reproducible environment Auto documented

    Easy scalable Easy to create Dev env like a Prod env Sharing a git repo is sharing the whole env One man can manage 100 servers (1/1200 at Google) 3 Why you need it
  4. Provisioning Rules Use a strong naming convention (type-zone-provider-#-prod) Divide your

    servers by type Find common rules to all server Keep everything as generic as possible Prepare for multi zone and multi provider Small states 4 What you need
  5. Saltastack Saltstack history The author: Thomas S. Hatch What needed

    to be fix: the slowness The solution: ØMQ ! Fast, modular, async Dependencies need to be specified explicitly 5 The revolution
  6. Saltstack Design Execution Modules: States Grains Renderers (yaml, mako, jinja)

    Returners (redis, cassandra, etcd) Runners (custom scripts executed on master) ! Push/Pull, choose what you like 6 Components
  7. Saltstack State file {% set home = pillar.get('home', '/home/ubuntu') %}!

    {% set pythondir = pillar.get('pythondir', 'easyvote') %}! nginx-pkg:! pkg.installed:! - name: nginx! ! start-nginx-server:! service.running:! - name: nginx! - enable: True! - reload: True! - require:! - pkg: nginx-pkg! - watch:! - file: python-nginx-conf! ! python-nginx-conf:! file.managed:! - name: /etc/nginx/sites-available/easyvote-python.conf! - source: salt://nginx/conf/easyvote-python.conf! - template: jinja! - defaults:! pythondir: {{ home }}/{{ pythondir }}! - require: ! - pkg: nginx-pkg! 7 State
  8. Saltstack Grains of Salt Info from the minion host: easyvotegandi01!

    id: easyvote-01-staging! ip_interfaces: {'lo': [''], 'eth0': ['']}! ipv4:!!! ipv6:! 2001:4b98:dc0:43:216:3eff:fe34:72fe! ::1! fe80::216:3eff:fe34:72fe! kernel: Linux! kernelrelease: 3.2.53-xenU-8869-x86_64! localhost: easyvotegandi01! lsb_distrib_codename: precise! lsb_distrib_description: Ubuntu 12.04.3 LTS! lsb_distrib_id: Ubuntu! lsb_distrib_release: 12.04! 8 Grains
  9. Saltstack Pillar, push variables to the minion Pillars are dictionaries

    (json, yaml, etc.) #!py! ! def run():! c9 = {}! config = subprocess.check_output("/usr/local/bin/node -e ‘console.log(! JSON.stringify(require(\"./deploy\")({revision: \"abcdefg\"}), null, 2))'", ! shell=True, cwd="%s/cloud9infra-config" % dirname)! c9['c9'] = json.loads(config)! return c9! ! ! ! mysql:! ----------! id:! 1! log-basename:! master! user:! ubuntu 9 Pillar
  10. Saltstack File templating Jinja + YAML, Mako + YAML, Wempy

    + YAML, Jinja + json, Mako + json, Wempy + json. ! daemonize yes! pidfile {{ directory }}/redis-infra.pid! port {{ redis.port }}! bind {{ bind }}! timeout {{ redis.timeout }}! loglevel notice! logfile {{ logdir }}/redis-infra.log! databases {{ redis.databases }}! save 900 100! save 300 10000! save 60 100000! stop-writes-on-bgsave-error yes! rdbcompression yes! rdbchecksum yes! dbfilename {{ redis.dbfilename }}! dir {{ directory }}! 10 Renderers
  11. Saltastack Save your data output salt '*' test.ping --return mongo_return,redis_return,cassandra_return

    Write your own import redis! import json! ! def returner(ret):! '''! Return information to a redis server! '''! serv = redis.Redis(! host='redis-serv.example.com',! port=6379,! db='0')! serv.sadd("%(id)s:jobs" % ret, ret['jid'])! serv.set("%(jid)s:%(id)s" % ret, json.dumps(ret['return']))! serv.sadd('jobs', ret['jid'])! serv.sadd(ret['jid'], ret['id'])! 11 Returners
  12. Saltstack Targeting minion servers salt 'web[1-5]' test.ping salt ‘*-prod' test.ping

    base: 'web1-(prod|devel)': - match: pcre - nginx ! nodegroups: group1: 'L@foo.domain.com,bar.domain.com,baz.domain.com or bl*.domain.com' group2: 'G@os:Debian and foo.domain.com’ ! salt -N group1 test.ping ! G is globbing, L list of minions, S subnet, etc. 12 Targeting
  13. Saltstack Runners and cli You can define your own commands

    or run common ones ! salt -L foo.bar.baz,quo.qux cmd.run ‘pgrep foo’ salt -G 'os:Ubuntu' test.ping salt '*' cmd.run 'echo "Hello: $FIRST_NAME"' env='{FIRST_NAME: "Joe"}' 13 Runners
  14. Saltstack Highstates and keys Accept a key salt-key -L salt-key

    -a ! Run all the states salt ‘*’ state.highstate 14 Keys and auto provision
  15. Fabric Fabric Set of python libraries to run command remotely

    ! Parallel execution Remote terminal output Interactive remote shell Plain python 15 From bash to fabric
  16. Fabric @task! def install_salt(master='salt.c9.io', name=None, set_hostname=True):! '''! Usage install_salt:master='master.salt.hostname',name='minion-name',set_hostname='yes'! '''!

    distro = run('lsb_release -s -c')! if distro == 'wheezy':! run('echo deb http://debian.saltstack.com/debian wheezy-saltstack main\! | sudo tee /etc/apt/sources.list.d/saltstack.list')! run('wget -q -O- “http://debian.saltstack.com/debian-salt-team-joehealy.gpg.key”\! | sudo apt-key add -')! ! run('sudo apt-get update')! run('sudo apt-get -y install salt-minion')! ! files.uncomment('/etc/salt/minion', 'master:.*', use_sudo=True)! files.sed('/etc/salt/minion', 'master:.*', 'master: %s' % master, use_sudo=True)! if name:! files.uncomment('/etc/salt/minion', 'id:.*', use_sudo=True)! files.sed('/etc/salt/minion', 'id:.*', 'id: %s' % name, use_sudo=True)! ! if set_hostname:! sudo('hostname %s' % name)! sudo('echo %s > /etc/hostname' % name)! ! run('sudo service salt-minion restart')! 16 fabric for salt
  17. Fabric Fabric CLI ! $ fab -l ! Install and

    configure saltstack minion in a new fresh server! fab -H user@your.server.name install_salt! ! Available commands:! ! install_salt Usage install_salt:master=‘master.salt.hostname',name='...! ! ! Live Demo! 17 Option
  18. Fabric Bash + Fabric + Salt = Power KISS: Use

    bash for script Use fabric for complex script with external software interaction, and for deploy Use Salt to provision ! Play with them, some example: Jenkins uses Fabric to deploy (legacy code) Jenkins uses Salt to deploy (new code) to staging 18 Bash + Fabric
  19. Continous delivery The github workflow Develop locally (Vagrant) Solve the

    issue or implement a feature in a new branch Make a pull request Social review (we are humans, right?) Test it Merge if passes all the steps Deploy to staging automatically 19 Workflow
  20. Continous delivery Con. delivery != Con. Deployment Use continuous deployment

    for staging env Use manual deploy for production env Monitor it Revert if everything is on fire Correlate deploy with metrics Use datadog, now, really! Demo! 20 Manual deploy?
  21. Sys App Architect @sysapp @mastrolinux http://sysapparchitect.com we are scaling too