Saltstack, Fabric and the Deploy problem

Introduction to the Salt provisioning system and the usafe of Fabric as a bridge between old bash script and the new way of provisioning servers.

Transcript

1. Saltstack
   Provisioning Systems And
   Scalable Deploy Strategies
   Sys App Architect 1
   

   View Slide

2. Provisioning
   Old Sys Admin Style
   Manual conf by hand
   QA test it and (sometimes) breaks it
   Writing all the doc to create the env again
   Forgot something? Fix it
   QA breaks it again
   Forgot something? Fix it
   QA breaks it again
   …
   2
   Why you need it
   

   View Slide

3. Provisisioning
   It saves time and money
   Reproducible environment
   Auto documented
   Easy scalable
   Easy to create Dev env like a Prod env
   Sharing a git repo is sharing the whole env
   One man can manage 100 servers (1/1200 at
   Google) 3
   Why you need it
   

   View Slide

4. Provisioning
   Rules
   Use a strong naming convention
   (type-zone-provider-#-prod)
   Divide your servers by type
   Find common rules to all server
   Keep everything as generic as possible
   Prepare for multi zone and multi provider
   Small states 4
   What you need
   

   View Slide

5. Saltastack
   Saltstack history
   The author: Thomas S. Hatch
   What needed to be fix: the slowness
   The solution: ØMQ
   !
   Fast, modular, async
   Dependencies need to be specified explicitly
   5
   The revolution
   

   View Slide

6. Saltstack
   Design
   Execution Modules:
   States
   Grains
   Renderers (yaml, mako, jinja)
   Returners (redis, cassandra, etcd)
   Runners (custom scripts executed on master)
   !
   Push/Pull, choose what you like
   6
   Components
   

   View Slide

7. Saltstack
   State file
   {% set home = pillar.get('home', '/home/ubuntu') %}!
   {% set pythondir = pillar.get('pythondir', 'easyvote') %}!
   nginx-pkg:!
   pkg.installed:!
   - name: nginx!
   !
   start-nginx-server:!
   service.running:!
   - name: nginx!
   - enable: True!
   - reload: True!
   - require:!
   - pkg: nginx-pkg!
   - watch:!
   - file: python-nginx-conf!
   !
   python-nginx-conf:!
   file.managed:!
   - name: /etc/nginx/sites-available/easyvote-python.conf!
   - source: salt://nginx/conf/easyvote-python.conf!
   - template: jinja!
   - defaults:!
   pythondir: {{ home }}/{{ pythondir }}!
   - require: !
   - pkg: nginx-pkg!
   7
   State
   

   View Slide

8. Saltstack
   Grains of Salt
   Info from the minion
   host: easyvotegandi01!
   id: easyvote-01-staging!
   ip_interfaces: {'lo': ['127.0.0.1'], 'eth0': ['46.226.105.41']}!
   ipv4:!
   127.0.0.1!
   46.226.105.41!
   ipv6:!
   2001:4b98:dc0:43:216:3eff:fe34:72fe!
   ::1!
   fe80::216:3eff:fe34:72fe!
   kernel: Linux!
   kernelrelease: 3.2.53-xenU-8869-x86_64!
   localhost: easyvotegandi01!
   lsb_distrib_codename: precise!
   lsb_distrib_description: Ubuntu 12.04.3 LTS!
   lsb_distrib_id: Ubuntu!
   lsb_distrib_release: 12.04!
   8
   Grains
   

   View Slide

9. Saltstack
   Pillar, push variables to the minion
   Pillars are dictionaries (json, yaml, etc.)
   #!py!
   !
   def run():!
   c9 = {}!
   config = subprocess.check_output("/usr/local/bin/node -e ‘console.log(!
   JSON.stringify(require(\"./deploy\")({revision: \"abcdefg\"}), null, 2))'", !
   shell=True, cwd="%s/cloud9infra-config" % dirname)!
   c9['c9'] = json.loads(config)!
   return c9!
   !
   !
   !
   mysql:!
   ----------!
   id:!
   1!
   log-basename:!
   master!
   user:!
   ubuntu
   9
   Pillar
   

   View Slide

10. Saltstack
    File templating
    Jinja + YAML, Mako + YAML, Wempy + YAML,
    Jinja + json, Mako + json, Wempy + json.
    !
    daemonize yes!
    pidfile {{ directory }}/redis-infra.pid!
    port {{ redis.port }}!
    bind {{ bind }}!
    timeout {{ redis.timeout }}!
    loglevel notice!
    logfile {{ logdir }}/redis-infra.log!
    databases {{ redis.databases }}!
    save 900 100!
    save 300 10000!
    save 60 100000!
    stop-writes-on-bgsave-error yes!
    rdbcompression yes!
    rdbchecksum yes!
    dbfilename {{ redis.dbfilename }}!
    dir {{ directory }}!
    10
    Renderers
    

    View Slide

11. Saltastack
    Save your data output
    salt '*' test.ping --return mongo_return,redis_return,cassandra_return
    Write your own
    import redis!
    import json!
    !
    def returner(ret):!
    '''!
    Return information to a redis server!
    '''!
    serv = redis.Redis(!
    host='redis-serv.example.com',!
    port=6379,!
    db='0')!
    serv.sadd("%(id)s:jobs" % ret, ret['jid'])!
    serv.set("%(jid)s:%(id)s" % ret, json.dumps(ret['return']))!
    serv.sadd('jobs', ret['jid'])!
    serv.sadd(ret['jid'], ret['id'])!
    11
    Returners
    

    View Slide

12. Saltstack
    Targeting minion servers
    salt 'web[1-5]' test.ping
    salt ‘*-prod' test.ping
    base:
    'web1-(prod|devel)':
    - match: pcre
    - nginx
    !
    nodegroups:
    group1: 'L@foo.domain.com,bar.domain.com,baz.domain.com or bl*.domain.com'
    group2: 'G@os:Debian and foo.domain.com’
    !
    salt -N group1 test.ping
    !
    G is globbing, L list of minions, S subnet, etc.
    12
    Targeting
    

    View Slide

13. Saltstack
    Runners and cli
    You can define your own commands
    or run common ones
    !
    salt -L foo.bar.baz,quo.qux cmd.run ‘pgrep foo’
    salt -G 'os:Ubuntu' test.ping
    salt '*' cmd.run 'echo "Hello: $FIRST_NAME"'
    env='{FIRST_NAME: "Joe"}'
    13
    Runners
    

    View Slide

14. Saltstack
    Highstates and keys
    Accept a key
    salt-key -L
    salt-key -a
    !
    Run all the states
    salt ‘*’ state.highstate
    14
    Keys and auto provision
    

    View Slide

15. Fabric
    Fabric
    Set of python libraries to run command
    remotely
    !
    Parallel execution
    Remote terminal output
    Interactive remote shell
    Plain python
    15
    From bash to fabric
    

    View Slide

16. Fabric
    @task!
    def install_salt(master='salt.c9.io', name=None, set_hostname=True):!
    '''!
    Usage install_salt:master='master.salt.hostname',name='minion-name',set_hostname='yes'!
    '''!
    distro = run('lsb_release -s -c')!
    if distro == 'wheezy':!
    run('echo deb http://debian.saltstack.com/debian wheezy-saltstack main\!
    | sudo tee /etc/apt/sources.list.d/saltstack.list')!
    run('wget -q -O- “http://debian.saltstack.com/debian-salt-team-joehealy.gpg.key”\!
    | sudo apt-key add -')!
    !
    run('sudo apt-get update')!
    run('sudo apt-get -y install salt-minion')!
    !
    files.uncomment('/etc/salt/minion', 'master:.*', use_sudo=True)!
    files.sed('/etc/salt/minion', 'master:.*', 'master: %s' % master, use_sudo=True)!
    if name:!
    files.uncomment('/etc/salt/minion', 'id:.*', use_sudo=True)!
    files.sed('/etc/salt/minion', 'id:.*', 'id: %s' % name, use_sudo=True)!
    !
    if set_hostname:!
    sudo('hostname %s' % name)!
    sudo('echo %s > /etc/hostname' % name)!
    !
    run('sudo service salt-minion restart')!
    16
    fabric for salt
    

    View Slide

17. Fabric
    Fabric CLI
    !
    $ fab -l !
    Install and configure saltstack minion in a new fresh server!
    fab -H user@your.server.name install_salt!
    !
    Available commands:!
    !
    install_salt Usage install_salt:master=‘master.salt.hostname',name='...!
    !
    !
    Live Demo!
    17
    Option
    

    View Slide

18. Fabric
    Bash + Fabric + Salt = Power
    KISS:
    Use bash for script
    Use fabric for complex script with external software
    interaction, and for deploy
    Use Salt to provision
    !
    Play with them, some example:
    Jenkins uses Fabric to deploy (legacy code)
    Jenkins uses Salt to deploy (new code) to staging
    18
    Bash + Fabric
    

    View Slide

19. Continous delivery
    The github workflow
    Develop locally (Vagrant)
    Solve the issue or implement a feature in a new
    branch
    Make a pull request
    Social review (we are humans, right?)
    Test it
    Merge if passes all the steps
    Deploy to staging automatically
    19
    Workflow
    

    View Slide

20. Continous delivery
    Con. delivery != Con. Deployment
    Use continuous deployment for staging env
    Use manual deploy for production env
    Monitor it
    Revert if everything is on fire
    Correlate deploy with metrics
    Use datadog, now, really!
    Demo!
    20
    Manual deploy?
    

    View Slide

21. Sys App Architect
    @sysapp
    @mastrolinux
    http://sysapparchitect.com
    we are scaling too
    21
    

    View Slide