Upgrade to Pro — share decks privately, control downloads, hide ads and more …

fabric: control your linux nodes with python

fabric: control your linux nodes with python

pyconsg 2013

Avatar for Calvin Cheng

Calvin Cheng

June 13, 2013
Tweet

More Decks by Calvin Cheng

Other Decks in Technology

Transcript

  1. WHAT IS IT? • Command-line tool for deployment automation and

    system administration as a ssh wrapper • Executes to local or remote ✓ local: not a ssh connection. uses python subprocess module. ✓ run or sudo: when executing against remote, it can execute as the current user or as current user with sudo rights. Saturday, June 15, 13
  2. WHAT IS IT? • Simply specify python functions in fabfile.py

    • Install it ✓ pip install fabric ✓ Once installed, fab command is available in your system path. ✓ Start writing your python function in fabfile.py Saturday, June 15, 13
  3. AUTOMATION • Daily tasks • Server setup • System update

    and maintenance • New project creation • Deployment processes Saturday, June 15, 13
  4. GET STARTED • Download and install virtualbox https://www.virtualbox.org • Download

    and install vagrant http://www.vagrantup.com cd ~/VirtualBox\ VMs mkdir debsqueeze64 cd debsqueeze64 vagrant init debsqueeze64 http://www.emken.biz/vagrant- boxes/debsqueeze64.box vagrant up Saturday, June 15, 13
  5. GET STARTED • This downloads and creates an image of

    debian 64-bit on your computer. • We will use fabric on our local machine to interact with this debian virtual machine • See http://www.vagrantbox.es to download other prepared boxes. Saturday, June 15, 13
  6. PYTHON VIRTUALENV • Create an isolated python project environment Mac

    OS X: package management with port or brew sudo port -v install virtualenv_select py27-virtualenv sudo port -v install py27-virtualenvwrapper sudo port select --set virtualenv virtualenv27 (optional) Saturday, June 15, 13
  7. PYTHON VIRTUALENV • Edit your ~/.bash_profile export WORKON_HOME=$HOME/.virtualenvs export PROJECT_HOME=$HOME/work

    source `which virtualenvwrapper.sh` • $HOME -- this refers to your user home or “~” (i.e., on my computer, “~” means “/Users/calvin” directory) (optional) Saturday, June 15, 13
  8. PYTHON VIRTUALENV • Create our python virtualenv for our fabric

    tutorial mkvirtualenv --distribute --no-site-packages learnfabric cd $PROJECT_HOME git clone git://github.com/calvinchengx/learnfabric.git cd learnfabric pip install -r requirements.txt (optional) Saturday, June 15, 13
  9. LEARN FABRIC • Once we have our debian virtual machine

    and optionally our python virtualenv ready, we begin git clone git://github.com/calvinchengx/learnfabric.git cd learnfabric pip install -r requirements.txt cp -rf ~/VirtualBox\ VMs/debsqueeze64/.vagrant . • This installs fabric and installs python-vagrant (a wrapper around vagrant commands so we can programmatically control our vagrant instance) Saturday, June 15, 13
  10. LEARN FABRIC • Once we have our debian virtual machine

    and optionally our python virtualenv ready, we begin git clone git://github.com/calvinchengx/learnfabric.git cd learnfabric pip install -r requirements.txt cp -rf ~/VirtualBox\ VMs/debsqueeze64/.vagrant . • This installs fabric and installs python-vagrant (a wrapper around vagrant commands so we can programmatically control our vagrant instance) Saturday, June 15, 13
  11. FABRIC FUNCTION fab mytask [[email protected]:2222] Executing task 'mytask' [[email protected]:2222] run:

    echo $USER [[email protected]:2222] out: vagrant [[email protected]:2222] out: Done. Disconnecting from [email protected]:2222... done. Saturday, June 15, 13
  12. THE CODE import vagrant from fabric.api import env, task, run

    v = vagrant.Vagrant() v.up() env.hosts = [v.user_hostname_port()] env.key_filename = v.keyfile() env.disable_known_hosts = True # useful when vagrant box ip changes. # our first fabric function @task def mytask(): run('echo $USER') Saturday, June 15, 13
  13. FABRIC CORE • local() runs a command locally, via python

    subprocess • run() runs a command remotely, via ssh • sudo() runs a command remotely as sudo, via ssh • put() copies a file from local to remote, via ssh • get() copies a file from remote to local, via ssh Saturday, June 15, 13
  14. FABRIC AUTH • All authentication is ssh-based • SSH keys

    help us avoid typing in passwords - place fabric ssh user (env.user)’s public key in remote node’s authorized_keys • Avoid directly using root user • Give your fabric ssh user sudo rights instead (env.user) Saturday, June 15, 13
  15. FABRIC CONFIG • from fabric.api import env • fabric environment

    is simply a dictionary containing host information, roles, user (env.user); and • any other custom information you would like to include. • $HOME/.fabricrc allows custom configuration, e.g. user = ssh_user_name where “ssh_user_name” is any value you so desire to pass in to env.user when a fab function runs. Saturday, June 15, 13
  16. FABRIC ROLEDEFS # code from fabric.api import env env.roledefs =

    { ‘web’: [‘100.0.0.1’, ‘100.0.0.2’], ‘db’: [‘100.0.0.3’], ‘media’: [‘100.0.0.4’] } # command line fab -R web mytask Saturday, June 15, 13
  17. FABRIC HOST(S) # code from fabric.api import env env.roledefs =

    { ‘web’: [‘100.0.0.1’, ‘100.0.0.2’], ‘db’: [‘100.0.0.3’], ‘media’: [‘100.0.0.4’] } # command line fab -H 100.0.0.1,100.0.0.2 mytask Saturday, June 15, 13
  18. MAP ROLES TO TASKS # code from fabric.api import env

    from fabric.decorators import roles env.roledefs = { ‘web’: [‘100.0.0.1’, ‘100.0.0.2’], ‘db’: [‘100.0.0.3’], ‘media’: [‘100.0.0.4’] } @roles(‘web’) def mytask(): run(‘uptime’) # command line fab mytask # already mapped to -R web Saturday, June 15, 13
  19. HANDLING FAILURES • Do not abort upon failure; just give

    a warning # code from fabric.context_managers import settings def mytask(): with settings(warn_only=True): run(‘rm /var/www/proj/releases/current’) run(‘ln -s /var/www/proj/releases/deployed /var/www/proj/releases/current’) Saturday, June 15, 13
  20. LAZY SYSADMIN • A generic command # code @roles(‘all’) def

    invoke(command): “”” Invoke an arbitrary command “”” sudo(command) # command line fab invoke:“aptitude update” fab invoke:”aptitude upgrade” Saturday, June 15, 13
  21. PARALLEL EXECUTION • A generic command # code @roles(‘all’) def

    invoke(command): “”” Invoke an arbitrary command “”” sudo(command) # command line fab -P invoke:“aptitude update” fab -P invoke:”aptitude upgrade” Saturday, June 15, 13
  22. EXAMPLE DEPLOY @task(default=True) @set_target_env def deploy(email=False): """ `fab -R all

    deploy` or fab -H mysite.com deploy`. Execute a deployment to the given groups of hosts or host """ if not chk_req(): return if git_branch_check() or test_host_check(): manage_release('Deployment start') git_archive_and_upload_tar() pip_requirements() collectstatic(deploy=True) symlink_current() webserver() migrate_db() # post-deployment tasks manage_release('Deployment end') _releases_cleanup() email_on_success(trigger=email) # command line Saturday, June 15, 13
  23. QUICK REFERENCE • fab -H host1.com task • fab -H

    host1.com task:arg1,arg2 • fab -P -H localhost, host1.com, host2.com task • fab -R web task # roles defined by env.roledefs • fab -R web task:arg1,arg2 • fab -P -R web task • fab -l • more options explained via fab --help Saturday, June 15, 13
  24. DEVOPS • Bridging the gap between code implementation by developers

    and deployment to staging and/or live servers by sysadmin • Automation leads to continuous integration (e.g. Jenkins CI Server) and continuous delivery • Support heterogeneous development environment in combination with vagrant (developers test against vagrant, which has same distro as staging and production hosts) Saturday, June 15, 13
  25. REFERENCES • My reusable fabric functions https://github.com/fabric-colors/fabric-colors https://fabric-colors.readthedocs.org/en/latest/index.html • Code

    that accompanies this set of slides https://github.com/calvinchengx/learnfabric • Fabric code and documentation https://github.com/fabric/fabric http://fabric.readthedocs.org • Multiple vagrant boxes for testing your server config http://awaseroot.wordpress.com/2012/05/06/script-for- adding-multiple-vagrant-boxes/ Saturday, June 15, 13