fabric: control your linux nodes with python

pyconsg 2013

Transcript

1. FABRIC Control your linux nodes with python Saturday, June 15, 13
2. WHAT IS IT? • Command-line tool for deployment automation and system administration as a ssh wrapper • Executes to local or remote ✓ local: not a ssh connection. uses python subprocess module. ✓ run or sudo: when executing against remote, it can execute as the current user or as current user with sudo rights. Saturday, June 15, 13
3. WHAT IS IT? • Simply specify python functions in fabfile.py • Install it ✓ pip install fabric ✓ Once installed, fab command is available in your system path. ✓ Start writing your python function in fabfile.py Saturday, June 15, 13
4. AUTOMATION • Daily tasks • Server setup • System update and maintenance • New project creation • Deployment processes Saturday, June 15, 13
5. GET STARTED • Download and install virtualbox https://www.virtualbox.org • Download and install vagrant http://www.vagrantup.com cd ~/VirtualBox\ VMs mkdir debsqueeze64 cd debsqueeze64 vagrant init debsqueeze64 http://www.emken.biz/vagrant- boxes/debsqueeze64.box vagrant up Saturday, June 15, 13
6. GET STARTED • This downloads and creates an image of debian 64-bit on your computer. • We will use fabric on our local machine to interact with this debian virtual machine • See http://www.vagrantbox.es to download other prepared boxes. Saturday, June 15, 13
7. PYTHON VIRTUALENV • Create an isolated python project environment Mac OS X: package management with port or brew sudo port -v install virtualenv_select py27-virtualenv sudo port -v install py27-virtualenvwrapper sudo port select --set virtualenv virtualenv27 (optional) Saturday, June 15, 13
8. PYTHON VIRTUALENV • Edit your ~/.bash_profile export WORKON_HOME=$HOME/.virtualenvs export PROJECT_HOME=$HOME/work source `which virtualenvwrapper.sh` • $HOME -- this refers to your user home or “~” (i.e., on my computer, “~” means “/Users/calvin” directory) (optional) Saturday, June 15, 13
9. PYTHON VIRTUALENV • Create our python virtualenv for our fabric tutorial mkvirtualenv --distribute --no-site-packages learnfabric cd $PROJECT_HOME git clone git://github.com/calvinchengx/learnfabric.git cd learnfabric pip install -r requirements.txt (optional) Saturday, June 15, 13
10. LEARN FABRIC • Once we have our debian virtual machine and optionally our python virtualenv ready, we begin git clone git://github.com/calvinchengx/learnfabric.git cd learnfabric pip install -r requirements.txt cp -rf ~/VirtualBox\ VMs/debsqueeze64/.vagrant . • This installs fabric and installs python-vagrant (a wrapper around vagrant commands so we can programmatically control our vagrant instance) Saturday, June 15, 13
11. LEARN FABRIC • Once we have our debian virtual machine and optionally our python virtualenv ready, we begin git clone git://github.com/calvinchengx/learnfabric.git cd learnfabric pip install -r requirements.txt cp -rf ~/VirtualBox\ VMs/debsqueeze64/.vagrant . • This installs fabric and installs python-vagrant (a wrapper around vagrant commands so we can programmatically control our vagrant instance) Saturday, June 15, 13
12. FABRIC FUNCTION fab mytask [vagrant@127.0.0.1:2222] Executing task 'mytask' [vagrant@127.0.0.1:2222] run: echo $USER [vagrant@127.0.0.1:2222] out: vagrant [vagrant@127.0.0.1:2222] out: Done. Disconnecting from vagrant@127.0.0.1:2222... done. Saturday, June 15, 13
13. THE CODE import vagrant from fabric.api import env, task, run v = vagrant.Vagrant() v.up() env.hosts = [v.user_hostname_port()] env.key_filename = v.keyfile() env.disable_known_hosts = True # useful when vagrant box ip changes. # our first fabric function @task def mytask(): run('echo $USER') Saturday, June 15, 13
14. FABRIC CORE • local() runs a command locally, via python subprocess • run() runs a command remotely, via ssh • sudo() runs a command remotely as sudo, via ssh • put() copies a file from local to remote, via ssh • get() copies a file from remote to local, via ssh Saturday, June 15, 13
15. FABRIC AUTH • All authentication is ssh-based • SSH keys help us avoid typing in passwords - place fabric ssh user (env.user)’s public key in remote node’s authorized_keys • Avoid directly using root user • Give your fabric ssh user sudo rights instead (env.user) Saturday, June 15, 13
16. FABRIC CONFIG • from fabric.api import env • fabric environment is simply a dictionary containing host information, roles, user (env.user); and • any other custom information you would like to include. • $HOME/.fabricrc allows custom configuration, e.g. user = ssh_user_name where “ssh_user_name” is any value you so desire to pass in to env.user when a fab function runs. Saturday, June 15, 13
17. FABRIC ROLEDEFS # code from fabric.api import env env.roledefs = { ‘web’: [‘100.0.0.1’, ‘100.0.0.2’], ‘db’: [‘100.0.0.3’], ‘media’: [‘100.0.0.4’] } # command line fab -R web mytask Saturday, June 15, 13
18. FABRIC HOST(S) # code from fabric.api import env env.roledefs = { ‘web’: [‘100.0.0.1’, ‘100.0.0.2’], ‘db’: [‘100.0.0.3’], ‘media’: [‘100.0.0.4’] } # command line fab -H 100.0.0.1,100.0.0.2 mytask Saturday, June 15, 13
19. MAP ROLES TO TASKS # code from fabric.api import env from fabric.decorators import roles env.roledefs = { ‘web’: [‘100.0.0.1’, ‘100.0.0.2’], ‘db’: [‘100.0.0.3’], ‘media’: [‘100.0.0.4’] } @roles(‘web’) def mytask(): run(‘uptime’) # command line fab mytask # already mapped to -R web Saturday, June 15, 13
20. HANDLING FAILURES • Do not abort upon failure; just give a warning # code from fabric.context_managers import settings def mytask(): with settings(warn_only=True): run(‘rm /var/www/proj/releases/current’) run(‘ln -s /var/www/proj/releases/deployed /var/www/proj/releases/current’) Saturday, June 15, 13
21. LAZY SYSADMIN • A generic command # code @roles(‘all’) def invoke(command): “”” Invoke an arbitrary command “”” sudo(command) # command line fab invoke:“aptitude update” fab invoke:”aptitude upgrade” Saturday, June 15, 13
22. PARALLEL EXECUTION • A generic command # code @roles(‘all’) def invoke(command): “”” Invoke an arbitrary command “”” sudo(command) # command line fab -P invoke:“aptitude update” fab -P invoke:”aptitude upgrade” Saturday, June 15, 13
23. PYTHON WEB APP Saturday, June 15, 13
24. EXAMPLE DEPLOY @task(default=True) @set_target_env def deploy(email=False): """ `fab -R all deploy` or fab -H mysite.com deploy`. Execute a deployment to the given groups of hosts or host """ if not chk_req(): return if git_branch_check() or test_host_check(): manage_release('Deployment start') git_archive_and_upload_tar() pip_requirements() collectstatic(deploy=True) symlink_current() webserver() migrate_db() # post-deployment tasks manage_release('Deployment end') _releases_cleanup() email_on_success(trigger=email) # command line Saturday, June 15, 13
25. QUICK REFERENCE • fab -H host1.com task • fab -H host1.com task:arg1,arg2 • fab -P -H localhost, host1.com, host2.com task • fab -R web task # roles defined by env.roledefs • fab -R web task:arg1,arg2 • fab -P -R web task • fab -l • more options explained via fab --help Saturday, June 15, 13
26. DEVOPS • Bridging the gap between code implementation by developers and deployment to staging and/or live servers by sysadmin • Automation leads to continuous integration (e.g. Jenkins CI Server) and continuous delivery • Support heterogeneous development environment in combination with vagrant (developers test against vagrant, which has same distro as staging and production hosts) Saturday, June 15, 13
27. REFERENCES • My reusable fabric functions https://github.com/fabric-colors/fabric-colors https://fabric-colors.readthedocs.org/en/latest/index.html • Code that accompanies this set of slides https://github.com/calvinchengx/learnfabric • Fabric code and documentation https://github.com/fabric/fabric http://fabric.readthedocs.org • Multiple vagrant boxes for testing your server config http://awaseroot.wordpress.com/2012/05/06/script-for- adding-multiple-vagrant-boxes/ Saturday, June 15, 13
28. Q&A • Get in touch @calvinchengx • http://calvinx.com Saturday, June 15, 13