DEPLOYING WITH VAGRANT AND ANSIBLE YEUK HON WONG @yeukhon GLASS Lab, CUNY City 

REPO https://bitbucket.org/ideapiteam/repo-api-master https://bitbucket.org/yeukhon/vagrant-ansible-example 

GLASS LAB: Developing Graphyte and Aurum. Serving ~100 undergradutes. 

DEPLOYMENT: Deploying web service is just a subset of a bigger picture. 

BIGGER PICTURE: “Obama campaign AWS infrastructure in one diagram” (http://awsofa.info/) 

Some Ideal Work-flow http://goo.gl/pJtK7 

DEPLOY #TODO: O Development deployment O Testing deployment O Production deployment 

GET A MACHINE. Virtualbox is nice. 

VAGRANT FOR DEVELOPMENT 

Getting Started ● Install Virtualbox https://www.virtualbox.org/wiki/Downloads ● Install Vagrant http://docs.vagrantup.com/v2/installation 

Vagrant Box ● A Vagrant box is a packaged virtual machine. If you extract it you get a vagrant file, ovf and and a vmdk file. ● Vagrant has official boxes for download (distribute with a universal SSH key, so do not use it in production) 

Official Boxes Ubuntu Lucid 32 Bit http://files.vagrantup.com/lucid32.box Ubuntu Lucid 64 Bit http://files.vagrantup.com/lucid64.box Ubuntu Precise 32 Bit http://files.vagrantup.com/precise32.box Ubuntu Precise 64 Bit http://files.vagrantup.com/precise64.box 

Community Boxes http://www.vagrantbox.es/ 

Step 1: Create a box 1. Download a box 2. Initialize a Vagrant workspace $ mkdir vagrant-example $ cd vagrant-example $ vagrant init precise32 $ vagrant box add precise32 http://files.vagrantup.com/precise32.box 

Step 2: Vagrantfile Vagrantfile Vagrantfile is a ruby configuration file used to configure a vagrant instance. Vagrant.configure("2") do |config| config.vm.box = "precise32" config.vm.network :public_network config.vm.provider :virtualbox do |vb| vb.customize ["modifyvm", :id, "--memory", "400"] end end 

Step 3: vagrant up vagrant up vagrant up will create a new VM or boot up an existing Vagrant virtual machine. $ vagrant up $ vagrant up Bringing machine 'default' up with 'virtualbox' provider... Bringing machine 'default' up with 'virtualbox' provider... [default] Importing base box 'precise32'... [default] Importing base box 'precise32'... 

Step 3: vagrant ssh vagrant ssh vagrant ssh will allow you to automatically ssh into your vagrant instance if you are using an official box. $ vagrant ssh $ vagrant ssh Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic-pae i686) Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic-pae i686) * Documentation: https://help.ubuntu.com/ * Documentation: https://help.ubuntu.com/ Welcome to your Vagrant-built virtual machine. Welcome to your Vagrant-built virtual machine. Last login: Fri Sep 14 06:22:31 2012 from 10.0.2.2 Last login: Fri Sep 14 06:22:31 2012 from 10.0.2.2 

Vagrant Commands Usage: vagrant [-v] [-h] command [] Usage: vagrant [-v] [-h] command [] -v, --version Print the version and exit. -v, --version Print the version and exit. -h, --help Print this help. -h, --help Print this help. Available subcommands: Available subcommands: box box destroy destroy halt halt init init package package plugin plugin provision provision reload reload resume resume ssh ssh ssh-config ssh-config status status suspend suspend up up For help on any individual command run `vagrant COMMAND -h` For help on any individual command run `vagrant COMMAND -h` 

I don't want an official box You can use any ISO local or over the wire. $ mkdir vagrant-example $ mkdir vagrant-example $ cd vagrant-example $ cd vagrant-example $ vagrant init $ vagrant init $ vim Vagrantfile config.vm.box_url = "http://domain.com/path/to/above.box" 

I want my own SSH key Even for vagrant official boxes, you should consider changing to your own SSH key. 

DEMO? Later! 

CONFIGURE MACHINES Human, please. 

Treat your infrastructure like your software. O Version control your configuration and specs O Readable O Automated 

REQUIREMENTS: O Strong community support O Medium to little entry to barrier (installation, tutorial, customizability) O Readable configuration O Version control O As lightweight as possible 

So many, which to use? #!/bin/bash #!/usr/bin/python + Fabric #!/bin/bash #!/usr/bin/python + Fabric 

ANSIBLE FOR CONFIGURATION 

CONFIGURATION IN PLAINTEXT (YAML) - name: Install Nginx apt: pkg=nginx state=present 

STRONG COMMUNITY Started in 2012, has over 423 forks, 1,429 watcher, 2000+ tickets solved, weekly clearance, active IRC & mailing list 

PYTHON Core is in Python. Much smaller core than other competitors. 

5 REQUIRED DEPENDENCIES Python for running Python Simplejson/json for results Jinja2 for template Pyyaml for configuration Paramiko for ssh2 protocol 

#TODO We will configure a machine for repoapi live. O bootstrap machine with dev pkgs O setup nginx O setup database O clone and install master repo O run server 

Getting Started ● Install AnVsible http://ansible.cc/docs/ $ sudo pip install pyyaml jinja2 paramiko $ sudo pip install pyyaml jinja2 paramiko $ git clone git://github.com/ansible/ansible.git $ git clone git://github.com/ansible/ansible.git $ cd ansible $ cd ansible $ sudo make install $ sudo make install 

Modules ● Ansible community has a lot of modules that you can used to configure a server. Modules could be in ruby, python, perl, etc. Just executable. http://ansible.cc/docs/modules.html apt, yum, pip, gem, homebrew, user apt, yum, pip, gem, homebrew, user file, copy, template, django_manage file, copy, template, django_manage postgres_user, postgres_db, mysql*, mongodb*, rabbitmq*, Riak postgres_user, postgres_db, mysql*, mongodb*, rabbitmq*, Riak ec2, ec2_facts, s3, ec2, ec2_facts, s3, hg, git, svn hg, git, svn many more... many more... 

Hosts / Inventory An inventory file contains a list of machine addresses. http://ansible.cc/docs/patterns.html 192.168.1.4 192.168.1.4 [webservers] [webservers] foo.example.com foo.example.com bar.example.com bar.example.com www[01:50].example.com www[01:50].example.com 

Ansible Playbook --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # You can use variables during configuration! # You can use variables during configuration! vars: vars: - APP_PORT: 8080 - APP_PORT: 8080 tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg=nginx apt: pkg=nginx 

Template for Play - name: Description of the task/play - name: Description of the task/play module_name: arguments option1=value1 option2=value2 module_name: arguments option1=value1 option2=value2 

apt 

apt get install nginx? --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True tasks: tasks: - name: Ensure Nginx is installed apt: pkg=nginx 

apt-get remove nginx? --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True tasks: tasks: - name: Ensure Nginx is removed apt: pkg=nginx state=absent 

Idempotence Idempotence means for the same configuration settings, f(f(x)) = f(x) f(f(x)) = f(x) HTTP's GET, HEAD, PUT, DELETE, OPTIONS, TRACE satisfy idempotence. 

TEMPLATE, VARIABLE, FACTS 

Variable --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # You can use variables during configuration! # You can use variables during configuration! vars: - APP_PORT: 6543 tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg=nginx apt: pkg=nginx 

Variable --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # You can use variables during configuration! # You can use variables during configuration! vars: - NGINX_NAME: nginx tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg= apt: pkg=$NGINX_NAME $NGINX_NAME 

Variable --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # You can use variables during configuration! # You can use variables during configuration! vars: - NGINX_NAME: nginx tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg= apt: pkg=$NGINX_NAME $NGINX_NAME 

Template + Variable --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # You can use variables during configuration! # You can use variables during configuration! vars: - HELLO_WORLD: “Hi, World” tasks: tasks: - name: Configure a foo.j2 - name: Configure a foo.j2 template: src=foo.j2 dest=/home/vagrant/foo.txt # call this foo.j2 - a Jinja template file # call this foo.j2 - a Jinja template file {{ {{ HELLO_WORLD }} }} # result # result $ cat foo.txt $ cat foo.txt Hi, World Hi, World 

Gather Facts Ansible can gather basic facts about the target machine. By default, this feature is on. If you don't need it, set gather_facts: no gather_facts: no to speed up configuration. --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True gather_facts: no gather_facts: no … …. . 

Gather Facts ansible -m setup 127.0.0.1 -u vagrant -k ansible -m setup 127.0.0.1 -u vagrant -k 127.0.0.1 | success >> { 127.0.0.1 | success >> { "ansible_facts": { "ansible_facts": { …. …. "ansible_lsb": { "ansible_lsb": { "codename": "precise", "codename": "precise", "description": "Ubuntu 12.04.1 LTS", "description": "Ubuntu 12.04.1 LTS", "id": "Ubuntu", "id": "Ubuntu", "major_release": "12", "major_release": "12", "release": "12.04" "release": "12.04" }, }, "ansible_pkg_mgr": "apt", "ansible_pkg_mgr": "apt", "ansible_processor": [ "ansible_processor": [ "Intel(R) Pentium(R) 4 CPU 3.20GHz" "Intel(R) Pentium(R) 4 CPU 3.20GHz" ], ], "ansible_processor_cores": 1, "ansible_processor_cores": 1, "ansible_processor_count": "ansible_processor_count": 1 1 "ansible_python_version": "2.7.3", "ansible_python_version": "2.7.3", …. …. } } see https://gist.github.com/yeukhon/4689592 see https://gist.github.com/yeukhon/4689592 

Templating Nginx nginx.conf and sites-available/default require configuration. o worker counts o application port and proxy settings o SSL (although we won't do it here) o many more 

nginx.conf.j2 # nginx.conf.j2 # nginx.conf.j2 user www-data; user www-data; worker_processes worker_processes {{ WORKER_COUNT }} {{ WORKER_COUNT }}; ; pid /var/run/nginx.pid; pid /var/run/nginx.pid; events { events { worker_connections worker_connections {{ WORKER_COUNT * 1024 }} {{ WORKER_COUNT * 1024 }}; ; # multi_accept on; # multi_accept on; } } 

Configure nginx.conf.j2 --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True vars: - WORKER_COUNT: ${ansible_processor_count} tasks: tasks: - name: Configure nginx.conf - name: Configure nginx.conf template: src=nginx.conf.j2 path=/etc/nginx/nginx.conf 

Var files --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # vars: # - WORKER_COUNT: ${ansible_processor_count} var_files: var_files: - nginx_vars.yml - nginx_vars.yml tasks: tasks: - name: Configure nginx.conf - name: Configure nginx.conf template: src=nginx.conf.j2 path=/etc/nginx/nginx.conf # nginx_vars.yml # nginx_vars.yml WORKER_COUNT: ${ansible_processor_count} 

DRY? LOOP! WITH_ITEMS 

with_items - name: Install Debian pkgs - name: Install Debian pkgs apt: pkg= apt: pkg=$item $item with_items: with_items: - build-essential - build-essential - python-dev - python-dev - python-pip - python-pip - python-setuptools - python-setuptools - name: Pip install Python tools - name: Pip install Python tools pip: name= pip: name=$item $item with_items: with_items: - distribute - distribute - virtualenv - virtualenv 

with_items 

A FEW MORE MODULES.... 

hg vars: vars: - URL: - URL: https://bitbucket.org/ideapiteam/repo-api-master https://bitbucket.org/ideapiteam/repo-api-master tasks: tasks: - name: Clone Master - name: Clone Master hg: repo=$URL dest=/home/vagrant/repoapi hg: repo=$URL dest=/home/vagrant/repoapi 

service tasks: tasks: - name: Ensure Nginx has started - name: Ensure Nginx has started service: name=nginx state=started service: name=nginx state=started # started, restarted, stopped 

file tasks: tasks: - name: Ensure /home/vagrant/repoapi exists - name: Ensure /home/vagrant/repoapi exists file: path=/home/vagrant/repoapi state=directory file: path=/home/vagrant/repoapi state=directory # state=directory/file/link 

user tasks: tasks: - name: Ensure user vagrant actually exist - name: Ensure user vagrant actually exist user: name=vagrant uid=1001 state=present user: name=vagrant uid=1001 state=present 

postgres_* tasks: tasks: - name: Create a Postgres database - name: Create a Postgres database postgres_db: db=mydb owner=user123 login_password: $md5_pwd postgres_db: db=mydb owner=user123 login_password: $md5_pwd - name: Create a Postgres user - name: Create a Postgres user postgres_user: user=user123 password=$hash_pwd priv=ALL postgres_user: user=user123 password=$hash_pwd priv=ALL 

ROLES? TAG 

Role You can have a cluster of webservice and dbserver. You can narrow down to a cluster of mongo instances, project1 webservice, project2 webservice, etc. Group them inside inventory. 

Inventory groups (roles) [webservers] [webservers] 192.168.1.1 192.168.1.1 192.168.1.2 192.168.1.2 [mongoservers] 192.168.1.4 192.168.1.4 192.168.1.5 192.168.1.5 192.168.1.7 192.168.1.7 [all] [all] Webservers Webservers mongoservers mongoservers 

Inventory name can be role # production # production [webservers] [webservers] www[01:20].example.com www[01:20].example.com [mongoservers] db[01:20].example.com db[01:20].example.com [all] [all] webservers webservers mongoservers mongoservers # development # development [webservers] [webservers] 192.168.1.1 192.168.1.1 192.168.1.2 192.168.1.2 [mongoservers] 192.168.1.4 192.168.1.4 192.168.1.5 192.168.1.5 192.168.1.7 192.168.1.7 [all] [all] webservers webservers mongoservers mongoservers ansible-playbook -i ansible-playbook -i development setup.yml setup.yml ansible-playbook -i ansible-playbook -i production setup.yml setup.yml 

Slow? --- --- # setup.yml # setup.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg=nginx apt: pkg=nginx - name: Update web service - name: Update web service hg: repo=$URL dest=/home/vagrant/repoapi hg: repo=$URL dest=/home/vagrant/repoapi … … more tasks x 100 ... more tasks x 100 ... 

Tags? --- --- # setup.yml # setup.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True tags: webserver tags: webserver tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg=nginx apt: pkg=nginx tags: tags: - nginx - nginx - name: Update web service - name: Update web service hg: repo=$URL dest=/home/vagrant/repoapi hg: repo=$URL dest=/home/vagrant/repoapi tags: tags: - repo - repo ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml --tags webserver --tags webserver ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml --tags nginx --tags nginx ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml --tags repo --tags repo ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml --tags nginx,repo --tags nginx,repo 

BEST PRACTICES? Use whatever works for you. 

BEST PRACTICES? http://ansible.cc/docs/patterns.html (inventory, tag, role by inventory, role by tag, role by file name, etc) 

PULL? To do full automated deploymet, you probably want to do pull rather than push. Ansible can do that too. http://jpmens.net/2012/07/14/ansible-pull-instead-of-push/ 

BUILD CUSTOM BOX Ansible is great for configuration. Vagrant is great for manging development vm. But building a base box to begin with? Use veewee. 

ANSIBLE HAS VAGRANT MODULES I haven't tried it, but in theory you could configure a bunch of vagrant instances using ansible and vagrant module! 

DEMO TIME! 

Questions? @yeukhon yeukhon.bitbucket.org speakerdeck.com/yeukhon 

IRC: #ansible github.com/ansible/ansible groups.google.com/group/ansible-project/ 

Ansible IRC: #ansible github.com/ansible/ansible groups.google.com/group/ansible-project/ 

:p Thank you!