Deploying with Vagrant and Ansible

April 17, 2013
Adv Tps Intrnet Prog: CSC I4330 3GG [2337] (CCNY )

1. DEPLOYING WITH VAGRANT AND ANSIBLE YEUK HON WONG @yeukhon GLASS Lab, CUNY City
2. REPO https://bitbucket.org/ideapiteam/repo-api-master https://bitbucket.org/yeukhon/vagrant-ansible-example
3. GLASS LAB: Developing Graphyte and Aurum. Serving ~100 undergradutes.
4. DEPLOYMENT: Deploying web service is just a subset of a bigger picture.
5. BIGGER PICTURE: “Obama campaign AWS infrastructure in one diagram” (http://awsofa.info/)
6. Some Ideal Work-flow http://goo.gl/pJtK7
7. DEPLOY #TODO: O Development deployment O Testing deployment O Production deployment
8. GET A MACHINE. Virtualbox is nice.
9. VAGRANT FOR DEVELOPMENT
10. Getting Started • Install Virtualbox https://www.virtualbox.org/wiki/Downloads • Install Vagrant http://docs.vagrantup.com/v2/installation
11. Vagrant Box • A Vagrant box is a packaged virtual machine. If you extract it you get a vagrant file, ovf and and a vmdk file. • Vagrant has official boxes for download (distribute with a universal SSH key, so do not use it in production)
12. Official Boxes Ubuntu Lucid 32 Bit http://files.vagrantup.com/lucid32.box Ubuntu Lucid 64 Bit http://files.vagrantup.com/lucid64.box Ubuntu Precise 32 Bit http://files.vagrantup.com/precise32.box Ubuntu Precise 64 Bit http://files.vagrantup.com/precise64.box
13. Community Boxes http://www.vagrantbox.es/
14. Step 1: Create a box 1. Download a box 2. Initialize a Vagrant workspace $ mkdir vagrant-example $ cd vagrant-example $ vagrant init precise32 $ vagrant box add precise32 http://files.vagrantup.com/precise32.box
15. Step 2: Vagrantfile Vagrantfile Vagrantfile is a ruby configuration file used to configure a vagrant instance. Vagrant.configure("2") do |config| config.vm.box = "precise32" config.vm.network :public_network config.vm.provider :virtualbox do |vb| vb.customize ["modifyvm", :id, "--memory", "400"] end end
16. Step 3: vagrant up vagrant up vagrant up will create a new VM or boot up an existing Vagrant virtual machine. $ vagrant up $ vagrant up Bringing machine 'default' up with 'virtualbox' provider... Bringing machine 'default' up with 'virtualbox' provider... [default] Importing base box 'precise32'... [default] Importing base box 'precise32'...
17. Step 3: vagrant ssh vagrant ssh vagrant ssh will allow you to automatically ssh into your vagrant instance if you are using an official box. $ vagrant ssh $ vagrant ssh Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic-pae i686) Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic-pae i686) * Documentation: https://help.ubuntu.com/ * Documentation: https://help.ubuntu.com/ Welcome to your Vagrant-built virtual machine. Welcome to your Vagrant-built virtual machine. Last login: Fri Sep 14 06:22:31 2012 from 10.0.2.2 Last login: Fri Sep 14 06:22:31 2012 from 10.0.2.2
18. Vagrant Commands Usage: vagrant [-v] [-h] command [<args>] Usage: vagrant [-v] [-h] command [<args>] -v, --version Print the version and exit. -v, --version Print the version and exit. -h, --help Print this help. -h, --help Print this help. Available subcommands: Available subcommands: box box destroy destroy halt halt init init package package plugin plugin provision provision reload reload resume resume ssh ssh ssh-config ssh-config status status suspend suspend up up For help on any individual command run `vagrant COMMAND -h` For help on any individual command run `vagrant COMMAND -h`
19. I don't want an official box You can use any ISO local or over the wire. $ mkdir vagrant-example $ mkdir vagrant-example $ cd vagrant-example $ cd vagrant-example $ vagrant init $ vagrant init $ vim Vagrantfile config.vm.box_url = "http://domain.com/path/to/above.box"
20. I want my own SSH key Even for vagrant official boxes, you should consider changing to your own SSH key.
21. DEMO? Later!
22. CONFIGURE MACHINES Human, please.
23. Treat your infrastructure like your software. O Version control your configuration and specs O Readable O Automated
24. REQUIREMENTS: O Strong community support O Medium to little entry to barrier (installation, tutorial, customizability) O Readable configuration O Version control O As lightweight as possible
25. So many, which to use? #!/bin/bash #!/usr/bin/python + Fabric #!/bin/bash #!/usr/bin/python + Fabric
26. ANSIBLE FOR CONFIGURATION
27. CONFIGURATION IN PLAINTEXT (YAML) - name: Install Nginx apt: pkg=nginx state=present
28. STRONG COMMUNITY Started in 2012, has over 423 forks, 1,429 watcher, 2000+ tickets solved, weekly clearance, active IRC & mailing list
29. PYTHON Core is in Python. Much smaller core than other competitors.
30. 5 REQUIRED DEPENDENCIES Python for running Python Simplejson/json for results Jinja2 for template Pyyaml for configuration Paramiko for ssh2 protocol
31. #TODO We will configure a machine for repoapi live. O bootstrap machine with dev pkgs O setup nginx O setup database O clone and install master repo O run server
32. Getting Started • Install AnVsible http://ansible.cc/docs/ $ sudo pip install pyyaml jinja2 paramiko $ sudo pip install pyyaml jinja2 paramiko $ git clone git://github.com/ansible/ansible.git $ git clone git://github.com/ansible/ansible.git $ cd ansible $ cd ansible $ sudo make install $ sudo make install
33. Modules • Ansible community has a lot of modules that you can used to configure a server. Modules could be in ruby, python, perl, etc. Just executable. http://ansible.cc/docs/modules.html apt, yum, pip, gem, homebrew, user apt, yum, pip, gem, homebrew, user file, copy, template, django_manage file, copy, template, django_manage postgres_user, postgres_db, mysql*, mongodb*, rabbitmq*, Riak postgres_user, postgres_db, mysql*, mongodb*, rabbitmq*, Riak ec2, ec2_facts, s3, ec2, ec2_facts, s3, hg, git, svn hg, git, svn many more... many more...
34. Hosts / Inventory An inventory file contains a list of machine addresses. http://ansible.cc/docs/patterns.html 192.168.1.4 192.168.1.4 [webservers] [webservers] foo.example.com foo.example.com bar.example.com bar.example.com www[01:50].example.com www[01:50].example.com
35. Ansible Playbook --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # You can use variables during configuration! # You can use variables during configuration! vars: vars: - APP_PORT: 8080 - APP_PORT: 8080 tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg=nginx apt: pkg=nginx
36. Template for Play - name: Description of the task/play - name: Description of the task/play module_name: arguments option1=value1 option2=value2 module_name: arguments option1=value1 option2=value2
37. apt
38. apt get install nginx? --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True tasks: tasks: - name: Ensure Nginx is installed apt: pkg=nginx
39. apt-get remove nginx? --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True tasks: tasks: - name: Ensure Nginx is removed apt: pkg=nginx state=absent
40. Idempotence Idempotence means for the same configuration settings, f(f(x)) = f(x) f(f(x)) = f(x) HTTP's GET, HEAD, PUT, DELETE, OPTIONS, TRACE satisfy idempotence.
41. TEMPLATE, VARIABLE, FACTS
42. Variable --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # You can use variables during configuration! # You can use variables during configuration! vars: - APP_PORT: 6543 tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg=nginx apt: pkg=nginx
43. Variable --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # You can use variables during configuration! # You can use variables during configuration! vars: - NGINX_NAME: nginx tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg= apt: pkg=$NGINX_NAME $NGINX_NAME
44. Variable --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # You can use variables during configuration! # You can use variables during configuration! vars: - NGINX_NAME: nginx tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg= apt: pkg=$NGINX_NAME $NGINX_NAME
45. Template + Variable --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # You can use variables during configuration! # You can use variables during configuration! vars: - HELLO_WORLD: “Hi, World” tasks: tasks: - name: Configure a foo.j2 - name: Configure a foo.j2 template: src=foo.j2 dest=/home/vagrant/foo.txt # call this foo.j2 - a Jinja template file # call this foo.j2 - a Jinja template file {{ {{ HELLO_WORLD }} }} # result # result $ cat foo.txt $ cat foo.txt Hi, World Hi, World
46. Gather Facts Ansible can gather basic facts about the target machine. By default, this feature is on. If you don't need it, set gather_facts: no gather_facts: no to speed up configuration. --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True gather_facts: no gather_facts: no … …. .
47. Gather Facts ansible -m setup 127.0.0.1 -u vagrant -k ansible -m setup 127.0.0.1 -u vagrant -k 127.0.0.1 | success >> { 127.0.0.1 | success >> { "ansible_facts": { "ansible_facts": { …. …. "ansible_lsb": { "ansible_lsb": { "codename": "precise", "codename": "precise", "description": "Ubuntu 12.04.1 LTS", "description": "Ubuntu 12.04.1 LTS", "id": "Ubuntu", "id": "Ubuntu", "major_release": "12", "major_release": "12", "release": "12.04" "release": "12.04" }, }, "ansible_pkg_mgr": "apt", "ansible_pkg_mgr": "apt", "ansible_processor": [ "ansible_processor": [ "Intel(R) Pentium(R) 4 CPU 3.20GHz" "Intel(R) Pentium(R) 4 CPU 3.20GHz" ], ], "ansible_processor_cores": 1, "ansible_processor_cores": 1, "ansible_processor_count": "ansible_processor_count": 1 1 "ansible_python_version": "2.7.3", "ansible_python_version": "2.7.3", …. …. } } see https://gist.github.com/yeukhon/4689592 see https://gist.github.com/yeukhon/4689592
48. Templating Nginx nginx.conf and sites-available/default require configuration. o worker counts o application port and proxy settings o SSL (although we won't do it here) o many more
49. nginx.conf.j2 # nginx.conf.j2 # nginx.conf.j2 user www-data; user www-data; worker_processes worker_processes {{ WORKER_COUNT }} {{ WORKER_COUNT }}; ; pid /var/run/nginx.pid; pid /var/run/nginx.pid; events { events { worker_connections worker_connections {{ WORKER_COUNT * 1024 }} {{ WORKER_COUNT * 1024 }}; ; # multi_accept on; # multi_accept on; } }
50. Configure nginx.conf.j2 --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True vars: - WORKER_COUNT: ${ansible_processor_count} tasks: tasks: - name: Configure nginx.conf - name: Configure nginx.conf template: src=nginx.conf.j2 path=/etc/nginx/nginx.conf
51. Var files --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # vars: # - WORKER_COUNT: ${ansible_processor_count} var_files: var_files: - nginx_vars.yml - nginx_vars.yml tasks: tasks: - name: Configure nginx.conf - name: Configure nginx.conf template: src=nginx.conf.j2 path=/etc/nginx/nginx.conf # nginx_vars.yml # nginx_vars.yml WORKER_COUNT: ${ansible_processor_count}
52. DRY? LOOP! WITH_ITEMS
53. with_items - name: Install Debian pkgs - name: Install Debian pkgs apt: pkg= apt: pkg=$item $item with_items: with_items: - build-essential - build-essential - python-dev - python-dev - python-pip - python-pip - python-setuptools - python-setuptools - name: Pip install Python tools - name: Pip install Python tools pip: name= pip: name=$item $item with_items: with_items: - distribute - distribute - virtualenv - virtualenv
54. with_items
55. A FEW MORE MODULES....
56. hg vars: vars: - URL: - URL: https://bitbucket.org/ideapiteam/repo-api-master https://bitbucket.org/ideapiteam/repo-api-master tasks: tasks: - name: Clone Master - name: Clone Master hg: repo=$URL dest=/home/vagrant/repoapi hg: repo=$URL dest=/home/vagrant/repoapi
57. service tasks: tasks: - name: Ensure Nginx has started - name: Ensure Nginx has started service: name=nginx state=started service: name=nginx state=started # started, restarted, stopped
58. file tasks: tasks: - name: Ensure /home/vagrant/repoapi exists - name: Ensure /home/vagrant/repoapi exists file: path=/home/vagrant/repoapi state=directory file: path=/home/vagrant/repoapi state=directory # state=directory/file/link
59. user tasks: tasks: - name: Ensure user vagrant actually exist - name: Ensure user vagrant actually exist user: name=vagrant uid=1001 state=present user: name=vagrant uid=1001 state=present
60. postgres_* tasks: tasks: - name: Create a Postgres database - name: Create a Postgres database postgres_db: db=mydb owner=user123 login_password: $md5_pwd postgres_db: db=mydb owner=user123 login_password: $md5_pwd - name: Create a Postgres user - name: Create a Postgres user postgres_user: user=user123 password=$hash_pwd priv=ALL postgres_user: user=user123 password=$hash_pwd priv=ALL
61. ROLES? TAG
62. Role You can have a cluster of webservice and dbserver. You can narrow down to a cluster of mongo instances, project1 webservice, project2 webservice, etc. Group them inside inventory.
63. Inventory groups (roles) [webservers] [webservers] 192.168.1.1 192.168.1.1 192.168.1.2 192.168.1.2 [mongoservers] 192.168.1.4 192.168.1.4 192.168.1.5 192.168.1.5 192.168.1.7 192.168.1.7 [all] [all] Webservers Webservers mongoservers mongoservers
64. Inventory name can be role # production # production [webservers] [webservers] www[01:20].example.com www[01:20].example.com [mongoservers] db[01:20].example.com db[01:20].example.com [all] [all] webservers webservers mongoservers mongoservers # development # development [webservers] [webservers] 192.168.1.1 192.168.1.1 192.168.1.2 192.168.1.2 [mongoservers] 192.168.1.4 192.168.1.4 192.168.1.5 192.168.1.5 192.168.1.7 192.168.1.7 [all] [all] webservers webservers mongoservers mongoservers ansible-playbook -i ansible-playbook -i development setup.yml setup.yml ansible-playbook -i ansible-playbook -i production setup.yml setup.yml
65. Slow? --- --- # setup.yml # setup.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg=nginx apt: pkg=nginx - name: Update web service - name: Update web service hg: repo=$URL dest=/home/vagrant/repoapi hg: repo=$URL dest=/home/vagrant/repoapi … … more tasks x 100 ... more tasks x 100 ...
66. Tags? --- --- # setup.yml # setup.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True tags: webserver tags: webserver tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg=nginx apt: pkg=nginx tags: tags: - nginx - nginx - name: Update web service - name: Update web service hg: repo=$URL dest=/home/vagrant/repoapi hg: repo=$URL dest=/home/vagrant/repoapi tags: tags: - repo - repo ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml --tags webserver --tags webserver ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml --tags nginx --tags nginx ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml --tags repo --tags repo ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml --tags nginx,repo --tags nginx,repo
67. BEST PRACTICES? Use whatever works for you.
68. BEST PRACTICES? http://ansible.cc/docs/patterns.html (inventory, tag, role by inventory, role by tag, role by file name, etc)
69. PULL? To do full automated deploymet, you probably want to do pull rather than push. Ansible can do that too. http://jpmens.net/2012/07/14/ansible-pull-instead-of-push/
70. BUILD CUSTOM BOX Ansible is great for configuration. Vagrant is great for manging development vm. But building a base box to begin with? Use veewee.
71. ANSIBLE HAS VAGRANT MODULES I haven't tried it, but in theory you could configure a bunch of vagrant instances using ansible and vagrant module!
72. DEMO TIME!
73. Questions? @yeukhon yeukhon.bitbucket.org speakerdeck.com/yeukhon
74. IRC: #ansible github.com/ansible/ansible groups.google.com/group/ansible-project/
75. Ansible IRC: #ansible github.com/ansible/ansible groups.google.com/group/ansible-project/
76. :p Thank you!