Deploying with Vagrant and Ansible

Transcript

1. DEPLOYING WITH VAGRANT AND ANSIBLE YEUK HON WONG @yeukhon GLASS

   Lab, CUNY City

2. REPO https://bitbucket.org/ideapiteam/repo-api-master https://bitbucket.org/yeukhon/vagrant-ansible-example

3. GLASS LAB: Developing Graphyte and Aurum. Serving ~100 undergradutes.

4. DEPLOYMENT: Deploying web service is just a subset of a

   bigger picture.

5. BIGGER PICTURE: “Obama campaign AWS infrastructure in one diagram” (http://awsofa.info/)

6. Some Ideal Work-flow http://goo.gl/pJtK7

7. DEPLOY #TODO: O Development deployment O Testing deployment O Production

   deployment

8. GET A MACHINE. Virtualbox is nice.

9. VAGRANT FOR DEVELOPMENT

10. Getting Started • Install Virtualbox https://www.virtualbox.org/wiki/Downloads • Install Vagrant http://docs.vagrantup.com/v2/installation

11. Vagrant Box • A Vagrant box is a packaged virtual

    machine. If you extract it you get a vagrant file, ovf and and a vmdk file. • Vagrant has official boxes for download (distribute with a universal SSH key, so do not use it in production)

12. Official Boxes Ubuntu Lucid 32 Bit http://files.vagrantup.com/lucid32.box Ubuntu Lucid 64

    Bit http://files.vagrantup.com/lucid64.box Ubuntu Precise 32 Bit http://files.vagrantup.com/precise32.box Ubuntu Precise 64 Bit http://files.vagrantup.com/precise64.box

13. Community Boxes http://www.vagrantbox.es/

14. Step 1: Create a box 1. Download a box 2.

    Initialize a Vagrant workspace $ mkdir vagrant-example $ cd vagrant-example $ vagrant init precise32 $ vagrant box add precise32 http://files.vagrantup.com/precise32.box

15. Step 2: Vagrantfile Vagrantfile Vagrantfile is a ruby configuration file

    used to configure a vagrant instance. Vagrant.configure("2") do |config| config.vm.box = "precise32" config.vm.network :public_network config.vm.provider :virtualbox do |vb| vb.customize ["modifyvm", :id, "--memory", "400"] end end

16. Step 3: vagrant up vagrant up vagrant up will create

    a new VM or boot up an existing Vagrant virtual machine. $ vagrant up $ vagrant up Bringing machine 'default' up with 'virtualbox' provider... Bringing machine 'default' up with 'virtualbox' provider... [default] Importing base box 'precise32'... [default] Importing base box 'precise32'...

17. Step 3: vagrant ssh vagrant ssh vagrant ssh will allow

    you to automatically ssh into your vagrant instance if you are using an official box. $ vagrant ssh $ vagrant ssh Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic-pae i686) Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic-pae i686) * Documentation: https://help.ubuntu.com/ * Documentation: https://help.ubuntu.com/ Welcome to your Vagrant-built virtual machine. Welcome to your Vagrant-built virtual machine. Last login: Fri Sep 14 06:22:31 2012 from 10.0.2.2 Last login: Fri Sep 14 06:22:31 2012 from 10.0.2.2

18. Vagrant Commands Usage: vagrant [-v] [-h] command [<args>] Usage: vagrant

    [-v] [-h] command [<args>] -v, --version Print the version and exit. -v, --version Print the version and exit. -h, --help Print this help. -h, --help Print this help. Available subcommands: Available subcommands: box box destroy destroy halt halt init init package package plugin plugin provision provision reload reload resume resume ssh ssh ssh-config ssh-config status status suspend suspend up up For help on any individual command run `vagrant COMMAND -h` For help on any individual command run `vagrant COMMAND -h`

19. I don't want an official box You can use any

    ISO local or over the wire. $ mkdir vagrant-example $ mkdir vagrant-example $ cd vagrant-example $ cd vagrant-example $ vagrant init $ vagrant init $ vim Vagrantfile config.vm.box_url = "http://domain.com/path/to/above.box"

20. I want my own SSH key Even for vagrant official

    boxes, you should consider changing to your own SSH key.

21. DEMO? Later!

22. CONFIGURE MACHINES Human, please.

23. Treat your infrastructure like your software. O Version control your

    configuration and specs O Readable O Automated

24. REQUIREMENTS: O Strong community support O Medium to little entry

    to barrier (installation, tutorial, customizability) O Readable configuration O Version control O As lightweight as possible

25. So many, which to use? #!/bin/bash #!/usr/bin/python + Fabric #!/bin/bash

    #!/usr/bin/python + Fabric

26. ANSIBLE FOR CONFIGURATION

27. CONFIGURATION IN PLAINTEXT (YAML) - name: Install Nginx apt: pkg=nginx

    state=present

28. STRONG COMMUNITY Started in 2012, has over 423 forks, 1,429

    watcher, 2000+ tickets solved, weekly clearance, active IRC & mailing list

29. PYTHON Core is in Python. Much smaller core than other

    competitors.

30. 5 REQUIRED DEPENDENCIES Python for running Python Simplejson/json for results

    Jinja2 for template Pyyaml for configuration Paramiko for ssh2 protocol

31. #TODO We will configure a machine for repoapi live. O

    bootstrap machine with dev pkgs O setup nginx O setup database O clone and install master repo O run server

32. Getting Started • Install AnVsible http://ansible.cc/docs/ $ sudo pip install

    pyyaml jinja2 paramiko $ sudo pip install pyyaml jinja2 paramiko $ git clone git://github.com/ansible/ansible.git $ git clone git://github.com/ansible/ansible.git $ cd ansible $ cd ansible $ sudo make install $ sudo make install

33. Modules • Ansible community has a lot of modules that

    you can used to configure a server. Modules could be in ruby, python, perl, etc. Just executable. http://ansible.cc/docs/modules.html apt, yum, pip, gem, homebrew, user apt, yum, pip, gem, homebrew, user file, copy, template, django_manage file, copy, template, django_manage postgres_user, postgres_db, mysql*, mongodb*, rabbitmq*, Riak postgres_user, postgres_db, mysql*, mongodb*, rabbitmq*, Riak ec2, ec2_facts, s3, ec2, ec2_facts, s3, hg, git, svn hg, git, svn many more... many more...

34. Hosts / Inventory An inventory file contains a list of

    machine addresses. http://ansible.cc/docs/patterns.html 192.168.1.4 192.168.1.4 [webservers] [webservers] foo.example.com foo.example.com bar.example.com bar.example.com www[01:50].example.com www[01:50].example.com

35. Ansible Playbook --- --- # nginx.yml # nginx.yml - hosts:

    dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # You can use variables during configuration! # You can use variables during configuration! vars: vars: - APP_PORT: 8080 - APP_PORT: 8080 tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg=nginx apt: pkg=nginx

36. Template for Play - name: Description of the task/play -

    name: Description of the task/play module_name: arguments option1=value1 option2=value2 module_name: arguments option1=value1 option2=value2

37. apt

38. apt get install nginx? --- --- # nginx.yml # nginx.yml

    - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True tasks: tasks: - name: Ensure Nginx is installed apt: pkg=nginx

39. apt-get remove nginx? --- --- # nginx.yml # nginx.yml -

    hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True tasks: tasks: - name: Ensure Nginx is removed apt: pkg=nginx state=absent

40. Idempotence Idempotence means for the same configuration settings, f(f(x)) =

    f(x) f(f(x)) = f(x) HTTP's GET, HEAD, PUT, DELETE, OPTIONS, TRACE satisfy idempotence.

41. TEMPLATE, VARIABLE, FACTS

42. Variable --- --- # nginx.yml # nginx.yml - hosts: dev-box

    - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # You can use variables during configuration! # You can use variables during configuration! vars: - APP_PORT: 6543 tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg=nginx apt: pkg=nginx

43. Variable --- --- # nginx.yml # nginx.yml - hosts: dev-box

    - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # You can use variables during configuration! # You can use variables during configuration! vars: - NGINX_NAME: nginx tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg= apt: pkg=$NGINX_NAME $NGINX_NAME

44. Variable --- --- # nginx.yml # nginx.yml - hosts: dev-box

    - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # You can use variables during configuration! # You can use variables during configuration! vars: - NGINX_NAME: nginx tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg= apt: pkg=$NGINX_NAME $NGINX_NAME

45. Template + Variable --- --- # nginx.yml # nginx.yml -

    hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # You can use variables during configuration! # You can use variables during configuration! vars: - HELLO_WORLD: “Hi, World” tasks: tasks: - name: Configure a foo.j2 - name: Configure a foo.j2 template: src=foo.j2 dest=/home/vagrant/foo.txt # call this foo.j2 - a Jinja template file # call this foo.j2 - a Jinja template file {{ {{ HELLO_WORLD }} }} # result # result $ cat foo.txt $ cat foo.txt Hi, World Hi, World

46. Gather Facts Ansible can gather basic facts about the target

    machine. By default, this feature is on. If you don't need it, set gather_facts: no gather_facts: no to speed up configuration. --- --- # nginx.yml # nginx.yml - hosts: dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True gather_facts: no gather_facts: no … …. .

47. Gather Facts ansible -m setup 127.0.0.1 -u vagrant -k ansible

    -m setup 127.0.0.1 -u vagrant -k 127.0.0.1 | success >> { 127.0.0.1 | success >> { "ansible_facts": { "ansible_facts": { …. …. "ansible_lsb": { "ansible_lsb": { "codename": "precise", "codename": "precise", "description": "Ubuntu 12.04.1 LTS", "description": "Ubuntu 12.04.1 LTS", "id": "Ubuntu", "id": "Ubuntu", "major_release": "12", "major_release": "12", "release": "12.04" "release": "12.04" }, }, "ansible_pkg_mgr": "apt", "ansible_pkg_mgr": "apt", "ansible_processor": [ "ansible_processor": [ "Intel(R) Pentium(R) 4 CPU 3.20GHz" "Intel(R) Pentium(R) 4 CPU 3.20GHz" ], ], "ansible_processor_cores": 1, "ansible_processor_cores": 1, "ansible_processor_count": "ansible_processor_count": 1 1 "ansible_python_version": "2.7.3", "ansible_python_version": "2.7.3", …. …. } } see https://gist.github.com/yeukhon/4689592 see https://gist.github.com/yeukhon/4689592

48. Templating Nginx nginx.conf and sites-available/default require configuration. o worker counts

    o application port and proxy settings o SSL (although we won't do it here) o many more

49. nginx.conf.j2 # nginx.conf.j2 # nginx.conf.j2 user www-data; user www-data; worker_processes

    worker_processes {{ WORKER_COUNT }} {{ WORKER_COUNT }}; ; pid /var/run/nginx.pid; pid /var/run/nginx.pid; events { events { worker_connections worker_connections {{ WORKER_COUNT * 1024 }} {{ WORKER_COUNT * 1024 }}; ; # multi_accept on; # multi_accept on; } }

50. Configure nginx.conf.j2 --- --- # nginx.yml # nginx.yml - hosts:

    dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True vars: - WORKER_COUNT: ${ansible_processor_count} tasks: tasks: - name: Configure nginx.conf - name: Configure nginx.conf template: src=nginx.conf.j2 path=/etc/nginx/nginx.conf

51. Var files --- --- # nginx.yml # nginx.yml - hosts:

    dev-box - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True # vars: # - WORKER_COUNT: ${ansible_processor_count} var_files: var_files: - nginx_vars.yml - nginx_vars.yml tasks: tasks: - name: Configure nginx.conf - name: Configure nginx.conf template: src=nginx.conf.j2 path=/etc/nginx/nginx.conf # nginx_vars.yml # nginx_vars.yml WORKER_COUNT: ${ansible_processor_count}

52. DRY? LOOP! WITH_ITEMS

53. with_items - name: Install Debian pkgs - name: Install Debian

    pkgs apt: pkg= apt: pkg=$item $item with_items: with_items: - build-essential - build-essential - python-dev - python-dev - python-pip - python-pip - python-setuptools - python-setuptools - name: Pip install Python tools - name: Pip install Python tools pip: name= pip: name=$item $item with_items: with_items: - distribute - distribute - virtualenv - virtualenv

54. with_items

55. A FEW MORE MODULES....

56. hg vars: vars: - URL: - URL: https://bitbucket.org/ideapiteam/repo-api-master https://bitbucket.org/ideapiteam/repo-api-master tasks:

    tasks: - name: Clone Master - name: Clone Master hg: repo=$URL dest=/home/vagrant/repoapi hg: repo=$URL dest=/home/vagrant/repoapi

57. service tasks: tasks: - name: Ensure Nginx has started -

    name: Ensure Nginx has started service: name=nginx state=started service: name=nginx state=started # started, restarted, stopped

58. file tasks: tasks: - name: Ensure /home/vagrant/repoapi exists - name:

    Ensure /home/vagrant/repoapi exists file: path=/home/vagrant/repoapi state=directory file: path=/home/vagrant/repoapi state=directory # state=directory/file/link

59. user tasks: tasks: - name: Ensure user vagrant actually exist

    - name: Ensure user vagrant actually exist user: name=vagrant uid=1001 state=present user: name=vagrant uid=1001 state=present

60. postgres_* tasks: tasks: - name: Create a Postgres database -

    name: Create a Postgres database postgres_db: db=mydb owner=user123 login_password: $md5_pwd postgres_db: db=mydb owner=user123 login_password: $md5_pwd - name: Create a Postgres user - name: Create a Postgres user postgres_user: user=user123 password=$hash_pwd priv=ALL postgres_user: user=user123 password=$hash_pwd priv=ALL

61. ROLES? TAG

62. Role You can have a cluster of webservice and dbserver.

    You can narrow down to a cluster of mongo instances, project1 webservice, project2 webservice, etc. Group them inside inventory.

63. Inventory groups (roles) [webservers] [webservers] 192.168.1.1 192.168.1.1 192.168.1.2 192.168.1.2 [mongoservers]

    192.168.1.4 192.168.1.4 192.168.1.5 192.168.1.5 192.168.1.7 192.168.1.7 [all] [all] Webservers Webservers mongoservers mongoservers

64. Inventory name can be role # production # production [webservers]

    [webservers] www[01:20].example.com www[01:20].example.com [mongoservers] db[01:20].example.com db[01:20].example.com [all] [all] webservers webservers mongoservers mongoservers # development # development [webservers] [webservers] 192.168.1.1 192.168.1.1 192.168.1.2 192.168.1.2 [mongoservers] 192.168.1.4 192.168.1.4 192.168.1.5 192.168.1.5 192.168.1.7 192.168.1.7 [all] [all] webservers webservers mongoservers mongoservers ansible-playbook -i ansible-playbook -i development setup.yml setup.yml ansible-playbook -i ansible-playbook -i production setup.yml setup.yml

65. Slow? --- --- # setup.yml # setup.yml - hosts: dev-box

    - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg=nginx apt: pkg=nginx - name: Update web service - name: Update web service hg: repo=$URL dest=/home/vagrant/repoapi hg: repo=$URL dest=/home/vagrant/repoapi … … more tasks x 100 ... more tasks x 100 ...

66. Tags? --- --- # setup.yml # setup.yml - hosts: dev-box

    - hosts: dev-box user: vagrant user: vagrant sudo: True sudo: True tags: webserver tags: webserver tasks: tasks: - name: Ensure Nginx is installed - name: Ensure Nginx is installed apt: pkg=nginx apt: pkg=nginx tags: tags: - nginx - nginx - name: Update web service - name: Update web service hg: repo=$URL dest=/home/vagrant/repoapi hg: repo=$URL dest=/home/vagrant/repoapi tags: tags: - repo - repo ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml --tags webserver --tags webserver ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml --tags nginx --tags nginx ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml --tags repo --tags repo ansible-playbook -i development setup.yml ansible-playbook -i development setup.yml --tags nginx,repo --tags nginx,repo

67. BEST PRACTICES? Use whatever works for you.

68. BEST PRACTICES? http://ansible.cc/docs/patterns.html (inventory, tag, role by inventory, role by

    tag, role by file name, etc)

69. PULL? To do full automated deploymet, you probably want to

    do pull rather than push. Ansible can do that too. http://jpmens.net/2012/07/14/ansible-pull-instead-of-push/

70. BUILD CUSTOM BOX Ansible is great for configuration. Vagrant is

    great for manging development vm. But building a base box to begin with? Use veewee.

71. ANSIBLE HAS VAGRANT MODULES I haven't tried it, but in

    theory you could configure a bunch of vagrant instances using ansible and vagrant module!

72. DEMO TIME!

73. Questions? @yeukhon yeukhon.bitbucket.org speakerdeck.com/yeukhon

74. IRC: #ansible github.com/ansible/ansible groups.google.com/group/ansible-project/

75. Ansible IRC: #ansible github.com/ansible/ansible groups.google.com/group/ansible-project/

76. :p Thank you!