Deploying with Vagrant and Ansible

April 17, 2013
Adv Tps Intrnet Prog: CSC I4330 3GG [2337] (CCNY )

Transcript

1. DEPLOYING WITH
   VAGRANT AND ANSIBLE
   YEUK HON WONG @yeukhon
   GLASS Lab, CUNY City
   

   View Slide

2. REPO
   https://bitbucket.org/ideapiteam/repo-api-master
   https://bitbucket.org/yeukhon/vagrant-ansible-example
   

   View Slide

3. GLASS LAB:
   Developing Graphyte and Aurum.
   Serving ~100 undergradutes.
   

   View Slide

4. DEPLOYMENT:
   Deploying web service is just a subset
   of a bigger picture.
   

   View Slide

5. BIGGER PICTURE:
   “Obama campaign AWS infrastructure in one
   diagram” (http://awsofa.info/)
   

   View Slide

6. Some Ideal Work-flow
   http://goo.gl/pJtK7
   

   View Slide

7. DEPLOY #TODO:
   O Development deployment
   O Testing deployment
   O Production deployment
   

   View Slide

8. GET A MACHINE.
   Virtualbox is nice.
   

   View Slide

9. VAGRANT
   FOR DEVELOPMENT
   

   View Slide

10. Getting Started
    ●
    Install Virtualbox
    https://www.virtualbox.org/wiki/Downloads
    ●
    Install Vagrant
    http://docs.vagrantup.com/v2/installation
    

    View Slide

11. Vagrant Box
    ●
    A Vagrant box is a packaged virtual machine.
    If you extract it you get a vagrant file, ovf and
    and a vmdk file.
    ●
    Vagrant has official boxes for download
    (distribute with a universal SSH key, so do not
    use it in production)
    

    View Slide

12. Official Boxes
    Ubuntu Lucid 32 Bit
    http://files.vagrantup.com/lucid32.box
    Ubuntu Lucid 64 Bit
    http://files.vagrantup.com/lucid64.box
    Ubuntu Precise 32 Bit
    http://files.vagrantup.com/precise32.box
    Ubuntu Precise 64 Bit
    http://files.vagrantup.com/precise64.box
    

    View Slide

13. Community Boxes
    http://www.vagrantbox.es/
    

    View Slide

14. Step 1: Create a box
    1. Download a box
    2. Initialize a Vagrant workspace
    $ mkdir vagrant-example
    $ cd vagrant-example
    $ vagrant init precise32
    $ vagrant box add precise32 http://files.vagrantup.com/precise32.box
    

    View Slide

15. Step 2: Vagrantfile
    Vagrantfile
    Vagrantfile is a ruby configuration file used to
    configure a vagrant instance.
    Vagrant.configure("2") do |config|
    config.vm.box = "precise32"
    config.vm.network :public_network
    config.vm.provider :virtualbox do |vb|
    vb.customize ["modifyvm", :id, "--memory", "400"]
    end
    end
    

    View Slide

16. Step 3: vagrant up
    vagrant up
    vagrant up will create a new VM or boot up an existing
    Vagrant virtual machine.
    $ vagrant up
    $ vagrant up
    Bringing machine 'default' up with 'virtualbox' provider...
    Bringing machine 'default' up with 'virtualbox' provider...
    [default] Importing base box 'precise32'...
    [default] Importing base box 'precise32'...
    

    View Slide

17. Step 3: vagrant ssh
    vagrant ssh
    vagrant ssh will allow you to automatically ssh into
    your vagrant instance if you are using an official box.
    $ vagrant ssh
    $ vagrant ssh
    Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic-pae i686)
    Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic-pae i686)
    * Documentation: https://help.ubuntu.com/
    * Documentation: https://help.ubuntu.com/
    Welcome to your Vagrant-built virtual machine.
    Welcome to your Vagrant-built virtual machine.
    Last login: Fri Sep 14 06:22:31 2012 from 10.0.2.2
    Last login: Fri Sep 14 06:22:31 2012 from 10.0.2.2
    

    View Slide

18. Vagrant Commands
    Usage: vagrant [-v] [-h] command []
    Usage: vagrant [-v] [-h] command []
    -v, --version Print the version and exit.
    -v, --version Print the version and exit.
    -h, --help Print this help.
    -h, --help Print this help.
    Available subcommands:
    Available subcommands:
    box
    box
    destroy
    destroy
    halt
    halt
    init
    init
    package
    package
    plugin
    plugin
    provision
    provision
    reload
    reload
    resume
    resume
    ssh
    ssh
    ssh-config
    ssh-config
    status
    status
    suspend
    suspend
    up
    up
    For help on any individual command run `vagrant COMMAND -h`
    For help on any individual command run `vagrant COMMAND -h`
    

    View Slide

19. I don't want an official box
    You can use any ISO local or over the wire.
    $ mkdir vagrant-example
    $ mkdir vagrant-example
    $ cd vagrant-example
    $ cd vagrant-example
    $ vagrant init
    $ vagrant init
    $ vim Vagrantfile
    config.vm.box_url = "http://domain.com/path/to/above.box"
    

    View Slide

20. I want my own SSH key
    Even for vagrant official boxes, you should consider
    changing to your own SSH key.
    

    View Slide

21. DEMO?
    Later!
    

    View Slide

22. CONFIGURE MACHINES
    Human, please.
    

    View Slide

23. Treat your infrastructure like
    your software.
    O Version control your configuration and specs
    O Readable
    O Automated
    

    View Slide

24. REQUIREMENTS:
    O Strong community support
    O Medium to little entry to barrier (installation,
    tutorial, customizability)
    O Readable configuration
    O Version control
    O As lightweight as possible
    

    View Slide

25. So many, which to use?
    #!/bin/bash #!/usr/bin/python + Fabric
    #!/bin/bash #!/usr/bin/python + Fabric
    

    View Slide

26. ANSIBLE
    FOR CONFIGURATION
    

    View Slide

27. CONFIGURATION IN
    PLAINTEXT (YAML)
    - name: Install Nginx
    apt: pkg=nginx state=present
    

    View Slide

28. STRONG COMMUNITY
    Started in 2012, has over 423 forks, 1,429 watcher,
    2000+ tickets solved, weekly clearance, active IRC &
    mailing list
    

    View Slide

29. PYTHON
    Core is in Python. Much smaller core than other
    competitors.
    

    View Slide

30. 5 REQUIRED DEPENDENCIES
    Python for running Python
    Simplejson/json for results
    Jinja2 for template
    Pyyaml for configuration
    Paramiko for ssh2 protocol
    

    View Slide

31. #TODO
    We will configure a machine for repoapi live.
    O
    bootstrap machine with dev pkgs
    O
    setup nginx
    O
    setup database
    O
    clone and install master repo
    O
    run server
    

    View Slide

32. Getting Started
    ●
    Install AnVsible
    http://ansible.cc/docs/
    $ sudo pip install pyyaml jinja2 paramiko
    $ sudo pip install pyyaml jinja2 paramiko
    $ git clone git://github.com/ansible/ansible.git
    $ git clone git://github.com/ansible/ansible.git
    $ cd ansible
    $ cd ansible
    $ sudo make install
    $ sudo make install
    

    View Slide

33. Modules
    ●
    Ansible community has a lot of modules that you can used
    to configure a server. Modules could be in ruby, python,
    perl, etc. Just executable.
    http://ansible.cc/docs/modules.html
    apt, yum, pip, gem, homebrew, user
    apt, yum, pip, gem, homebrew, user
    file, copy, template, django_manage
    file, copy, template, django_manage
    postgres_user, postgres_db, mysql*, mongodb*, rabbitmq*, Riak
    postgres_user, postgres_db, mysql*, mongodb*, rabbitmq*, Riak
    ec2, ec2_facts, s3,
    ec2, ec2_facts, s3,
    hg, git, svn
    hg, git, svn
    many more...
    many more...
    

    View Slide

34. Hosts / Inventory
    An inventory file contains a list of machine
    addresses.
    http://ansible.cc/docs/patterns.html
    192.168.1.4
    192.168.1.4
    [webservers]
    [webservers]
    foo.example.com
    foo.example.com
    bar.example.com
    bar.example.com
    www[01:50].example.com
    www[01:50].example.com
    

    View Slide

35. Ansible Playbook
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # You can use variables during configuration!
    # You can use variables during configuration!
    vars:
    vars:
    - APP_PORT: 8080
    - APP_PORT: 8080
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=nginx
    apt: pkg=nginx
    

    View Slide

36. Template for Play
    - name: Description of the task/play
    - name: Description of the task/play
    module_name: arguments option1=value1 option2=value2
    module_name: arguments option1=value1 option2=value2
    

    View Slide

37. apt
    

    View Slide

38. apt get install nginx?
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    apt: pkg=nginx
    

    View Slide

39. apt-get remove nginx?
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    tasks:
    tasks:
    - name: Ensure Nginx is removed
    apt: pkg=nginx state=absent
    

    View Slide

40. Idempotence
    Idempotence means for the same configuration
    settings,
    f(f(x)) = f(x)
    f(f(x)) = f(x)
    HTTP's GET, HEAD, PUT, DELETE, OPTIONS, TRACE
    satisfy idempotence.
    

    View Slide

41. TEMPLATE,
    VARIABLE,
    FACTS
    

    View Slide

42. Variable
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # You can use variables during configuration!
    # You can use variables during configuration!
    vars:
    - APP_PORT: 6543
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=nginx
    apt: pkg=nginx
    

    View Slide

43. Variable
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # You can use variables during configuration!
    # You can use variables during configuration!
    vars:
    - NGINX_NAME: nginx
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=
    apt: pkg=$NGINX_NAME
    $NGINX_NAME
    

    View Slide

44. Variable
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # You can use variables during configuration!
    # You can use variables during configuration!
    vars:
    - NGINX_NAME: nginx
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=
    apt: pkg=$NGINX_NAME
    $NGINX_NAME
    

    View Slide

45. Template + Variable
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # You can use variables during configuration!
    # You can use variables during configuration!
    vars:
    - HELLO_WORLD: “Hi, World”
    tasks:
    tasks:
    - name: Configure a foo.j2
    - name: Configure a foo.j2
    template: src=foo.j2 dest=/home/vagrant/foo.txt
    # call this foo.j2 - a Jinja template file
    # call this foo.j2 - a Jinja template file
    {{
    {{ HELLO_WORLD }}
    }}
    # result
    # result
    $ cat foo.txt
    $ cat foo.txt
    Hi, World
    Hi, World
    

    View Slide

46. Gather Facts
    Ansible can gather basic facts about the target
    machine. By default, this feature is on. If you don't
    need it, set gather_facts: no
    gather_facts: no to speed up configuration.
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    gather_facts: no
    gather_facts: no
    …
    ….
    .
    

    View Slide

47. Gather Facts
    ansible -m setup 127.0.0.1 -u vagrant -k
    ansible -m setup 127.0.0.1 -u vagrant -k
    127.0.0.1 | success >> {
    127.0.0.1 | success >> {
    "ansible_facts": {
    "ansible_facts": {
    ….
    ….
    "ansible_lsb": {
    "ansible_lsb": {
    "codename": "precise",
    "codename": "precise",
    "description": "Ubuntu 12.04.1 LTS",
    "description": "Ubuntu 12.04.1 LTS",
    "id": "Ubuntu",
    "id": "Ubuntu",
    "major_release": "12",
    "major_release": "12",
    "release": "12.04"
    "release": "12.04"
    },
    },
    "ansible_pkg_mgr": "apt",
    "ansible_pkg_mgr": "apt",
    "ansible_processor": [
    "ansible_processor": [
    "Intel(R) Pentium(R) 4 CPU 3.20GHz"
    "Intel(R) Pentium(R) 4 CPU 3.20GHz"
    ],
    ],
    "ansible_processor_cores": 1,
    "ansible_processor_cores": 1,
    "ansible_processor_count":
    "ansible_processor_count": 1
    1
    "ansible_python_version": "2.7.3",
    "ansible_python_version": "2.7.3",
    ….
    ….
    }
    }
    see https://gist.github.com/yeukhon/4689592
    see https://gist.github.com/yeukhon/4689592
    

    View Slide

48. Templating Nginx
    nginx.conf and sites-available/default require
    configuration.
    o worker counts
    o application port and proxy settings
    o SSL (although we won't do it here)
    o many more
    

    View Slide

49. nginx.conf.j2
    # nginx.conf.j2
    # nginx.conf.j2
    user www-data;
    user www-data;
    worker_processes
    worker_processes {{ WORKER_COUNT }}
    {{ WORKER_COUNT }};
    ;
    pid /var/run/nginx.pid;
    pid /var/run/nginx.pid;
    events {
    events {
    worker_connections
    worker_connections {{ WORKER_COUNT * 1024 }}
    {{ WORKER_COUNT * 1024 }};
    ;
    # multi_accept on;
    # multi_accept on;
    }
    }
    

    View Slide

50. Configure nginx.conf.j2
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    vars:
    - WORKER_COUNT: ${ansible_processor_count}
    tasks:
    tasks:
    - name: Configure nginx.conf
    - name: Configure nginx.conf
    template: src=nginx.conf.j2 path=/etc/nginx/nginx.conf
    

    View Slide

51. Var files
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # vars:
    # - WORKER_COUNT: ${ansible_processor_count}
    var_files:
    var_files:
    - nginx_vars.yml
    - nginx_vars.yml
    tasks:
    tasks:
    - name: Configure nginx.conf
    - name: Configure nginx.conf
    template: src=nginx.conf.j2 path=/etc/nginx/nginx.conf
    # nginx_vars.yml
    # nginx_vars.yml
    WORKER_COUNT: ${ansible_processor_count}
    

    View Slide

52. DRY? LOOP!
    WITH_ITEMS
    

    View Slide

53. with_items
    - name: Install Debian pkgs
    - name: Install Debian pkgs
    apt: pkg=
    apt: pkg=$item
    $item
    with_items:
    with_items:
    - build-essential
    - build-essential
    - python-dev
    - python-dev
    - python-pip
    - python-pip
    - python-setuptools
    - python-setuptools
    - name: Pip install Python tools
    - name: Pip install Python tools
    pip: name=
    pip: name=$item
    $item
    with_items:
    with_items:
    - distribute
    - distribute
    - virtualenv
    - virtualenv
    

    View Slide

54. with_items
    

    View Slide

55. A FEW MORE
    MODULES....
    

    View Slide

56. hg
    vars:
    vars:
    - URL:
    - URL: https://bitbucket.org/ideapiteam/repo-api-master
    https://bitbucket.org/ideapiteam/repo-api-master
    tasks:
    tasks:
    - name: Clone Master
    - name: Clone Master
    hg: repo=$URL dest=/home/vagrant/repoapi
    hg: repo=$URL dest=/home/vagrant/repoapi
    

    View Slide

57. service
    tasks:
    tasks:
    - name: Ensure Nginx has started
    - name: Ensure Nginx has started
    service: name=nginx state=started
    service: name=nginx state=started
    # started, restarted, stopped
    

    View Slide

58. file
    tasks:
    tasks:
    - name: Ensure /home/vagrant/repoapi exists
    - name: Ensure /home/vagrant/repoapi exists
    file: path=/home/vagrant/repoapi state=directory
    file: path=/home/vagrant/repoapi state=directory
    # state=directory/file/link
    

    View Slide

59. user
    tasks:
    tasks:
    - name: Ensure user vagrant actually exist
    - name: Ensure user vagrant actually exist
    user: name=vagrant uid=1001 state=present
    user: name=vagrant uid=1001 state=present
    

    View Slide

60. postgres_*
    tasks:
    tasks:
    - name: Create a Postgres database
    - name: Create a Postgres database
    postgres_db: db=mydb owner=user123 login_password: $md5_pwd
    postgres_db: db=mydb owner=user123 login_password: $md5_pwd
    - name: Create a Postgres user
    - name: Create a Postgres user
    postgres_user: user=user123 password=$hash_pwd priv=ALL
    postgres_user: user=user123 password=$hash_pwd priv=ALL
    

    View Slide

61. ROLES?
    TAG
    

    View Slide

62. Role
    You can have a cluster of webservice and
    dbserver. You can narrow down to a cluster of
    mongo instances, project1 webservice,
    project2 webservice, etc.
    Group them inside inventory.
    

    View Slide

63. Inventory groups (roles)
    [webservers]
    [webservers]
    192.168.1.1
    192.168.1.1
    192.168.1.2
    192.168.1.2
    [mongoservers]
    192.168.1.4
    192.168.1.4
    192.168.1.5
    192.168.1.5
    192.168.1.7
    192.168.1.7
    [all]
    [all]
    Webservers
    Webservers
    mongoservers
    mongoservers
    

    View Slide

64. Inventory name can be role
    # production
    # production
    [webservers]
    [webservers]
    www[01:20].example.com
    www[01:20].example.com
    [mongoservers]
    db[01:20].example.com
    db[01:20].example.com
    [all]
    [all]
    webservers
    webservers
    mongoservers
    mongoservers
    # development
    # development
    [webservers]
    [webservers]
    192.168.1.1
    192.168.1.1
    192.168.1.2
    192.168.1.2
    [mongoservers]
    192.168.1.4
    192.168.1.4
    192.168.1.5
    192.168.1.5
    192.168.1.7
    192.168.1.7
    [all]
    [all]
    webservers
    webservers
    mongoservers
    mongoservers
    ansible-playbook -i
    ansible-playbook -i development setup.yml
    setup.yml ansible-playbook -i
    ansible-playbook -i production setup.yml
    setup.yml
    

    View Slide

65. Slow?
    ---
    ---
    # setup.yml
    # setup.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=nginx
    apt: pkg=nginx
    - name: Update web service
    - name: Update web service
    hg: repo=$URL dest=/home/vagrant/repoapi
    hg: repo=$URL dest=/home/vagrant/repoapi
    …
    … more tasks x 100 ...
    more tasks x 100 ...
    

    View Slide

66. Tags?
    ---
    ---
    # setup.yml
    # setup.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    tags: webserver
    tags: webserver
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=nginx
    apt: pkg=nginx
    tags:
    tags:
    - nginx
    - nginx
    - name: Update web service
    - name: Update web service
    hg: repo=$URL dest=/home/vagrant/repoapi
    hg: repo=$URL dest=/home/vagrant/repoapi
    tags:
    tags:
    - repo
    - repo
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml --tags webserver
    --tags webserver
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml --tags nginx
    --tags nginx
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml --tags repo
    --tags repo
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml --tags nginx,repo
    --tags nginx,repo
    

    View Slide

67. BEST PRACTICES?
    Use whatever works for you.
    

    View Slide

68. BEST PRACTICES?
    http://ansible.cc/docs/patterns.html
    (inventory, tag, role by inventory, role by tag, role by file
    name, etc)
    

    View Slide

69. PULL?
    To do full automated deploymet, you probably
    want to do pull rather than push. Ansible can do
    that too.
    http://jpmens.net/2012/07/14/ansible-pull-instead-of-push/
    

    View Slide

70. BUILD CUSTOM BOX
    Ansible is great for configuration. Vagrant is
    great for manging development vm. But
    building a base box to begin with? Use veewee.
    

    View Slide

71. ANSIBLE HAS VAGRANT
    MODULES
    I haven't tried it, but in theory you could
    configure a bunch of vagrant instances using
    ansible and vagrant module!
    

    View Slide

72. DEMO TIME!
    

    View Slide

73. Questions?
    @yeukhon
    yeukhon.bitbucket.org
    speakerdeck.com/yeukhon
    

    View Slide

74. IRC: #ansible
    github.com/ansible/ansible
    groups.google.com/group/ansible-project/
    

    View Slide

75. Ansible
    IRC: #ansible
    github.com/ansible/ansible
    groups.google.com/group/ansible-project/
    

    View Slide

76. :p
    Thank you!
    

    View Slide