Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deploying with Vagrant and Ansible

yeukhon
April 17, 2013

Deploying with Vagrant and Ansible

April 17, 2013
Adv Tps Intrnet Prog: CSC I4330 3GG [2337] (CCNY )

yeukhon

April 17, 2013
Tweet

More Decks by yeukhon

Other Decks in Programming

Transcript

  1. DEPLOYING WITH
    VAGRANT AND ANSIBLE
    YEUK HON WONG @yeukhon
    GLASS Lab, CUNY City

    View full-size slide

  2. REPO
    https://bitbucket.org/ideapiteam/repo-api-master
    https://bitbucket.org/yeukhon/vagrant-ansible-example

    View full-size slide

  3. GLASS LAB:
    Developing Graphyte and Aurum.
    Serving ~100 undergradutes.

    View full-size slide

  4. DEPLOYMENT:
    Deploying web service is just a subset
    of a bigger picture.

    View full-size slide

  5. BIGGER PICTURE:
    “Obama campaign AWS infrastructure in one
    diagram” (http://awsofa.info/)

    View full-size slide

  6. Some Ideal Work-flow
    http://goo.gl/pJtK7

    View full-size slide

  7. DEPLOY #TODO:
    O Development deployment
    O Testing deployment
    O Production deployment

    View full-size slide

  8. GET A MACHINE.
    Virtualbox is nice.

    View full-size slide

  9. VAGRANT
    FOR DEVELOPMENT

    View full-size slide

  10. Getting Started

    Install Virtualbox
    https://www.virtualbox.org/wiki/Downloads

    Install Vagrant
    http://docs.vagrantup.com/v2/installation

    View full-size slide

  11. Vagrant Box

    A Vagrant box is a packaged virtual machine.
    If you extract it you get a vagrant file, ovf and
    and a vmdk file.

    Vagrant has official boxes for download
    (distribute with a universal SSH key, so do not
    use it in production)

    View full-size slide

  12. Official Boxes
    Ubuntu Lucid 32 Bit
    http://files.vagrantup.com/lucid32.box
    Ubuntu Lucid 64 Bit
    http://files.vagrantup.com/lucid64.box
    Ubuntu Precise 32 Bit
    http://files.vagrantup.com/precise32.box
    Ubuntu Precise 64 Bit
    http://files.vagrantup.com/precise64.box

    View full-size slide

  13. Community Boxes
    http://www.vagrantbox.es/

    View full-size slide

  14. Step 1: Create a box
    1. Download a box
    2. Initialize a Vagrant workspace
    $ mkdir vagrant-example
    $ cd vagrant-example
    $ vagrant init precise32
    $ vagrant box add precise32 http://files.vagrantup.com/precise32.box

    View full-size slide

  15. Step 2: Vagrantfile
    Vagrantfile
    Vagrantfile is a ruby configuration file used to
    configure a vagrant instance.
    Vagrant.configure("2") do |config|
    config.vm.box = "precise32"
    config.vm.network :public_network
    config.vm.provider :virtualbox do |vb|
    vb.customize ["modifyvm", :id, "--memory", "400"]
    end
    end

    View full-size slide

  16. Step 3: vagrant up
    vagrant up
    vagrant up will create a new VM or boot up an existing
    Vagrant virtual machine.
    $ vagrant up
    $ vagrant up
    Bringing machine 'default' up with 'virtualbox' provider...
    Bringing machine 'default' up with 'virtualbox' provider...
    [default] Importing base box 'precise32'...
    [default] Importing base box 'precise32'...

    View full-size slide

  17. Step 3: vagrant ssh
    vagrant ssh
    vagrant ssh will allow you to automatically ssh into
    your vagrant instance if you are using an official box.
    $ vagrant ssh
    $ vagrant ssh
    Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic-pae i686)
    Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic-pae i686)
    * Documentation: https://help.ubuntu.com/
    * Documentation: https://help.ubuntu.com/
    Welcome to your Vagrant-built virtual machine.
    Welcome to your Vagrant-built virtual machine.
    Last login: Fri Sep 14 06:22:31 2012 from 10.0.2.2
    Last login: Fri Sep 14 06:22:31 2012 from 10.0.2.2

    View full-size slide

  18. Vagrant Commands
    Usage: vagrant [-v] [-h] command []
    Usage: vagrant [-v] [-h] command []
    -v, --version Print the version and exit.
    -v, --version Print the version and exit.
    -h, --help Print this help.
    -h, --help Print this help.
    Available subcommands:
    Available subcommands:
    box
    box
    destroy
    destroy
    halt
    halt
    init
    init
    package
    package
    plugin
    plugin
    provision
    provision
    reload
    reload
    resume
    resume
    ssh
    ssh
    ssh-config
    ssh-config
    status
    status
    suspend
    suspend
    up
    up
    For help on any individual command run `vagrant COMMAND -h`
    For help on any individual command run `vagrant COMMAND -h`

    View full-size slide

  19. I don't want an official box
    You can use any ISO local or over the wire.
    $ mkdir vagrant-example
    $ mkdir vagrant-example
    $ cd vagrant-example
    $ cd vagrant-example
    $ vagrant init
    $ vagrant init
    $ vim Vagrantfile
    config.vm.box_url = "http://domain.com/path/to/above.box"

    View full-size slide

  20. I want my own SSH key
    Even for vagrant official boxes, you should consider
    changing to your own SSH key.

    View full-size slide

  21. CONFIGURE MACHINES
    Human, please.

    View full-size slide

  22. Treat your infrastructure like
    your software.
    O Version control your configuration and specs
    O Readable
    O Automated

    View full-size slide

  23. REQUIREMENTS:
    O Strong community support
    O Medium to little entry to barrier (installation,
    tutorial, customizability)
    O Readable configuration
    O Version control
    O As lightweight as possible

    View full-size slide

  24. So many, which to use?
    #!/bin/bash #!/usr/bin/python + Fabric
    #!/bin/bash #!/usr/bin/python + Fabric

    View full-size slide

  25. ANSIBLE
    FOR CONFIGURATION

    View full-size slide

  26. CONFIGURATION IN
    PLAINTEXT (YAML)
    - name: Install Nginx
    apt: pkg=nginx state=present

    View full-size slide

  27. STRONG COMMUNITY
    Started in 2012, has over 423 forks, 1,429 watcher,
    2000+ tickets solved, weekly clearance, active IRC &
    mailing list

    View full-size slide

  28. PYTHON
    Core is in Python. Much smaller core than other
    competitors.

    View full-size slide

  29. 5 REQUIRED DEPENDENCIES
    Python for running Python
    Simplejson/json for results
    Jinja2 for template
    Pyyaml for configuration
    Paramiko for ssh2 protocol

    View full-size slide

  30. #TODO
    We will configure a machine for repoapi live.
    O
    bootstrap machine with dev pkgs
    O
    setup nginx
    O
    setup database
    O
    clone and install master repo
    O
    run server

    View full-size slide

  31. Getting Started

    Install AnVsible
    http://ansible.cc/docs/
    $ sudo pip install pyyaml jinja2 paramiko
    $ sudo pip install pyyaml jinja2 paramiko
    $ git clone git://github.com/ansible/ansible.git
    $ git clone git://github.com/ansible/ansible.git
    $ cd ansible
    $ cd ansible
    $ sudo make install
    $ sudo make install

    View full-size slide

  32. Modules

    Ansible community has a lot of modules that you can used
    to configure a server. Modules could be in ruby, python,
    perl, etc. Just executable.
    http://ansible.cc/docs/modules.html
    apt, yum, pip, gem, homebrew, user
    apt, yum, pip, gem, homebrew, user
    file, copy, template, django_manage
    file, copy, template, django_manage
    postgres_user, postgres_db, mysql*, mongodb*, rabbitmq*, Riak
    postgres_user, postgres_db, mysql*, mongodb*, rabbitmq*, Riak
    ec2, ec2_facts, s3,
    ec2, ec2_facts, s3,
    hg, git, svn
    hg, git, svn
    many more...
    many more...

    View full-size slide

  33. Hosts / Inventory
    An inventory file contains a list of machine
    addresses.
    http://ansible.cc/docs/patterns.html
    192.168.1.4
    192.168.1.4
    [webservers]
    [webservers]
    foo.example.com
    foo.example.com
    bar.example.com
    bar.example.com
    www[01:50].example.com
    www[01:50].example.com

    View full-size slide

  34. Ansible Playbook
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # You can use variables during configuration!
    # You can use variables during configuration!
    vars:
    vars:
    - APP_PORT: 8080
    - APP_PORT: 8080
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=nginx
    apt: pkg=nginx

    View full-size slide

  35. Template for Play
    - name: Description of the task/play
    - name: Description of the task/play
    module_name: arguments option1=value1 option2=value2
    module_name: arguments option1=value1 option2=value2

    View full-size slide

  36. apt get install nginx?
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    apt: pkg=nginx

    View full-size slide

  37. apt-get remove nginx?
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    tasks:
    tasks:
    - name: Ensure Nginx is removed
    apt: pkg=nginx state=absent

    View full-size slide

  38. Idempotence
    Idempotence means for the same configuration
    settings,
    f(f(x)) = f(x)
    f(f(x)) = f(x)
    HTTP's GET, HEAD, PUT, DELETE, OPTIONS, TRACE
    satisfy idempotence.

    View full-size slide

  39. TEMPLATE,
    VARIABLE,
    FACTS

    View full-size slide

  40. Variable
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # You can use variables during configuration!
    # You can use variables during configuration!
    vars:
    - APP_PORT: 6543
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=nginx
    apt: pkg=nginx

    View full-size slide

  41. Variable
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # You can use variables during configuration!
    # You can use variables during configuration!
    vars:
    - NGINX_NAME: nginx
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=
    apt: pkg=$NGINX_NAME
    $NGINX_NAME

    View full-size slide

  42. Variable
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # You can use variables during configuration!
    # You can use variables during configuration!
    vars:
    - NGINX_NAME: nginx
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=
    apt: pkg=$NGINX_NAME
    $NGINX_NAME

    View full-size slide

  43. Template + Variable
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # You can use variables during configuration!
    # You can use variables during configuration!
    vars:
    - HELLO_WORLD: “Hi, World”
    tasks:
    tasks:
    - name: Configure a foo.j2
    - name: Configure a foo.j2
    template: src=foo.j2 dest=/home/vagrant/foo.txt
    # call this foo.j2 - a Jinja template file
    # call this foo.j2 - a Jinja template file
    {{
    {{ HELLO_WORLD }}
    }}
    # result
    # result
    $ cat foo.txt
    $ cat foo.txt
    Hi, World
    Hi, World

    View full-size slide

  44. Gather Facts
    Ansible can gather basic facts about the target
    machine. By default, this feature is on. If you don't
    need it, set gather_facts: no
    gather_facts: no to speed up configuration.
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    gather_facts: no
    gather_facts: no

    ….
    .

    View full-size slide

  45. Gather Facts
    ansible -m setup 127.0.0.1 -u vagrant -k
    ansible -m setup 127.0.0.1 -u vagrant -k
    127.0.0.1 | success >> {
    127.0.0.1 | success >> {
    "ansible_facts": {
    "ansible_facts": {
    ….
    ….
    "ansible_lsb": {
    "ansible_lsb": {
    "codename": "precise",
    "codename": "precise",
    "description": "Ubuntu 12.04.1 LTS",
    "description": "Ubuntu 12.04.1 LTS",
    "id": "Ubuntu",
    "id": "Ubuntu",
    "major_release": "12",
    "major_release": "12",
    "release": "12.04"
    "release": "12.04"
    },
    },
    "ansible_pkg_mgr": "apt",
    "ansible_pkg_mgr": "apt",
    "ansible_processor": [
    "ansible_processor": [
    "Intel(R) Pentium(R) 4 CPU 3.20GHz"
    "Intel(R) Pentium(R) 4 CPU 3.20GHz"
    ],
    ],
    "ansible_processor_cores": 1,
    "ansible_processor_cores": 1,
    "ansible_processor_count":
    "ansible_processor_count": 1
    1
    "ansible_python_version": "2.7.3",
    "ansible_python_version": "2.7.3",
    ….
    ….
    }
    }
    see https://gist.github.com/yeukhon/4689592
    see https://gist.github.com/yeukhon/4689592

    View full-size slide

  46. Templating Nginx
    nginx.conf and sites-available/default require
    configuration.
    o worker counts
    o application port and proxy settings
    o SSL (although we won't do it here)
    o many more

    View full-size slide

  47. nginx.conf.j2
    # nginx.conf.j2
    # nginx.conf.j2
    user www-data;
    user www-data;
    worker_processes
    worker_processes {{ WORKER_COUNT }}
    {{ WORKER_COUNT }};
    ;
    pid /var/run/nginx.pid;
    pid /var/run/nginx.pid;
    events {
    events {
    worker_connections
    worker_connections {{ WORKER_COUNT * 1024 }}
    {{ WORKER_COUNT * 1024 }};
    ;
    # multi_accept on;
    # multi_accept on;
    }
    }

    View full-size slide

  48. Configure nginx.conf.j2
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    vars:
    - WORKER_COUNT: ${ansible_processor_count}
    tasks:
    tasks:
    - name: Configure nginx.conf
    - name: Configure nginx.conf
    template: src=nginx.conf.j2 path=/etc/nginx/nginx.conf

    View full-size slide

  49. Var files
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # vars:
    # - WORKER_COUNT: ${ansible_processor_count}
    var_files:
    var_files:
    - nginx_vars.yml
    - nginx_vars.yml
    tasks:
    tasks:
    - name: Configure nginx.conf
    - name: Configure nginx.conf
    template: src=nginx.conf.j2 path=/etc/nginx/nginx.conf
    # nginx_vars.yml
    # nginx_vars.yml
    WORKER_COUNT: ${ansible_processor_count}

    View full-size slide

  50. DRY? LOOP!
    WITH_ITEMS

    View full-size slide

  51. with_items
    - name: Install Debian pkgs
    - name: Install Debian pkgs
    apt: pkg=
    apt: pkg=$item
    $item
    with_items:
    with_items:
    - build-essential
    - build-essential
    - python-dev
    - python-dev
    - python-pip
    - python-pip
    - python-setuptools
    - python-setuptools
    - name: Pip install Python tools
    - name: Pip install Python tools
    pip: name=
    pip: name=$item
    $item
    with_items:
    with_items:
    - distribute
    - distribute
    - virtualenv
    - virtualenv

    View full-size slide

  52. A FEW MORE
    MODULES....

    View full-size slide

  53. hg
    vars:
    vars:
    - URL:
    - URL: https://bitbucket.org/ideapiteam/repo-api-master
    https://bitbucket.org/ideapiteam/repo-api-master
    tasks:
    tasks:
    - name: Clone Master
    - name: Clone Master
    hg: repo=$URL dest=/home/vagrant/repoapi
    hg: repo=$URL dest=/home/vagrant/repoapi

    View full-size slide

  54. service
    tasks:
    tasks:
    - name: Ensure Nginx has started
    - name: Ensure Nginx has started
    service: name=nginx state=started
    service: name=nginx state=started
    # started, restarted, stopped

    View full-size slide

  55. file
    tasks:
    tasks:
    - name: Ensure /home/vagrant/repoapi exists
    - name: Ensure /home/vagrant/repoapi exists
    file: path=/home/vagrant/repoapi state=directory
    file: path=/home/vagrant/repoapi state=directory
    # state=directory/file/link

    View full-size slide

  56. user
    tasks:
    tasks:
    - name: Ensure user vagrant actually exist
    - name: Ensure user vagrant actually exist
    user: name=vagrant uid=1001 state=present
    user: name=vagrant uid=1001 state=present

    View full-size slide

  57. postgres_*
    tasks:
    tasks:
    - name: Create a Postgres database
    - name: Create a Postgres database
    postgres_db: db=mydb owner=user123 login_password: $md5_pwd
    postgres_db: db=mydb owner=user123 login_password: $md5_pwd
    - name: Create a Postgres user
    - name: Create a Postgres user
    postgres_user: user=user123 password=$hash_pwd priv=ALL
    postgres_user: user=user123 password=$hash_pwd priv=ALL

    View full-size slide

  58. Role
    You can have a cluster of webservice and
    dbserver. You can narrow down to a cluster of
    mongo instances, project1 webservice,
    project2 webservice, etc.
    Group them inside inventory.

    View full-size slide

  59. Inventory groups (roles)
    [webservers]
    [webservers]
    192.168.1.1
    192.168.1.1
    192.168.1.2
    192.168.1.2
    [mongoservers]
    192.168.1.4
    192.168.1.4
    192.168.1.5
    192.168.1.5
    192.168.1.7
    192.168.1.7
    [all]
    [all]
    Webservers
    Webservers
    mongoservers
    mongoservers

    View full-size slide

  60. Inventory name can be role
    # production
    # production
    [webservers]
    [webservers]
    www[01:20].example.com
    www[01:20].example.com
    [mongoservers]
    db[01:20].example.com
    db[01:20].example.com
    [all]
    [all]
    webservers
    webservers
    mongoservers
    mongoservers
    # development
    # development
    [webservers]
    [webservers]
    192.168.1.1
    192.168.1.1
    192.168.1.2
    192.168.1.2
    [mongoservers]
    192.168.1.4
    192.168.1.4
    192.168.1.5
    192.168.1.5
    192.168.1.7
    192.168.1.7
    [all]
    [all]
    webservers
    webservers
    mongoservers
    mongoservers
    ansible-playbook -i
    ansible-playbook -i development setup.yml
    setup.yml ansible-playbook -i
    ansible-playbook -i production setup.yml
    setup.yml

    View full-size slide

  61. Slow?
    ---
    ---
    # setup.yml
    # setup.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=nginx
    apt: pkg=nginx
    - name: Update web service
    - name: Update web service
    hg: repo=$URL dest=/home/vagrant/repoapi
    hg: repo=$URL dest=/home/vagrant/repoapi

    … more tasks x 100 ...
    more tasks x 100 ...

    View full-size slide

  62. Tags?
    ---
    ---
    # setup.yml
    # setup.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    tags: webserver
    tags: webserver
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=nginx
    apt: pkg=nginx
    tags:
    tags:
    - nginx
    - nginx
    - name: Update web service
    - name: Update web service
    hg: repo=$URL dest=/home/vagrant/repoapi
    hg: repo=$URL dest=/home/vagrant/repoapi
    tags:
    tags:
    - repo
    - repo
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml --tags webserver
    --tags webserver
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml --tags nginx
    --tags nginx
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml --tags repo
    --tags repo
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml --tags nginx,repo
    --tags nginx,repo

    View full-size slide

  63. BEST PRACTICES?
    Use whatever works for you.

    View full-size slide

  64. BEST PRACTICES?
    http://ansible.cc/docs/patterns.html
    (inventory, tag, role by inventory, role by tag, role by file
    name, etc)

    View full-size slide

  65. PULL?
    To do full automated deploymet, you probably
    want to do pull rather than push. Ansible can do
    that too.
    http://jpmens.net/2012/07/14/ansible-pull-instead-of-push/

    View full-size slide

  66. BUILD CUSTOM BOX
    Ansible is great for configuration. Vagrant is
    great for manging development vm. But
    building a base box to begin with? Use veewee.

    View full-size slide

  67. ANSIBLE HAS VAGRANT
    MODULES
    I haven't tried it, but in theory you could
    configure a bunch of vagrant instances using
    ansible and vagrant module!

    View full-size slide

  68. Questions?
    @yeukhon
    yeukhon.bitbucket.org
    speakerdeck.com/yeukhon

    View full-size slide

  69. IRC: #ansible
    github.com/ansible/ansible
    groups.google.com/group/ansible-project/

    View full-size slide

  70. Ansible
    IRC: #ansible
    github.com/ansible/ansible
    groups.google.com/group/ansible-project/

    View full-size slide

  71. :p
    Thank you!

    View full-size slide