Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deploying with Vagrant and Ansible

yeukhon
April 17, 2013

Deploying with Vagrant and Ansible

April 17, 2013
Adv Tps Intrnet Prog: CSC I4330 3GG [2337] (CCNY )

yeukhon

April 17, 2013
Tweet

More Decks by yeukhon

Other Decks in Programming

Transcript

  1. DEPLOYING WITH
    VAGRANT AND ANSIBLE
    YEUK HON WONG @yeukhon
    GLASS Lab, CUNY City

    View Slide

  2. REPO
    https://bitbucket.org/ideapiteam/repo-api-master
    https://bitbucket.org/yeukhon/vagrant-ansible-example

    View Slide

  3. GLASS LAB:
    Developing Graphyte and Aurum.
    Serving ~100 undergradutes.

    View Slide

  4. DEPLOYMENT:
    Deploying web service is just a subset
    of a bigger picture.

    View Slide

  5. BIGGER PICTURE:
    “Obama campaign AWS infrastructure in one
    diagram” (http://awsofa.info/)

    View Slide

  6. Some Ideal Work-flow
    http://goo.gl/pJtK7

    View Slide

  7. DEPLOY #TODO:
    O Development deployment
    O Testing deployment
    O Production deployment

    View Slide

  8. GET A MACHINE.
    Virtualbox is nice.

    View Slide

  9. VAGRANT
    FOR DEVELOPMENT

    View Slide

  10. Getting Started

    Install Virtualbox
    https://www.virtualbox.org/wiki/Downloads

    Install Vagrant
    http://docs.vagrantup.com/v2/installation

    View Slide

  11. Vagrant Box

    A Vagrant box is a packaged virtual machine.
    If you extract it you get a vagrant file, ovf and
    and a vmdk file.

    Vagrant has official boxes for download
    (distribute with a universal SSH key, so do not
    use it in production)

    View Slide

  12. Official Boxes
    Ubuntu Lucid 32 Bit
    http://files.vagrantup.com/lucid32.box
    Ubuntu Lucid 64 Bit
    http://files.vagrantup.com/lucid64.box
    Ubuntu Precise 32 Bit
    http://files.vagrantup.com/precise32.box
    Ubuntu Precise 64 Bit
    http://files.vagrantup.com/precise64.box

    View Slide

  13. Community Boxes
    http://www.vagrantbox.es/

    View Slide

  14. Step 1: Create a box
    1. Download a box
    2. Initialize a Vagrant workspace
    $ mkdir vagrant-example
    $ cd vagrant-example
    $ vagrant init precise32
    $ vagrant box add precise32 http://files.vagrantup.com/precise32.box

    View Slide

  15. Step 2: Vagrantfile
    Vagrantfile
    Vagrantfile is a ruby configuration file used to
    configure a vagrant instance.
    Vagrant.configure("2") do |config|
    config.vm.box = "precise32"
    config.vm.network :public_network
    config.vm.provider :virtualbox do |vb|
    vb.customize ["modifyvm", :id, "--memory", "400"]
    end
    end

    View Slide

  16. Step 3: vagrant up
    vagrant up
    vagrant up will create a new VM or boot up an existing
    Vagrant virtual machine.
    $ vagrant up
    $ vagrant up
    Bringing machine 'default' up with 'virtualbox' provider...
    Bringing machine 'default' up with 'virtualbox' provider...
    [default] Importing base box 'precise32'...
    [default] Importing base box 'precise32'...

    View Slide

  17. Step 3: vagrant ssh
    vagrant ssh
    vagrant ssh will allow you to automatically ssh into
    your vagrant instance if you are using an official box.
    $ vagrant ssh
    $ vagrant ssh
    Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic-pae i686)
    Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic-pae i686)
    * Documentation: https://help.ubuntu.com/
    * Documentation: https://help.ubuntu.com/
    Welcome to your Vagrant-built virtual machine.
    Welcome to your Vagrant-built virtual machine.
    Last login: Fri Sep 14 06:22:31 2012 from 10.0.2.2
    Last login: Fri Sep 14 06:22:31 2012 from 10.0.2.2

    View Slide

  18. Vagrant Commands
    Usage: vagrant [-v] [-h] command []
    Usage: vagrant [-v] [-h] command []
    -v, --version Print the version and exit.
    -v, --version Print the version and exit.
    -h, --help Print this help.
    -h, --help Print this help.
    Available subcommands:
    Available subcommands:
    box
    box
    destroy
    destroy
    halt
    halt
    init
    init
    package
    package
    plugin
    plugin
    provision
    provision
    reload
    reload
    resume
    resume
    ssh
    ssh
    ssh-config
    ssh-config
    status
    status
    suspend
    suspend
    up
    up
    For help on any individual command run `vagrant COMMAND -h`
    For help on any individual command run `vagrant COMMAND -h`

    View Slide

  19. I don't want an official box
    You can use any ISO local or over the wire.
    $ mkdir vagrant-example
    $ mkdir vagrant-example
    $ cd vagrant-example
    $ cd vagrant-example
    $ vagrant init
    $ vagrant init
    $ vim Vagrantfile
    config.vm.box_url = "http://domain.com/path/to/above.box"

    View Slide

  20. I want my own SSH key
    Even for vagrant official boxes, you should consider
    changing to your own SSH key.

    View Slide

  21. DEMO?
    Later!

    View Slide

  22. CONFIGURE MACHINES
    Human, please.

    View Slide

  23. Treat your infrastructure like
    your software.
    O Version control your configuration and specs
    O Readable
    O Automated

    View Slide

  24. REQUIREMENTS:
    O Strong community support
    O Medium to little entry to barrier (installation,
    tutorial, customizability)
    O Readable configuration
    O Version control
    O As lightweight as possible

    View Slide

  25. So many, which to use?
    #!/bin/bash #!/usr/bin/python + Fabric
    #!/bin/bash #!/usr/bin/python + Fabric

    View Slide

  26. ANSIBLE
    FOR CONFIGURATION

    View Slide

  27. CONFIGURATION IN
    PLAINTEXT (YAML)
    - name: Install Nginx
    apt: pkg=nginx state=present

    View Slide

  28. STRONG COMMUNITY
    Started in 2012, has over 423 forks, 1,429 watcher,
    2000+ tickets solved, weekly clearance, active IRC &
    mailing list

    View Slide

  29. PYTHON
    Core is in Python. Much smaller core than other
    competitors.

    View Slide

  30. 5 REQUIRED DEPENDENCIES
    Python for running Python
    Simplejson/json for results
    Jinja2 for template
    Pyyaml for configuration
    Paramiko for ssh2 protocol

    View Slide

  31. #TODO
    We will configure a machine for repoapi live.
    O
    bootstrap machine with dev pkgs
    O
    setup nginx
    O
    setup database
    O
    clone and install master repo
    O
    run server

    View Slide

  32. Getting Started

    Install AnVsible
    http://ansible.cc/docs/
    $ sudo pip install pyyaml jinja2 paramiko
    $ sudo pip install pyyaml jinja2 paramiko
    $ git clone git://github.com/ansible/ansible.git
    $ git clone git://github.com/ansible/ansible.git
    $ cd ansible
    $ cd ansible
    $ sudo make install
    $ sudo make install

    View Slide

  33. Modules

    Ansible community has a lot of modules that you can used
    to configure a server. Modules could be in ruby, python,
    perl, etc. Just executable.
    http://ansible.cc/docs/modules.html
    apt, yum, pip, gem, homebrew, user
    apt, yum, pip, gem, homebrew, user
    file, copy, template, django_manage
    file, copy, template, django_manage
    postgres_user, postgres_db, mysql*, mongodb*, rabbitmq*, Riak
    postgres_user, postgres_db, mysql*, mongodb*, rabbitmq*, Riak
    ec2, ec2_facts, s3,
    ec2, ec2_facts, s3,
    hg, git, svn
    hg, git, svn
    many more...
    many more...

    View Slide

  34. Hosts / Inventory
    An inventory file contains a list of machine
    addresses.
    http://ansible.cc/docs/patterns.html
    192.168.1.4
    192.168.1.4
    [webservers]
    [webservers]
    foo.example.com
    foo.example.com
    bar.example.com
    bar.example.com
    www[01:50].example.com
    www[01:50].example.com

    View Slide

  35. Ansible Playbook
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # You can use variables during configuration!
    # You can use variables during configuration!
    vars:
    vars:
    - APP_PORT: 8080
    - APP_PORT: 8080
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=nginx
    apt: pkg=nginx

    View Slide

  36. Template for Play
    - name: Description of the task/play
    - name: Description of the task/play
    module_name: arguments option1=value1 option2=value2
    module_name: arguments option1=value1 option2=value2

    View Slide

  37. apt

    View Slide

  38. apt get install nginx?
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    apt: pkg=nginx

    View Slide

  39. apt-get remove nginx?
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    tasks:
    tasks:
    - name: Ensure Nginx is removed
    apt: pkg=nginx state=absent

    View Slide

  40. Idempotence
    Idempotence means for the same configuration
    settings,
    f(f(x)) = f(x)
    f(f(x)) = f(x)
    HTTP's GET, HEAD, PUT, DELETE, OPTIONS, TRACE
    satisfy idempotence.

    View Slide

  41. TEMPLATE,
    VARIABLE,
    FACTS

    View Slide

  42. Variable
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # You can use variables during configuration!
    # You can use variables during configuration!
    vars:
    - APP_PORT: 6543
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=nginx
    apt: pkg=nginx

    View Slide

  43. Variable
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # You can use variables during configuration!
    # You can use variables during configuration!
    vars:
    - NGINX_NAME: nginx
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=
    apt: pkg=$NGINX_NAME
    $NGINX_NAME

    View Slide

  44. Variable
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # You can use variables during configuration!
    # You can use variables during configuration!
    vars:
    - NGINX_NAME: nginx
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=
    apt: pkg=$NGINX_NAME
    $NGINX_NAME

    View Slide

  45. Template + Variable
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # You can use variables during configuration!
    # You can use variables during configuration!
    vars:
    - HELLO_WORLD: “Hi, World”
    tasks:
    tasks:
    - name: Configure a foo.j2
    - name: Configure a foo.j2
    template: src=foo.j2 dest=/home/vagrant/foo.txt
    # call this foo.j2 - a Jinja template file
    # call this foo.j2 - a Jinja template file
    {{
    {{ HELLO_WORLD }}
    }}
    # result
    # result
    $ cat foo.txt
    $ cat foo.txt
    Hi, World
    Hi, World

    View Slide

  46. Gather Facts
    Ansible can gather basic facts about the target
    machine. By default, this feature is on. If you don't
    need it, set gather_facts: no
    gather_facts: no to speed up configuration.
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    gather_facts: no
    gather_facts: no

    ….
    .

    View Slide

  47. Gather Facts
    ansible -m setup 127.0.0.1 -u vagrant -k
    ansible -m setup 127.0.0.1 -u vagrant -k
    127.0.0.1 | success >> {
    127.0.0.1 | success >> {
    "ansible_facts": {
    "ansible_facts": {
    ….
    ….
    "ansible_lsb": {
    "ansible_lsb": {
    "codename": "precise",
    "codename": "precise",
    "description": "Ubuntu 12.04.1 LTS",
    "description": "Ubuntu 12.04.1 LTS",
    "id": "Ubuntu",
    "id": "Ubuntu",
    "major_release": "12",
    "major_release": "12",
    "release": "12.04"
    "release": "12.04"
    },
    },
    "ansible_pkg_mgr": "apt",
    "ansible_pkg_mgr": "apt",
    "ansible_processor": [
    "ansible_processor": [
    "Intel(R) Pentium(R) 4 CPU 3.20GHz"
    "Intel(R) Pentium(R) 4 CPU 3.20GHz"
    ],
    ],
    "ansible_processor_cores": 1,
    "ansible_processor_cores": 1,
    "ansible_processor_count":
    "ansible_processor_count": 1
    1
    "ansible_python_version": "2.7.3",
    "ansible_python_version": "2.7.3",
    ….
    ….
    }
    }
    see https://gist.github.com/yeukhon/4689592
    see https://gist.github.com/yeukhon/4689592

    View Slide

  48. Templating Nginx
    nginx.conf and sites-available/default require
    configuration.
    o worker counts
    o application port and proxy settings
    o SSL (although we won't do it here)
    o many more

    View Slide

  49. nginx.conf.j2
    # nginx.conf.j2
    # nginx.conf.j2
    user www-data;
    user www-data;
    worker_processes
    worker_processes {{ WORKER_COUNT }}
    {{ WORKER_COUNT }};
    ;
    pid /var/run/nginx.pid;
    pid /var/run/nginx.pid;
    events {
    events {
    worker_connections
    worker_connections {{ WORKER_COUNT * 1024 }}
    {{ WORKER_COUNT * 1024 }};
    ;
    # multi_accept on;
    # multi_accept on;
    }
    }

    View Slide

  50. Configure nginx.conf.j2
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    vars:
    - WORKER_COUNT: ${ansible_processor_count}
    tasks:
    tasks:
    - name: Configure nginx.conf
    - name: Configure nginx.conf
    template: src=nginx.conf.j2 path=/etc/nginx/nginx.conf

    View Slide

  51. Var files
    ---
    ---
    # nginx.yml
    # nginx.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    # vars:
    # - WORKER_COUNT: ${ansible_processor_count}
    var_files:
    var_files:
    - nginx_vars.yml
    - nginx_vars.yml
    tasks:
    tasks:
    - name: Configure nginx.conf
    - name: Configure nginx.conf
    template: src=nginx.conf.j2 path=/etc/nginx/nginx.conf
    # nginx_vars.yml
    # nginx_vars.yml
    WORKER_COUNT: ${ansible_processor_count}

    View Slide

  52. DRY? LOOP!
    WITH_ITEMS

    View Slide

  53. with_items
    - name: Install Debian pkgs
    - name: Install Debian pkgs
    apt: pkg=
    apt: pkg=$item
    $item
    with_items:
    with_items:
    - build-essential
    - build-essential
    - python-dev
    - python-dev
    - python-pip
    - python-pip
    - python-setuptools
    - python-setuptools
    - name: Pip install Python tools
    - name: Pip install Python tools
    pip: name=
    pip: name=$item
    $item
    with_items:
    with_items:
    - distribute
    - distribute
    - virtualenv
    - virtualenv

    View Slide

  54. with_items

    View Slide

  55. A FEW MORE
    MODULES....

    View Slide

  56. hg
    vars:
    vars:
    - URL:
    - URL: https://bitbucket.org/ideapiteam/repo-api-master
    https://bitbucket.org/ideapiteam/repo-api-master
    tasks:
    tasks:
    - name: Clone Master
    - name: Clone Master
    hg: repo=$URL dest=/home/vagrant/repoapi
    hg: repo=$URL dest=/home/vagrant/repoapi

    View Slide

  57. service
    tasks:
    tasks:
    - name: Ensure Nginx has started
    - name: Ensure Nginx has started
    service: name=nginx state=started
    service: name=nginx state=started
    # started, restarted, stopped

    View Slide

  58. file
    tasks:
    tasks:
    - name: Ensure /home/vagrant/repoapi exists
    - name: Ensure /home/vagrant/repoapi exists
    file: path=/home/vagrant/repoapi state=directory
    file: path=/home/vagrant/repoapi state=directory
    # state=directory/file/link

    View Slide

  59. user
    tasks:
    tasks:
    - name: Ensure user vagrant actually exist
    - name: Ensure user vagrant actually exist
    user: name=vagrant uid=1001 state=present
    user: name=vagrant uid=1001 state=present

    View Slide

  60. postgres_*
    tasks:
    tasks:
    - name: Create a Postgres database
    - name: Create a Postgres database
    postgres_db: db=mydb owner=user123 login_password: $md5_pwd
    postgres_db: db=mydb owner=user123 login_password: $md5_pwd
    - name: Create a Postgres user
    - name: Create a Postgres user
    postgres_user: user=user123 password=$hash_pwd priv=ALL
    postgres_user: user=user123 password=$hash_pwd priv=ALL

    View Slide

  61. ROLES?
    TAG

    View Slide

  62. Role
    You can have a cluster of webservice and
    dbserver. You can narrow down to a cluster of
    mongo instances, project1 webservice,
    project2 webservice, etc.
    Group them inside inventory.

    View Slide

  63. Inventory groups (roles)
    [webservers]
    [webservers]
    192.168.1.1
    192.168.1.1
    192.168.1.2
    192.168.1.2
    [mongoservers]
    192.168.1.4
    192.168.1.4
    192.168.1.5
    192.168.1.5
    192.168.1.7
    192.168.1.7
    [all]
    [all]
    Webservers
    Webservers
    mongoservers
    mongoservers

    View Slide

  64. Inventory name can be role
    # production
    # production
    [webservers]
    [webservers]
    www[01:20].example.com
    www[01:20].example.com
    [mongoservers]
    db[01:20].example.com
    db[01:20].example.com
    [all]
    [all]
    webservers
    webservers
    mongoservers
    mongoservers
    # development
    # development
    [webservers]
    [webservers]
    192.168.1.1
    192.168.1.1
    192.168.1.2
    192.168.1.2
    [mongoservers]
    192.168.1.4
    192.168.1.4
    192.168.1.5
    192.168.1.5
    192.168.1.7
    192.168.1.7
    [all]
    [all]
    webservers
    webservers
    mongoservers
    mongoservers
    ansible-playbook -i
    ansible-playbook -i development setup.yml
    setup.yml ansible-playbook -i
    ansible-playbook -i production setup.yml
    setup.yml

    View Slide

  65. Slow?
    ---
    ---
    # setup.yml
    # setup.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=nginx
    apt: pkg=nginx
    - name: Update web service
    - name: Update web service
    hg: repo=$URL dest=/home/vagrant/repoapi
    hg: repo=$URL dest=/home/vagrant/repoapi

    … more tasks x 100 ...
    more tasks x 100 ...

    View Slide

  66. Tags?
    ---
    ---
    # setup.yml
    # setup.yml
    - hosts: dev-box
    - hosts: dev-box
    user: vagrant
    user: vagrant
    sudo: True
    sudo: True
    tags: webserver
    tags: webserver
    tasks:
    tasks:
    - name: Ensure Nginx is installed
    - name: Ensure Nginx is installed
    apt: pkg=nginx
    apt: pkg=nginx
    tags:
    tags:
    - nginx
    - nginx
    - name: Update web service
    - name: Update web service
    hg: repo=$URL dest=/home/vagrant/repoapi
    hg: repo=$URL dest=/home/vagrant/repoapi
    tags:
    tags:
    - repo
    - repo
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml --tags webserver
    --tags webserver
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml --tags nginx
    --tags nginx
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml --tags repo
    --tags repo
    ansible-playbook -i development setup.yml
    ansible-playbook -i development setup.yml --tags nginx,repo
    --tags nginx,repo

    View Slide

  67. BEST PRACTICES?
    Use whatever works for you.

    View Slide

  68. BEST PRACTICES?
    http://ansible.cc/docs/patterns.html
    (inventory, tag, role by inventory, role by tag, role by file
    name, etc)

    View Slide

  69. PULL?
    To do full automated deploymet, you probably
    want to do pull rather than push. Ansible can do
    that too.
    http://jpmens.net/2012/07/14/ansible-pull-instead-of-push/

    View Slide

  70. BUILD CUSTOM BOX
    Ansible is great for configuration. Vagrant is
    great for manging development vm. But
    building a base box to begin with? Use veewee.

    View Slide

  71. ANSIBLE HAS VAGRANT
    MODULES
    I haven't tried it, but in theory you could
    configure a bunch of vagrant instances using
    ansible and vagrant module!

    View Slide

  72. DEMO TIME!

    View Slide

  73. Questions?
    @yeukhon
    yeukhon.bitbucket.org
    speakerdeck.com/yeukhon

    View Slide

  74. IRC: #ansible
    github.com/ansible/ansible
    groups.google.com/group/ansible-project/

    View Slide

  75. Ansible
    IRC: #ansible
    github.com/ansible/ansible
    groups.google.com/group/ansible-project/

    View Slide

  76. :p
    Thank you!

    View Slide