Upgrade to Pro — share decks privately, control downloads, hide ads and more …

But, it works on my machine! - Virtual Development Environments

But, it works on my machine! - Virtual Development Environments

ou cannot develop the way you have always done it. The days of setting up a single development environment on your laptop are long gone. Even running multiple virtual hosts on the same machine with a common database server is not enough in today’s environment. These days you need matching environments; development, testing, production. They have to match down to the service and version number.

Managing virtual environments is a whole new skill you have to master. Virtualizing your development is here to stay. But with so many options, what are the best tools, techniques and practices for virtualizing development? How do you virtualize your development process? How do you keep everything up to date? How do you do all of this and still have time to do development?

Vranac Srdjan

October 21, 2014
Tweet

More Decks by Vranac Srdjan

Other Decks in Programming

Transcript

  1. But, it works on my
    machine!
    Virtual Development Environments
    Srdjan Vranac // code4hire.com // @vranac

    View full-size slide

  2. business owner,
    developer
    , consultant, mercenary, writing
    terrible code that performs exceptionally, wrangling ele
    PHP
    ants
    and
    Python
    s, obsessed with
    process automation
    , interested in
    continuous integration and delivery, clean code, testing, best
    practices and distributed systems

    View full-size slide

  3. A long time ago in a galaxy far, far away...

    View full-size slide

  4. Works on my machine™

    View full-size slide

  5. In the Beginning...
    Developers wrote code
    System Administrators deployed code

    View full-size slide

  6. ©2012-2013 MokonalovesMochi

    View full-size slide

  7. ...until one day...

    View full-size slide

  8. I'll write code that tells computer how to
    set itself up
    #!bin/sh
    sudo apt-get update
    sudo apt-get -y install build-essential
    sudo apt-get install apache2
    sudo a2enmon rewrite
    sudo a2enmod vhost_alias
    sudo tee /etc/apache2/sites-available/mysite <ServerAdmin webmaster@localhost
    DocumentRoot /var/www
    Options FollowSymLinks
    AllowOverride None
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Order allow,deny
    allow from all
    ENDOFFILE

    View full-size slide

  9. Soooo.... What is the problem?

    View full-size slide

  10. Idempotence
    (/ˌaɪdɨmˈpoʊtəns/ eye-dəm-poh-təns)
    "Idempotence is the property of certain
    operations in mathematics and computer
    science, that can be applied multiple times
    without changing the result beyond the initial
    application."

    View full-size slide

  11. Not "Robust"

    View full-size slide

  12. Everybody is rolling their own

    View full-size slide

  13. Single & Multiple VM

    View full-size slide

  14. Single VM
    config.vm.define :devbox do |devbox|
    ...
    end

    View full-size slide

  15. Multiple VM
    config.vm.define :devbox do |devbox|
    ...
    end
    config.vm.define :dbbox do |dbbox|
    ...
    end

    View full-size slide

  16. Two step process
    (provide & provision)

    View full-size slide

  17. Providing:
    Download basebox
    Create a VM from basebox
    Configure VM
    Boot the VM

    View full-size slide

  18. Provisioning:
    Puppet (remote puppet master)
    Shell script
    Chef (solo or master)
    Ansible

    View full-size slide

  19. VagrantFile
    # Vagrantfile API/syntax version.
    VAGRANTFILE_API_VERSION = "2"
    Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
    config.vm.box = "trusty"
    config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cl
    config.vm.network "private_network", ip: "33.33.33.100"
    config.vm.synced_folder "./", "/var/www", type: "nfs"
    config.vm.boot_timeout = 9000
    config.vm.provider "virtualbox" do |vb|
    # Boot with headless mode
    # vb.gui = true
    # Use VBoxManage to customize the VM. For example to change memory:
    vb.customize ["modifyvm", :id, "--memory", "2048"]
    end
    config.vm.provision "ansible" do |ansible|
    ansible.playbook = "provisioning/vagrant.yml"
    # output as much as you can, or comment this out for silence
    ansible.verbose = "vvvv"
    ansible.sudo = true
    end
    end

    View full-size slide

  20. VagrantFile
    ...
    config.vm.box = "trusty"
    config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cl
    ...

    View full-size slide

  21. VagrantFile
    ...
    config.vm.network "private_network", ip: "33.33.33.100"
    config.vm.synced_folder "./", "/var/www", type: "nfs"
    config.vm.boot_timeout = 9000
    ...

    View full-size slide

  22. VagrantFile
    ...
    config.vm.provider "virtualbox" do |vb|
    # Boot with headless mode
    # vb.gui = true
    # Use VBoxManage to customize the VM. For example to change memory:
    vb.customize ["modifyvm", :id, "--memory", "2048"]
    end
    ...

    View full-size slide

  23. VagrantFile
    ...
    config.vm.provision "puppet" do |puppet|
    puppet.manifests_path = "manifests"
    puppet.manifest_file = "dev.pp"
    # Uncomment this for a torrent of puppet info
    # puppet.options = "--verbose --debug"
    end
    ...

    View full-size slide

  24. VagrantFile
    ...
    config.vm.provision "ansible" do |ansible|
    ansible.playbook = "provisioning/vagrant.yml"
    # output as much as you can, or comment this out for silence
    ansible.verbose = "vvvv"
    ansible.sudo = true
    end
    ...

    View full-size slide

  25. Different Vagrantfile syntax
    v1
    Vagrant::Config.run do |config|
    ....
    end
    v2
    Vagrant.configure("2") do |config|
    ....
    end

    View full-size slide

  26. Shared resources
    vboxfs can be slow
    NFS extensions, performance is ok(ish)
    they do work on windows...
    kinda... sorta...
    Windows

    View full-size slide

  27. Speed
    VM are booted sequentially, that can take time

    View full-size slide

  28. Trifecta
    Package/File/Service
    Learn it, live it, love it.
    If you can use only this, you can still do a lot

    View full-size slide

  29. base.pp
    node base {
    ...
    file {'/etc/apt/sources.list.d/php55.list':
    ensure => present,
    content => "deb http://packages.dotdeb.org wheezy-php55 all"
    }
    ...
    exec { "authorize-php55":
    command => "sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com E9C74FEEA2098A6E",
    require => File["/etc/apt/sources.list.d/php55.list"],
    }
    ...
    # because puppet command are not run sequentially, ensure that packages are
    # up to date before installing packages, services, files, etc.
    Package { require => Exec["apt-get update"] }
    ...
    package { "nginx":
    ensure => present,
    require => Package["apache2.2-common"]
    }
    # starts the nginx service once the packages installed, and monitors changes
    # to its configuration files and reloads if necessary
    service { "nginx":
    ensure => running,
    enable => true,
    provider => 'upstart',
    require => [Package['nginx'], Package['php5-fpm']],
    subscribe => File["/etc/nginx/sites-available/default"],
    }
    }

    View full-size slide

  30. File (configure)
    node base {
    ...
    file {'/etc/apt/sources.list.d/php55.list':
    ensure => present,
    content => "deb http://packages.dotdeb.org wheezy-php55 all"
    }
    ...
    }

    View full-size slide

  31. Package (install)
    node base {
    ...
    # because puppet command are not run sequentially, ensure that packages are
    # up to date before installing packages, services, files, etc.
    Package { require => Exec["apt-get update"] }
    ...
    package { "nginx":
    ensure => present,
    require => Package["apache2.2-common"]
    }
    }

    View full-size slide

  32. Service (startup)
    node base {
    # starts the nginx service once the packages installed, and monitors changes
    # to its configuration files and reloads if necessary
    service { "nginx":
    ensure => running,
    enable => true,
    provider => 'upstart',
    require => [Package['nginx'], Package['php5-fpm']],
    subscribe => File["/etc/nginx/sites-available/default"],
    }
    }

    View full-size slide

  33. Exec shell command
    node base {
    ...
    exec { "authorize-php55":
    command => "sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com E9C74FEEA2098A6E
    require => File["/etc/apt/sources.list.d/php55.list"],
    }
    ...
    }

    View full-size slide

  34. Speed
    Package installation can take time,
    especially if apt-get (or similar) is executed before each
    command

    View full-size slide

  35. Debug:
    Passing arguments to puppet from console is almost impossible

    View full-size slide

  36. Debug:
    Commands in manifest files are not being executed in known
    order
    exec { "download-composer":
    ...
    require => Package["make", "curl", "git", "php5"]
    }

    View full-size slide

  37. Debug:
    Error messages can be completelly useless

    View full-size slide

  38. Debug:
    Puppet files can be linted,
    but without running them you do not know
    if/when they will fail (and where)
    Puppet unit testing with
    rspec-puppet and rspec-system-serverspec

    View full-size slide

  39. Automation should not require
    programming experience
    It MUST be easy
    We all have other stuff to do, don't we?

    View full-size slide

  40. "I wrote Ansible because none of the existing
    tools fit my brain. I wanted a tool that I could not
    use for 6 months, come back later, and still
    remember how it worked."
    Michael DeHaan
    Ansible
    project founder

    View full-size slide

  41. “An ansible is a fictional machine capable of instantaneous or
    superluminal communication”

    View full-size slide

  42. What is it?
    IT Automation tool
    Push based (Pull possible)
    Agentless, no agent on the client, uses SSH
    Scalable
    No databases or daemons added after install
    No Root permissions required, sudo is available
    Supported package managers for RHEL, CentOS, Fedora,
    Debian or Ubuntu

    View full-size slide

  43. Why use it?
    Consistent
    Predictable
    Repeatable
    Easy PERIOD

    View full-size slide

  44. Requirements
    Python 2.7
    (Python 2.5 + simplejson possible)
    Paramiko(ssh), PyYaml, Jinja2
    SSHD
    Possible Module Dependencies

    View full-size slide

  45. Installation?
    pip install ansible
    DONE

    View full-size slide

  46. controller → remotes
    controller
    remote5
    ssh
    remote4
    ssh
    remote3
    ssh
    remote2
    ssh
    remote1
    ssh

    View full-size slide

  47. Inventory
    [localhost]
    127.0.0.1
    [webservers]
    www.example.com ntp=ntp1.pool.ntp.org
    web[10-23].example.com
    vagrant ansible_ssh_host=127.0.0.1 ansible_ssh_port=222
    [dbservers]
    db.example.com
    [production:children]
    webservers
    dbservers

    View full-size slide

  48. Dynamic Inventory
    Amazon EC2
    Digital Ocean
    Linode
    Cobbler
    Google Compute Engine
    ...

    View full-size slide

  49. Hello, World!
    $ ansible localhost -m ping
    localhost | success >>
    {
    "changed": false,
    "ping": "pong"
    }

    View full-size slide

  50. Facts
    $ ansible localhost -m setup
    localhost | success >>
    {
    "ansible_facts": {
    "ansible_all_ipv4_addresses": [
    "33.33.33.100",
    ],
    "ansible_architecture": "x86_64",
    "ansible_default_ipv4": {
    "address": "192.168.1.194",
    "gateway": "192.168.1.1",
    "interface": "eth0",
    "macaddress": "22:54:00:02:8e:0f",
    },
    "ansible_distribution": "CentOS",
    "ansible_distribution_version": "6.2",
    ...
    }
    Plus ohai and facter if installed on remote

    View full-size slide

  51. Modules
    accelerate acl, add_host, airbrake_deployment, alternatives, apache2_module, apt, apt_key, apt_repository, apt_rpm, arista_interface,
    arista_l2interface, arista_lag, arista_vlan, assemble, assert, async_status, at, authorized_key, azure, bigip_facts, bigip_monitor_http,
    bigip_monitor_tcp, bigip_node, bigip_pool, bigip_pool_member, boundary_meter, bzr, campfire, capabilities, cloudformation, command,
    composer, copy, cpanm, cron, datadog_event, debconf, debug, digital_ocean, digital_ocean_domain, digital_ocean_sshkey, django_manage,
    dnsimple, dnsmadeeasy, docker, docker_image, easy_install, ec2, ec2_ami, ec2_ami_search, ec2_asg, ec2_eip, ec2_elb, ec2_elb_lb, ec2_facts,
    ec2_group, ec2_key, ec2_lc, ec2_metric_alarm, ec2_scaling_policy, ec2_snapshot, ec2_tag, ec2_vol, ec2_vpc, ejabberd_user, elasticache,
    facter, fail, fetch, file, filesystem, fireball, firewalld, flowdock, gc_storage, gce, gce_lb, gce_net, gce_pd, gem, get_url, git, github_hooks,
    glance_image, group, group_by, grove, hg, hipchat, homebrew, homebrew_cask, homebrew_tap, hostname, htpasswd, include_vars, ini_file,
    irc, jabber, jboss, jira, kernel_blacklist, keystone_user, layman, librato_annotation, lineinfile, linode, lldp, locale_gen, logentries, lvg, lvol,
    macports, mail, modprobe, mongodb_user, monit, mount, mqtt, mysql_db, mysql_replication, mysql_user, mysql_variables, nagios,
    netscaler, newrelic_deployment, nexmo, nova_compute, nova_keypair, npm, ohai, open_iscsi, openbsd_pkg, openvswitch_bridge,
    openvswitch_port, opkg, osx_say, ovirt, pacman, pagerduty, pause, ping, pingdom, pip, pkgin, pkgng, pkgutil, portage, portinstall,
    postgresql_db, postgresql_privs, postgresql_user, quantum_floating_ip, quantum_floating_ip_associate, quantum_network,
    quantum_router, quantum_router_gateway, quantum_router_interface, quantum_subnet, rabbitmq_parameter, rabbitmq_plugin,
    rabbitmq_policy, rabbitmq_user, rabbitmq_vhost, raw, rax, rax_cbs, rax_cbs_attachments, rax_clb, rax_clb_nodes, rax_dns, rax_dns_record,
    rax_facts, rax_files, rax_files_objects, rax_identity, rax_keypair, rax_meta, rax_network, rax_queue, rax_scaling_group, rax_scaling_policy,
    rds, rds_param_group, rds_subnet_group, redhat_subscription, redis, replace, rhn_channel, rhn_register, riak, rollbar_deployment, route53,
    rpm_key, s3, script, seboolean, selinux, service, set_fact, setup, shell, slack, slurp, sns, stackdriver, stat, subversion, supervisorctl, svr4pkg,
    swdepot, synchronize, sysctl, template, twilio, typetalk, ufw, unarchive, uri, urpmi, user, virt, vsphere_guest, wait_for, win_feature,
    win_get_url, win_group, win_msi, win_ping, win_service, win_stat, win_user, xattr, yum, zfs, zypper, zypper_repository
    230+ modules and growing

    View full-size slide

  52. Ad-Hoc commands
    $ ansible webservers -m copy -a 'src=resolv.conf
    dest=/etc/resolv.conf'
    www.example.com | success >> {
    "changed": true,
    "dest": "/etc/resolv.conf",
    "group": "adm",
    "md5sum": "c6fce6e28c46be0512eaf3b7cfdb66d7",
    "mode": "0644",
    "owner": "ubuntu",
    "path": "resolv.conf",
    "src": "/home/ubuntu/.ansible/tmp/ansible-322091977449/resolv.conf",
    "state": "file"
    }

    View full-size slide

  53. Playbooks
    YAML Files
    Decleratively define your OS/App configuration
    Collection of tasks using modules
    Each group of tasks is a play

    View full-size slide

  54. Tasks
    ---
    # tasks/foo.yml
    # This is a task
    - name: Placeholder foo
    command: /bin/foo
    # This is another task
    - name: Placeholder bar
    command: /bin/bar

    View full-size slide

  55. Tasks
    ---
    - name: Installing supervisor task for snapshot worker
    template: src=supervisor.conf.j2
    dest={{ SUPERVISOR_CONFIG_DIR }}/{{ item['filename'] }}.conf
    backup=yes
    owner=root
    group=root
    mode=0644
    # located in defaults/main.yml
    with_items: snapshot_worker_configuration
    when: snapshot_worker_configuration|lower != 'none'
    notify:
    - reload supervisor
    tags: [supervisor, configuration]

    View full-size slide

  56. Tasks
    ---
    - name: Installing supervisor task for snapshot worker
    template: src=supervisor.conf.j2
    dest={{ SUPERVISOR_CONFIG_DIR }}/{{ item['filename'] }}.conf
    backup=yes
    owner=root
    group=root
    mode=0644

    View full-size slide

  57. Tasks
    # located in defaults/main.yml
    with_items: snapshot_worker_configuration
    when: snapshot_worker_configuration|lower != 'none'

    View full-size slide

  58. Tasks
    notify:
    - reload supervisor

    View full-size slide

  59. Variables
    From inventory
    In playbooks
    From host_vars/ files
    From group_vars/ files

    View full-size slide

  60. Variables
    ---
    - hosts: localhost
    vars:
    - greeting: Hello
    tasks:
    - command: echo "{{greeting}}, {{inventory_hostname}}"

    View full-size slide

  61. Variables
    host_vars/production
    ---
    snapshot_worker_configuration:
    - filename: snapshot_worker
    name: process_snapshot_report_worker
    command: "php process_snapshot_report_worker.php"
    process_name: process_snapshot_report_worker_%(process_num)02d
    numprocs: 1
    directory: "/var/www/scripts/utils/"
    autostart: true
    autorestart: true
    user: ubuntu
    stdout_logfile: "/var/log/app/utils_process_snapshot_report_worker.log"
    stdout_logfile_maxbytes: 1MB
    stderr_logfile: "/var/log/app/supervisor_error_log"
    stderr_logfile_maxbytes: 1MB

    View full-size slide

  62. {{ templates }}
    ;{{ ansible_managed }}
    [program:{{ item.name }}]
    {% for directive, value in item.iteritems() if directive != "name" and direct
    ive != "filename" and value != None %}
    {{ directive }}={{ value }}
    {% endfor %}

    View full-size slide

  63. {{ templates }}
    ;{{ ansible_managed }}
    [program:{{ item.name }}]

    View full-size slide

  64. {{ templates }}
    {% for directive, value in item.iteritems() if directive != "name" and direct
    ive != "filename" and value != None %}
    {{ directive }}={{ value }}
    {% endfor %}

    View full-size slide

  65. {{ templates }}
    ;Ansible managed: /Users/vranac/dev/playground-ansible/vagrant-ansible-php/ro
    les/supervisor/templates/supervisor.conf.j2 modified on 2014-06-19 10:38:31 b
    y vranac on vurunica
    [program:process_snapshot_report_worker]
    stderr_logfile_maxbytes=1MB
    autorestart=True
    stderr_logfile=/var/log/app/supervisor_error_log
    process_name=process_snapshot_report_worker_%(process_num)02d
    stdout_logfile_maxbytes=1MB
    numprocs=1
    command=php process_snapshot_report_worker.php
    user=ubuntu
    autostart=True
    directory=/var/www/scripts/utils/
    stdout_logfile=/var/log/app/utils_process_snapshot_report_worker.log

    View full-size slide

  66. Roles
    roles/
    nginx/
    files/
    handlers/main.yml
    meta/main.yml
    tasks/main.yml
    templates/
    vars/main.yml

    View full-size slide

  67. Roles
    ---
    - hosts: all
    roles:
    - nginx
    - mysql
    - { role: app, dir: '/etc/app', ntp: 'n1.example.org' }
    - { role: special, when: "ansible_os_family == 'RedHat'" }
    tasks:
    ...

    View full-size slide

  68. Roles
    ...
    - { role: app, dir: '/etc/app', ntp: 'n1.example.org' }
    - { role: special, when: "ansible_os_family == 'RedHat'" }
    ...

    View full-size slide

  69. Ansible Galaxy
    http://galaxy.ansible.com/
    ansible-galaxy

    View full-size slide

  70. Asynchronous Actions and Polling
    ---
    - hosts: all
    tasks:
    - name: "simulate long running op (15 sec), wait for up to 45 sec,
    poll every 5 sec"
    command: /bin/sleep 15
    async: 45
    poll: 5

    View full-size slide

  71. Check Mode (“Dry Run”)
    Running a task in check mode --check
    Showing Differences with --diff

    View full-size slide

  72. Compare to X
    https://devopsu.com/books/taste-test-grid.html

    View full-size slide

  73. The End
    Thank You!
    Questions?

    View full-size slide