But, it works on my machine! - Virtual Development Environments

But, it works on my machine! - Virtual Development Environments

ou cannot develop the way you have always done it. The days of setting up a single development environment on your laptop are long gone. Even running multiple virtual hosts on the same machine with a common database server is not enough in today’s environment. These days you need matching environments; development, testing, production. They have to match down to the service and version number.

Managing virtual environments is a whole new skill you have to master. Virtualizing your development is here to stay. But with so many options, what are the best tools, techniques and practices for virtualizing development? How do you virtualize your development process? How do you keep everything up to date? How do you do all of this and still have time to do development?

F2d82b268a7cbccc9809c939428df64f?s=128

Vranac Srdjan

October 21, 2014
Tweet

Transcript

  1. But, it works on my machine! Virtual Development Environments Srdjan

    Vranac // code4hire.com // @vranac
  2. business owner, developer , consultant, mercenary, writing terrible code that

    performs exceptionally, wrangling ele PHP ants and Python s, obsessed with process automation , interested in continuous integration and delivery, clean code, testing, best practices and distributed systems
  3. A long time ago in a galaxy far, far away...

  4. Works on my machine™

  5. None
  6. None
  7. In the Beginning... Developers wrote code System Administrators deployed code

  8. ©2012-2013 MokonalovesMochi

  9. ...until one day...

  10. I'll write code that tells computer how to set itself

    up #!bin/sh sudo apt-get update sudo apt-get -y install build-essential sudo apt-get install apache2 sudo a2enmon rewrite sudo a2enmod vhost_alias sudo tee /etc/apache2/sites-available/mysite <<ENDOFFILE ServerAdmin webmaster@localhost DocumentRoot /var/www Options FollowSymLinks AllowOverride None Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny allow from all ENDOFFILE
  11. None
  12. Soooo.... What is the problem?

  13. Idempotence (/ˌaɪdɨmˈpoʊtəns/ eye-dəm-poh-təns) "Idempotence is the property of certain operations

    in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application."
  14. Not "Robust"

  15. Everybody is rolling their own

  16. Present

  17. None
  18. Single & Multiple VM

  19. Single VM config.vm.define :devbox do |devbox| ... end

  20. Multiple VM config.vm.define :devbox do |devbox| ... end config.vm.define :dbbox

    do |dbbox| ... end
  21. Two step process (provide & provision)

  22. Providing: Download basebox Create a VM from basebox Configure VM

    Boot the VM
  23. Provisioning: Puppet (remote puppet master) Shell script Chef (solo or

    master) Ansible
  24. VagrantFile # Vagrantfile API/syntax version. VAGRANTFILE_API_VERSION = "2" Vagrant.configure(VAGRANTFILE_API_VERSION) do

    |config| config.vm.box = "trusty" config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cl config.vm.network "private_network", ip: "33.33.33.100" config.vm.synced_folder "./", "/var/www", type: "nfs" config.vm.boot_timeout = 9000 config.vm.provider "virtualbox" do |vb| # Boot with headless mode # vb.gui = true # Use VBoxManage to customize the VM. For example to change memory: vb.customize ["modifyvm", :id, "--memory", "2048"] end config.vm.provision "ansible" do |ansible| ansible.playbook = "provisioning/vagrant.yml" # output as much as you can, or comment this out for silence ansible.verbose = "vvvv" ansible.sudo = true end end
  25. VagrantFile ... config.vm.box = "trusty" config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cl ...

  26. VagrantFile ... config.vm.network "private_network", ip: "33.33.33.100" config.vm.synced_folder "./", "/var/www", type:

    "nfs" config.vm.boot_timeout = 9000 ...
  27. VagrantFile ... config.vm.provider "virtualbox" do |vb| # Boot with headless

    mode # vb.gui = true # Use VBoxManage to customize the VM. For example to change memory: vb.customize ["modifyvm", :id, "--memory", "2048"] end ...
  28. VagrantFile ... config.vm.provision "puppet" do |puppet| puppet.manifests_path = "manifests" puppet.manifest_file

    = "dev.pp" # Uncomment this for a torrent of puppet info # puppet.options = "--verbose --debug" end ...
  29. VagrantFile ... config.vm.provision "ansible" do |ansible| ansible.playbook = "provisioning/vagrant.yml" #

    output as much as you can, or comment this out for silence ansible.verbose = "vvvv" ansible.sudo = true end ...
  30. Problems:

  31. Different Vagrantfile syntax v1 Vagrant::Config.run do |config| .... end v2

    Vagrant.configure("2") do |config| .... end
  32. Shared resources vboxfs can be slow NFS extensions, performance is

    ok(ish) they do work on windows... kinda... sorta... Windows
  33. Speed VM are booted sequentially, that can take time

  34. None
  35. Trifecta Package/File/Service Learn it, live it, love it. If you

    can use only this, you can still do a lot
  36. base.pp node base { ... file {'/etc/apt/sources.list.d/php55.list': ensure => present,

    content => "deb http://packages.dotdeb.org wheezy-php55 all" } ... exec { "authorize-php55": command => "sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com E9C74FEEA2098A6E", require => File["/etc/apt/sources.list.d/php55.list"], } ... # because puppet command are not run sequentially, ensure that packages are # up to date before installing packages, services, files, etc. Package { require => Exec["apt-get update"] } ... package { "nginx": ensure => present, require => Package["apache2.2-common"] } # starts the nginx service once the packages installed, and monitors changes # to its configuration files and reloads if necessary service { "nginx": ensure => running, enable => true, provider => 'upstart', require => [Package['nginx'], Package['php5-fpm']], subscribe => File["/etc/nginx/sites-available/default"], } }
  37. File (configure) node base { ... file {'/etc/apt/sources.list.d/php55.list': ensure =>

    present, content => "deb http://packages.dotdeb.org wheezy-php55 all" } ... }
  38. Package (install) node base { ... # because puppet command

    are not run sequentially, ensure that packages are # up to date before installing packages, services, files, etc. Package { require => Exec["apt-get update"] } ... package { "nginx": ensure => present, require => Package["apache2.2-common"] } }
  39. Service (startup) node base { # starts the nginx service

    once the packages installed, and monitors changes # to its configuration files and reloads if necessary service { "nginx": ensure => running, enable => true, provider => 'upstart', require => [Package['nginx'], Package['php5-fpm']], subscribe => File["/etc/nginx/sites-available/default"], } }
  40. Exec shell command node base { ... exec { "authorize-php55":

    command => "sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com E9C74FEEA2098A6E require => File["/etc/apt/sources.list.d/php55.list"], } ... }
  41. Problems:

  42. Speed Package installation can take time, especially if apt-get (or

    similar) is executed before each command
  43. Debug: Passing arguments to puppet from console is almost impossible

  44. Debug: Commands in manifest files are not being executed in

    known order exec { "download-composer": ... require => Package["make", "curl", "git", "php5"] }
  45. Debug: Error messages can be completelly useless

  46. Debug: Puppet files can be linted, but without running them

    you do not know if/when they will fail (and where) Puppet unit testing with rspec-puppet and rspec-system-serverspec
  47. Automation should not require programming experience It MUST be easy

    We all have other stuff to do, don't we?
  48. None
  49. "I wrote Ansible because none of the existing tools fit

    my brain. I wanted a tool that I could not use for 6 months, come back later, and still remember how it worked." Michael DeHaan Ansible project founder
  50. “An ansible is a fictional machine capable of instantaneous or

    superluminal communication”
  51. What is it? IT Automation tool Push based (Pull possible)

    Agentless, no agent on the client, uses SSH Scalable No databases or daemons added after install No Root permissions required, sudo is available Supported package managers for RHEL, CentOS, Fedora, Debian or Ubuntu
  52. Why use it? Consistent Predictable Repeatable Easy PERIOD

  53. Requirements Python 2.7 (Python 2.5 + simplejson possible) Paramiko(ssh), PyYaml,

    Jinja2 SSHD Possible Module Dependencies
  54. Installation? pip install ansible DONE

  55. controller → remotes controller remote5 ssh remote4 ssh remote3 ssh

    remote2 ssh remote1 ssh
  56. Inventory [localhost] 127.0.0.1 [webservers] www.example.com ntp=ntp1.pool.ntp.org web[10-23].example.com vagrant ansible_ssh_host=127.0.0.1 ansible_ssh_port=222

    [dbservers] db.example.com [production:children] webservers dbservers
  57. Dynamic Inventory Amazon EC2 Digital Ocean Linode Cobbler Google Compute

    Engine ...
  58. Hello, World! $ ansible localhost -m ping localhost | success

    >> { "changed": false, "ping": "pong" }
  59. Facts $ ansible localhost -m setup localhost | success >>

    { "ansible_facts": { "ansible_all_ipv4_addresses": [ "33.33.33.100", ], "ansible_architecture": "x86_64", "ansible_default_ipv4": { "address": "192.168.1.194", "gateway": "192.168.1.1", "interface": "eth0", "macaddress": "22:54:00:02:8e:0f", }, "ansible_distribution": "CentOS", "ansible_distribution_version": "6.2", ... } Plus ohai and facter if installed on remote
  60. Modules accelerate acl, add_host, airbrake_deployment, alternatives, apache2_module, apt, apt_key, apt_repository,

    apt_rpm, arista_interface, arista_l2interface, arista_lag, arista_vlan, assemble, assert, async_status, at, authorized_key, azure, bigip_facts, bigip_monitor_http, bigip_monitor_tcp, bigip_node, bigip_pool, bigip_pool_member, boundary_meter, bzr, campfire, capabilities, cloudformation, command, composer, copy, cpanm, cron, datadog_event, debconf, debug, digital_ocean, digital_ocean_domain, digital_ocean_sshkey, django_manage, dnsimple, dnsmadeeasy, docker, docker_image, easy_install, ec2, ec2_ami, ec2_ami_search, ec2_asg, ec2_eip, ec2_elb, ec2_elb_lb, ec2_facts, ec2_group, ec2_key, ec2_lc, ec2_metric_alarm, ec2_scaling_policy, ec2_snapshot, ec2_tag, ec2_vol, ec2_vpc, ejabberd_user, elasticache, facter, fail, fetch, file, filesystem, fireball, firewalld, flowdock, gc_storage, gce, gce_lb, gce_net, gce_pd, gem, get_url, git, github_hooks, glance_image, group, group_by, grove, hg, hipchat, homebrew, homebrew_cask, homebrew_tap, hostname, htpasswd, include_vars, ini_file, irc, jabber, jboss, jira, kernel_blacklist, keystone_user, layman, librato_annotation, lineinfile, linode, lldp, locale_gen, logentries, lvg, lvol, macports, mail, modprobe, mongodb_user, monit, mount, mqtt, mysql_db, mysql_replication, mysql_user, mysql_variables, nagios, netscaler, newrelic_deployment, nexmo, nova_compute, nova_keypair, npm, ohai, open_iscsi, openbsd_pkg, openvswitch_bridge, openvswitch_port, opkg, osx_say, ovirt, pacman, pagerduty, pause, ping, pingdom, pip, pkgin, pkgng, pkgutil, portage, portinstall, postgresql_db, postgresql_privs, postgresql_user, quantum_floating_ip, quantum_floating_ip_associate, quantum_network, quantum_router, quantum_router_gateway, quantum_router_interface, quantum_subnet, rabbitmq_parameter, rabbitmq_plugin, rabbitmq_policy, rabbitmq_user, rabbitmq_vhost, raw, rax, rax_cbs, rax_cbs_attachments, rax_clb, rax_clb_nodes, rax_dns, rax_dns_record, rax_facts, rax_files, rax_files_objects, rax_identity, rax_keypair, rax_meta, rax_network, rax_queue, rax_scaling_group, rax_scaling_policy, rds, rds_param_group, rds_subnet_group, redhat_subscription, redis, replace, rhn_channel, rhn_register, riak, rollbar_deployment, route53, rpm_key, s3, script, seboolean, selinux, service, set_fact, setup, shell, slack, slurp, sns, stackdriver, stat, subversion, supervisorctl, svr4pkg, swdepot, synchronize, sysctl, template, twilio, typetalk, ufw, unarchive, uri, urpmi, user, virt, vsphere_guest, wait_for, win_feature, win_get_url, win_group, win_msi, win_ping, win_service, win_stat, win_user, xattr, yum, zfs, zypper, zypper_repository 230+ modules and growing
  61. Ad-Hoc commands $ ansible webservers -m copy -a 'src=resolv.conf dest=/etc/resolv.conf'

    www.example.com | success >> { "changed": true, "dest": "/etc/resolv.conf", "group": "adm", "md5sum": "c6fce6e28c46be0512eaf3b7cfdb66d7", "mode": "0644", "owner": "ubuntu", "path": "resolv.conf", "src": "/home/ubuntu/.ansible/tmp/ansible-322091977449/resolv.conf", "state": "file" }
  62. Playbooks YAML Files Decleratively define your OS/App configuration Collection of

    tasks using modules Each group of tasks is a play
  63. Tasks --- # tasks/foo.yml # This is a task -

    name: Placeholder foo command: /bin/foo # This is another task - name: Placeholder bar command: /bin/bar
  64. Tasks --- - name: Installing supervisor task for snapshot worker

    template: src=supervisor.conf.j2 dest={{ SUPERVISOR_CONFIG_DIR }}/{{ item['filename'] }}.conf backup=yes owner=root group=root mode=0644 # located in defaults/main.yml with_items: snapshot_worker_configuration when: snapshot_worker_configuration|lower != 'none' notify: - reload supervisor tags: [supervisor, configuration]
  65. Tasks --- - name: Installing supervisor task for snapshot worker

    template: src=supervisor.conf.j2 dest={{ SUPERVISOR_CONFIG_DIR }}/{{ item['filename'] }}.conf backup=yes owner=root group=root mode=0644
  66. Tasks # located in defaults/main.yml with_items: snapshot_worker_configuration when: snapshot_worker_configuration|lower !=

    'none'
  67. Tasks notify: - reload supervisor

  68. Variables From inventory In playbooks From host_vars/ files From group_vars/

    files
  69. Variables --- - hosts: localhost vars: - greeting: Hello tasks:

    - command: echo "{{greeting}}, {{inventory_hostname}}"
  70. Variables

  71. Variables host_vars/production --- snapshot_worker_configuration: - filename: snapshot_worker name: process_snapshot_report_worker command:

    "php process_snapshot_report_worker.php" process_name: process_snapshot_report_worker_%(process_num)02d numprocs: 1 directory: "/var/www/scripts/utils/" autostart: true autorestart: true user: ubuntu stdout_logfile: "/var/log/app/utils_process_snapshot_report_worker.log" stdout_logfile_maxbytes: 1MB stderr_logfile: "/var/log/app/supervisor_error_log" stderr_logfile_maxbytes: 1MB
  72. {{ templates }} ;{{ ansible_managed }} [program:{{ item.name }}] {%

    for directive, value in item.iteritems() if directive != "name" and direct ive != "filename" and value != None %} {{ directive }}={{ value }} {% endfor %}
  73. {{ templates }} ;{{ ansible_managed }} [program:{{ item.name }}]

  74. {{ templates }} {% for directive, value in item.iteritems() if

    directive != "name" and direct ive != "filename" and value != None %} {{ directive }}={{ value }} {% endfor %}
  75. {{ templates }} ;Ansible managed: /Users/vranac/dev/playground-ansible/vagrant-ansible-php/ro les/supervisor/templates/supervisor.conf.j2 modified on 2014-06-19

    10:38:31 b y vranac on vurunica [program:process_snapshot_report_worker] stderr_logfile_maxbytes=1MB autorestart=True stderr_logfile=/var/log/app/supervisor_error_log process_name=process_snapshot_report_worker_%(process_num)02d stdout_logfile_maxbytes=1MB numprocs=1 command=php process_snapshot_report_worker.php user=ubuntu autostart=True directory=/var/www/scripts/utils/ stdout_logfile=/var/log/app/utils_process_snapshot_report_worker.log
  76. None
  77. Roles

  78. Roles roles/ nginx/ files/ handlers/main.yml meta/main.yml tasks/main.yml templates/ vars/main.yml

  79. Roles --- - hosts: all roles: - nginx - mysql

    - { role: app, dir: '/etc/app', ntp: 'n1.example.org' } - { role: special, when: "ansible_os_family == 'RedHat'" } tasks: ...
  80. Roles ... - { role: app, dir: '/etc/app', ntp: 'n1.example.org'

    } - { role: special, when: "ansible_os_family == 'RedHat'" } ...
  81. Ansible Galaxy http://galaxy.ansible.com/ ansible-galaxy

  82. Asynchronous Actions and Polling --- - hosts: all tasks: -

    name: "simulate long running op (15 sec), wait for up to 45 sec, poll every 5 sec" command: /bin/sleep 15 async: 45 poll: 5
  83. Check Mode (“Dry Run”) Running a task in check mode

    --check Showing Differences with --diff
  84. Compare to X https://devopsu.com/books/taste-test-grid.html

  85. None
  86. The End Thank You! Questions?