But, it works on my machine! - Virtual Development Environments

ou cannot develop the way you have always done it. The days of setting up a single development environment on your laptop are long gone. Even running multiple virtual hosts on the same machine with a common database server is not enough in today’s environment. These days you need matching environments; development, testing, production. They have to match down to the service and version number.

Managing virtual environments is a whole new skill you have to master. Virtualizing your development is here to stay. But with so many options, what are the best tools, techniques and practices for virtualizing development? How do you virtualize your development process? How do you keep everything up to date? How do you do all of this and still have time to do development?

Transcript

1. But, it works on my machine! Virtual Development Environments Srdjan Vranac // code4hire.com // @vranac
2. business owner, developer , consultant, mercenary, writing terrible code that performs exceptionally, wrangling ele PHP ants and Python s, obsessed with process automation , interested in continuous integration and delivery, clean code, testing, best practices and distributed systems
3. A long time ago in a galaxy far, far away...
4. Works on my machine™
7. In the Beginning... Developers wrote code System Administrators deployed code
8. ©2012-2013 MokonalovesMochi
9. ...until one day...
10. I'll write code that tells computer how to set itself up #!bin/sh sudo apt-get update sudo apt-get -y install build-essential sudo apt-get install apache2 sudo a2enmon rewrite sudo a2enmod vhost_alias sudo tee /etc/apache2/sites-available/mysite <<ENDOFFILE ServerAdmin webmaster@localhost DocumentRoot /var/www Options FollowSymLinks AllowOverride None Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny allow from all ENDOFFILE
12. Soooo.... What is the problem?
13. Idempotence (/ˌaɪdɨmˈpoʊtəns/ eye-dəm-poh-təns) "Idempotence is the property of certain operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application."
14. Not "Robust"
15. Everybody is rolling their own
16. Present
18. Single & Multiple VM
19. Single VM config.vm.define :devbox do |devbox| ... end
20. Multiple VM config.vm.define :devbox do |devbox| ... end config.vm.define :dbbox do |dbbox| ... end
21. Two step process (provide & provision)
22. Providing: Download basebox Create a VM from basebox Configure VM Boot the VM
23. Provisioning: Puppet (remote puppet master) Shell script Chef (solo or master) Ansible
24. VagrantFile # Vagrantfile API/syntax version. VAGRANTFILE_API_VERSION = "2" Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| config.vm.box = "trusty" config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cl config.vm.network "private_network", ip: "33.33.33.100" config.vm.synced_folder "./", "/var/www", type: "nfs" config.vm.boot_timeout = 9000 config.vm.provider "virtualbox" do |vb| # Boot with headless mode # vb.gui = true # Use VBoxManage to customize the VM. For example to change memory: vb.customize ["modifyvm", :id, "--memory", "2048"] end config.vm.provision "ansible" do |ansible| ansible.playbook = "provisioning/vagrant.yml" # output as much as you can, or comment this out for silence ansible.verbose = "vvvv" ansible.sudo = true end end
25. VagrantFile ... config.vm.box = "trusty" config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cl ...
26. VagrantFile ... config.vm.network "private_network", ip: "33.33.33.100" config.vm.synced_folder "./", "/var/www", type: "nfs" config.vm.boot_timeout = 9000 ...
27. VagrantFile ... config.vm.provider "virtualbox" do |vb| # Boot with headless mode # vb.gui = true # Use VBoxManage to customize the VM. For example to change memory: vb.customize ["modifyvm", :id, "--memory", "2048"] end ...
28. VagrantFile ... config.vm.provision "puppet" do |puppet| puppet.manifests_path = "manifests" puppet.manifest_file = "dev.pp" # Uncomment this for a torrent of puppet info # puppet.options = "--verbose --debug" end ...
29. VagrantFile ... config.vm.provision "ansible" do |ansible| ansible.playbook = "provisioning/vagrant.yml" # output as much as you can, or comment this out for silence ansible.verbose = "vvvv" ansible.sudo = true end ...
30. Problems:
31. Different Vagrantfile syntax v1 Vagrant::Config.run do |config| .... end v2 Vagrant.configure("2") do |config| .... end
32. Shared resources vboxfs can be slow NFS extensions, performance is ok(ish) they do work on windows... kinda... sorta... Windows
33. Speed VM are booted sequentially, that can take time
35. Trifecta Package/File/Service Learn it, live it, love it. If you can use only this, you can still do a lot
36. base.pp node base { ... file {'/etc/apt/sources.list.d/php55.list': ensure => present, content => "deb http://packages.dotdeb.org wheezy-php55 all" } ... exec { "authorize-php55": command => "sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com E9C74FEEA2098A6E", require => File["/etc/apt/sources.list.d/php55.list"], } ... # because puppet command are not run sequentially, ensure that packages are # up to date before installing packages, services, files, etc. Package { require => Exec["apt-get update"] } ... package { "nginx": ensure => present, require => Package["apache2.2-common"] } # starts the nginx service once the packages installed, and monitors changes # to its configuration files and reloads if necessary service { "nginx": ensure => running, enable => true, provider => 'upstart', require => [Package['nginx'], Package['php5-fpm']], subscribe => File["/etc/nginx/sites-available/default"], } }
37. File (configure) node base { ... file {'/etc/apt/sources.list.d/php55.list': ensure => present, content => "deb http://packages.dotdeb.org wheezy-php55 all" } ... }
38. Package (install) node base { ... # because puppet command are not run sequentially, ensure that packages are # up to date before installing packages, services, files, etc. Package { require => Exec["apt-get update"] } ... package { "nginx": ensure => present, require => Package["apache2.2-common"] } }
39. Service (startup) node base { # starts the nginx service once the packages installed, and monitors changes # to its configuration files and reloads if necessary service { "nginx": ensure => running, enable => true, provider => 'upstart', require => [Package['nginx'], Package['php5-fpm']], subscribe => File["/etc/nginx/sites-available/default"], } }
40. Exec shell command node base { ... exec { "authorize-php55": command => "sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com E9C74FEEA2098A6E require => File["/etc/apt/sources.list.d/php55.list"], } ... }
41. Problems:
42. Speed Package installation can take time, especially if apt-get (or similar) is executed before each command
43. Debug: Passing arguments to puppet from console is almost impossible
44. Debug: Commands in manifest files are not being executed in known order exec { "download-composer": ... require => Package["make", "curl", "git", "php5"] }
45. Debug: Error messages can be completelly useless
46. Debug: Puppet files can be linted, but without running them you do not know if/when they will fail (and where) Puppet unit testing with rspec-puppet and rspec-system-serverspec
47. Automation should not require programming experience It MUST be easy We all have other stuff to do, don't we?
49. "I wrote Ansible because none of the existing tools fit my brain. I wanted a tool that I could not use for 6 months, come back later, and still remember how it worked." Michael DeHaan Ansible project founder
50. “An ansible is a fictional machine capable of instantaneous or superluminal communication”
51. What is it? IT Automation tool Push based (Pull possible) Agentless, no agent on the client, uses SSH Scalable No databases or daemons added after install No Root permissions required, sudo is available Supported package managers for RHEL, CentOS, Fedora, Debian or Ubuntu
52. Why use it? Consistent Predictable Repeatable Easy PERIOD
53. Requirements Python 2.7 (Python 2.5 + simplejson possible) Paramiko(ssh), PyYaml, Jinja2 SSHD Possible Module Dependencies
54. Installation? pip install ansible DONE
55. controller → remotes controller remote5 ssh remote4 ssh remote3 ssh remote2 ssh remote1 ssh
56. Inventory [localhost] 127.0.0.1 [webservers] www.example.com ntp=ntp1.pool.ntp.org web[10-23].example.com vagrant ansible_ssh_host=127.0.0.1 ansible_ssh_port=222 [dbservers] db.example.com [production:children] webservers dbservers
57. Dynamic Inventory Amazon EC2 Digital Ocean Linode Cobbler Google Compute Engine ...
58. Hello, World! $ ansible localhost -m ping localhost | success >> { "changed": false, "ping": "pong" }
59. Facts $ ansible localhost -m setup localhost | success >> { "ansible_facts": { "ansible_all_ipv4_addresses": [ "33.33.33.100", ], "ansible_architecture": "x86_64", "ansible_default_ipv4": { "address": "192.168.1.194", "gateway": "192.168.1.1", "interface": "eth0", "macaddress": "22:54:00:02:8e:0f", }, "ansible_distribution": "CentOS", "ansible_distribution_version": "6.2", ... } Plus ohai and facter if installed on remote
60. Modules accelerate acl, add_host, airbrake_deployment, alternatives, apache2_module, apt, apt_key, apt_repository, apt_rpm, arista_interface, arista_l2interface, arista_lag, arista_vlan, assemble, assert, async_status, at, authorized_key, azure, bigip_facts, bigip_monitor_http, bigip_monitor_tcp, bigip_node, bigip_pool, bigip_pool_member, boundary_meter, bzr, campfire, capabilities, cloudformation, command, composer, copy, cpanm, cron, datadog_event, debconf, debug, digital_ocean, digital_ocean_domain, digital_ocean_sshkey, django_manage, dnsimple, dnsmadeeasy, docker, docker_image, easy_install, ec2, ec2_ami, ec2_ami_search, ec2_asg, ec2_eip, ec2_elb, ec2_elb_lb, ec2_facts, ec2_group, ec2_key, ec2_lc, ec2_metric_alarm, ec2_scaling_policy, ec2_snapshot, ec2_tag, ec2_vol, ec2_vpc, ejabberd_user, elasticache, facter, fail, fetch, file, filesystem, fireball, firewalld, flowdock, gc_storage, gce, gce_lb, gce_net, gce_pd, gem, get_url, git, github_hooks, glance_image, group, group_by, grove, hg, hipchat, homebrew, homebrew_cask, homebrew_tap, hostname, htpasswd, include_vars, ini_file, irc, jabber, jboss, jira, kernel_blacklist, keystone_user, layman, librato_annotation, lineinfile, linode, lldp, locale_gen, logentries, lvg, lvol, macports, mail, modprobe, mongodb_user, monit, mount, mqtt, mysql_db, mysql_replication, mysql_user, mysql_variables, nagios, netscaler, newrelic_deployment, nexmo, nova_compute, nova_keypair, npm, ohai, open_iscsi, openbsd_pkg, openvswitch_bridge, openvswitch_port, opkg, osx_say, ovirt, pacman, pagerduty, pause, ping, pingdom, pip, pkgin, pkgng, pkgutil, portage, portinstall, postgresql_db, postgresql_privs, postgresql_user, quantum_floating_ip, quantum_floating_ip_associate, quantum_network, quantum_router, quantum_router_gateway, quantum_router_interface, quantum_subnet, rabbitmq_parameter, rabbitmq_plugin, rabbitmq_policy, rabbitmq_user, rabbitmq_vhost, raw, rax, rax_cbs, rax_cbs_attachments, rax_clb, rax_clb_nodes, rax_dns, rax_dns_record, rax_facts, rax_files, rax_files_objects, rax_identity, rax_keypair, rax_meta, rax_network, rax_queue, rax_scaling_group, rax_scaling_policy, rds, rds_param_group, rds_subnet_group, redhat_subscription, redis, replace, rhn_channel, rhn_register, riak, rollbar_deployment, route53, rpm_key, s3, script, seboolean, selinux, service, set_fact, setup, shell, slack, slurp, sns, stackdriver, stat, subversion, supervisorctl, svr4pkg, swdepot, synchronize, sysctl, template, twilio, typetalk, ufw, unarchive, uri, urpmi, user, virt, vsphere_guest, wait_for, win_feature, win_get_url, win_group, win_msi, win_ping, win_service, win_stat, win_user, xattr, yum, zfs, zypper, zypper_repository 230+ modules and growing
61. Ad-Hoc commands $ ansible webservers -m copy -a 'src=resolv.conf dest=/etc/resolv.conf' www.example.com | success >> { "changed": true, "dest": "/etc/resolv.conf", "group": "adm", "md5sum": "c6fce6e28c46be0512eaf3b7cfdb66d7", "mode": "0644", "owner": "ubuntu", "path": "resolv.conf", "src": "/home/ubuntu/.ansible/tmp/ansible-322091977449/resolv.conf", "state": "file" }
62. Playbooks YAML Files Decleratively define your OS/App configuration Collection of tasks using modules Each group of tasks is a play
63. Tasks --- # tasks/foo.yml # This is a task - name: Placeholder foo command: /bin/foo # This is another task - name: Placeholder bar command: /bin/bar
64. Tasks --- - name: Installing supervisor task for snapshot worker template: src=supervisor.conf.j2 dest={{ SUPERVISOR_CONFIG_DIR }}/{{ item['filename'] }}.conf backup=yes owner=root group=root mode=0644 # located in defaults/main.yml with_items: snapshot_worker_configuration when: snapshot_worker_configuration|lower != 'none' notify: - reload supervisor tags: [supervisor, configuration]
65. Tasks --- - name: Installing supervisor task for snapshot worker template: src=supervisor.conf.j2 dest={{ SUPERVISOR_CONFIG_DIR }}/{{ item['filename'] }}.conf backup=yes owner=root group=root mode=0644
66. Tasks # located in defaults/main.yml with_items: snapshot_worker_configuration when: snapshot_worker_configuration|lower != 'none'
67. Tasks notify: - reload supervisor
68. Variables From inventory In playbooks From host_vars/ files From group_vars/ files
69. Variables --- - hosts: localhost vars: - greeting: Hello tasks: - command: echo "{{greeting}}, {{inventory_hostname}}"
70. Variables
71. Variables host_vars/production --- snapshot_worker_configuration: - filename: snapshot_worker name: process_snapshot_report_worker command: "php process_snapshot_report_worker.php" process_name: process_snapshot_report_worker_%(process_num)02d numprocs: 1 directory: "/var/www/scripts/utils/" autostart: true autorestart: true user: ubuntu stdout_logfile: "/var/log/app/utils_process_snapshot_report_worker.log" stdout_logfile_maxbytes: 1MB stderr_logfile: "/var/log/app/supervisor_error_log" stderr_logfile_maxbytes: 1MB
72. {{ templates }} ;{{ ansible_managed }} [program:{{ item.name }}] {% for directive, value in item.iteritems() if directive != "name" and direct ive != "filename" and value != None %} {{ directive }}={{ value }} {% endfor %}
73. {{ templates }} ;{{ ansible_managed }} [program:{{ item.name }}]
74. {{ templates }} {% for directive, value in item.iteritems() if directive != "name" and direct ive != "filename" and value != None %} {{ directive }}={{ value }} {% endfor %}
75. {{ templates }} ;Ansible managed: /Users/vranac/dev/playground-ansible/vagrant-ansible-php/ro les/supervisor/templates/supervisor.conf.j2 modified on 2014-06-19 10:38:31 b y vranac on vurunica [program:process_snapshot_report_worker] stderr_logfile_maxbytes=1MB autorestart=True stderr_logfile=/var/log/app/supervisor_error_log process_name=process_snapshot_report_worker_%(process_num)02d stdout_logfile_maxbytes=1MB numprocs=1 command=php process_snapshot_report_worker.php user=ubuntu autostart=True directory=/var/www/scripts/utils/ stdout_logfile=/var/log/app/utils_process_snapshot_report_worker.log
77. Roles
78. Roles roles/ nginx/ files/ handlers/main.yml meta/main.yml tasks/main.yml templates/ vars/main.yml
79. Roles --- - hosts: all roles: - nginx - mysql - { role: app, dir: '/etc/app', ntp: 'n1.example.org' } - { role: special, when: "ansible_os_family == 'RedHat'" } tasks: ...
80. Roles ... - { role: app, dir: '/etc/app', ntp: 'n1.example.org' } - { role: special, when: "ansible_os_family == 'RedHat'" } ...
81. Ansible Galaxy http://galaxy.ansible.com/ ansible-galaxy
82. Asynchronous Actions and Polling --- - hosts: all tasks: - name: "simulate long running op (15 sec), wait for up to 45 sec, poll every 5 sec" command: /bin/sleep 15 async: 45 poll: 5
83. Check Mode (“Dry Run”) Running a task in check mode --check Showing Differences with --diff
84. Compare to X https://devopsu.com/books/taste-test-grid.html
86. The End Thank You! Questions?