Save 37% off PRO during our Black Friday Sale! »

Using Vagrant, Docker, & Terraform for environments

A19936fa5955e8d2fc29b16c0a74a9b0?s=47 Nikhil Vaze
September 28, 2015

Using Vagrant, Docker, & Terraform for environments

Talk given at HashiConf 2015 on 9/28/2015 in Portland, Oregon.

This was a 50 minute talk centered around how Electric Cloud identified certain problems, came up with requirements and found solutions.

There was a focus on how to gather consensus within an organization and how I would explain these technologies to others.

A19936fa5955e8d2fc29b16c0a74a9b0?s=128

Nikhil Vaze

September 28, 2015
Tweet

Transcript

  1. © Electric Cloud | electric-cloud.com Nikhil Vaze | @therealnikhil |

    HashiConf 2015 USING VAGRANT, DOCKER AND TERRAFORM TO STREAMLINE YOUR DEV AND DEMO ENVIRONMENTS
  2. © Electric Cloud | electric-cloud.com Introductions Nikhil Vaze https://github.com/nikhilv @therealnikhil

    Senior Software Engineer at Electric Cloud Full stack engineer and loves to hack on things. Johns Hopkins University Alum
  3. © Electric Cloud | electric-cloud.com I love tools that make

    my life easier! Vagrant Docker Terraform
  4. © Electric Cloud | electric-cloud.com DEV OPS Electric Cloud: Powering

    Continuous Delivery ElectricFlow Deploy Pipeline Management and Visibility ElectricAccelerator Build & Test Acceleration DevOps Automation Platform (ElectricFlow) Build DELIVERY Plugs right in to your existing tools CONTINUOUS Build Test
  5. © Electric Cloud | electric-cloud.com I spend my time on

    integrations Source Control, Defect Tracking, Databases, Clouds, Testing tools , Application Servers….
  6. © Electric Cloud | electric-cloud.com The dark days before Vagrant...

    2010-2013
  7. © Electric Cloud | electric-cloud.com Our situation in 2010 •

    One engineering site (Sunnyvale, CA) • All engineers were in one room, on the same schedule. • Code reviews were done informally by calling someone over to your desk. • Engineers documented setup on wiki. Other employees expected to follow.
  8. © Electric Cloud | electric-cloud.com Show of hands Who has

    been burned following a wiki document?
  9. © Electric Cloud | electric-cloud.com How to start a code

    review in 2010?
  10. © Electric Cloud | electric-cloud.com The company grows - 2012

    • We were lucky enough to grow. • Added three new sites in three different time zones. • New employees? Read this wiki!
  11. © Electric Cloud | electric-cloud.com “I tried reading the wiki…”

    • Information out of date • Incomplete information • Couldn’t find it • Person who can answer the question to unblock you is asleep.
  12. © Electric Cloud | electric-cloud.com No Problem! I’ll just update

    the wiki
  13. © Electric Cloud | electric-cloud.com How to install our product

    on AWS EC2?
  14. © Electric Cloud | electric-cloud.com “I tried reading the wiki…”

    • Information out of date • Incomplete information • Couldn’t find it • Person who can answer the question to unblock you is asleep.
  15. © Electric Cloud | electric-cloud.com What were the problems? •

    Wiki was out of date. • Our software was getting harder to setup.  For example: we introduced support for clustering • Found ourselves spending more time troubleshooting individual environments instead of product issues.
  16. © Electric Cloud | electric-cloud.com After spending a day on

    a configuration issue
  17. © Electric Cloud | electric-cloud.com There has to be a

    better way ~ 2013 How do we create development environments such that they can be: • reconstructed • shared • local to each tester • deployed into an automated test environment
  18. © Electric Cloud | electric-cloud.com Side note: Cultural Observation •

    What worked well and got you this far may not be the thing to get you the next level.  It’s hard because the people suffering may be new to the organization  It’s very natural to think “But it has worked for us so well!”
  19. © Electric Cloud | electric-cloud.com Partly Cloudy Days: Adopting better

    tooling ~2013
  20. © Electric Cloud | electric-cloud.com “If it hurts, do it

    more frequently, and bring the pain forward.” Jez Humble
  21. © Electric Cloud | electric-cloud.com Bringing the pain forward: •

    It hurts to construct these environments so….let’s do it more often! • How can we run through the process regularly? • How can we have documentation that is always up to date?
  22. © Electric Cloud | electric-cloud.com Introducing Vagrant  https://www.vagrantup.com/ 

    Open Source  Made by HashiCorp $ vagrant up (start or create the VM) $ vagrant ssh (ssh to the VM) $ vagrant halt (stop the VM) $ vagrant destroy (destroy the VM) $ vagrant package (snapshot the VM)
  23. © Electric Cloud | electric-cloud.com Vagrant up

  24. © Electric Cloud | electric-cloud.com Vagrant ssh

  25. © Electric Cloud | electric-cloud.com Vagrant status

  26. © Electric Cloud | electric-cloud.com Vagrant destroy

  27. © Electric Cloud | electric-cloud.com So…. I know what you’re

    all thinking “Big deal. You just described VMs”
  28. © Electric Cloud | electric-cloud.com Step 1: Vanilla Vagrantfile $

    vagrant up
  29. © Electric Cloud | electric-cloud.com Step 1.5: Vanilla Vagrantfile $

    vagrant up
  30. © Electric Cloud | electric-cloud.com Step 2: Install software within

    Vagrantfile
  31. © Electric Cloud | electric-cloud.com Step 2: Install software within

    Vagrantfile
  32. © Electric Cloud | electric-cloud.com What problems did Vagrant solve?

    • Simple commands to instantiate and destroy • Documentation via code inside Vagrantfile (or shell script invoked by Vagrantfile) • Check in Vagrantfile. Now available to all developers • Each developer can run Vagrant on their laptop • Same process can be used in automated testing
  33. © Electric Cloud | electric-cloud.com How do I get Vagrant

    and what do I need to use it? • Install Vagrant on each machine where vagrant will be used. • Install virtualization provider (VirtualBox). • Supported Host Operating Systems:  Mac OSX  Windows  Linux (Debian, RedHat) • Vagrant  http://downloads.vagrantup.com/ • Oracle VirtualBox  https://www.virtualbox.org/wiki/Downloads
  34. © Electric Cloud | electric-cloud.com Vagrant Tips • Your relationship

    with VirtualBox will be love/hate • At a high level, Vagrant uses Ruby to wrap VBoxManage. So if you are running into any issues you can open up the VirtualBox UI or instrument the ruby code (add puts/print statements) • As with all Open Source: read the source, check out the open issues. https://github.com/mitchellh/vagrant/ • Stackoverflow also helpful
  35. © Electric Cloud | electric-cloud.com Side note: Cultural Observation •

    When trying to gather consensus or persuade an organization to adopt change:  Identify a problem, formulate a hypothesis and then look for a tool/solution. • Avoid  “Oooh shiny technology!”  “All the cool kids are doing it” • Even if you have followed the problem solving steps people may still feel like you’re just chasing technology.
  36. © Electric Cloud | electric-cloud.com Real emails showing Vagrant adoption

  37. © Electric Cloud | electric-cloud.com Next use case: Multi App

    Server, Multi Host Deployments
  38. © Electric Cloud | electric-cloud.com Use case : Creating a

    demo environment  Jboss/Wildfly  Domain mode, 3 machines  Business on the line! Fast fast fast!
  39. © Electric Cloud | electric-cloud.com Diagram

  40. © Electric Cloud | electric-cloud.com Requirements for environment creation 

    Definition based  Reproducible  Shareable  Each invocation must be unique to an individual
  41. © Electric Cloud | electric-cloud.com Cue Terraform

  42. © Electric Cloud | electric-cloud.com Introducing Terraform  https://terraform.io/ 

    Open Source  Made by HashiCorp $ terraform plan $ terraform apply $ terraform destroy
  43. © Electric Cloud | electric-cloud.com Let’s read some code! https://gist.github.com/nikhilv/74a107af7e44866e3f14

    Note: these days terraform’s doc have improved. A lot of this is available in the aws-two-tier example https://github.com/hashicorp/terraf orm/tree/master/examples/aws-two- tier
  44. © Electric Cloud | electric-cloud.com Plan $ terraform plan -var

    'name=nikhilDemo' -var 'key_name=<keyName>' –var 'key_path=<path>'
  45. © Electric Cloud | electric-cloud.com Terraform Graph $ terraform graph

    | dot –Tpng > graph.png
  46. © Electric Cloud | electric-cloud.com Apply $ terraform apply -var

    'name=nikhilDemo' -var 'key_name=<keyName>' –var 'key_path=<path>'
  47. © Electric Cloud | electric-cloud.com Apply $ terraform apply -var

    'name=nikhilDemo' -var 'key_name=<keyName>' –var 'key_path=<path>'
  48. © Electric Cloud | electric-cloud.com Destroy $ terraform destroy -var

    'name=nikhilDemo' -var 'key_name=<keyName>' –var 'key_path=<path>'
  49. © Electric Cloud | electric-cloud.com Review: Requirements for environment creation

     Definition based .tf files  Reproducible Plan -> Apply – Destroy.  Shareable Check into source control  Each invocation must be unique to an individual Parametrized scripts
  50. © Electric Cloud | electric-cloud.com What problems did Terraform solve?

    • Simple commands to instantiate and destroy cloud VMs • Documentation via code inside Terraform files • Check in Terraform files. Now available to all developers • Each developer can run Terraform on their laptop • Save those UI clicks! Write it once in Terraform and build on it later
  51. © Electric Cloud | electric-cloud.com Terraform Tips • Super easy

    to install and upgrade (directory of executables) • Caution: Using Terraform will cost you money with your cloud provider! • Some syntax is not intuitive. But the documentation has improved. • Parameterize and use the graph • IMO: Good for demo and short term work. But I wouldn’t use it alone to deploy to production
  52. © Electric Cloud | electric-cloud.com Next use case: Integrated Software

    Delivery Scenario
  53. © Electric Cloud | electric-cloud.com Integrated Software Delivery Scenario

  54. © Electric Cloud | electric-cloud.com Delivery Pipeline Commit Pipelines

  55. © Electric Cloud | electric-cloud.com Interlude: Docker overview

  56. © Electric Cloud | electric-cloud.com Docker • https://docker.io/ • Open

    Source • Used to wrap LXC, now has its own container implementation • Really, really fast. Start measuring in milliseconds
  57. © Electric Cloud | electric-cloud.com Docker 102 – Different drivers

  58. © Electric Cloud | electric-cloud.com Docker 102 containers vs. VMs

  59. © Electric Cloud | electric-cloud.com Back to the use case

  60. © Electric Cloud | electric-cloud.com Use case : Simulating ‘Big

    Data Software Team’  Pieces  Front End  Dashing Dashboard  Database  Redis  Backend  Ratpack Server
  61. © Electric Cloud | electric-cloud.com Use case : Simulating ‘Big

    Data Software Team’ Communication paths  ELB-> instance(s)  Backend -> DB  Frontend -> Backend  IoT -> ELB / Backend  Mobile Device -> ELB / Backend
  62. © Electric Cloud | electric-cloud.com Use case : Simulating ‘Big

    Data Software Team’  How does the Front end know the IP the backend gets spawned at?  How does the Back end know the IP the database gets spawned at?  How does the ELB know about the two availability zones? What is the criteria for the health check?
  63. © Electric Cloud | electric-cloud.com Technologies used  Didn’t use

    Terraform  Regretted it. All my setup is now bespoke and won’t be easy to re-use  Lost: ELB (Health Check, Port forward and instances), VPC, Subnets, EC2 instance data  Decided Docker would be the way to go from “dev” to “production”  Easy to simulate pieces locally on dev laptop and then run same thing in AWS.  Was easy to share with others working on the same demo  Feel confident that I could go back to the source for a future demo
  64. © Electric Cloud | electric-cloud.com “Demo”

  65. © Electric Cloud | electric-cloud.com Pictures because I didn’t want

    to bring this on a plane
  66. © Electric Cloud | electric-cloud.com Docker Tips • Where do

    you save, version and test your containers? • Your disk usage will skyrocket. One reason Docker is fast is because it caches intermediate layers. • As with all Open Source: read the source, check out the open issues. https://github.com/docker/docker • Check out Greg Osuri’s guide on Guide to automating a multi- tiered application securely on AWS with Docker and Terraform https://www.airpair.com/aws/posts/ntiered-aws-docker-terraform-guide
  67. © Electric Cloud | electric-cloud.com Wrap up

  68. © Electric Cloud | electric-cloud.com Vagrant, Terraform, Docker FTW! •

    It’s an exciting time to be in software • Warning bells should go off if you keep clicking on a UI. • Organization and structure increase velocity and confidence • Drift is minimized if the ‘documentation’ is exercised regularly • You will find bugs or areas where documentation could be improved
  69. © Electric Cloud | electric-cloud.com Thank you Networking Other Systems

    ISV, Internet & Entertainment Heavy Industry Financial Services Automotive Medical Devices Semiconductor Aerospace / Defense Mobile Devices Q&A